You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Tibor Digana <ti...@apache.org> on 2022/03/14 01:55:22 UTC
issues with the user property "maven.test.failure.ignore" and new proposals
In case of the user property *maven.test.failure.ignore* the MOJO must not
throw any exception which is interpreted by the Maven Core as BUILD SUCCESS.
We have received an internal requirement to change the behavior of the user
property *maven.test.failure.ignore* so that the behavior will have one
exception.
Suppose that you have JDK 1.8 but you use:
/jre/java --add-reads ...
The outcome is JVM exit with an error message.
I agree with Herve who also says that *maven.test.failure.ignore* should
not allow the MOJO to throw the exception. It is not a typical JVM
segmentation fault or another JVM crash where we cannot do anything about
it, and where the entire build would crash in the command line which
of course means that the build cannot normally be interpreted as BUILD
SUCCESS. So we are still on the same level of failures related to the test
purposes.
On the other hand, Olivier has reopened the issues SUREFIRE-1426 and
SUREFIRE-1681 where the exceptional behavior of the feature is expected.
This is exactly the reason why I closed these tickets several years ago and
my proposal was to not to have any exceptions in the feature behavior and
the proposal was to introduce a new user property for exact use cases.
The next idea, which comes from two developers, would provide us with the
same non exceptional and exact behavior of the user property
*maven.test.failure.ignore* but it would also provide us with new user
property in the case with fine grade control of the build errors, e.g.
*maven.test.jvm.error.ignore*.
I would like to open the discussion on this topic. You're welcome!
Cheers
Tibor
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Romain Manni-Bucau <rm...@gmail.com>.
@Tibor: I never said it was a bug, my point is that you never needs it so
we should keep that deprecated/not recommended and enable people having too
constrained env to work on it through a SPI as proposed by Guillaume. So
proposal was to drop that property support and add a SPI if anyone think
this property is needed.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le lun. 14 mars 2022 à 10:57, Christoph Läubrich <ma...@laeubi-soft.de> a
écrit :
> Just wondering but maybe it would be better to configure the severity
> instead of a true/false, something like:
>
> maven.test.failure=WARN
> maven.test.error=ERROR
>
> would only warn about failing tests but thrw exception if starting the
> test fails?
>
> Am 14.03.22 um 10:52 schrieb Tibor Digana:
> > Romain, it is not a bug.
> > Don't consider this as a bug. It was a feature request for change by
> > Olivier, and not a bug.
> > I closed both issues years ago but not because of ignorance but because
> the
> > appearance of the exceptional behavior is a wrong compromise and which
> does
> > not help anyone and even it makes the situation worse because typically
> > other group of users would fire a new Jira tickets. You would not be able
> > to satisfy two contradictory parties which have completely different
> > opinions, and so we use to introduce new params and this way we satisfy
> > both parties, they may combine the params as they wish, and everybody
> would
> > be happy and nobody would claim. Many technical solutions might exist,
> e.g.
> > param=boolean|string or new param=boolean, whatever.
> >
> > The truth is that this problem with (java --add-reads ...) happened in
> our
> > ASF environment on Java 8 which in good configuration would not happen
> and
> > should not.
> > It is not right way that we abuse "maven.test.failure.ignore" which would
> > tell us "Hey, you have illegal configuration in your build system and it
> > would not work by JDK design", it is not the goal of the plugin to tell
> you
> > that you have configured the build wrong because it won't and the same
> > configuration of the plugin (not the build) says "ignore the error".
> > Yesterday I discussed this problem with Herve and we independently
> observed
> > equal opinions and that's not everything because we understood that the
> > purpose of the config parameter is to not throw MOJO exception which is
> > right, but the exceptional behavior should have an exact new param for
> the
> > exact use case.
> > SPI for this parameter is too much because no user would implement it
> for a
> > trivial parameter;; the SPI is used to be implemented by frameworks or
> > systems or application servers but this is not our case.
> >
> >
> >
> >
> > On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <
> rmannibucau@gmail.com>
> > wrote:
> >
> >> +1
> >> if it is to investigate a CI issue, it is generally easy to add debug
> >> insights (by code or agent) so a SPI sounds like the sanest for the
> plugin
> >> to me.
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> >> <https://rmannibucau.metawerx.net/> | Old Blog
> >> <http://rmannibucau.wordpress.com> | Github <
> >> https://github.com/rmannibucau> |
> >> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> >> <
> >>
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >>>
> >>
> >>
> >> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <gn...@apache.org> a
> écrit
> >> :
> >>
> >>> If that's not currently possible, maybe a SPI should be provided so
> that
> >>> people can use plug in extensions to analyze the test result and
> override
> >>> it if necessary (transforming an error into a warning, storing results
> >> in a
> >>> way which is easier to use by other tools later...) ?
> >>>
> >>> Guillaume
> >>>
> >>> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <ma...@laeubi-soft.de>
> a
> >>> écrit :
> >>>
> >>>> I also agree that the test at least should run, we use this property
> to
> >>>> run the test and produce result and later on have a buildstep that
> >>>> analyze the results (and probably fail the build job).
> >>>>
> >>>> As it is not recommend, I wonder what is the recommended way to
> archive
> >>>> something similar?
> >>>>
> >>>> Am 14.03.22 um 06:29 schrieb Olivier Lamy:
> >>>>> On Mon, 14 Mar 2022 at 11:55, Tibor Digana <ti...@apache.org>
> >>>> wrote:
> >>>>>
> >>>>>> In case of the user property *maven.test.failure.ignore* the MOJO
> >> must
> >>>> not
> >>>>>> throw any exception which is interpreted by the Maven Core as BUILD
> >>>>>> SUCCESS.
> >>>>>>
> >>>>>
> >>>>> This is a very simple reduction of the problem description.
> >>>>> The documentation here
> >>>>>
> >>>>
> >>>
> >>
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
> >>>>> says
> >>>>> "Set this to "true" to ignore a failure during testing. Its use is
> >> NOT
> >>>>> RECOMMENDED, but quite convenient on occasion"
> >>>>>
> >>>>> Personally, I understand this to ignore failures in junit results BUT
> >>> at
> >>>>> least I want the tests to run.
> >>>>> I guess this is how our users use this feature (feature we do not
> >>>> recommend
> >>>>> by the way...)
> >>>>> And this is perfectly explained here
> >>>>>
> >>>>
> >>>
> >>
> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
> >>>>> there is a difference between ignoring some junit failures and
> >>> ignoring a
> >>>>> configuration error because some jvm args has been misconfigured for
> >>> many
> >>>>> reasons and surefire cannot start.
> >>>>>
> >>>>> If I follow your reasoning ("MOJO must not throw any exception ") we
> >>>> should
> >>>>> ignore every surefire configuration error and keep running the build
> >>>> until
> >>>>> the end and says BUILD SUCCESS
> >>>>> what about
> >>>>>
> >>>>> mvn test -Dsurefire.rerunFailingTestsCount=notanumber
> >>>>> -Dmaven.test.failure.ignore=true
> >>>>>
> >>>>> we should not throw any exceptions as you said below and keep saying
> >>>> BUILD
> >>>>> SUCCESS?
> >>>>> argLine is a configuration element as any others so it should fail
> >>>> directly
> >>>>> and not be ignored especially when the surefire plugin cannot run.
> >>>>>
> >>>>>
> >>>>>> We have received an internal requirement to change the behavior of
> >> the
> >>>> user
> >>>>>> property *maven.test.failure.ignore* so that the behavior will have
> >>> one
> >>>>>> exception.
> >>>>>
> >>>>> Suppose that you have JDK 1.8 but you use:
> >>>>>> /jre/java --add-reads ...
> >>>>>> The outcome is JVM exit with an error message.
> >>>>>> I agree with Herve who also says that *maven.test.failure.ignore*
> >>>> should
> >>>>>> not allow the MOJO to throw the exception. It is not a typical JVM
> >>>>>> segmentation fault or another JVM crash where we cannot do anything
> >>>> about
> >>>>>> it, and where the entire build would crash in the command line which
> >>>>>> of course means that the build cannot normally be interpreted as
> >> BUILD
> >>>>>> SUCCESS. So we are still on the same level of failures related to
> >> the
> >>>> test
> >>>>>> purposes.
> >>>>>>
> >>>>>> On the other hand, Olivier has reopened the issues SUREFIRE-1426 and
> >>>>>> SUREFIRE-1681 where the exceptional behavior of the feature is
> >>> expected.
> >>>>>> This is exactly the reason why I closed these tickets several years
> >>> ago
> >>>> and
> >>>>>> my proposal was to not to have any exceptions in the feature
> >> behavior
> >>>> and
> >>>>>> the proposal was to introduce a new user property for exact use
> >> cases.
> >>>>>> The next idea, which comes from two developers, would provide us
> >> with
> >>>> the
> >>>>>> same non exceptional and exact behavior of the user property
> >>>>>> *maven.test.failure.ignore* but it would also provide us with new
> >> user
> >>>>>> property in the case with fine grade control of the build errors,
> >> e.g.
> >>>>>> *maven.test.jvm.error.ignore*.
> >>>>>>
> >>>>>
> >>>>> with a default of?
> >>>>> honestly I just see this new parameter as introducing more complexity
> >>> in
> >>>> an
> >>>>> already very complex code and not worth it.
> >>>>> again read both issue's comments and my comments.
> >>>>> Please try to have a user POV and think about making our users'
> >>>> experience
> >>>>> more simple.
> >>>>>
> >>>>> This should be very simple:
> >>>>> If surefire forked jvm cannot start it's build error and cannot run
> >> any
> >>>>> tests, it's a problem users want to know immediately because it can
> >> be
> >>>> for
> >>>>> a lot of reasons: wrong argLine, not enough memory on the system
> >> etc...
> >>>>>
> >>>>> AND AGAIN it is very different from ignoring junit result failures.
> >>>>>
> >>>>> Try to look at a user point of view and think about what is the use
> >>> case
> >>>> of
> >>>>> the property maven.test.failure.ignore=true, I guess 99% of the time,
> >>>> users
> >>>>> wants to run all their tests (even on a CI with different matrix) so
> >>> they
> >>>>> can collect all the tests results which has runned to see if there is
> >>> any
> >>>>> issue for some combination of their matrix and eventually turn the
> >>> result
> >>>>> as unstable (this is a very typical use case in Jenkins and was even
> >> a
> >>>>> built in feature of the previous Jenkins Maven plugin)
> >>>>> BUT if for any reasons one of the module do not start his tests
> >> because
> >>>> the
> >>>>> jvm cannot be start the users will not see that and will be totally
> >>> blind
> >>>>> until maybe someone look inside a very very large log file (which
> >> means
> >>>>> probably never)
> >>>>>
> >>>>> Long story short as my experience as a user facing this problem/bug:
> >>>>> I had the case on a very large multi modules (~250 modules) build of
> >> a
> >>>> very
> >>>>> used open source project.
> >>>>> I was using this maven.test.failure.ignore property but one of the
> >>>> modules
> >>>>> had a bad jpms configuration for a jdk17 profile on the CI.
> >>>>> The build has a matrix of 2 os and 4 jdks and different maven run
> >> which
> >>>>> means around ~60k tests and a Jenkins log file about 40M
> >>>>> So because of this property the build was not failing and kept saying
> >>>> BUILD
> >>>>> SUCCESS for weeks/months and basically not testing one module with
> >> jdk
> >>>> 17...
> >>>>> And frankly you do not dig into a log file of 32M after each run
> >>>> especially
> >>>>> when it says success...
> >>>>> 3 days after the first release claiming supporting jdk 17 we
> >> received a
> >>>> bug
> >>>>> report about a something not working with jdk17....
> >>>>> and guess what? Where was this feature supposed to be tested?
> >>>>>
> >>>>> so I frankly believe we do not need a complex new property, in this
> >>> case
> >>>>> this should fail directly because this will improve user experience.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> I would like to open the discussion on this topic. You're welcome!
> >>>>>>
> >>>>>>
> >>>>>> Cheers
> >>>>>> Tibor
> >>>>>>
> >>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>> For additional commands, e-mail: dev-help@maven.apache.org
> >>>>
> >>>>
> >>>
> >>> --
> >>> ------------------------
> >>> Guillaume Nodet
> >>>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Olivier Lamy <ol...@apache.org>.
On Thu, 24 Mar 2022 at 01:18, Tibor Digana <ti...@apache.org> wrote:
> Hi Olivier, I have used the Maven branch according to your instructions. I
> have used the snapshot version of the plugin and JDK 1.8.0u05.
> Both implementations end up with the same build result. The only
> difference is the number of IF-Else branches in the implementation.
>
> I started Maven build twice for your and mine impl with these two configs:
>
> <argLine>-Xmx1m -XX:MaxPermSize=1m</argLine>
>
> <argLine>-FFFOOOOO</argLine>
>
>
> 1. The first config would end up with BUILD SUCCESS and it is expected
> because JUnit caught the OutOfMemoryError "during testing" and the plugin
> has reported it. The OOM did not happen in Surefire with my use case which
> we cannot guarantee what the JVM would do in another real scenario.
> 2. The second config is the *JVM initialization error* which happens
> "before testing" and the build fails with a failure as expected, BUILD
> FAILURE.
>
> Pls comment on these build results. These are expected by us, we can agree
> on them, right?
>
it's what I except yes
but for case 2 this is failing directly for me (which is really fine as I
definitely prefer this behaviour :) )
[*INFO*] *--- *maven-compiler-plugin:3.10.0:testCompile
*(default-testCompile)* @ maven-model* ---*
[*INFO*] Changes detected - recompiling the module!
[*INFO*] Compiling 38 source files to
/Volumes/workspace/dev/sources/maven/maven-core/maven-model/target/test-classes
[*INFO*]
[*INFO*] *--- *maven-surefire-plugin:3.0.0-M6-SNAPSHOT:test *(default-test)*
@ maven-model* ---*
[*INFO*] Using auto detected provider
org.apache.maven.surefire.junit4.JUnit4Provider
[*INFO*]
[*INFO*] -------------------------------------------------------
[*INFO*] T E S T S
[*INFO*] -------------------------------------------------------
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1m; support
was removed in 8.0
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "main"
[*INFO*]
[*INFO*] Results:
[*INFO*]
[*INFO*] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
my config is (but the difference looks to be platform specific)
mvn -v
*Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)*
Maven home: /Users/olamy/softs/maven/apache-maven-3.8.4
Java version: 1.8.0_322, vendor: Temurin, runtime:
/Library/Java/JavaVirtualMachines/temurin-8.jdk/Contents/Home/jre
Default locale: en_AU, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac"
you definitely need to add IT test to your PR anyway the changes looks good
to me.
> 1. case
> *$ mvn verify -nsu -Dmaven.test.failure.ignore*
>
> [INFO] --- maven-surefire-plugin:3.0.0-M6-SNAPSHOT:test (default-test) @
> maven-model ---
> [INFO] Using auto detected provider
> org.apache.maven.surefire.junit4.JUnit4Provider
> [INFO]
> [INFO] -------------------------------------------------------
> [INFO] T E S T S
> [INFO] -------------------------------------------------------
> [INFO] Running org.apache.maven.model.ActivationFileTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.583 s - in org.apache.maven.model.ActivationFileTest
> [INFO] Running org.apache.maven.model.ActivationOSTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.099 s - in org.apache.maven.model.ActivationOSTest
> [INFO] Running org.apache.maven.model.ActivationPropertyTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.136 s - in org.apache.maven.model.ActivationPropertyTest
> [INFO] Running org.apache.maven.model.ActivationTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.118 s - in org.apache.maven.model.ActivationTest
> [INFO] Running org.apache.maven.model.BuildTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.177 s - in org.apache.maven.model.BuildTest
> [INFO] Running org.apache.maven.model.CiManagementTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.22 s - in org.apache.maven.model.CiManagementTest
> [INFO] Running org.apache.maven.model.ContributorTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.171 s - in org.apache.maven.model.ContributorTest
> [INFO] Running org.apache.maven.model.DependencyManagementTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.211 s - in org.apache.maven.model.DependencyManagementTest
> [INFO] Running org.apache.maven.model.DependencyTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.163 s - in org.apache.maven.model.DependencyTest
> [INFO] Running org.apache.maven.model.DeploymentRepositoryTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.198 s - in org.apache.maven.model.DeploymentRepositoryTest
> [INFO] Running org.apache.maven.model.DeveloperTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.201 s - in org.apache.maven.model.DeveloperTest
> [INFO] Running org.apache.maven.model.DistributionManagementTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.193 s - in org.apache.maven.model.DistributionManagementTest
> [INFO] Running org.apache.maven.model.ExclusionTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.218 s - in org.apache.maven.model.ExclusionTest
> [INFO] Running org.apache.maven.model.ExtensionTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.194 s - in org.apache.maven.model.ExtensionTest
> [INFO] Running org.apache.maven.model.IssueManagementTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.206 s - in org.apache.maven.model.IssueManagementTest
> [INFO] Running org.apache.maven.model.LicenseTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.216 s - in org.apache.maven.model.LicenseTest
> [INFO] Running org.apache.maven.model.MailingListTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.257 s - in org.apache.maven.model.MailingListTest
> [INFO] Running org.apache.maven.model.merge.ModelMergerTest
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 6.742 s <<< FAILURE! - in org.apache.maven.model.merge.ModelMergerTest
> [ERROR]
> org.apache.maven.model.merge.ModelMergerTest.testMergedModelSerialization
> Time elapsed: 2.502 s <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.maven.model.merge.ModelMergerTest.testMergedModelSerialization(ModelMergerTest.java:43)
>
> [INFO] Running org.apache.maven.model.ModelTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 2.079 s - in org.apache.maven.model.ModelTest
> [INFO] Running org.apache.maven.model.NotifierTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.655 s - in org.apache.maven.model.NotifierTest
> [INFO] Running org.apache.maven.model.OrganizationTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.512 s - in org.apache.maven.model.OrganizationTest
> [INFO] Running org.apache.maven.model.ParentTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.367 s - in org.apache.maven.model.ParentTest
> [INFO] Running org.apache.maven.model.PluginConfigurationTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.715 s - in org.apache.maven.model.PluginConfigurationTest
> [INFO] Running org.apache.maven.model.PluginContainerTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.316 s - in org.apache.maven.model.PluginContainerTest
> [INFO] Running org.apache.maven.model.PluginExecutionTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.451 s - in org.apache.maven.model.PluginExecutionTest
> [INFO] Running org.apache.maven.model.PluginManagementTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.972 s - in org.apache.maven.model.PluginManagementTest
> [INFO] Running org.apache.maven.model.PluginTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.502 s - in org.apache.maven.model.PluginTest
> [INFO] Running org.apache.maven.model.PrerequisitesTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.663 s - in org.apache.maven.model.PrerequisitesTest
> [INFO] Running org.apache.maven.model.ProfileTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.202 s - in org.apache.maven.model.ProfileTest
> [INFO] Running org.apache.maven.model.RelocationTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.562 s - in org.apache.maven.model.RelocationTest
> [INFO] Running org.apache.maven.model.ReportingTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.487 s - in org.apache.maven.model.ReportingTest
> [INFO] Running org.apache.maven.model.ReportPluginTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.599 s - in org.apache.maven.model.ReportPluginTest
> [INFO] Running org.apache.maven.model.ReportSetTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.287 s - in org.apache.maven.model.ReportSetTest
> [INFO] Running org.apache.maven.model.RepositoryPolicyTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.1
> s - in org.apache.maven.model.RepositoryPolicyTest
> [INFO] Running org.apache.maven.model.RepositoryTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.102 s - in org.apache.maven.model.RepositoryTest
> [INFO] Running org.apache.maven.model.ResourceTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.85 s - in org.apache.maven.model.ResourceTest
> [INFO] Running org.apache.maven.model.ScmTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.237 s - in org.apache.maven.model.ScmTest
> [INFO] Running org.apache.maven.model.SiteTest
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 1.074 s - in org.apache.maven.model.SiteTest
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1m;
> support was removed in 8.0
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] ModelMergerTest.testMergedModelSerialization:43 » OutOfMemory
> Java heap space
> [INFO]
> [ERROR] Tests run: 149, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [ERROR]
>
> Please refer to
> C:\vcs\github\maven-core\maven-model\target\surefire-reports for the
> individual test results.
> Please refer to dump files (if any exist) [date].dump,
> [date]-jvmRun[N].dump and [date].dumpstream.
> [INFO]
> [INFO] --- maven-jar-plugin:3.2.2:jar (default-jar) @ maven-model ---
> [INFO] Building jar:
> C:\vcs\github\maven-core\maven-model\target\maven-model-3.8.6-SNAPSHOT.jar
> ...
> ...
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary for Apache Maven 3.8.6-SNAPSHOT:
> [INFO]
> [INFO] Apache Maven ....................................... SUCCESS [
> 16.395 s]
> [INFO] Maven Model ........................................ SUCCESS [
> 57.064 s]
> [INFO] Maven Artifact ..................................... SUCCESS [
> 6.102 s]
> [INFO] Maven Plugin API ................................... SUCCESS [
> 7.027 s]
> [INFO] Maven Builder Support .............................. SUCCESS [
> 2.486 s]
> [INFO] Maven Model Builder ................................ SUCCESS [
> 15.773 s]
> [INFO] Maven Settings ..................................... SUCCESS [
> 2.945 s]
> [INFO] Maven Settings Builder ............................. SUCCESS [
> 5.509 s]
> [INFO] Maven Repository Metadata Model .................... SUCCESS [
> 3.903 s]
> [INFO] Maven Artifact Resolver Provider ................... SUCCESS [
> 13.878 s]
> [INFO] Maven Core ......................................... SUCCESS [01:19
> min]
> [INFO] Maven SLF4J Simple Provider ........................ SUCCESS [
> 3.678 s]
> [INFO] Maven Embedder ..................................... SUCCESS [
> 15.397 s]
> [INFO] Maven Compat ....................................... SUCCESS [
> 53.385 s]
> [INFO] Apache Maven Distribution .......................... SUCCESS [
> 27.397 s]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 05:11 min
> [INFO] Finished at: 2022-03-23T01:15:10+01:00
> [INFO]
> ------------------------------------------------------------------------
>
> 2. case
> *$ mvn verify -nsu -Dmaven.test.failure.ignore*
>
> [INFO] -------------------------------------------------------
> [INFO] T E S T S
> [INFO] -------------------------------------------------------
> Unrecognized option: -FFFOOOOO
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary for Apache Maven 3.8.6-SNAPSHOT:
> [INFO]
> [INFO] Apache Maven ....................................... SUCCESS [
> 18.186 s]
> [INFO] Maven Model ........................................ FAILURE [
> 12.388 s]
> [INFO] Maven Artifact ..................................... SKIPPED
> ... ... ...
> [INFO] Apache Maven Distribution .......................... SKIPPED
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
>
>
> On Tue, Mar 22, 2022 at 11:44 AM Olivier Lamy <ol...@apache.org> wrote:
>
>> On Tue, 22 Mar 2022 at 14:40, Tibor Digana <ti...@apache.org>
>> wrote:
>>
>> > Sorry for late reply.
>> >
>> > I have created a demo project
>> > https://github.com/Tibor17/maven.test.failure.ignore/ which simulates
>> > OOM.
>> >
>> > According to the definition of the parameter "maven.failure.test.ignore"
>> > the failures (also errors) should be ignored during testing.
>> > The word "during" is important because the time when it was a failure
>> > before JVM startup it was not a test failure - it is MOJO failure. The
>> > author who wrote this config param knew this difference.
>> > This perfectly fits JVM illegal arguments like -Xxxx.
>> > But I have investigated OOM for the same reason with one JVM. My
>> suspicion
>> > was not confirmed, see the next...
>> > If there are two JVMs, we are breaking the JVM during testing. It is a
>> very
>> > unlikely situation that the JVM argument would not fail in JVM1 but it
>> > would fail in JVM2 later. It may happen only if the user
>> > uses @surefire.forkCount@ interpolation but it is again very unlikely
>> and
>> > the users use it in current working directory configuration.
>> >
>> > I understand Olivier's PR but I have one objection to the code
>> branching in
>> > the class *SurefireHelper*. It is based on 3 levels but I have to
>> disagree
>> > due to there still should be 2 levels as before:
>> > 1. ignored failures (my change is only here - yellow - only added
>> throwing
>> > exception)
>> > 2. processes failures
>> >
>> > and so, I have created a PR
>> > https://github.com/apache/maven-surefire/pull/496
>> > Pls have a look into it, thx.
>> > This diff looks very simple:
>> >
>> > if ( reportParameters.isTestFailureIgnore() )
>> > {
>> > String errorMessage = createErrorMessage( reportParameters,
>> > result, firstForkException );
>> >
>> > if ( firstForkException instanceof SurefireBooterForkException )
>> > {
>> > throw new MojoExecutionException( errorMessage,
>> firstForkException
>> > );
>> >
>> > }
>> >
>> > log.error( errorMessage );
>> > }
>> > else
>> > {
>> > throwException( reportParameters, result, firstForkException );
>> > }
>> >
>> >
>> >
>> > *$ mvn test*
>> >
>> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
>> > 2.445 s <<< FAILURE! - in org.apache.maven.OOMTest
>> > [ERROR] org.apache.maven.OOMTest.test Time elapsed: 2.063 s <<< ERROR!
>> > java.lang.OutOfMemoryError: Java heap space
>> > at org.apache.maven.OOMTest.test(OOMTest.java:15)
>> > [INFO]
>> > [INFO] Results:
>> > [INFO]
>> > [ERROR] Errors:
>> > [ERROR] OOMTest.test:15 » OutOfMemory Java heap space
>> > [INFO]
>> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
>> > [INFO]
>> > [INFO]
>> > ------------------------------------------------------------------------
>> > [INFO] BUILD FAILURE
>> > [INFO]
>> > ------------------------------------------------------------------------
>> >
>> > *$ mvn test -Dmaven.test.failure.ignore*
>> >
>> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
>> > 2.472 s <<< FAILURE! - in org.apache.maven.OOMTest
>> > [ERROR] org.apache.maven.OOMTest.test Time elapsed: 2.262 s <<< ERROR!
>> > java.lang.OutOfMemoryError: Java heap space
>> > at org.apache.maven.OOMTest.test(OOMTest.java:15)
>> > [INFO]
>> > [INFO] Results:
>> > [INFO]
>> > [ERROR] Errors:
>> > [ERROR] OOMTest.test:15 » OutOfMemory Java heap space
>> > [INFO]
>> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
>> > [INFO]
>> > [ERROR]
>> >
>> > Please refer to
>> > C:\vcs\github\maven.test.failure.ignore\target\surefire-reports for the
>> > individual test results.
>> > Please refer to dump files (if any exist) [date].dump,
>> > [date]-jvmRun[N].dump and [date].dumpstream.
>> > [INFO]
>> > ------------------------------------------------------------------------
>> > [INFO] BUILD SUCCESS
>> > [INFO]
>> > ------------------------------------------------------------------------
>> >
>>
>> Please use a project with multiple modules, not a single one where you can
>> see the line with OOME.
>> Any project with a huge number of modules will hide this and nobody will
>> ever notice it.
>>
>>
>> git clone -b first-module-fail https://github.com/apache/maven
>> maven-core
>>
>> cd maven-core
>>
>>
>> [INFO] --- maven-assembly-plugin:3.3.0:single (create-distro-packages) @
>> apache-maven ---
>> [INFO] Reading assembly descriptor: src/main/assembly/bin.xml
>> [INFO] Building zip:
>>
>> /Volumes/workspace/dev/sources/maven/maven-core/target/maven-core/apache-maven/target/apache-maven-3.8.6-SNAPSHOT-bin.zip
>> [INFO] Building tar:
>>
>> /Volumes/workspace/dev/sources/maven/maven-core/target/maven-core/apache-maven/target/apache-maven-3.8.6-SNAPSHOT-bin.tar.gz
>> [INFO]
>> [INFO] --- maven-checkstyle-plugin:3.0.0:check (checkstyle-check) @
>> apache-maven ---
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Reactor Summary for Apache Maven 3.8.6-SNAPSHOT:
>> [INFO]
>> [INFO] Apache Maven ....................................... SUCCESS [
>> 3.220 s]
>> [INFO] Maven Model ........................................ SUCCESS [
>> 4.734 s]
>> [INFO] Maven Artifact ..................................... SUCCESS [
>> 3.142 s]
>> [INFO] Maven Plugin API ................................... SUCCESS [
>> 1.763 s]
>> [INFO] Maven Builder Support .............................. SUCCESS [
>> 1.288 s]
>> [INFO] Maven Model Builder ................................ SUCCESS [
>> 5.129 s]
>> [INFO] Maven Settings ..................................... SUCCESS [
>> 0.694 s]
>> [INFO] Maven Settings Builder ............................. SUCCESS [
>> 1.730 s]
>> [INFO] Maven Repository Metadata Model .................... SUCCESS [
>> 1.470 s]
>> [INFO] Maven Artifact Resolver Provider ................... SUCCESS [
>> 4.255 s]
>> [INFO] Maven Core ......................................... SUCCESS [
>> 19.075 s]
>> [INFO] Maven SLF4J Simple Provider ........................ SUCCESS [
>> 0.972 s]
>> [INFO] Maven Embedder ..................................... SUCCESS [
>> 4.293 s]
>> [INFO] Maven Compat ....................................... SUCCESS [
>> 10.350 s]
>> [INFO] Apache Maven Distribution .......................... SUCCESS [
>> 4.898 s]
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD SUCCESS
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 01:07 min
>> [INFO] Finished at: 2022-03-22T20:19:24+10:00
>> [INFO]
>> ------------------------------------------------------------------------
>>
>>
>>
>>
>> even CI is green
>>
>> https://ci-maven.apache.org/job/Maven/job/maven-box/job/maven/job/first-module-fail/
>> so all looks good to merge a PR...
>> but hey in fact a module has not run any tests...
>> Do you really think it's right? (yes this problem exists for a very very
>> long time but we can choose to fix problems even after a long period)
>> Does your PR fix that?
>>
>>
>>
>> >
>> >
>> >
>> > On Sat, Mar 19, 2022 at 12:17 PM Slawomir Jaranowski <
>> > s.jaranowski@gmail.com>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > I start to feel that we first try to define technical details but we
>> > don't
>> > > know for what.
>> > > Each business requirement should be resolved as simply as possible
>> from a
>> > > technical perspective.
>> > > In other ways we will start to build complicated and many features
>> which
>> > > probably were not needed at all.
>> > >
>> > > Please first define business requirements with examples, next we can
>> find
>> > > technical ways to cover it.
>> > >
>> > > Example scenario:
>> > >
>> > > Businesses requirement:
>> > > We want to collect history of test result in order to detect flaky
>> > tests,
>> > > analize tests execution time, etc
>> > >
>> > > Given:
>> > > - build is done by CI system
>> > > - there is system which collect reports for executed tests and build
>> > > history of it
>> > > - there are broken tests in build - we know them and accept (eg, on
>> some
>> > > os)
>> > >
>> > > When:
>> > > - build finished with success on CI
>> > >
>> > > Then:
>> > > - all tests result are reported
>> > > - failed tests are reported
>> > >
>> > >
>> > >
>> > >
>> > > sob., 19 mar 2022 o 06:27 Christoph Läubrich <ma...@laeubi-soft.de>
>> > > napisał(a):
>> > >
>> > > > Hi Tibor,
>> > > >
>> > > > just to make it clear, I don't talkin about log-levels here but how
>> to
>> > > > intepret/handle a failure/failing/crashing test.
>> > > >
>> > > >
>> > > > Am 18.03.22 um 21:54 schrieb Tibor Digaňa:
>> > > > > Christoph, just let me briefly explain the log level hierarchy.
>> > > > > If you select WARN log level, then ERROR can be printed too.
>> > > > > Similar with INFO, means that WARN and ERROR would be printed as
>> > well.
>> > > > >
>> > > > > The real decision making in the plugin is a bit more complicated,
>> see
>> > > the
>> > > > > pull request https://github.com/apache/maven-surefire/pull/478
>> > > > > After I have replied to this email, I would verify the behavior of
>> > the
>> > > > > plugin simulating OOM.
>> > > > >
>> > > > >
>> > > > > T
>> > > > >
>> > > > >
>> > > > > On Fri, Mar 18, 2022 at 5:42 AM Christoph Läubrich <
>> > > mail@laeubi-soft.de>
>> > > > > wrote:
>> > > > >
>> > > > >> No I think more about
>> > > > >>
>> > > > >> if (severity == "WARN") {
>> > > > >> log.warn("There is something wrong");
>> > > > >> } else if (severity == "ERROR") {
>> > > > >> throw new MojoFailureException("...") {
>> > > > >> } else {
>> > > > >> throw new MojoExecutionException("...") {
>> > > > >> }
>> > > > >>
>> > > > >> That way the plugin can handle any error/failure/... in a way
>> that
>> > the
>> > > > >> user can "downgrade" a certain problem if desired.
>> > > > >>
>> > > > >>
>> > > > >> Am 18.03.22 um 00:06 schrieb Tibor Digana:
>> > > > >>> @Christoph
>> > > > >>> FATAL , WARN , ERROR
>> > > > >>> They are log levels?
>> > > > >>> The plugin does not control the log level after caught
>> exception in
>> > > > Maven
>> > > > >>> core. The Maven Core does!
>> > > > >>> I think it's time to make a demo.
>> > > > >>>
>> > > > >>> On Thu, Mar 17, 2022 at 6:21 AM Christoph Läubrich <
>> > > > mail@laeubi-soft.de>
>> > > > >>> wrote:
>> > > > >>>
>> > > > >>>> Hi Tibor,
>> > > > >>>>
>> > > > >>>> it shouldn't be to hard to guess another property like
>> > > > >>>>
>> > > > >>>> maven.test.jvmcrash=FATAL
>> > > > >>>> maven.test.failure=WARN
>> > > > >>>> maven.test.error=ERROR
>> > > > >>>>
>> > > > >>>> :-)
>> > > > >>>>
>> > > > >>>> Am 16.03.22 um 12:25 schrieb Tibor Digana:
>> > > > >>>>> Hi Christoph,
>> > > > >>>>>
>> > > > >>>>> Such a granularity with error/failure might be also an
>> additional
>> > > > >>>>> requirement but still you miss the third option to JVM error
>> > which
>> > > is
>> > > > >>>>> different from test error/failure - they don;t have the same
>> > > meaning.
>> > > > >>>>>
>> > > > >>>>> T
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>> On Mon, Mar 14, 2022 at 10:57 AM Christoph Läubrich <
>> > > > >> mail@laeubi-soft.de
>> > > > >>>>>
>> > > > >>>>> wrote:
>> > > > >>>>>
>> > > > >>>>>> Just wondering but maybe it would be better to configure the
>> > > > severity
>> > > > >>>>>> instead of a true/false, something like:
>> > > > >>>>>>
>> > > > >>>>>> maven.test.failure=WARN
>> > > > >>>>>> maven.test.error=ERROR
>> > > > >>>>>>
>> > > > >>>>>> would only warn about failing tests but thrw exception if
>> > starting
>> > > > the
>> > > > >>>>>> test fails?
>> > > > >>>>>>
>> > > > >>>>>> Am 14.03.22 um 10:52 schrieb Tibor Digana:
>> > > > >>>>>>> Romain, it is not a bug.
>> > > > >>>>>>> Don't consider this as a bug. It was a feature request for
>> > change
>> > > > by
>> > > > >>>>>>> Olivier, and not a bug.
>> > > > >>>>>>> I closed both issues years ago but not because of ignorance
>> but
>> > > > >> because
>> > > > >>>>>> the
>> > > > >>>>>>> appearance of the exceptional behavior is a wrong compromise
>> > and
>> > > > >> which
>> > > > >>>>>> does
>> > > > >>>>>>> not help anyone and even it makes the situation worse
>> because
>> > > > >> typically
>> > > > >>>>>>> other group of users would fire a new Jira tickets. You
>> would
>> > not
>> > > > be
>> > > > >>>> able
>> > > > >>>>>>> to satisfy two contradictory parties which have completely
>> > > > different
>> > > > >>>>>>> opinions, and so we use to introduce new params and this
>> way we
>> > > > >> satisfy
>> > > > >>>>>>> both parties, they may combine the params as they wish, and
>> > > > everybody
>> > > > >>>>>> would
>> > > > >>>>>>> be happy and nobody would claim. Many technical solutions
>> might
>> > > > >> exist,
>> > > > >>>>>> e.g.
>> > > > >>>>>>> param=boolean|string or new param=boolean, whatever.
>> > > > >>>>>>>
>> > > > >>>>>>> The truth is that this problem with (java --add-reads ...)
>> > > happened
>> > > > >> in
>> > > > >>>>>> our
>> > > > >>>>>>> ASF environment on Java 8 which in good configuration would
>> not
>> > > > >> happen
>> > > > >>>>>> and
>> > > > >>>>>>> should not.
>> > > > >>>>>>> It is not right way that we abuse
>> "maven.test.failure.ignore"
>> > > which
>> > > > >>>> would
>> > > > >>>>>>> tell us "Hey, you have illegal configuration in your build
>> > system
>> > > > and
>> > > > >>>> it
>> > > > >>>>>>> would not work by JDK design", it is not the goal of the
>> plugin
>> > > to
>> > > > >> tell
>> > > > >>>>>> you
>> > > > >>>>>>> that you have configured the build wrong because it won't
>> and
>> > the
>> > > > >> same
>> > > > >>>>>>> configuration of the plugin (not the build) says "ignore
>> the
>> > > > error".
>> > > > >>>>>>> Yesterday I discussed this problem with Herve and we
>> > > independently
>> > > > >>>>>> observed
>> > > > >>>>>>> equal opinions and that's not everything because we
>> understood
>> > > that
>> > > > >> the
>> > > > >>>>>>> purpose of the config parameter is to not throw MOJO
>> exception
>> > > > which
>> > > > >> is
>> > > > >>>>>>> right, but the exceptional behavior should have an exact new
>> > > param
>> > > > >> for
>> > > > >>>>>> the
>> > > > >>>>>>> exact use case.
>> > > > >>>>>>> SPI for this parameter is too much because no user would
>> > > implement
>> > > > it
>> > > > >>>>>> for a
>> > > > >>>>>>> trivial parameter;; the SPI is used to be implemented by
>> > > frameworks
>> > > > >> or
>> > > > >>>>>>> systems or application servers but this is not our case.
>> > > > >>>>>>>
>> > > > >>>>>>>
>> > > > >>>>>>>
>> > > > >>>>>>>
>> > > > >>>>>>> On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <
>> > > > >>>>>> rmannibucau@gmail.com>
>> > > > >>>>>>> wrote:
>> > > > >>>>>>>
>> > > > >>>>>>>> +1
>> > > > >>>>>>>> if it is to investigate a CI issue, it is generally easy to
>> > add
>> > > > >> debug
>> > > > >>>>>>>> insights (by code or agent) so a SPI sounds like the sanest
>> > for
>> > > > the
>> > > > >>>>>> plugin
>> > > > >>>>>>>> to me.
>> > > > >>>>>>>>
>> > > > >>>>>>>> Romain Manni-Bucau
>> > > > >>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> > > > >>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>> > > > >>>>>>>> <http://rmannibucau.wordpress.com> | Github <
>> > > > >>>>>>>> https://github.com/rmannibucau> |
>> > > > >>>>>>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
>> > > > >>>>>>>> <
>> > > > >>>>>>>>
>> > > > >>>>>>
>> > > > >>>>
>> > > > >>
>> > > >
>> > >
>> >
>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>> > > > >>>>>>>>>
>> > > > >>>>>>>>
>> > > > >>>>>>>>
>> > > > >>>>>>>> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <
>> > > gnodet@apache.org>
>> > > > a
>> > > > >>>>>> écrit
>> > > > >>>>>>>> :
>> > > > >>>>>>>>
>> > > > >>>>>>>>> If that's not currently possible, maybe a SPI should be
>> > > provided
>> > > > so
>> > > > >>>>>> that
>> > > > >>>>>>>>> people can use plug in extensions to analyze the test
>> result
>> > > and
>> > > > >>>>>> override
>> > > > >>>>>>>>> it if necessary (transforming an error into a warning,
>> > storing
>> > > > >>>> results
>> > > > >>>>>>>> in a
>> > > > >>>>>>>>> way which is easier to use by other tools later...) ?
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> Guillaume
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <
>> > > > >>>> mail@laeubi-soft.de>
>> > > > >>>>>> a
>> > > > >>>>>>>>> écrit :
>> > > > >>>>>>>>>
>> > > > >>>>>>>>>> I also agree that the test at least should run, we use
>> this
>> > > > >> property
>> > > > >>>>>> to
>> > > > >>>>>>>>>> run the test and produce result and later on have a
>> > buildstep
>> > > > that
>> > > > >>>>>>>>>> analyze the results (and probably fail the build job).
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> As it is not recommend, I wonder what is the recommended
>> way
>> > > to
>> > > > >>>>>> archive
>> > > > >>>>>>>>>> something similar?
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>> Am 14.03.22 um 06:29 schrieb Olivier Lamy:
>> > > > >>>>>>>>>>> On Mon, 14 Mar 2022 at 11:55, Tibor Digana <
>> > > > >> tibordigana@apache.org
>> > > > >>>>>
>> > > > >>>>>>>>>> wrote:
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>>> In case of the user property
>> *maven.test.failure.ignore*
>> > the
>> > > > >> MOJO
>> > > > >>>>>>>> must
>> > > > >>>>>>>>>> not
>> > > > >>>>>>>>>>>> throw any exception which is interpreted by the Maven
>> Core
>> > > as
>> > > > >>>> BUILD
>> > > > >>>>>>>>>>>> SUCCESS.
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> This is a very simple reduction of the problem
>> description.
>> > > > >>>>>>>>>>> The documentation here
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>
>> > > > >>>>>>>>
>> > > > >>>>>>
>> > > > >>>>
>> > > > >>
>> > > >
>> > >
>> >
>> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
>> > > > >>>>>>>>>>> says
>> > > > >>>>>>>>>>> "Set this to "true" to ignore a failure during testing.
>> Its
>> > > use
>> > > > >> is
>> > > > >>>>>>>> NOT
>> > > > >>>>>>>>>>> RECOMMENDED, but quite convenient on occasion"
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> Personally, I understand this to ignore failures in
>> junit
>> > > > results
>> > > > >>>> BUT
>> > > > >>>>>>>>> at
>> > > > >>>>>>>>>>> least I want the tests to run.
>> > > > >>>>>>>>>>> I guess this is how our users use this feature (feature
>> we
>> > do
>> > > > not
>> > > > >>>>>>>>>> recommend
>> > > > >>>>>>>>>>> by the way...)
>> > > > >>>>>>>>>>> And this is perfectly explained here
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>
>> > > > >>>>>>>>
>> > > > >>>>>>
>> > > > >>>>
>> > > > >>
>> > > >
>> > >
>> >
>> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
>> > > > >>>>>>>>>>> there is a difference between ignoring some junit
>> failures
>> > > and
>> > > > >>>>>>>>> ignoring a
>> > > > >>>>>>>>>>> configuration error because some jvm args has been
>> > > > misconfigured
>> > > > >>>> for
>> > > > >>>>>>>>> many
>> > > > >>>>>>>>>>> reasons and surefire cannot start.
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> If I follow your reasoning ("MOJO must not throw any
>> > > exception
>> > > > ")
>> > > > >>>> we
>> > > > >>>>>>>>>> should
>> > > > >>>>>>>>>>> ignore every surefire configuration error and keep
>> running
>> > > the
>> > > > >>>> build
>> > > > >>>>>>>>>> until
>> > > > >>>>>>>>>>> the end and says BUILD SUCCESS
>> > > > >>>>>>>>>>> what about
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> mvn test -Dsurefire.rerunFailingTestsCount=notanumber
>> > > > >>>>>>>>>>> -Dmaven.test.failure.ignore=true
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> we should not throw any exceptions as you said below and
>> > keep
>> > > > >>>> saying
>> > > > >>>>>>>>>> BUILD
>> > > > >>>>>>>>>>> SUCCESS?
>> > > > >>>>>>>>>>> argLine is a configuration element as any others so it
>> > should
>> > > > >> fail
>> > > > >>>>>>>>>> directly
>> > > > >>>>>>>>>>> and not be ignored especially when the surefire plugin
>> > cannot
>> > > > >> run.
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>>> We have received an internal requirement to change the
>> > > > behavior
>> > > > >> of
>> > > > >>>>>>>> the
>> > > > >>>>>>>>>> user
>> > > > >>>>>>>>>>>> property *maven.test.failure.ignore* so that the
>> behavior
>> > > will
>> > > > >>>> have
>> > > > >>>>>>>>> one
>> > > > >>>>>>>>>>>> exception.
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> Suppose that you have JDK 1.8 but you use:
>> > > > >>>>>>>>>>>> /jre/java --add-reads ...
>> > > > >>>>>>>>>>>> The outcome is JVM exit with an error message.
>> > > > >>>>>>>>>>>> I agree with Herve who also says that
>> > > > >> *maven.test.failure.ignore*
>> > > > >>>>>>>>>> should
>> > > > >>>>>>>>>>>> not allow the MOJO to throw the exception. It is not a
>> > > typical
>> > > > >> JVM
>> > > > >>>>>>>>>>>> segmentation fault or another JVM crash where we
>> cannot do
>> > > > >>>> anything
>> > > > >>>>>>>>>> about
>> > > > >>>>>>>>>>>> it, and where the entire build would crash in the
>> command
>> > > line
>> > > > >>>> which
>> > > > >>>>>>>>>>>> of course means that the build cannot normally be
>> > > interpreted
>> > > > as
>> > > > >>>>>>>> BUILD
>> > > > >>>>>>>>>>>> SUCCESS. So we are still on the same level of failures
>> > > related
>> > > > >> to
>> > > > >>>>>>>> the
>> > > > >>>>>>>>>> test
>> > > > >>>>>>>>>>>> purposes.
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> On the other hand, Olivier has reopened the issues
>> > > > SUREFIRE-1426
>> > > > >>>> and
>> > > > >>>>>>>>>>>> SUREFIRE-1681 where the exceptional behavior of the
>> > feature
>> > > is
>> > > > >>>>>>>>> expected.
>> > > > >>>>>>>>>>>> This is exactly the reason why I closed these tickets
>> > > several
>> > > > >>>> years
>> > > > >>>>>>>>> ago
>> > > > >>>>>>>>>> and
>> > > > >>>>>>>>>>>> my proposal was to not to have any exceptions in the
>> > feature
>> > > > >>>>>>>> behavior
>> > > > >>>>>>>>>> and
>> > > > >>>>>>>>>>>> the proposal was to introduce a new user property for
>> > exact
>> > > > use
>> > > > >>>>>>>> cases.
>> > > > >>>>>>>>>>>> The next idea, which comes from two developers, would
>> > > provide
>> > > > us
>> > > > >>>>>>>> with
>> > > > >>>>>>>>>> the
>> > > > >>>>>>>>>>>> same non exceptional and exact behavior of the user
>> > property
>> > > > >>>>>>>>>>>> *maven.test.failure.ignore* but it would also provide
>> us
>> > > with
>> > > > >> new
>> > > > >>>>>>>> user
>> > > > >>>>>>>>>>>> property in the case with fine grade control of the
>> build
>> > > > >> errors,
>> > > > >>>>>>>> e.g.
>> > > > >>>>>>>>>>>> *maven.test.jvm.error.ignore*.
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> with a default of?
>> > > > >>>>>>>>>>> honestly I just see this new parameter as introducing
>> more
>> > > > >>>> complexity
>> > > > >>>>>>>>> in
>> > > > >>>>>>>>>> an
>> > > > >>>>>>>>>>> already very complex code and not worth it.
>> > > > >>>>>>>>>>> again read both issue's comments and my comments.
>> > > > >>>>>>>>>>> Please try to have a user POV and think about making our
>> > > users'
>> > > > >>>>>>>>>> experience
>> > > > >>>>>>>>>>> more simple.
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> This should be very simple:
>> > > > >>>>>>>>>>> If surefire forked jvm cannot start it's build error and
>> > > cannot
>> > > > >> run
>> > > > >>>>>>>> any
>> > > > >>>>>>>>>>> tests, it's a problem users want to know immediately
>> > because
>> > > it
>> > > > >> can
>> > > > >>>>>>>> be
>> > > > >>>>>>>>>> for
>> > > > >>>>>>>>>>> a lot of reasons: wrong argLine, not enough memory on
>> the
>> > > > system
>> > > > >>>>>>>> etc...
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> AND AGAIN it is very different from ignoring junit
>> result
>> > > > >> failures.
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> Try to look at a user point of view and think about
>> what is
>> > > the
>> > > > >> use
>> > > > >>>>>>>>> case
>> > > > >>>>>>>>>> of
>> > > > >>>>>>>>>>> the property maven.test.failure.ignore=true, I guess
>> 99% of
>> > > the
>> > > > >>>> time,
>> > > > >>>>>>>>>> users
>> > > > >>>>>>>>>>> wants to run all their tests (even on a CI with
>> different
>> > > > matrix)
>> > > > >>>> so
>> > > > >>>>>>>>> they
>> > > > >>>>>>>>>>> can collect all the tests results which has runned to
>> see
>> > if
>> > > > >> there
>> > > > >>>> is
>> > > > >>>>>>>>> any
>> > > > >>>>>>>>>>> issue for some combination of their matrix and
>> eventually
>> > > turn
>> > > > >> the
>> > > > >>>>>>>>> result
>> > > > >>>>>>>>>>> as unstable (this is a very typical use case in Jenkins
>> and
>> > > was
>> > > > >>>> even
>> > > > >>>>>>>> a
>> > > > >>>>>>>>>>> built in feature of the previous Jenkins Maven plugin)
>> > > > >>>>>>>>>>> BUT if for any reasons one of the module do not start
>> his
>> > > tests
>> > > > >>>>>>>> because
>> > > > >>>>>>>>>> the
>> > > > >>>>>>>>>>> jvm cannot be start the users will not see that and
>> will be
>> > > > >> totally
>> > > > >>>>>>>>> blind
>> > > > >>>>>>>>>>> until maybe someone look inside a very very large log
>> file
>> > > > (which
>> > > > >>>>>>>> means
>> > > > >>>>>>>>>>> probably never)
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> Long story short as my experience as a user facing this
>> > > > >>>> problem/bug:
>> > > > >>>>>>>>>>> I had the case on a very large multi modules (~250
>> modules)
>> > > > build
>> > > > >>>> of
>> > > > >>>>>>>> a
>> > > > >>>>>>>>>> very
>> > > > >>>>>>>>>>> used open source project.
>> > > > >>>>>>>>>>> I was using this maven.test.failure.ignore property but
>> one
>> > > of
>> > > > >> the
>> > > > >>>>>>>>>> modules
>> > > > >>>>>>>>>>> had a bad jpms configuration for a jdk17 profile on the
>> CI.
>> > > > >>>>>>>>>>> The build has a matrix of 2 os and 4 jdks and different
>> > maven
>> > > > run
>> > > > >>>>>>>> which
>> > > > >>>>>>>>>>> means around ~60k tests and a Jenkins log file about 40M
>> > > > >>>>>>>>>>> So because of this property the build was not failing
>> and
>> > > kept
>> > > > >>>> saying
>> > > > >>>>>>>>>> BUILD
>> > > > >>>>>>>>>>> SUCCESS for weeks/months and basically not testing one
>> > module
>> > > > >> with
>> > > > >>>>>>>> jdk
>> > > > >>>>>>>>>> 17...
>> > > > >>>>>>>>>>> And frankly you do not dig into a log file of 32M after
>> > each
>> > > > run
>> > > > >>>>>>>>>> especially
>> > > > >>>>>>>>>>> when it says success...
>> > > > >>>>>>>>>>> 3 days after the first release claiming supporting jdk
>> 17
>> > we
>> > > > >>>>>>>> received a
>> > > > >>>>>>>>>> bug
>> > > > >>>>>>>>>>> report about a something not working with jdk17....
>> > > > >>>>>>>>>>> and guess what? Where was this feature supposed to be
>> > tested?
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>> so I frankly believe we do not need a complex new
>> property,
>> > > in
>> > > > >> this
>> > > > >>>>>>>>> case
>> > > > >>>>>>>>>>> this should fail directly because this will improve user
>> > > > >>>> experience.
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> I would like to open the discussion on this topic.
>> You're
>> > > > >> welcome!
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>> Cheers
>> > > > >>>>>>>>>>>> Tibor
>> > > > >>>>>>>>>>>>
>> > > > >>>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>
>> > > ---------------------------------------------------------------------
>> > > > >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> > > > >>>>>>>>>> For additional commands, e-mail:
>> dev-help@maven.apache.org
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>>
>> > > > >>>>>>>>>
>> > > > >>>>>>>>> --
>> > > > >>>>>>>>> ------------------------
>> > > > >>>>>>>>> Guillaume Nodet
>> > > > >>>>>>>>>
>> > > > >>>>>>>>
>> > > > >>>>>>>
>> > > > >>>>>>
>> > > > >>>>>>
>> > > >
>> ---------------------------------------------------------------------
>> > > > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> > > > >>>>>> For additional commands, e-mail: dev-help@maven.apache.org
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>
>> > > > >>>>
>> > > > >>>>
>> > > ---------------------------------------------------------------------
>> > > > >>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> > > > >>>> For additional commands, e-mail: dev-help@maven.apache.org
>> > > > >>>>
>> > > > >>>>
>> > > > >>>
>> > > > >>
>> > > > >>
>> > ---------------------------------------------------------------------
>> > > > >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> > > > >> For additional commands, e-mail: dev-help@maven.apache.org
>> > > > >>
>> > > > >>
>> > > > >
>> > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> > > > For additional commands, e-mail: dev-help@maven.apache.org
>> > > >
>> > > >
>> > >
>> > > --
>> > > Sławomir Jaranowski
>> > >
>> >
>>
>
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Tibor Digana <ti...@apache.org>.
Hi Olivier, I have used the Maven branch according to your instructions. I
have used the snapshot version of the plugin and JDK 1.8.0u05.
Both implementations end up with the same build result. The only difference
is the number of IF-Else branches in the implementation.
I started Maven build twice for your and mine impl with these two configs:
<argLine>-Xmx1m -XX:MaxPermSize=1m</argLine>
<argLine>-FFFOOOOO</argLine>
1. The first config would end up with BUILD SUCCESS and it is expected
because JUnit caught the OutOfMemoryError "during testing" and the plugin
has reported it. The OOM did not happen in Surefire with my use case which
we cannot guarantee what the JVM would do in another real scenario.
2. The second config is the *JVM initialization error* which happens
"before testing" and the build fails with a failure as expected, BUILD
FAILURE.
Pls comment on these build results. These are expected by us, we can agree
on them, right?
1. case
*$ mvn verify -nsu -Dmaven.test.failure.ignore*
[INFO] --- maven-surefire-plugin:3.0.0-M6-SNAPSHOT:test (default-test) @
maven-model ---
[INFO] Using auto detected provider
org.apache.maven.surefire.junit4.JUnit4Provider
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.maven.model.ActivationFileTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.583 s - in org.apache.maven.model.ActivationFileTest
[INFO] Running org.apache.maven.model.ActivationOSTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.099 s - in org.apache.maven.model.ActivationOSTest
[INFO] Running org.apache.maven.model.ActivationPropertyTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.136 s - in org.apache.maven.model.ActivationPropertyTest
[INFO] Running org.apache.maven.model.ActivationTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.118 s - in org.apache.maven.model.ActivationTest
[INFO] Running org.apache.maven.model.BuildTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.177 s - in org.apache.maven.model.BuildTest
[INFO] Running org.apache.maven.model.CiManagementTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.22
s - in org.apache.maven.model.CiManagementTest
[INFO] Running org.apache.maven.model.ContributorTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.171 s - in org.apache.maven.model.ContributorTest
[INFO] Running org.apache.maven.model.DependencyManagementTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.211 s - in org.apache.maven.model.DependencyManagementTest
[INFO] Running org.apache.maven.model.DependencyTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.163 s - in org.apache.maven.model.DependencyTest
[INFO] Running org.apache.maven.model.DeploymentRepositoryTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.198 s - in org.apache.maven.model.DeploymentRepositoryTest
[INFO] Running org.apache.maven.model.DeveloperTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.201 s - in org.apache.maven.model.DeveloperTest
[INFO] Running org.apache.maven.model.DistributionManagementTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.193 s - in org.apache.maven.model.DistributionManagementTest
[INFO] Running org.apache.maven.model.ExclusionTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.218 s - in org.apache.maven.model.ExclusionTest
[INFO] Running org.apache.maven.model.ExtensionTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.194 s - in org.apache.maven.model.ExtensionTest
[INFO] Running org.apache.maven.model.IssueManagementTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.206 s - in org.apache.maven.model.IssueManagementTest
[INFO] Running org.apache.maven.model.LicenseTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.216 s - in org.apache.maven.model.LicenseTest
[INFO] Running org.apache.maven.model.MailingListTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.257 s - in org.apache.maven.model.MailingListTest
[INFO] Running org.apache.maven.model.merge.ModelMergerTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
6.742 s <<< FAILURE! - in org.apache.maven.model.merge.ModelMergerTest
[ERROR]
org.apache.maven.model.merge.ModelMergerTest.testMergedModelSerialization
Time elapsed: 2.502 s <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
at
org.apache.maven.model.merge.ModelMergerTest.testMergedModelSerialization(ModelMergerTest.java:43)
[INFO] Running org.apache.maven.model.ModelTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
2.079 s - in org.apache.maven.model.ModelTest
[INFO] Running org.apache.maven.model.NotifierTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.655 s - in org.apache.maven.model.NotifierTest
[INFO] Running org.apache.maven.model.OrganizationTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.512 s - in org.apache.maven.model.OrganizationTest
[INFO] Running org.apache.maven.model.ParentTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.367 s - in org.apache.maven.model.ParentTest
[INFO] Running org.apache.maven.model.PluginConfigurationTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.715 s - in org.apache.maven.model.PluginConfigurationTest
[INFO] Running org.apache.maven.model.PluginContainerTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.316 s - in org.apache.maven.model.PluginContainerTest
[INFO] Running org.apache.maven.model.PluginExecutionTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.451 s - in org.apache.maven.model.PluginExecutionTest
[INFO] Running org.apache.maven.model.PluginManagementTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.972 s - in org.apache.maven.model.PluginManagementTest
[INFO] Running org.apache.maven.model.PluginTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.502 s - in org.apache.maven.model.PluginTest
[INFO] Running org.apache.maven.model.PrerequisitesTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.663 s - in org.apache.maven.model.PrerequisitesTest
[INFO] Running org.apache.maven.model.ProfileTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.202 s - in org.apache.maven.model.ProfileTest
[INFO] Running org.apache.maven.model.RelocationTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.562 s - in org.apache.maven.model.RelocationTest
[INFO] Running org.apache.maven.model.ReportingTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.487 s - in org.apache.maven.model.ReportingTest
[INFO] Running org.apache.maven.model.ReportPluginTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.599 s - in org.apache.maven.model.ReportPluginTest
[INFO] Running org.apache.maven.model.ReportSetTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.287 s - in org.apache.maven.model.ReportSetTest
[INFO] Running org.apache.maven.model.RepositoryPolicyTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.1
s - in org.apache.maven.model.RepositoryPolicyTest
[INFO] Running org.apache.maven.model.RepositoryTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.102 s - in org.apache.maven.model.RepositoryTest
[INFO] Running org.apache.maven.model.ResourceTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.85
s - in org.apache.maven.model.ResourceTest
[INFO] Running org.apache.maven.model.ScmTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.237 s - in org.apache.maven.model.ScmTest
[INFO] Running org.apache.maven.model.SiteTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.074 s - in org.apache.maven.model.SiteTest
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1m;
support was removed in 8.0
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ModelMergerTest.testMergedModelSerialization:43 » OutOfMemory
Java heap space
[INFO]
[ERROR] Tests run: 149, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[ERROR]
Please refer to
C:\vcs\github\maven-core\maven-model\target\surefire-reports for the
individual test results.
Please refer to dump files (if any exist) [date].dump,
[date]-jvmRun[N].dump and [date].dumpstream.
[INFO]
[INFO] --- maven-jar-plugin:3.2.2:jar (default-jar) @ maven-model ---
[INFO] Building jar:
C:\vcs\github\maven-core\maven-model\target\maven-model-3.8.6-SNAPSHOT.jar
...
...
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Maven 3.8.6-SNAPSHOT:
[INFO]
[INFO] Apache Maven ....................................... SUCCESS [
16.395 s]
[INFO] Maven Model ........................................ SUCCESS [
57.064 s]
[INFO] Maven Artifact ..................................... SUCCESS [
6.102 s]
[INFO] Maven Plugin API ................................... SUCCESS [
7.027 s]
[INFO] Maven Builder Support .............................. SUCCESS [
2.486 s]
[INFO] Maven Model Builder ................................ SUCCESS [
15.773 s]
[INFO] Maven Settings ..................................... SUCCESS [
2.945 s]
[INFO] Maven Settings Builder ............................. SUCCESS [
5.509 s]
[INFO] Maven Repository Metadata Model .................... SUCCESS [
3.903 s]
[INFO] Maven Artifact Resolver Provider ................... SUCCESS [
13.878 s]
[INFO] Maven Core ......................................... SUCCESS [01:19
min]
[INFO] Maven SLF4J Simple Provider ........................ SUCCESS [
3.678 s]
[INFO] Maven Embedder ..................................... SUCCESS [
15.397 s]
[INFO] Maven Compat ....................................... SUCCESS [
53.385 s]
[INFO] Apache Maven Distribution .......................... SUCCESS [
27.397 s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 05:11 min
[INFO] Finished at: 2022-03-23T01:15:10+01:00
[INFO]
------------------------------------------------------------------------
2. case
*$ mvn verify -nsu -Dmaven.test.failure.ignore*
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
Unrecognized option: -FFFOOOOO
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Maven 3.8.6-SNAPSHOT:
[INFO]
[INFO] Apache Maven ....................................... SUCCESS [
18.186 s]
[INFO] Maven Model ........................................ FAILURE [
12.388 s]
[INFO] Maven Artifact ..................................... SKIPPED
... ... ...
[INFO] Apache Maven Distribution .......................... SKIPPED
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
On Tue, Mar 22, 2022 at 11:44 AM Olivier Lamy <ol...@apache.org> wrote:
> On Tue, 22 Mar 2022 at 14:40, Tibor Digana <ti...@apache.org> wrote:
>
> > Sorry for late reply.
> >
> > I have created a demo project
> > https://github.com/Tibor17/maven.test.failure.ignore/ which simulates
> > OOM.
> >
> > According to the definition of the parameter "maven.failure.test.ignore"
> > the failures (also errors) should be ignored during testing.
> > The word "during" is important because the time when it was a failure
> > before JVM startup it was not a test failure - it is MOJO failure. The
> > author who wrote this config param knew this difference.
> > This perfectly fits JVM illegal arguments like -Xxxx.
> > But I have investigated OOM for the same reason with one JVM. My
> suspicion
> > was not confirmed, see the next...
> > If there are two JVMs, we are breaking the JVM during testing. It is a
> very
> > unlikely situation that the JVM argument would not fail in JVM1 but it
> > would fail in JVM2 later. It may happen only if the user
> > uses @surefire.forkCount@ interpolation but it is again very unlikely
> and
> > the users use it in current working directory configuration.
> >
> > I understand Olivier's PR but I have one objection to the code branching
> in
> > the class *SurefireHelper*. It is based on 3 levels but I have to
> disagree
> > due to there still should be 2 levels as before:
> > 1. ignored failures (my change is only here - yellow - only added
> throwing
> > exception)
> > 2. processes failures
> >
> > and so, I have created a PR
> > https://github.com/apache/maven-surefire/pull/496
> > Pls have a look into it, thx.
> > This diff looks very simple:
> >
> > if ( reportParameters.isTestFailureIgnore() )
> > {
> > String errorMessage = createErrorMessage( reportParameters,
> > result, firstForkException );
> >
> > if ( firstForkException instanceof SurefireBooterForkException )
> > {
> > throw new MojoExecutionException( errorMessage,
> firstForkException
> > );
> >
> > }
> >
> > log.error( errorMessage );
> > }
> > else
> > {
> > throwException( reportParameters, result, firstForkException );
> > }
> >
> >
> >
> > *$ mvn test*
> >
> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> > 2.445 s <<< FAILURE! - in org.apache.maven.OOMTest
> > [ERROR] org.apache.maven.OOMTest.test Time elapsed: 2.063 s <<< ERROR!
> > java.lang.OutOfMemoryError: Java heap space
> > at org.apache.maven.OOMTest.test(OOMTest.java:15)
> > [INFO]
> > [INFO] Results:
> > [INFO]
> > [ERROR] Errors:
> > [ERROR] OOMTest.test:15 » OutOfMemory Java heap space
> > [INFO]
> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> > [INFO]
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] BUILD FAILURE
> > [INFO]
> > ------------------------------------------------------------------------
> >
> > *$ mvn test -Dmaven.test.failure.ignore*
> >
> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> > 2.472 s <<< FAILURE! - in org.apache.maven.OOMTest
> > [ERROR] org.apache.maven.OOMTest.test Time elapsed: 2.262 s <<< ERROR!
> > java.lang.OutOfMemoryError: Java heap space
> > at org.apache.maven.OOMTest.test(OOMTest.java:15)
> > [INFO]
> > [INFO] Results:
> > [INFO]
> > [ERROR] Errors:
> > [ERROR] OOMTest.test:15 » OutOfMemory Java heap space
> > [INFO]
> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> > [INFO]
> > [ERROR]
> >
> > Please refer to
> > C:\vcs\github\maven.test.failure.ignore\target\surefire-reports for the
> > individual test results.
> > Please refer to dump files (if any exist) [date].dump,
> > [date]-jvmRun[N].dump and [date].dumpstream.
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] BUILD SUCCESS
> > [INFO]
> > ------------------------------------------------------------------------
> >
>
> Please use a project with multiple modules, not a single one where you can
> see the line with OOME.
> Any project with a huge number of modules will hide this and nobody will
> ever notice it.
>
>
> git clone -b first-module-fail https://github.com/apache/maven maven-core
>
> cd maven-core
>
>
> [INFO] --- maven-assembly-plugin:3.3.0:single (create-distro-packages) @
> apache-maven ---
> [INFO] Reading assembly descriptor: src/main/assembly/bin.xml
> [INFO] Building zip:
>
> /Volumes/workspace/dev/sources/maven/maven-core/target/maven-core/apache-maven/target/apache-maven-3.8.6-SNAPSHOT-bin.zip
> [INFO] Building tar:
>
> /Volumes/workspace/dev/sources/maven/maven-core/target/maven-core/apache-maven/target/apache-maven-3.8.6-SNAPSHOT-bin.tar.gz
> [INFO]
> [INFO] --- maven-checkstyle-plugin:3.0.0:check (checkstyle-check) @
> apache-maven ---
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary for Apache Maven 3.8.6-SNAPSHOT:
> [INFO]
> [INFO] Apache Maven ....................................... SUCCESS [
> 3.220 s]
> [INFO] Maven Model ........................................ SUCCESS [
> 4.734 s]
> [INFO] Maven Artifact ..................................... SUCCESS [
> 3.142 s]
> [INFO] Maven Plugin API ................................... SUCCESS [
> 1.763 s]
> [INFO] Maven Builder Support .............................. SUCCESS [
> 1.288 s]
> [INFO] Maven Model Builder ................................ SUCCESS [
> 5.129 s]
> [INFO] Maven Settings ..................................... SUCCESS [
> 0.694 s]
> [INFO] Maven Settings Builder ............................. SUCCESS [
> 1.730 s]
> [INFO] Maven Repository Metadata Model .................... SUCCESS [
> 1.470 s]
> [INFO] Maven Artifact Resolver Provider ................... SUCCESS [
> 4.255 s]
> [INFO] Maven Core ......................................... SUCCESS [
> 19.075 s]
> [INFO] Maven SLF4J Simple Provider ........................ SUCCESS [
> 0.972 s]
> [INFO] Maven Embedder ..................................... SUCCESS [
> 4.293 s]
> [INFO] Maven Compat ....................................... SUCCESS [
> 10.350 s]
> [INFO] Apache Maven Distribution .......................... SUCCESS [
> 4.898 s]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 01:07 min
> [INFO] Finished at: 2022-03-22T20:19:24+10:00
> [INFO]
> ------------------------------------------------------------------------
>
>
>
>
> even CI is green
>
> https://ci-maven.apache.org/job/Maven/job/maven-box/job/maven/job/first-module-fail/
> so all looks good to merge a PR...
> but hey in fact a module has not run any tests...
> Do you really think it's right? (yes this problem exists for a very very
> long time but we can choose to fix problems even after a long period)
> Does your PR fix that?
>
>
>
> >
> >
> >
> > On Sat, Mar 19, 2022 at 12:17 PM Slawomir Jaranowski <
> > s.jaranowski@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I start to feel that we first try to define technical details but we
> > don't
> > > know for what.
> > > Each business requirement should be resolved as simply as possible
> from a
> > > technical perspective.
> > > In other ways we will start to build complicated and many features
> which
> > > probably were not needed at all.
> > >
> > > Please first define business requirements with examples, next we can
> find
> > > technical ways to cover it.
> > >
> > > Example scenario:
> > >
> > > Businesses requirement:
> > > We want to collect history of test result in order to detect flaky
> > tests,
> > > analize tests execution time, etc
> > >
> > > Given:
> > > - build is done by CI system
> > > - there is system which collect reports for executed tests and build
> > > history of it
> > > - there are broken tests in build - we know them and accept (eg, on
> some
> > > os)
> > >
> > > When:
> > > - build finished with success on CI
> > >
> > > Then:
> > > - all tests result are reported
> > > - failed tests are reported
> > >
> > >
> > >
> > >
> > > sob., 19 mar 2022 o 06:27 Christoph Läubrich <ma...@laeubi-soft.de>
> > > napisał(a):
> > >
> > > > Hi Tibor,
> > > >
> > > > just to make it clear, I don't talkin about log-levels here but how
> to
> > > > intepret/handle a failure/failing/crashing test.
> > > >
> > > >
> > > > Am 18.03.22 um 21:54 schrieb Tibor Digaňa:
> > > > > Christoph, just let me briefly explain the log level hierarchy.
> > > > > If you select WARN log level, then ERROR can be printed too.
> > > > > Similar with INFO, means that WARN and ERROR would be printed as
> > well.
> > > > >
> > > > > The real decision making in the plugin is a bit more complicated,
> see
> > > the
> > > > > pull request https://github.com/apache/maven-surefire/pull/478
> > > > > After I have replied to this email, I would verify the behavior of
> > the
> > > > > plugin simulating OOM.
> > > > >
> > > > >
> > > > > T
> > > > >
> > > > >
> > > > > On Fri, Mar 18, 2022 at 5:42 AM Christoph Läubrich <
> > > mail@laeubi-soft.de>
> > > > > wrote:
> > > > >
> > > > >> No I think more about
> > > > >>
> > > > >> if (severity == "WARN") {
> > > > >> log.warn("There is something wrong");
> > > > >> } else if (severity == "ERROR") {
> > > > >> throw new MojoFailureException("...") {
> > > > >> } else {
> > > > >> throw new MojoExecutionException("...") {
> > > > >> }
> > > > >>
> > > > >> That way the plugin can handle any error/failure/... in a way that
> > the
> > > > >> user can "downgrade" a certain problem if desired.
> > > > >>
> > > > >>
> > > > >> Am 18.03.22 um 00:06 schrieb Tibor Digana:
> > > > >>> @Christoph
> > > > >>> FATAL , WARN , ERROR
> > > > >>> They are log levels?
> > > > >>> The plugin does not control the log level after caught exception
> in
> > > > Maven
> > > > >>> core. The Maven Core does!
> > > > >>> I think it's time to make a demo.
> > > > >>>
> > > > >>> On Thu, Mar 17, 2022 at 6:21 AM Christoph Läubrich <
> > > > mail@laeubi-soft.de>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Hi Tibor,
> > > > >>>>
> > > > >>>> it shouldn't be to hard to guess another property like
> > > > >>>>
> > > > >>>> maven.test.jvmcrash=FATAL
> > > > >>>> maven.test.failure=WARN
> > > > >>>> maven.test.error=ERROR
> > > > >>>>
> > > > >>>> :-)
> > > > >>>>
> > > > >>>> Am 16.03.22 um 12:25 schrieb Tibor Digana:
> > > > >>>>> Hi Christoph,
> > > > >>>>>
> > > > >>>>> Such a granularity with error/failure might be also an
> additional
> > > > >>>>> requirement but still you miss the third option to JVM error
> > which
> > > is
> > > > >>>>> different from test error/failure - they don;t have the same
> > > meaning.
> > > > >>>>>
> > > > >>>>> T
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Mon, Mar 14, 2022 at 10:57 AM Christoph Läubrich <
> > > > >> mail@laeubi-soft.de
> > > > >>>>>
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Just wondering but maybe it would be better to configure the
> > > > severity
> > > > >>>>>> instead of a true/false, something like:
> > > > >>>>>>
> > > > >>>>>> maven.test.failure=WARN
> > > > >>>>>> maven.test.error=ERROR
> > > > >>>>>>
> > > > >>>>>> would only warn about failing tests but thrw exception if
> > starting
> > > > the
> > > > >>>>>> test fails?
> > > > >>>>>>
> > > > >>>>>> Am 14.03.22 um 10:52 schrieb Tibor Digana:
> > > > >>>>>>> Romain, it is not a bug.
> > > > >>>>>>> Don't consider this as a bug. It was a feature request for
> > change
> > > > by
> > > > >>>>>>> Olivier, and not a bug.
> > > > >>>>>>> I closed both issues years ago but not because of ignorance
> but
> > > > >> because
> > > > >>>>>> the
> > > > >>>>>>> appearance of the exceptional behavior is a wrong compromise
> > and
> > > > >> which
> > > > >>>>>> does
> > > > >>>>>>> not help anyone and even it makes the situation worse because
> > > > >> typically
> > > > >>>>>>> other group of users would fire a new Jira tickets. You would
> > not
> > > > be
> > > > >>>> able
> > > > >>>>>>> to satisfy two contradictory parties which have completely
> > > > different
> > > > >>>>>>> opinions, and so we use to introduce new params and this way
> we
> > > > >> satisfy
> > > > >>>>>>> both parties, they may combine the params as they wish, and
> > > > everybody
> > > > >>>>>> would
> > > > >>>>>>> be happy and nobody would claim. Many technical solutions
> might
> > > > >> exist,
> > > > >>>>>> e.g.
> > > > >>>>>>> param=boolean|string or new param=boolean, whatever.
> > > > >>>>>>>
> > > > >>>>>>> The truth is that this problem with (java --add-reads ...)
> > > happened
> > > > >> in
> > > > >>>>>> our
> > > > >>>>>>> ASF environment on Java 8 which in good configuration would
> not
> > > > >> happen
> > > > >>>>>> and
> > > > >>>>>>> should not.
> > > > >>>>>>> It is not right way that we abuse "maven.test.failure.ignore"
> > > which
> > > > >>>> would
> > > > >>>>>>> tell us "Hey, you have illegal configuration in your build
> > system
> > > > and
> > > > >>>> it
> > > > >>>>>>> would not work by JDK design", it is not the goal of the
> plugin
> > > to
> > > > >> tell
> > > > >>>>>> you
> > > > >>>>>>> that you have configured the build wrong because it won't and
> > the
> > > > >> same
> > > > >>>>>>> configuration of the plugin (not the build) says "ignore the
> > > > error".
> > > > >>>>>>> Yesterday I discussed this problem with Herve and we
> > > independently
> > > > >>>>>> observed
> > > > >>>>>>> equal opinions and that's not everything because we
> understood
> > > that
> > > > >> the
> > > > >>>>>>> purpose of the config parameter is to not throw MOJO
> exception
> > > > which
> > > > >> is
> > > > >>>>>>> right, but the exceptional behavior should have an exact new
> > > param
> > > > >> for
> > > > >>>>>> the
> > > > >>>>>>> exact use case.
> > > > >>>>>>> SPI for this parameter is too much because no user would
> > > implement
> > > > it
> > > > >>>>>> for a
> > > > >>>>>>> trivial parameter;; the SPI is used to be implemented by
> > > frameworks
> > > > >> or
> > > > >>>>>>> systems or application servers but this is not our case.
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <
> > > > >>>>>> rmannibucau@gmail.com>
> > > > >>>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> +1
> > > > >>>>>>>> if it is to investigate a CI issue, it is generally easy to
> > add
> > > > >> debug
> > > > >>>>>>>> insights (by code or agent) so a SPI sounds like the sanest
> > for
> > > > the
> > > > >>>>>> plugin
> > > > >>>>>>>> to me.
> > > > >>>>>>>>
> > > > >>>>>>>> Romain Manni-Bucau
> > > > >>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > > > >>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
> > > > >>>>>>>> <http://rmannibucau.wordpress.com> | Github <
> > > > >>>>>>>> https://github.com/rmannibucau> |
> > > > >>>>>>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> > > > >>>>>>>> <
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>
> > > >
> > >
> >
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <
> > > gnodet@apache.org>
> > > > a
> > > > >>>>>> écrit
> > > > >>>>>>>> :
> > > > >>>>>>>>
> > > > >>>>>>>>> If that's not currently possible, maybe a SPI should be
> > > provided
> > > > so
> > > > >>>>>> that
> > > > >>>>>>>>> people can use plug in extensions to analyze the test
> result
> > > and
> > > > >>>>>> override
> > > > >>>>>>>>> it if necessary (transforming an error into a warning,
> > storing
> > > > >>>> results
> > > > >>>>>>>> in a
> > > > >>>>>>>>> way which is easier to use by other tools later...) ?
> > > > >>>>>>>>>
> > > > >>>>>>>>> Guillaume
> > > > >>>>>>>>>
> > > > >>>>>>>>> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <
> > > > >>>> mail@laeubi-soft.de>
> > > > >>>>>> a
> > > > >>>>>>>>> écrit :
> > > > >>>>>>>>>
> > > > >>>>>>>>>> I also agree that the test at least should run, we use
> this
> > > > >> property
> > > > >>>>>> to
> > > > >>>>>>>>>> run the test and produce result and later on have a
> > buildstep
> > > > that
> > > > >>>>>>>>>> analyze the results (and probably fail the build job).
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> As it is not recommend, I wonder what is the recommended
> way
> > > to
> > > > >>>>>> archive
> > > > >>>>>>>>>> something similar?
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Am 14.03.22 um 06:29 schrieb Olivier Lamy:
> > > > >>>>>>>>>>> On Mon, 14 Mar 2022 at 11:55, Tibor Digana <
> > > > >> tibordigana@apache.org
> > > > >>>>>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> In case of the user property *maven.test.failure.ignore*
> > the
> > > > >> MOJO
> > > > >>>>>>>> must
> > > > >>>>>>>>>> not
> > > > >>>>>>>>>>>> throw any exception which is interpreted by the Maven
> Core
> > > as
> > > > >>>> BUILD
> > > > >>>>>>>>>>>> SUCCESS.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> This is a very simple reduction of the problem
> description.
> > > > >>>>>>>>>>> The documentation here
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>
> > > >
> > >
> >
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
> > > > >>>>>>>>>>> says
> > > > >>>>>>>>>>> "Set this to "true" to ignore a failure during testing.
> Its
> > > use
> > > > >> is
> > > > >>>>>>>> NOT
> > > > >>>>>>>>>>> RECOMMENDED, but quite convenient on occasion"
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Personally, I understand this to ignore failures in junit
> > > > results
> > > > >>>> BUT
> > > > >>>>>>>>> at
> > > > >>>>>>>>>>> least I want the tests to run.
> > > > >>>>>>>>>>> I guess this is how our users use this feature (feature
> we
> > do
> > > > not
> > > > >>>>>>>>>> recommend
> > > > >>>>>>>>>>> by the way...)
> > > > >>>>>>>>>>> And this is perfectly explained here
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
> > > > >>>>>>>>>>> there is a difference between ignoring some junit
> failures
> > > and
> > > > >>>>>>>>> ignoring a
> > > > >>>>>>>>>>> configuration error because some jvm args has been
> > > > misconfigured
> > > > >>>> for
> > > > >>>>>>>>> many
> > > > >>>>>>>>>>> reasons and surefire cannot start.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> If I follow your reasoning ("MOJO must not throw any
> > > exception
> > > > ")
> > > > >>>> we
> > > > >>>>>>>>>> should
> > > > >>>>>>>>>>> ignore every surefire configuration error and keep
> running
> > > the
> > > > >>>> build
> > > > >>>>>>>>>> until
> > > > >>>>>>>>>>> the end and says BUILD SUCCESS
> > > > >>>>>>>>>>> what about
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> mvn test -Dsurefire.rerunFailingTestsCount=notanumber
> > > > >>>>>>>>>>> -Dmaven.test.failure.ignore=true
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> we should not throw any exceptions as you said below and
> > keep
> > > > >>>> saying
> > > > >>>>>>>>>> BUILD
> > > > >>>>>>>>>>> SUCCESS?
> > > > >>>>>>>>>>> argLine is a configuration element as any others so it
> > should
> > > > >> fail
> > > > >>>>>>>>>> directly
> > > > >>>>>>>>>>> and not be ignored especially when the surefire plugin
> > cannot
> > > > >> run.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> We have received an internal requirement to change the
> > > > behavior
> > > > >> of
> > > > >>>>>>>> the
> > > > >>>>>>>>>> user
> > > > >>>>>>>>>>>> property *maven.test.failure.ignore* so that the
> behavior
> > > will
> > > > >>>> have
> > > > >>>>>>>>> one
> > > > >>>>>>>>>>>> exception.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Suppose that you have JDK 1.8 but you use:
> > > > >>>>>>>>>>>> /jre/java --add-reads ...
> > > > >>>>>>>>>>>> The outcome is JVM exit with an error message.
> > > > >>>>>>>>>>>> I agree with Herve who also says that
> > > > >> *maven.test.failure.ignore*
> > > > >>>>>>>>>> should
> > > > >>>>>>>>>>>> not allow the MOJO to throw the exception. It is not a
> > > typical
> > > > >> JVM
> > > > >>>>>>>>>>>> segmentation fault or another JVM crash where we cannot
> do
> > > > >>>> anything
> > > > >>>>>>>>>> about
> > > > >>>>>>>>>>>> it, and where the entire build would crash in the
> command
> > > line
> > > > >>>> which
> > > > >>>>>>>>>>>> of course means that the build cannot normally be
> > > interpreted
> > > > as
> > > > >>>>>>>> BUILD
> > > > >>>>>>>>>>>> SUCCESS. So we are still on the same level of failures
> > > related
> > > > >> to
> > > > >>>>>>>> the
> > > > >>>>>>>>>> test
> > > > >>>>>>>>>>>> purposes.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On the other hand, Olivier has reopened the issues
> > > > SUREFIRE-1426
> > > > >>>> and
> > > > >>>>>>>>>>>> SUREFIRE-1681 where the exceptional behavior of the
> > feature
> > > is
> > > > >>>>>>>>> expected.
> > > > >>>>>>>>>>>> This is exactly the reason why I closed these tickets
> > > several
> > > > >>>> years
> > > > >>>>>>>>> ago
> > > > >>>>>>>>>> and
> > > > >>>>>>>>>>>> my proposal was to not to have any exceptions in the
> > feature
> > > > >>>>>>>> behavior
> > > > >>>>>>>>>> and
> > > > >>>>>>>>>>>> the proposal was to introduce a new user property for
> > exact
> > > > use
> > > > >>>>>>>> cases.
> > > > >>>>>>>>>>>> The next idea, which comes from two developers, would
> > > provide
> > > > us
> > > > >>>>>>>> with
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>>> same non exceptional and exact behavior of the user
> > property
> > > > >>>>>>>>>>>> *maven.test.failure.ignore* but it would also provide us
> > > with
> > > > >> new
> > > > >>>>>>>> user
> > > > >>>>>>>>>>>> property in the case with fine grade control of the
> build
> > > > >> errors,
> > > > >>>>>>>> e.g.
> > > > >>>>>>>>>>>> *maven.test.jvm.error.ignore*.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> with a default of?
> > > > >>>>>>>>>>> honestly I just see this new parameter as introducing
> more
> > > > >>>> complexity
> > > > >>>>>>>>> in
> > > > >>>>>>>>>> an
> > > > >>>>>>>>>>> already very complex code and not worth it.
> > > > >>>>>>>>>>> again read both issue's comments and my comments.
> > > > >>>>>>>>>>> Please try to have a user POV and think about making our
> > > users'
> > > > >>>>>>>>>> experience
> > > > >>>>>>>>>>> more simple.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> This should be very simple:
> > > > >>>>>>>>>>> If surefire forked jvm cannot start it's build error and
> > > cannot
> > > > >> run
> > > > >>>>>>>> any
> > > > >>>>>>>>>>> tests, it's a problem users want to know immediately
> > because
> > > it
> > > > >> can
> > > > >>>>>>>> be
> > > > >>>>>>>>>> for
> > > > >>>>>>>>>>> a lot of reasons: wrong argLine, not enough memory on the
> > > > system
> > > > >>>>>>>> etc...
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> AND AGAIN it is very different from ignoring junit result
> > > > >> failures.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Try to look at a user point of view and think about what
> is
> > > the
> > > > >> use
> > > > >>>>>>>>> case
> > > > >>>>>>>>>> of
> > > > >>>>>>>>>>> the property maven.test.failure.ignore=true, I guess 99%
> of
> > > the
> > > > >>>> time,
> > > > >>>>>>>>>> users
> > > > >>>>>>>>>>> wants to run all their tests (even on a CI with different
> > > > matrix)
> > > > >>>> so
> > > > >>>>>>>>> they
> > > > >>>>>>>>>>> can collect all the tests results which has runned to see
> > if
> > > > >> there
> > > > >>>> is
> > > > >>>>>>>>> any
> > > > >>>>>>>>>>> issue for some combination of their matrix and eventually
> > > turn
> > > > >> the
> > > > >>>>>>>>> result
> > > > >>>>>>>>>>> as unstable (this is a very typical use case in Jenkins
> and
> > > was
> > > > >>>> even
> > > > >>>>>>>> a
> > > > >>>>>>>>>>> built in feature of the previous Jenkins Maven plugin)
> > > > >>>>>>>>>>> BUT if for any reasons one of the module do not start his
> > > tests
> > > > >>>>>>>> because
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>> jvm cannot be start the users will not see that and will
> be
> > > > >> totally
> > > > >>>>>>>>> blind
> > > > >>>>>>>>>>> until maybe someone look inside a very very large log
> file
> > > > (which
> > > > >>>>>>>> means
> > > > >>>>>>>>>>> probably never)
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Long story short as my experience as a user facing this
> > > > >>>> problem/bug:
> > > > >>>>>>>>>>> I had the case on a very large multi modules (~250
> modules)
> > > > build
> > > > >>>> of
> > > > >>>>>>>> a
> > > > >>>>>>>>>> very
> > > > >>>>>>>>>>> used open source project.
> > > > >>>>>>>>>>> I was using this maven.test.failure.ignore property but
> one
> > > of
> > > > >> the
> > > > >>>>>>>>>> modules
> > > > >>>>>>>>>>> had a bad jpms configuration for a jdk17 profile on the
> CI.
> > > > >>>>>>>>>>> The build has a matrix of 2 os and 4 jdks and different
> > maven
> > > > run
> > > > >>>>>>>> which
> > > > >>>>>>>>>>> means around ~60k tests and a Jenkins log file about 40M
> > > > >>>>>>>>>>> So because of this property the build was not failing and
> > > kept
> > > > >>>> saying
> > > > >>>>>>>>>> BUILD
> > > > >>>>>>>>>>> SUCCESS for weeks/months and basically not testing one
> > module
> > > > >> with
> > > > >>>>>>>> jdk
> > > > >>>>>>>>>> 17...
> > > > >>>>>>>>>>> And frankly you do not dig into a log file of 32M after
> > each
> > > > run
> > > > >>>>>>>>>> especially
> > > > >>>>>>>>>>> when it says success...
> > > > >>>>>>>>>>> 3 days after the first release claiming supporting jdk 17
> > we
> > > > >>>>>>>> received a
> > > > >>>>>>>>>> bug
> > > > >>>>>>>>>>> report about a something not working with jdk17....
> > > > >>>>>>>>>>> and guess what? Where was this feature supposed to be
> > tested?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> so I frankly believe we do not need a complex new
> property,
> > > in
> > > > >> this
> > > > >>>>>>>>> case
> > > > >>>>>>>>>>> this should fail directly because this will improve user
> > > > >>>> experience.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I would like to open the discussion on this topic.
> You're
> > > > >> welcome!
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Cheers
> > > > >>>>>>>>>>>> Tibor
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>
> > > ---------------------------------------------------------------------
> > > > >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > > >>>>>>>>>> For additional commands, e-mail:
> dev-help@maven.apache.org
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> --
> > > > >>>>>>>>> ------------------------
> > > > >>>>>>>>> Guillaume Nodet
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > ---------------------------------------------------------------------
> > > > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > > >>>>>> For additional commands, e-mail: dev-help@maven.apache.org
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > ---------------------------------------------------------------------
> > > > >>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > > >>>> For additional commands, e-mail: dev-help@maven.apache.org
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > > >> For additional commands, e-mail: dev-help@maven.apache.org
> > > > >>
> > > > >>
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > > For additional commands, e-mail: dev-help@maven.apache.org
> > > >
> > > >
> > >
> > > --
> > > Sławomir Jaranowski
> > >
> >
>
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Olivier Lamy <ol...@apache.org>.
On Tue, 22 Mar 2022 at 14:40, Tibor Digana <ti...@apache.org> wrote:
> Sorry for late reply.
>
> I have created a demo project
> https://github.com/Tibor17/maven.test.failure.ignore/ which simulates
> OOM.
>
> According to the definition of the parameter "maven.failure.test.ignore"
> the failures (also errors) should be ignored during testing.
> The word "during" is important because the time when it was a failure
> before JVM startup it was not a test failure - it is MOJO failure. The
> author who wrote this config param knew this difference.
> This perfectly fits JVM illegal arguments like -Xxxx.
> But I have investigated OOM for the same reason with one JVM. My suspicion
> was not confirmed, see the next...
> If there are two JVMs, we are breaking the JVM during testing. It is a very
> unlikely situation that the JVM argument would not fail in JVM1 but it
> would fail in JVM2 later. It may happen only if the user
> uses @surefire.forkCount@ interpolation but it is again very unlikely and
> the users use it in current working directory configuration.
>
> I understand Olivier's PR but I have one objection to the code branching in
> the class *SurefireHelper*. It is based on 3 levels but I have to disagree
> due to there still should be 2 levels as before:
> 1. ignored failures (my change is only here - yellow - only added throwing
> exception)
> 2. processes failures
>
> and so, I have created a PR
> https://github.com/apache/maven-surefire/pull/496
> Pls have a look into it, thx.
> This diff looks very simple:
>
> if ( reportParameters.isTestFailureIgnore() )
> {
> String errorMessage = createErrorMessage( reportParameters,
> result, firstForkException );
>
> if ( firstForkException instanceof SurefireBooterForkException )
> {
> throw new MojoExecutionException( errorMessage, firstForkException
> );
>
> }
>
> log.error( errorMessage );
> }
> else
> {
> throwException( reportParameters, result, firstForkException );
> }
>
>
>
> *$ mvn test*
>
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 2.445 s <<< FAILURE! - in org.apache.maven.OOMTest
> [ERROR] org.apache.maven.OOMTest.test Time elapsed: 2.063 s <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.maven.OOMTest.test(OOMTest.java:15)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] OOMTest.test:15 » OutOfMemory Java heap space
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
>
> *$ mvn test -Dmaven.test.failure.ignore*
>
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 2.472 s <<< FAILURE! - in org.apache.maven.OOMTest
> [ERROR] org.apache.maven.OOMTest.test Time elapsed: 2.262 s <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.maven.OOMTest.test(OOMTest.java:15)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] OOMTest.test:15 » OutOfMemory Java heap space
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [ERROR]
>
> Please refer to
> C:\vcs\github\maven.test.failure.ignore\target\surefire-reports for the
> individual test results.
> Please refer to dump files (if any exist) [date].dump,
> [date]-jvmRun[N].dump and [date].dumpstream.
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
>
Please use a project with multiple modules, not a single one where you can
see the line with OOME.
Any project with a huge number of modules will hide this and nobody will
ever notice it.
git clone -b first-module-fail https://github.com/apache/maven maven-core
cd maven-core
[INFO] --- maven-assembly-plugin:3.3.0:single (create-distro-packages) @
apache-maven ---
[INFO] Reading assembly descriptor: src/main/assembly/bin.xml
[INFO] Building zip:
/Volumes/workspace/dev/sources/maven/maven-core/target/maven-core/apache-maven/target/apache-maven-3.8.6-SNAPSHOT-bin.zip
[INFO] Building tar:
/Volumes/workspace/dev/sources/maven/maven-core/target/maven-core/apache-maven/target/apache-maven-3.8.6-SNAPSHOT-bin.tar.gz
[INFO]
[INFO] --- maven-checkstyle-plugin:3.0.0:check (checkstyle-check) @
apache-maven ---
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Maven 3.8.6-SNAPSHOT:
[INFO]
[INFO] Apache Maven ....................................... SUCCESS [
3.220 s]
[INFO] Maven Model ........................................ SUCCESS [
4.734 s]
[INFO] Maven Artifact ..................................... SUCCESS [
3.142 s]
[INFO] Maven Plugin API ................................... SUCCESS [
1.763 s]
[INFO] Maven Builder Support .............................. SUCCESS [
1.288 s]
[INFO] Maven Model Builder ................................ SUCCESS [
5.129 s]
[INFO] Maven Settings ..................................... SUCCESS [
0.694 s]
[INFO] Maven Settings Builder ............................. SUCCESS [
1.730 s]
[INFO] Maven Repository Metadata Model .................... SUCCESS [
1.470 s]
[INFO] Maven Artifact Resolver Provider ................... SUCCESS [
4.255 s]
[INFO] Maven Core ......................................... SUCCESS [
19.075 s]
[INFO] Maven SLF4J Simple Provider ........................ SUCCESS [
0.972 s]
[INFO] Maven Embedder ..................................... SUCCESS [
4.293 s]
[INFO] Maven Compat ....................................... SUCCESS [
10.350 s]
[INFO] Apache Maven Distribution .......................... SUCCESS [
4.898 s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 01:07 min
[INFO] Finished at: 2022-03-22T20:19:24+10:00
[INFO]
------------------------------------------------------------------------
even CI is green
https://ci-maven.apache.org/job/Maven/job/maven-box/job/maven/job/first-module-fail/
so all looks good to merge a PR...
but hey in fact a module has not run any tests...
Do you really think it's right? (yes this problem exists for a very very
long time but we can choose to fix problems even after a long period)
Does your PR fix that?
>
>
>
> On Sat, Mar 19, 2022 at 12:17 PM Slawomir Jaranowski <
> s.jaranowski@gmail.com>
> wrote:
>
> > Hi,
> >
> > I start to feel that we first try to define technical details but we
> don't
> > know for what.
> > Each business requirement should be resolved as simply as possible from a
> > technical perspective.
> > In other ways we will start to build complicated and many features which
> > probably were not needed at all.
> >
> > Please first define business requirements with examples, next we can find
> > technical ways to cover it.
> >
> > Example scenario:
> >
> > Businesses requirement:
> > We want to collect history of test result in order to detect flaky
> tests,
> > analize tests execution time, etc
> >
> > Given:
> > - build is done by CI system
> > - there is system which collect reports for executed tests and build
> > history of it
> > - there are broken tests in build - we know them and accept (eg, on some
> > os)
> >
> > When:
> > - build finished with success on CI
> >
> > Then:
> > - all tests result are reported
> > - failed tests are reported
> >
> >
> >
> >
> > sob., 19 mar 2022 o 06:27 Christoph Läubrich <ma...@laeubi-soft.de>
> > napisał(a):
> >
> > > Hi Tibor,
> > >
> > > just to make it clear, I don't talkin about log-levels here but how to
> > > intepret/handle a failure/failing/crashing test.
> > >
> > >
> > > Am 18.03.22 um 21:54 schrieb Tibor Digaňa:
> > > > Christoph, just let me briefly explain the log level hierarchy.
> > > > If you select WARN log level, then ERROR can be printed too.
> > > > Similar with INFO, means that WARN and ERROR would be printed as
> well.
> > > >
> > > > The real decision making in the plugin is a bit more complicated, see
> > the
> > > > pull request https://github.com/apache/maven-surefire/pull/478
> > > > After I have replied to this email, I would verify the behavior of
> the
> > > > plugin simulating OOM.
> > > >
> > > >
> > > > T
> > > >
> > > >
> > > > On Fri, Mar 18, 2022 at 5:42 AM Christoph Läubrich <
> > mail@laeubi-soft.de>
> > > > wrote:
> > > >
> > > >> No I think more about
> > > >>
> > > >> if (severity == "WARN") {
> > > >> log.warn("There is something wrong");
> > > >> } else if (severity == "ERROR") {
> > > >> throw new MojoFailureException("...") {
> > > >> } else {
> > > >> throw new MojoExecutionException("...") {
> > > >> }
> > > >>
> > > >> That way the plugin can handle any error/failure/... in a way that
> the
> > > >> user can "downgrade" a certain problem if desired.
> > > >>
> > > >>
> > > >> Am 18.03.22 um 00:06 schrieb Tibor Digana:
> > > >>> @Christoph
> > > >>> FATAL , WARN , ERROR
> > > >>> They are log levels?
> > > >>> The plugin does not control the log level after caught exception in
> > > Maven
> > > >>> core. The Maven Core does!
> > > >>> I think it's time to make a demo.
> > > >>>
> > > >>> On Thu, Mar 17, 2022 at 6:21 AM Christoph Läubrich <
> > > mail@laeubi-soft.de>
> > > >>> wrote:
> > > >>>
> > > >>>> Hi Tibor,
> > > >>>>
> > > >>>> it shouldn't be to hard to guess another property like
> > > >>>>
> > > >>>> maven.test.jvmcrash=FATAL
> > > >>>> maven.test.failure=WARN
> > > >>>> maven.test.error=ERROR
> > > >>>>
> > > >>>> :-)
> > > >>>>
> > > >>>> Am 16.03.22 um 12:25 schrieb Tibor Digana:
> > > >>>>> Hi Christoph,
> > > >>>>>
> > > >>>>> Such a granularity with error/failure might be also an additional
> > > >>>>> requirement but still you miss the third option to JVM error
> which
> > is
> > > >>>>> different from test error/failure - they don;t have the same
> > meaning.
> > > >>>>>
> > > >>>>> T
> > > >>>>>
> > > >>>>>
> > > >>>>> On Mon, Mar 14, 2022 at 10:57 AM Christoph Läubrich <
> > > >> mail@laeubi-soft.de
> > > >>>>>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Just wondering but maybe it would be better to configure the
> > > severity
> > > >>>>>> instead of a true/false, something like:
> > > >>>>>>
> > > >>>>>> maven.test.failure=WARN
> > > >>>>>> maven.test.error=ERROR
> > > >>>>>>
> > > >>>>>> would only warn about failing tests but thrw exception if
> starting
> > > the
> > > >>>>>> test fails?
> > > >>>>>>
> > > >>>>>> Am 14.03.22 um 10:52 schrieb Tibor Digana:
> > > >>>>>>> Romain, it is not a bug.
> > > >>>>>>> Don't consider this as a bug. It was a feature request for
> change
> > > by
> > > >>>>>>> Olivier, and not a bug.
> > > >>>>>>> I closed both issues years ago but not because of ignorance but
> > > >> because
> > > >>>>>> the
> > > >>>>>>> appearance of the exceptional behavior is a wrong compromise
> and
> > > >> which
> > > >>>>>> does
> > > >>>>>>> not help anyone and even it makes the situation worse because
> > > >> typically
> > > >>>>>>> other group of users would fire a new Jira tickets. You would
> not
> > > be
> > > >>>> able
> > > >>>>>>> to satisfy two contradictory parties which have completely
> > > different
> > > >>>>>>> opinions, and so we use to introduce new params and this way we
> > > >> satisfy
> > > >>>>>>> both parties, they may combine the params as they wish, and
> > > everybody
> > > >>>>>> would
> > > >>>>>>> be happy and nobody would claim. Many technical solutions might
> > > >> exist,
> > > >>>>>> e.g.
> > > >>>>>>> param=boolean|string or new param=boolean, whatever.
> > > >>>>>>>
> > > >>>>>>> The truth is that this problem with (java --add-reads ...)
> > happened
> > > >> in
> > > >>>>>> our
> > > >>>>>>> ASF environment on Java 8 which in good configuration would not
> > > >> happen
> > > >>>>>> and
> > > >>>>>>> should not.
> > > >>>>>>> It is not right way that we abuse "maven.test.failure.ignore"
> > which
> > > >>>> would
> > > >>>>>>> tell us "Hey, you have illegal configuration in your build
> system
> > > and
> > > >>>> it
> > > >>>>>>> would not work by JDK design", it is not the goal of the plugin
> > to
> > > >> tell
> > > >>>>>> you
> > > >>>>>>> that you have configured the build wrong because it won't and
> the
> > > >> same
> > > >>>>>>> configuration of the plugin (not the build) says "ignore the
> > > error".
> > > >>>>>>> Yesterday I discussed this problem with Herve and we
> > independently
> > > >>>>>> observed
> > > >>>>>>> equal opinions and that's not everything because we understood
> > that
> > > >> the
> > > >>>>>>> purpose of the config parameter is to not throw MOJO exception
> > > which
> > > >> is
> > > >>>>>>> right, but the exceptional behavior should have an exact new
> > param
> > > >> for
> > > >>>>>> the
> > > >>>>>>> exact use case.
> > > >>>>>>> SPI for this parameter is too much because no user would
> > implement
> > > it
> > > >>>>>> for a
> > > >>>>>>> trivial parameter;; the SPI is used to be implemented by
> > frameworks
> > > >> or
> > > >>>>>>> systems or application servers but this is not our case.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <
> > > >>>>>> rmannibucau@gmail.com>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> +1
> > > >>>>>>>> if it is to investigate a CI issue, it is generally easy to
> add
> > > >> debug
> > > >>>>>>>> insights (by code or agent) so a SPI sounds like the sanest
> for
> > > the
> > > >>>>>> plugin
> > > >>>>>>>> to me.
> > > >>>>>>>>
> > > >>>>>>>> Romain Manni-Bucau
> > > >>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > > >>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
> > > >>>>>>>> <http://rmannibucau.wordpress.com> | Github <
> > > >>>>>>>> https://github.com/rmannibucau> |
> > > >>>>>>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> > > >>>>>>>> <
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> >
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <
> > gnodet@apache.org>
> > > a
> > > >>>>>> écrit
> > > >>>>>>>> :
> > > >>>>>>>>
> > > >>>>>>>>> If that's not currently possible, maybe a SPI should be
> > provided
> > > so
> > > >>>>>> that
> > > >>>>>>>>> people can use plug in extensions to analyze the test result
> > and
> > > >>>>>> override
> > > >>>>>>>>> it if necessary (transforming an error into a warning,
> storing
> > > >>>> results
> > > >>>>>>>> in a
> > > >>>>>>>>> way which is easier to use by other tools later...) ?
> > > >>>>>>>>>
> > > >>>>>>>>> Guillaume
> > > >>>>>>>>>
> > > >>>>>>>>> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <
> > > >>>> mail@laeubi-soft.de>
> > > >>>>>> a
> > > >>>>>>>>> écrit :
> > > >>>>>>>>>
> > > >>>>>>>>>> I also agree that the test at least should run, we use this
> > > >> property
> > > >>>>>> to
> > > >>>>>>>>>> run the test and produce result and later on have a
> buildstep
> > > that
> > > >>>>>>>>>> analyze the results (and probably fail the build job).
> > > >>>>>>>>>>
> > > >>>>>>>>>> As it is not recommend, I wonder what is the recommended way
> > to
> > > >>>>>> archive
> > > >>>>>>>>>> something similar?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Am 14.03.22 um 06:29 schrieb Olivier Lamy:
> > > >>>>>>>>>>> On Mon, 14 Mar 2022 at 11:55, Tibor Digana <
> > > >> tibordigana@apache.org
> > > >>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> In case of the user property *maven.test.failure.ignore*
> the
> > > >> MOJO
> > > >>>>>>>> must
> > > >>>>>>>>>> not
> > > >>>>>>>>>>>> throw any exception which is interpreted by the Maven Core
> > as
> > > >>>> BUILD
> > > >>>>>>>>>>>> SUCCESS.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> This is a very simple reduction of the problem description.
> > > >>>>>>>>>>> The documentation here
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> >
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
> > > >>>>>>>>>>> says
> > > >>>>>>>>>>> "Set this to "true" to ignore a failure during testing. Its
> > use
> > > >> is
> > > >>>>>>>> NOT
> > > >>>>>>>>>>> RECOMMENDED, but quite convenient on occasion"
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Personally, I understand this to ignore failures in junit
> > > results
> > > >>>> BUT
> > > >>>>>>>>> at
> > > >>>>>>>>>>> least I want the tests to run.
> > > >>>>>>>>>>> I guess this is how our users use this feature (feature we
> do
> > > not
> > > >>>>>>>>>> recommend
> > > >>>>>>>>>>> by the way...)
> > > >>>>>>>>>>> And this is perfectly explained here
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> >
> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
> > > >>>>>>>>>>> there is a difference between ignoring some junit failures
> > and
> > > >>>>>>>>> ignoring a
> > > >>>>>>>>>>> configuration error because some jvm args has been
> > > misconfigured
> > > >>>> for
> > > >>>>>>>>> many
> > > >>>>>>>>>>> reasons and surefire cannot start.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> If I follow your reasoning ("MOJO must not throw any
> > exception
> > > ")
> > > >>>> we
> > > >>>>>>>>>> should
> > > >>>>>>>>>>> ignore every surefire configuration error and keep running
> > the
> > > >>>> build
> > > >>>>>>>>>> until
> > > >>>>>>>>>>> the end and says BUILD SUCCESS
> > > >>>>>>>>>>> what about
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> mvn test -Dsurefire.rerunFailingTestsCount=notanumber
> > > >>>>>>>>>>> -Dmaven.test.failure.ignore=true
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> we should not throw any exceptions as you said below and
> keep
> > > >>>> saying
> > > >>>>>>>>>> BUILD
> > > >>>>>>>>>>> SUCCESS?
> > > >>>>>>>>>>> argLine is a configuration element as any others so it
> should
> > > >> fail
> > > >>>>>>>>>> directly
> > > >>>>>>>>>>> and not be ignored especially when the surefire plugin
> cannot
> > > >> run.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> We have received an internal requirement to change the
> > > behavior
> > > >> of
> > > >>>>>>>> the
> > > >>>>>>>>>> user
> > > >>>>>>>>>>>> property *maven.test.failure.ignore* so that the behavior
> > will
> > > >>>> have
> > > >>>>>>>>> one
> > > >>>>>>>>>>>> exception.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Suppose that you have JDK 1.8 but you use:
> > > >>>>>>>>>>>> /jre/java --add-reads ...
> > > >>>>>>>>>>>> The outcome is JVM exit with an error message.
> > > >>>>>>>>>>>> I agree with Herve who also says that
> > > >> *maven.test.failure.ignore*
> > > >>>>>>>>>> should
> > > >>>>>>>>>>>> not allow the MOJO to throw the exception. It is not a
> > typical
> > > >> JVM
> > > >>>>>>>>>>>> segmentation fault or another JVM crash where we cannot do
> > > >>>> anything
> > > >>>>>>>>>> about
> > > >>>>>>>>>>>> it, and where the entire build would crash in the command
> > line
> > > >>>> which
> > > >>>>>>>>>>>> of course means that the build cannot normally be
> > interpreted
> > > as
> > > >>>>>>>> BUILD
> > > >>>>>>>>>>>> SUCCESS. So we are still on the same level of failures
> > related
> > > >> to
> > > >>>>>>>> the
> > > >>>>>>>>>> test
> > > >>>>>>>>>>>> purposes.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On the other hand, Olivier has reopened the issues
> > > SUREFIRE-1426
> > > >>>> and
> > > >>>>>>>>>>>> SUREFIRE-1681 where the exceptional behavior of the
> feature
> > is
> > > >>>>>>>>> expected.
> > > >>>>>>>>>>>> This is exactly the reason why I closed these tickets
> > several
> > > >>>> years
> > > >>>>>>>>> ago
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>> my proposal was to not to have any exceptions in the
> feature
> > > >>>>>>>> behavior
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>> the proposal was to introduce a new user property for
> exact
> > > use
> > > >>>>>>>> cases.
> > > >>>>>>>>>>>> The next idea, which comes from two developers, would
> > provide
> > > us
> > > >>>>>>>> with
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>> same non exceptional and exact behavior of the user
> property
> > > >>>>>>>>>>>> *maven.test.failure.ignore* but it would also provide us
> > with
> > > >> new
> > > >>>>>>>> user
> > > >>>>>>>>>>>> property in the case with fine grade control of the build
> > > >> errors,
> > > >>>>>>>> e.g.
> > > >>>>>>>>>>>> *maven.test.jvm.error.ignore*.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> with a default of?
> > > >>>>>>>>>>> honestly I just see this new parameter as introducing more
> > > >>>> complexity
> > > >>>>>>>>> in
> > > >>>>>>>>>> an
> > > >>>>>>>>>>> already very complex code and not worth it.
> > > >>>>>>>>>>> again read both issue's comments and my comments.
> > > >>>>>>>>>>> Please try to have a user POV and think about making our
> > users'
> > > >>>>>>>>>> experience
> > > >>>>>>>>>>> more simple.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> This should be very simple:
> > > >>>>>>>>>>> If surefire forked jvm cannot start it's build error and
> > cannot
> > > >> run
> > > >>>>>>>> any
> > > >>>>>>>>>>> tests, it's a problem users want to know immediately
> because
> > it
> > > >> can
> > > >>>>>>>> be
> > > >>>>>>>>>> for
> > > >>>>>>>>>>> a lot of reasons: wrong argLine, not enough memory on the
> > > system
> > > >>>>>>>> etc...
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> AND AGAIN it is very different from ignoring junit result
> > > >> failures.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Try to look at a user point of view and think about what is
> > the
> > > >> use
> > > >>>>>>>>> case
> > > >>>>>>>>>> of
> > > >>>>>>>>>>> the property maven.test.failure.ignore=true, I guess 99% of
> > the
> > > >>>> time,
> > > >>>>>>>>>> users
> > > >>>>>>>>>>> wants to run all their tests (even on a CI with different
> > > matrix)
> > > >>>> so
> > > >>>>>>>>> they
> > > >>>>>>>>>>> can collect all the tests results which has runned to see
> if
> > > >> there
> > > >>>> is
> > > >>>>>>>>> any
> > > >>>>>>>>>>> issue for some combination of their matrix and eventually
> > turn
> > > >> the
> > > >>>>>>>>> result
> > > >>>>>>>>>>> as unstable (this is a very typical use case in Jenkins and
> > was
> > > >>>> even
> > > >>>>>>>> a
> > > >>>>>>>>>>> built in feature of the previous Jenkins Maven plugin)
> > > >>>>>>>>>>> BUT if for any reasons one of the module do not start his
> > tests
> > > >>>>>>>> because
> > > >>>>>>>>>> the
> > > >>>>>>>>>>> jvm cannot be start the users will not see that and will be
> > > >> totally
> > > >>>>>>>>> blind
> > > >>>>>>>>>>> until maybe someone look inside a very very large log file
> > > (which
> > > >>>>>>>> means
> > > >>>>>>>>>>> probably never)
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Long story short as my experience as a user facing this
> > > >>>> problem/bug:
> > > >>>>>>>>>>> I had the case on a very large multi modules (~250 modules)
> > > build
> > > >>>> of
> > > >>>>>>>> a
> > > >>>>>>>>>> very
> > > >>>>>>>>>>> used open source project.
> > > >>>>>>>>>>> I was using this maven.test.failure.ignore property but one
> > of
> > > >> the
> > > >>>>>>>>>> modules
> > > >>>>>>>>>>> had a bad jpms configuration for a jdk17 profile on the CI.
> > > >>>>>>>>>>> The build has a matrix of 2 os and 4 jdks and different
> maven
> > > run
> > > >>>>>>>> which
> > > >>>>>>>>>>> means around ~60k tests and a Jenkins log file about 40M
> > > >>>>>>>>>>> So because of this property the build was not failing and
> > kept
> > > >>>> saying
> > > >>>>>>>>>> BUILD
> > > >>>>>>>>>>> SUCCESS for weeks/months and basically not testing one
> module
> > > >> with
> > > >>>>>>>> jdk
> > > >>>>>>>>>> 17...
> > > >>>>>>>>>>> And frankly you do not dig into a log file of 32M after
> each
> > > run
> > > >>>>>>>>>> especially
> > > >>>>>>>>>>> when it says success...
> > > >>>>>>>>>>> 3 days after the first release claiming supporting jdk 17
> we
> > > >>>>>>>> received a
> > > >>>>>>>>>> bug
> > > >>>>>>>>>>> report about a something not working with jdk17....
> > > >>>>>>>>>>> and guess what? Where was this feature supposed to be
> tested?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> so I frankly believe we do not need a complex new property,
> > in
> > > >> this
> > > >>>>>>>>> case
> > > >>>>>>>>>>> this should fail directly because this will improve user
> > > >>>> experience.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I would like to open the discussion on this topic. You're
> > > >> welcome!
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Cheers
> > > >>>>>>>>>>>> Tibor
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>
> > ---------------------------------------------------------------------
> > > >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > >>>>>>>>>> For additional commands, e-mail: dev-help@maven.apache.org
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> --
> > > >>>>>>>>> ------------------------
> > > >>>>>>>>> Guillaume Nodet
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > ---------------------------------------------------------------------
> > > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > >>>>>> For additional commands, e-mail: dev-help@maven.apache.org
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > ---------------------------------------------------------------------
> > > >>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > >>>> For additional commands, e-mail: dev-help@maven.apache.org
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > >> For additional commands, e-mail: dev-help@maven.apache.org
> > > >>
> > > >>
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > For additional commands, e-mail: dev-help@maven.apache.org
> > >
> > >
> >
> > --
> > Sławomir Jaranowski
> >
>
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Olivier Lamy <ol...@apache.org>.
On Tue, 22 Mar 2022 at 14:40, Tibor Digana <ti...@apache.org> wrote:
> Sorry for late reply.
>
> I have created a demo project
> https://github.com/Tibor17/maven.test.failure.ignore/ which simulates
> OOM.
>
> According to the definition of the parameter "maven.failure.test.ignore"
> the failures (also errors) should be ignored during testing.
>
https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
says
"Set this to "true" to ignore a failure during testing. Its use is NOT
RECOMMENDED, but quite convenient on occasion."
"during" which means tests have been runned.
Please try to have a user point of view.
Users are using this property to run ALL their tests and collect data at
the end.
but here some tests will not be running and they will not have the
information and believe everything is working properly....
Which is totally wrong!
> The word "during" is important because the time when it was a failure
> before JVM startup it was not a test failure - it is MOJO failure. The
> author who wrote this config param knew this difference.
> This perfectly fits JVM illegal arguments like -Xxxx.
> But I have investigated OOM for the same reason with one JVM. My suspicion
> was not confirmed, see the next...
> If there are two JVMs, we are breaking the JVM during testing. It is a very
> unlikely situation that the JVM argument would not fail in JVM1 but it
> would fail in JVM2 later. It may happen only if the user
> uses @surefire.forkCount@ interpolation but it is again very unlikely and
> the users use it in current working directory configuration.
>
> I understand Olivier's PR but I have one objection to the code branching in
> the class *SurefireHelper*. It is based on 3 levels but I have to disagree
> due to there still should be 2 levels as before:
> 1. ignored failures (my change is only here - yellow - only added throwing
> exception)
> 2. processes failures
>
> and so, I have created a PR
> https://github.com/apache/maven-surefire/pull/496
> Pls have a look into it, thx.
> This diff looks very simple:
>
> if ( reportParameters.isTestFailureIgnore() )
> {
> String errorMessage = createErrorMessage( reportParameters,
> result, firstForkException );
>
> if ( firstForkException instanceof SurefireBooterForkException )
> {
> throw new MojoExecutionException( errorMessage, firstForkException
> );
>
> }
>
> log.error( errorMessage );
> }
> else
> {
> throwException( reportParameters, result, firstForkException );
> }
>
>
>
> *$ mvn test*
>
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 2.445 s <<< FAILURE! - in org.apache.maven.OOMTest
> [ERROR] org.apache.maven.OOMTest.test Time elapsed: 2.063 s <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.maven.OOMTest.test(OOMTest.java:15)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] OOMTest.test:15 » OutOfMemory Java heap space
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
>
> *$ mvn test -Dmaven.test.failure.ignore*
>
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 2.472 s <<< FAILURE! - in org.apache.maven.OOMTest
> [ERROR] org.apache.maven.OOMTest.test Time elapsed: 2.262 s <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.maven.OOMTest.test(OOMTest.java:15)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] OOMTest.test:15 » OutOfMemory Java heap space
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [ERROR]
>
> Please refer to
> C:\vcs\github\maven.test.failure.ignore\target\surefire-reports for the
> individual test results.
> Please refer to dump files (if any exist) [date].dump,
> [date]-jvmRun[N].dump and [date].dumpstream.
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
>
>
>
>
> On Sat, Mar 19, 2022 at 12:17 PM Slawomir Jaranowski <
> s.jaranowski@gmail.com>
> wrote:
>
> > Hi,
> >
> > I start to feel that we first try to define technical details but we
> don't
> > know for what.
> > Each business requirement should be resolved as simply as possible from a
> > technical perspective.
> > In other ways we will start to build complicated and many features which
> > probably were not needed at all.
> >
> > Please first define business requirements with examples, next we can find
> > technical ways to cover it.
> >
> > Example scenario:
> >
> > Businesses requirement:
> > We want to collect history of test result in order to detect flaky
> tests,
> > analize tests execution time, etc
> >
> > Given:
> > - build is done by CI system
> > - there is system which collect reports for executed tests and build
> > history of it
> > - there are broken tests in build - we know them and accept (eg, on some
> > os)
> >
> > When:
> > - build finished with success on CI
> >
> > Then:
> > - all tests result are reported
> > - failed tests are reported
> >
> >
> >
> >
> > sob., 19 mar 2022 o 06:27 Christoph Läubrich <ma...@laeubi-soft.de>
> > napisał(a):
> >
> > > Hi Tibor,
> > >
> > > just to make it clear, I don't talkin about log-levels here but how to
> > > intepret/handle a failure/failing/crashing test.
> > >
> > >
> > > Am 18.03.22 um 21:54 schrieb Tibor Digaňa:
> > > > Christoph, just let me briefly explain the log level hierarchy.
> > > > If you select WARN log level, then ERROR can be printed too.
> > > > Similar with INFO, means that WARN and ERROR would be printed as
> well.
> > > >
> > > > The real decision making in the plugin is a bit more complicated, see
> > the
> > > > pull request https://github.com/apache/maven-surefire/pull/478
> > > > After I have replied to this email, I would verify the behavior of
> the
> > > > plugin simulating OOM.
> > > >
> > > >
> > > > T
> > > >
> > > >
> > > > On Fri, Mar 18, 2022 at 5:42 AM Christoph Läubrich <
> > mail@laeubi-soft.de>
> > > > wrote:
> > > >
> > > >> No I think more about
> > > >>
> > > >> if (severity == "WARN") {
> > > >> log.warn("There is something wrong");
> > > >> } else if (severity == "ERROR") {
> > > >> throw new MojoFailureException("...") {
> > > >> } else {
> > > >> throw new MojoExecutionException("...") {
> > > >> }
> > > >>
> > > >> That way the plugin can handle any error/failure/... in a way that
> the
> > > >> user can "downgrade" a certain problem if desired.
> > > >>
> > > >>
> > > >> Am 18.03.22 um 00:06 schrieb Tibor Digana:
> > > >>> @Christoph
> > > >>> FATAL , WARN , ERROR
> > > >>> They are log levels?
> > > >>> The plugin does not control the log level after caught exception in
> > > Maven
> > > >>> core. The Maven Core does!
> > > >>> I think it's time to make a demo.
> > > >>>
> > > >>> On Thu, Mar 17, 2022 at 6:21 AM Christoph Läubrich <
> > > mail@laeubi-soft.de>
> > > >>> wrote:
> > > >>>
> > > >>>> Hi Tibor,
> > > >>>>
> > > >>>> it shouldn't be to hard to guess another property like
> > > >>>>
> > > >>>> maven.test.jvmcrash=FATAL
> > > >>>> maven.test.failure=WARN
> > > >>>> maven.test.error=ERROR
> > > >>>>
> > > >>>> :-)
> > > >>>>
> > > >>>> Am 16.03.22 um 12:25 schrieb Tibor Digana:
> > > >>>>> Hi Christoph,
> > > >>>>>
> > > >>>>> Such a granularity with error/failure might be also an additional
> > > >>>>> requirement but still you miss the third option to JVM error
> which
> > is
> > > >>>>> different from test error/failure - they don;t have the same
> > meaning.
> > > >>>>>
> > > >>>>> T
> > > >>>>>
> > > >>>>>
> > > >>>>> On Mon, Mar 14, 2022 at 10:57 AM Christoph Läubrich <
> > > >> mail@laeubi-soft.de
> > > >>>>>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Just wondering but maybe it would be better to configure the
> > > severity
> > > >>>>>> instead of a true/false, something like:
> > > >>>>>>
> > > >>>>>> maven.test.failure=WARN
> > > >>>>>> maven.test.error=ERROR
> > > >>>>>>
> > > >>>>>> would only warn about failing tests but thrw exception if
> starting
> > > the
> > > >>>>>> test fails?
> > > >>>>>>
> > > >>>>>> Am 14.03.22 um 10:52 schrieb Tibor Digana:
> > > >>>>>>> Romain, it is not a bug.
> > > >>>>>>> Don't consider this as a bug. It was a feature request for
> change
> > > by
> > > >>>>>>> Olivier, and not a bug.
> > > >>>>>>> I closed both issues years ago but not because of ignorance but
> > > >> because
> > > >>>>>> the
> > > >>>>>>> appearance of the exceptional behavior is a wrong compromise
> and
> > > >> which
> > > >>>>>> does
> > > >>>>>>> not help anyone and even it makes the situation worse because
> > > >> typically
> > > >>>>>>> other group of users would fire a new Jira tickets. You would
> not
> > > be
> > > >>>> able
> > > >>>>>>> to satisfy two contradictory parties which have completely
> > > different
> > > >>>>>>> opinions, and so we use to introduce new params and this way we
> > > >> satisfy
> > > >>>>>>> both parties, they may combine the params as they wish, and
> > > everybody
> > > >>>>>> would
> > > >>>>>>> be happy and nobody would claim. Many technical solutions might
> > > >> exist,
> > > >>>>>> e.g.
> > > >>>>>>> param=boolean|string or new param=boolean, whatever.
> > > >>>>>>>
> > > >>>>>>> The truth is that this problem with (java --add-reads ...)
> > happened
> > > >> in
> > > >>>>>> our
> > > >>>>>>> ASF environment on Java 8 which in good configuration would not
> > > >> happen
> > > >>>>>> and
> > > >>>>>>> should not.
> > > >>>>>>> It is not right way that we abuse "maven.test.failure.ignore"
> > which
> > > >>>> would
> > > >>>>>>> tell us "Hey, you have illegal configuration in your build
> system
> > > and
> > > >>>> it
> > > >>>>>>> would not work by JDK design", it is not the goal of the plugin
> > to
> > > >> tell
> > > >>>>>> you
> > > >>>>>>> that you have configured the build wrong because it won't and
> the
> > > >> same
> > > >>>>>>> configuration of the plugin (not the build) says "ignore the
> > > error".
> > > >>>>>>> Yesterday I discussed this problem with Herve and we
> > independently
> > > >>>>>> observed
> > > >>>>>>> equal opinions and that's not everything because we understood
> > that
> > > >> the
> > > >>>>>>> purpose of the config parameter is to not throw MOJO exception
> > > which
> > > >> is
> > > >>>>>>> right, but the exceptional behavior should have an exact new
> > param
> > > >> for
> > > >>>>>> the
> > > >>>>>>> exact use case.
> > > >>>>>>> SPI for this parameter is too much because no user would
> > implement
> > > it
> > > >>>>>> for a
> > > >>>>>>> trivial parameter;; the SPI is used to be implemented by
> > frameworks
> > > >> or
> > > >>>>>>> systems or application servers but this is not our case.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <
> > > >>>>>> rmannibucau@gmail.com>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> +1
> > > >>>>>>>> if it is to investigate a CI issue, it is generally easy to
> add
> > > >> debug
> > > >>>>>>>> insights (by code or agent) so a SPI sounds like the sanest
> for
> > > the
> > > >>>>>> plugin
> > > >>>>>>>> to me.
> > > >>>>>>>>
> > > >>>>>>>> Romain Manni-Bucau
> > > >>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > > >>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
> > > >>>>>>>> <http://rmannibucau.wordpress.com> | Github <
> > > >>>>>>>> https://github.com/rmannibucau> |
> > > >>>>>>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> > > >>>>>>>> <
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> >
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <
> > gnodet@apache.org>
> > > a
> > > >>>>>> écrit
> > > >>>>>>>> :
> > > >>>>>>>>
> > > >>>>>>>>> If that's not currently possible, maybe a SPI should be
> > provided
> > > so
> > > >>>>>> that
> > > >>>>>>>>> people can use plug in extensions to analyze the test result
> > and
> > > >>>>>> override
> > > >>>>>>>>> it if necessary (transforming an error into a warning,
> storing
> > > >>>> results
> > > >>>>>>>> in a
> > > >>>>>>>>> way which is easier to use by other tools later...) ?
> > > >>>>>>>>>
> > > >>>>>>>>> Guillaume
> > > >>>>>>>>>
> > > >>>>>>>>> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <
> > > >>>> mail@laeubi-soft.de>
> > > >>>>>> a
> > > >>>>>>>>> écrit :
> > > >>>>>>>>>
> > > >>>>>>>>>> I also agree that the test at least should run, we use this
> > > >> property
> > > >>>>>> to
> > > >>>>>>>>>> run the test and produce result and later on have a
> buildstep
> > > that
> > > >>>>>>>>>> analyze the results (and probably fail the build job).
> > > >>>>>>>>>>
> > > >>>>>>>>>> As it is not recommend, I wonder what is the recommended way
> > to
> > > >>>>>> archive
> > > >>>>>>>>>> something similar?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Am 14.03.22 um 06:29 schrieb Olivier Lamy:
> > > >>>>>>>>>>> On Mon, 14 Mar 2022 at 11:55, Tibor Digana <
> > > >> tibordigana@apache.org
> > > >>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> In case of the user property *maven.test.failure.ignore*
> the
> > > >> MOJO
> > > >>>>>>>> must
> > > >>>>>>>>>> not
> > > >>>>>>>>>>>> throw any exception which is interpreted by the Maven Core
> > as
> > > >>>> BUILD
> > > >>>>>>>>>>>> SUCCESS.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> This is a very simple reduction of the problem description.
> > > >>>>>>>>>>> The documentation here
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> >
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
> > > >>>>>>>>>>> says
> > > >>>>>>>>>>> "Set this to "true" to ignore a failure during testing. Its
> > use
> > > >> is
> > > >>>>>>>> NOT
> > > >>>>>>>>>>> RECOMMENDED, but quite convenient on occasion"
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Personally, I understand this to ignore failures in junit
> > > results
> > > >>>> BUT
> > > >>>>>>>>> at
> > > >>>>>>>>>>> least I want the tests to run.
> > > >>>>>>>>>>> I guess this is how our users use this feature (feature we
> do
> > > not
> > > >>>>>>>>>> recommend
> > > >>>>>>>>>>> by the way...)
> > > >>>>>>>>>>> And this is perfectly explained here
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > >
> >
> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
> > > >>>>>>>>>>> there is a difference between ignoring some junit failures
> > and
> > > >>>>>>>>> ignoring a
> > > >>>>>>>>>>> configuration error because some jvm args has been
> > > misconfigured
> > > >>>> for
> > > >>>>>>>>> many
> > > >>>>>>>>>>> reasons and surefire cannot start.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> If I follow your reasoning ("MOJO must not throw any
> > exception
> > > ")
> > > >>>> we
> > > >>>>>>>>>> should
> > > >>>>>>>>>>> ignore every surefire configuration error and keep running
> > the
> > > >>>> build
> > > >>>>>>>>>> until
> > > >>>>>>>>>>> the end and says BUILD SUCCESS
> > > >>>>>>>>>>> what about
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> mvn test -Dsurefire.rerunFailingTestsCount=notanumber
> > > >>>>>>>>>>> -Dmaven.test.failure.ignore=true
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> we should not throw any exceptions as you said below and
> keep
> > > >>>> saying
> > > >>>>>>>>>> BUILD
> > > >>>>>>>>>>> SUCCESS?
> > > >>>>>>>>>>> argLine is a configuration element as any others so it
> should
> > > >> fail
> > > >>>>>>>>>> directly
> > > >>>>>>>>>>> and not be ignored especially when the surefire plugin
> cannot
> > > >> run.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> We have received an internal requirement to change the
> > > behavior
> > > >> of
> > > >>>>>>>> the
> > > >>>>>>>>>> user
> > > >>>>>>>>>>>> property *maven.test.failure.ignore* so that the behavior
> > will
> > > >>>> have
> > > >>>>>>>>> one
> > > >>>>>>>>>>>> exception.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Suppose that you have JDK 1.8 but you use:
> > > >>>>>>>>>>>> /jre/java --add-reads ...
> > > >>>>>>>>>>>> The outcome is JVM exit with an error message.
> > > >>>>>>>>>>>> I agree with Herve who also says that
> > > >> *maven.test.failure.ignore*
> > > >>>>>>>>>> should
> > > >>>>>>>>>>>> not allow the MOJO to throw the exception. It is not a
> > typical
> > > >> JVM
> > > >>>>>>>>>>>> segmentation fault or another JVM crash where we cannot do
> > > >>>> anything
> > > >>>>>>>>>> about
> > > >>>>>>>>>>>> it, and where the entire build would crash in the command
> > line
> > > >>>> which
> > > >>>>>>>>>>>> of course means that the build cannot normally be
> > interpreted
> > > as
> > > >>>>>>>> BUILD
> > > >>>>>>>>>>>> SUCCESS. So we are still on the same level of failures
> > related
> > > >> to
> > > >>>>>>>> the
> > > >>>>>>>>>> test
> > > >>>>>>>>>>>> purposes.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On the other hand, Olivier has reopened the issues
> > > SUREFIRE-1426
> > > >>>> and
> > > >>>>>>>>>>>> SUREFIRE-1681 where the exceptional behavior of the
> feature
> > is
> > > >>>>>>>>> expected.
> > > >>>>>>>>>>>> This is exactly the reason why I closed these tickets
> > several
> > > >>>> years
> > > >>>>>>>>> ago
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>> my proposal was to not to have any exceptions in the
> feature
> > > >>>>>>>> behavior
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>> the proposal was to introduce a new user property for
> exact
> > > use
> > > >>>>>>>> cases.
> > > >>>>>>>>>>>> The next idea, which comes from two developers, would
> > provide
> > > us
> > > >>>>>>>> with
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>> same non exceptional and exact behavior of the user
> property
> > > >>>>>>>>>>>> *maven.test.failure.ignore* but it would also provide us
> > with
> > > >> new
> > > >>>>>>>> user
> > > >>>>>>>>>>>> property in the case with fine grade control of the build
> > > >> errors,
> > > >>>>>>>> e.g.
> > > >>>>>>>>>>>> *maven.test.jvm.error.ignore*.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> with a default of?
> > > >>>>>>>>>>> honestly I just see this new parameter as introducing more
> > > >>>> complexity
> > > >>>>>>>>> in
> > > >>>>>>>>>> an
> > > >>>>>>>>>>> already very complex code and not worth it.
> > > >>>>>>>>>>> again read both issue's comments and my comments.
> > > >>>>>>>>>>> Please try to have a user POV and think about making our
> > users'
> > > >>>>>>>>>> experience
> > > >>>>>>>>>>> more simple.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> This should be very simple:
> > > >>>>>>>>>>> If surefire forked jvm cannot start it's build error and
> > cannot
> > > >> run
> > > >>>>>>>> any
> > > >>>>>>>>>>> tests, it's a problem users want to know immediately
> because
> > it
> > > >> can
> > > >>>>>>>> be
> > > >>>>>>>>>> for
> > > >>>>>>>>>>> a lot of reasons: wrong argLine, not enough memory on the
> > > system
> > > >>>>>>>> etc...
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> AND AGAIN it is very different from ignoring junit result
> > > >> failures.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Try to look at a user point of view and think about what is
> > the
> > > >> use
> > > >>>>>>>>> case
> > > >>>>>>>>>> of
> > > >>>>>>>>>>> the property maven.test.failure.ignore=true, I guess 99% of
> > the
> > > >>>> time,
> > > >>>>>>>>>> users
> > > >>>>>>>>>>> wants to run all their tests (even on a CI with different
> > > matrix)
> > > >>>> so
> > > >>>>>>>>> they
> > > >>>>>>>>>>> can collect all the tests results which has runned to see
> if
> > > >> there
> > > >>>> is
> > > >>>>>>>>> any
> > > >>>>>>>>>>> issue for some combination of their matrix and eventually
> > turn
> > > >> the
> > > >>>>>>>>> result
> > > >>>>>>>>>>> as unstable (this is a very typical use case in Jenkins and
> > was
> > > >>>> even
> > > >>>>>>>> a
> > > >>>>>>>>>>> built in feature of the previous Jenkins Maven plugin)
> > > >>>>>>>>>>> BUT if for any reasons one of the module do not start his
> > tests
> > > >>>>>>>> because
> > > >>>>>>>>>> the
> > > >>>>>>>>>>> jvm cannot be start the users will not see that and will be
> > > >> totally
> > > >>>>>>>>> blind
> > > >>>>>>>>>>> until maybe someone look inside a very very large log file
> > > (which
> > > >>>>>>>> means
> > > >>>>>>>>>>> probably never)
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Long story short as my experience as a user facing this
> > > >>>> problem/bug:
> > > >>>>>>>>>>> I had the case on a very large multi modules (~250 modules)
> > > build
> > > >>>> of
> > > >>>>>>>> a
> > > >>>>>>>>>> very
> > > >>>>>>>>>>> used open source project.
> > > >>>>>>>>>>> I was using this maven.test.failure.ignore property but one
> > of
> > > >> the
> > > >>>>>>>>>> modules
> > > >>>>>>>>>>> had a bad jpms configuration for a jdk17 profile on the CI.
> > > >>>>>>>>>>> The build has a matrix of 2 os and 4 jdks and different
> maven
> > > run
> > > >>>>>>>> which
> > > >>>>>>>>>>> means around ~60k tests and a Jenkins log file about 40M
> > > >>>>>>>>>>> So because of this property the build was not failing and
> > kept
> > > >>>> saying
> > > >>>>>>>>>> BUILD
> > > >>>>>>>>>>> SUCCESS for weeks/months and basically not testing one
> module
> > > >> with
> > > >>>>>>>> jdk
> > > >>>>>>>>>> 17...
> > > >>>>>>>>>>> And frankly you do not dig into a log file of 32M after
> each
> > > run
> > > >>>>>>>>>> especially
> > > >>>>>>>>>>> when it says success...
> > > >>>>>>>>>>> 3 days after the first release claiming supporting jdk 17
> we
> > > >>>>>>>> received a
> > > >>>>>>>>>> bug
> > > >>>>>>>>>>> report about a something not working with jdk17....
> > > >>>>>>>>>>> and guess what? Where was this feature supposed to be
> tested?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> so I frankly believe we do not need a complex new property,
> > in
> > > >> this
> > > >>>>>>>>> case
> > > >>>>>>>>>>> this should fail directly because this will improve user
> > > >>>> experience.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I would like to open the discussion on this topic. You're
> > > >> welcome!
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Cheers
> > > >>>>>>>>>>>> Tibor
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>
> > ---------------------------------------------------------------------
> > > >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > >>>>>>>>>> For additional commands, e-mail: dev-help@maven.apache.org
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> --
> > > >>>>>>>>> ------------------------
> > > >>>>>>>>> Guillaume Nodet
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > ---------------------------------------------------------------------
> > > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > >>>>>> For additional commands, e-mail: dev-help@maven.apache.org
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > ---------------------------------------------------------------------
> > > >>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > >>>> For additional commands, e-mail: dev-help@maven.apache.org
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > >> For additional commands, e-mail: dev-help@maven.apache.org
> > > >>
> > > >>
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > For additional commands, e-mail: dev-help@maven.apache.org
> > >
> > >
> >
> > --
> > Sławomir Jaranowski
> >
>
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Tibor Digana <ti...@apache.org>.
Sorry for late reply.
I have created a demo project
https://github.com/Tibor17/maven.test.failure.ignore/ which simulates OOM.
According to the definition of the parameter "maven.failure.test.ignore"
the failures (also errors) should be ignored during testing.
The word "during" is important because the time when it was a failure
before JVM startup it was not a test failure - it is MOJO failure. The
author who wrote this config param knew this difference.
This perfectly fits JVM illegal arguments like -Xxxx.
But I have investigated OOM for the same reason with one JVM. My suspicion
was not confirmed, see the next...
If there are two JVMs, we are breaking the JVM during testing. It is a very
unlikely situation that the JVM argument would not fail in JVM1 but it
would fail in JVM2 later. It may happen only if the user
uses @surefire.forkCount@ interpolation but it is again very unlikely and
the users use it in current working directory configuration.
I understand Olivier's PR but I have one objection to the code branching in
the class *SurefireHelper*. It is based on 3 levels but I have to disagree
due to there still should be 2 levels as before:
1. ignored failures (my change is only here - yellow - only added throwing
exception)
2. processes failures
and so, I have created a PR
https://github.com/apache/maven-surefire/pull/496
Pls have a look into it, thx.
This diff looks very simple:
if ( reportParameters.isTestFailureIgnore() )
{
String errorMessage = createErrorMessage( reportParameters,
result, firstForkException );
if ( firstForkException instanceof SurefireBooterForkException )
{
throw new MojoExecutionException( errorMessage, firstForkException );
}
log.error( errorMessage );
}
else
{
throwException( reportParameters, result, firstForkException );
}
*$ mvn test*
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
2.445 s <<< FAILURE! - in org.apache.maven.OOMTest
[ERROR] org.apache.maven.OOMTest.test Time elapsed: 2.063 s <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
at org.apache.maven.OOMTest.test(OOMTest.java:15)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] OOMTest.test:15 » OutOfMemory Java heap space
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
*$ mvn test -Dmaven.test.failure.ignore*
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
2.472 s <<< FAILURE! - in org.apache.maven.OOMTest
[ERROR] org.apache.maven.OOMTest.test Time elapsed: 2.262 s <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
at org.apache.maven.OOMTest.test(OOMTest.java:15)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] OOMTest.test:15 » OutOfMemory Java heap space
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[ERROR]
Please refer to
C:\vcs\github\maven.test.failure.ignore\target\surefire-reports for the
individual test results.
Please refer to dump files (if any exist) [date].dump,
[date]-jvmRun[N].dump and [date].dumpstream.
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
On Sat, Mar 19, 2022 at 12:17 PM Slawomir Jaranowski <s....@gmail.com>
wrote:
> Hi,
>
> I start to feel that we first try to define technical details but we don't
> know for what.
> Each business requirement should be resolved as simply as possible from a
> technical perspective.
> In other ways we will start to build complicated and many features which
> probably were not needed at all.
>
> Please first define business requirements with examples, next we can find
> technical ways to cover it.
>
> Example scenario:
>
> Businesses requirement:
> We want to collect history of test result in order to detect flaky tests,
> analize tests execution time, etc
>
> Given:
> - build is done by CI system
> - there is system which collect reports for executed tests and build
> history of it
> - there are broken tests in build - we know them and accept (eg, on some
> os)
>
> When:
> - build finished with success on CI
>
> Then:
> - all tests result are reported
> - failed tests are reported
>
>
>
>
> sob., 19 mar 2022 o 06:27 Christoph Läubrich <ma...@laeubi-soft.de>
> napisał(a):
>
> > Hi Tibor,
> >
> > just to make it clear, I don't talkin about log-levels here but how to
> > intepret/handle a failure/failing/crashing test.
> >
> >
> > Am 18.03.22 um 21:54 schrieb Tibor Digaňa:
> > > Christoph, just let me briefly explain the log level hierarchy.
> > > If you select WARN log level, then ERROR can be printed too.
> > > Similar with INFO, means that WARN and ERROR would be printed as well.
> > >
> > > The real decision making in the plugin is a bit more complicated, see
> the
> > > pull request https://github.com/apache/maven-surefire/pull/478
> > > After I have replied to this email, I would verify the behavior of the
> > > plugin simulating OOM.
> > >
> > >
> > > T
> > >
> > >
> > > On Fri, Mar 18, 2022 at 5:42 AM Christoph Läubrich <
> mail@laeubi-soft.de>
> > > wrote:
> > >
> > >> No I think more about
> > >>
> > >> if (severity == "WARN") {
> > >> log.warn("There is something wrong");
> > >> } else if (severity == "ERROR") {
> > >> throw new MojoFailureException("...") {
> > >> } else {
> > >> throw new MojoExecutionException("...") {
> > >> }
> > >>
> > >> That way the plugin can handle any error/failure/... in a way that the
> > >> user can "downgrade" a certain problem if desired.
> > >>
> > >>
> > >> Am 18.03.22 um 00:06 schrieb Tibor Digana:
> > >>> @Christoph
> > >>> FATAL , WARN , ERROR
> > >>> They are log levels?
> > >>> The plugin does not control the log level after caught exception in
> > Maven
> > >>> core. The Maven Core does!
> > >>> I think it's time to make a demo.
> > >>>
> > >>> On Thu, Mar 17, 2022 at 6:21 AM Christoph Läubrich <
> > mail@laeubi-soft.de>
> > >>> wrote:
> > >>>
> > >>>> Hi Tibor,
> > >>>>
> > >>>> it shouldn't be to hard to guess another property like
> > >>>>
> > >>>> maven.test.jvmcrash=FATAL
> > >>>> maven.test.failure=WARN
> > >>>> maven.test.error=ERROR
> > >>>>
> > >>>> :-)
> > >>>>
> > >>>> Am 16.03.22 um 12:25 schrieb Tibor Digana:
> > >>>>> Hi Christoph,
> > >>>>>
> > >>>>> Such a granularity with error/failure might be also an additional
> > >>>>> requirement but still you miss the third option to JVM error which
> is
> > >>>>> different from test error/failure - they don;t have the same
> meaning.
> > >>>>>
> > >>>>> T
> > >>>>>
> > >>>>>
> > >>>>> On Mon, Mar 14, 2022 at 10:57 AM Christoph Läubrich <
> > >> mail@laeubi-soft.de
> > >>>>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Just wondering but maybe it would be better to configure the
> > severity
> > >>>>>> instead of a true/false, something like:
> > >>>>>>
> > >>>>>> maven.test.failure=WARN
> > >>>>>> maven.test.error=ERROR
> > >>>>>>
> > >>>>>> would only warn about failing tests but thrw exception if starting
> > the
> > >>>>>> test fails?
> > >>>>>>
> > >>>>>> Am 14.03.22 um 10:52 schrieb Tibor Digana:
> > >>>>>>> Romain, it is not a bug.
> > >>>>>>> Don't consider this as a bug. It was a feature request for change
> > by
> > >>>>>>> Olivier, and not a bug.
> > >>>>>>> I closed both issues years ago but not because of ignorance but
> > >> because
> > >>>>>> the
> > >>>>>>> appearance of the exceptional behavior is a wrong compromise and
> > >> which
> > >>>>>> does
> > >>>>>>> not help anyone and even it makes the situation worse because
> > >> typically
> > >>>>>>> other group of users would fire a new Jira tickets. You would not
> > be
> > >>>> able
> > >>>>>>> to satisfy two contradictory parties which have completely
> > different
> > >>>>>>> opinions, and so we use to introduce new params and this way we
> > >> satisfy
> > >>>>>>> both parties, they may combine the params as they wish, and
> > everybody
> > >>>>>> would
> > >>>>>>> be happy and nobody would claim. Many technical solutions might
> > >> exist,
> > >>>>>> e.g.
> > >>>>>>> param=boolean|string or new param=boolean, whatever.
> > >>>>>>>
> > >>>>>>> The truth is that this problem with (java --add-reads ...)
> happened
> > >> in
> > >>>>>> our
> > >>>>>>> ASF environment on Java 8 which in good configuration would not
> > >> happen
> > >>>>>> and
> > >>>>>>> should not.
> > >>>>>>> It is not right way that we abuse "maven.test.failure.ignore"
> which
> > >>>> would
> > >>>>>>> tell us "Hey, you have illegal configuration in your build system
> > and
> > >>>> it
> > >>>>>>> would not work by JDK design", it is not the goal of the plugin
> to
> > >> tell
> > >>>>>> you
> > >>>>>>> that you have configured the build wrong because it won't and the
> > >> same
> > >>>>>>> configuration of the plugin (not the build) says "ignore the
> > error".
> > >>>>>>> Yesterday I discussed this problem with Herve and we
> independently
> > >>>>>> observed
> > >>>>>>> equal opinions and that's not everything because we understood
> that
> > >> the
> > >>>>>>> purpose of the config parameter is to not throw MOJO exception
> > which
> > >> is
> > >>>>>>> right, but the exceptional behavior should have an exact new
> param
> > >> for
> > >>>>>> the
> > >>>>>>> exact use case.
> > >>>>>>> SPI for this parameter is too much because no user would
> implement
> > it
> > >>>>>> for a
> > >>>>>>> trivial parameter;; the SPI is used to be implemented by
> frameworks
> > >> or
> > >>>>>>> systems or application servers but this is not our case.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <
> > >>>>>> rmannibucau@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> +1
> > >>>>>>>> if it is to investigate a CI issue, it is generally easy to add
> > >> debug
> > >>>>>>>> insights (by code or agent) so a SPI sounds like the sanest for
> > the
> > >>>>>> plugin
> > >>>>>>>> to me.
> > >>>>>>>>
> > >>>>>>>> Romain Manni-Bucau
> > >>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > >>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
> > >>>>>>>> <http://rmannibucau.wordpress.com> | Github <
> > >>>>>>>> https://github.com/rmannibucau> |
> > >>>>>>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> > >>>>>>>> <
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <
> gnodet@apache.org>
> > a
> > >>>>>> écrit
> > >>>>>>>> :
> > >>>>>>>>
> > >>>>>>>>> If that's not currently possible, maybe a SPI should be
> provided
> > so
> > >>>>>> that
> > >>>>>>>>> people can use plug in extensions to analyze the test result
> and
> > >>>>>> override
> > >>>>>>>>> it if necessary (transforming an error into a warning, storing
> > >>>> results
> > >>>>>>>> in a
> > >>>>>>>>> way which is easier to use by other tools later...) ?
> > >>>>>>>>>
> > >>>>>>>>> Guillaume
> > >>>>>>>>>
> > >>>>>>>>> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <
> > >>>> mail@laeubi-soft.de>
> > >>>>>> a
> > >>>>>>>>> écrit :
> > >>>>>>>>>
> > >>>>>>>>>> I also agree that the test at least should run, we use this
> > >> property
> > >>>>>> to
> > >>>>>>>>>> run the test and produce result and later on have a buildstep
> > that
> > >>>>>>>>>> analyze the results (and probably fail the build job).
> > >>>>>>>>>>
> > >>>>>>>>>> As it is not recommend, I wonder what is the recommended way
> to
> > >>>>>> archive
> > >>>>>>>>>> something similar?
> > >>>>>>>>>>
> > >>>>>>>>>> Am 14.03.22 um 06:29 schrieb Olivier Lamy:
> > >>>>>>>>>>> On Mon, 14 Mar 2022 at 11:55, Tibor Digana <
> > >> tibordigana@apache.org
> > >>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> In case of the user property *maven.test.failure.ignore* the
> > >> MOJO
> > >>>>>>>> must
> > >>>>>>>>>> not
> > >>>>>>>>>>>> throw any exception which is interpreted by the Maven Core
> as
> > >>>> BUILD
> > >>>>>>>>>>>> SUCCESS.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> This is a very simple reduction of the problem description.
> > >>>>>>>>>>> The documentation here
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
> > >>>>>>>>>>> says
> > >>>>>>>>>>> "Set this to "true" to ignore a failure during testing. Its
> use
> > >> is
> > >>>>>>>> NOT
> > >>>>>>>>>>> RECOMMENDED, but quite convenient on occasion"
> > >>>>>>>>>>>
> > >>>>>>>>>>> Personally, I understand this to ignore failures in junit
> > results
> > >>>> BUT
> > >>>>>>>>> at
> > >>>>>>>>>>> least I want the tests to run.
> > >>>>>>>>>>> I guess this is how our users use this feature (feature we do
> > not
> > >>>>>>>>>> recommend
> > >>>>>>>>>>> by the way...)
> > >>>>>>>>>>> And this is perfectly explained here
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
> > >>>>>>>>>>> there is a difference between ignoring some junit failures
> and
> > >>>>>>>>> ignoring a
> > >>>>>>>>>>> configuration error because some jvm args has been
> > misconfigured
> > >>>> for
> > >>>>>>>>> many
> > >>>>>>>>>>> reasons and surefire cannot start.
> > >>>>>>>>>>>
> > >>>>>>>>>>> If I follow your reasoning ("MOJO must not throw any
> exception
> > ")
> > >>>> we
> > >>>>>>>>>> should
> > >>>>>>>>>>> ignore every surefire configuration error and keep running
> the
> > >>>> build
> > >>>>>>>>>> until
> > >>>>>>>>>>> the end and says BUILD SUCCESS
> > >>>>>>>>>>> what about
> > >>>>>>>>>>>
> > >>>>>>>>>>> mvn test -Dsurefire.rerunFailingTestsCount=notanumber
> > >>>>>>>>>>> -Dmaven.test.failure.ignore=true
> > >>>>>>>>>>>
> > >>>>>>>>>>> we should not throw any exceptions as you said below and keep
> > >>>> saying
> > >>>>>>>>>> BUILD
> > >>>>>>>>>>> SUCCESS?
> > >>>>>>>>>>> argLine is a configuration element as any others so it should
> > >> fail
> > >>>>>>>>>> directly
> > >>>>>>>>>>> and not be ignored especially when the surefire plugin cannot
> > >> run.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> We have received an internal requirement to change the
> > behavior
> > >> of
> > >>>>>>>> the
> > >>>>>>>>>> user
> > >>>>>>>>>>>> property *maven.test.failure.ignore* so that the behavior
> will
> > >>>> have
> > >>>>>>>>> one
> > >>>>>>>>>>>> exception.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Suppose that you have JDK 1.8 but you use:
> > >>>>>>>>>>>> /jre/java --add-reads ...
> > >>>>>>>>>>>> The outcome is JVM exit with an error message.
> > >>>>>>>>>>>> I agree with Herve who also says that
> > >> *maven.test.failure.ignore*
> > >>>>>>>>>> should
> > >>>>>>>>>>>> not allow the MOJO to throw the exception. It is not a
> typical
> > >> JVM
> > >>>>>>>>>>>> segmentation fault or another JVM crash where we cannot do
> > >>>> anything
> > >>>>>>>>>> about
> > >>>>>>>>>>>> it, and where the entire build would crash in the command
> line
> > >>>> which
> > >>>>>>>>>>>> of course means that the build cannot normally be
> interpreted
> > as
> > >>>>>>>> BUILD
> > >>>>>>>>>>>> SUCCESS. So we are still on the same level of failures
> related
> > >> to
> > >>>>>>>> the
> > >>>>>>>>>> test
> > >>>>>>>>>>>> purposes.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On the other hand, Olivier has reopened the issues
> > SUREFIRE-1426
> > >>>> and
> > >>>>>>>>>>>> SUREFIRE-1681 where the exceptional behavior of the feature
> is
> > >>>>>>>>> expected.
> > >>>>>>>>>>>> This is exactly the reason why I closed these tickets
> several
> > >>>> years
> > >>>>>>>>> ago
> > >>>>>>>>>> and
> > >>>>>>>>>>>> my proposal was to not to have any exceptions in the feature
> > >>>>>>>> behavior
> > >>>>>>>>>> and
> > >>>>>>>>>>>> the proposal was to introduce a new user property for exact
> > use
> > >>>>>>>> cases.
> > >>>>>>>>>>>> The next idea, which comes from two developers, would
> provide
> > us
> > >>>>>>>> with
> > >>>>>>>>>> the
> > >>>>>>>>>>>> same non exceptional and exact behavior of the user property
> > >>>>>>>>>>>> *maven.test.failure.ignore* but it would also provide us
> with
> > >> new
> > >>>>>>>> user
> > >>>>>>>>>>>> property in the case with fine grade control of the build
> > >> errors,
> > >>>>>>>> e.g.
> > >>>>>>>>>>>> *maven.test.jvm.error.ignore*.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> with a default of?
> > >>>>>>>>>>> honestly I just see this new parameter as introducing more
> > >>>> complexity
> > >>>>>>>>> in
> > >>>>>>>>>> an
> > >>>>>>>>>>> already very complex code and not worth it.
> > >>>>>>>>>>> again read both issue's comments and my comments.
> > >>>>>>>>>>> Please try to have a user POV and think about making our
> users'
> > >>>>>>>>>> experience
> > >>>>>>>>>>> more simple.
> > >>>>>>>>>>>
> > >>>>>>>>>>> This should be very simple:
> > >>>>>>>>>>> If surefire forked jvm cannot start it's build error and
> cannot
> > >> run
> > >>>>>>>> any
> > >>>>>>>>>>> tests, it's a problem users want to know immediately because
> it
> > >> can
> > >>>>>>>> be
> > >>>>>>>>>> for
> > >>>>>>>>>>> a lot of reasons: wrong argLine, not enough memory on the
> > system
> > >>>>>>>> etc...
> > >>>>>>>>>>>
> > >>>>>>>>>>> AND AGAIN it is very different from ignoring junit result
> > >> failures.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Try to look at a user point of view and think about what is
> the
> > >> use
> > >>>>>>>>> case
> > >>>>>>>>>> of
> > >>>>>>>>>>> the property maven.test.failure.ignore=true, I guess 99% of
> the
> > >>>> time,
> > >>>>>>>>>> users
> > >>>>>>>>>>> wants to run all their tests (even on a CI with different
> > matrix)
> > >>>> so
> > >>>>>>>>> they
> > >>>>>>>>>>> can collect all the tests results which has runned to see if
> > >> there
> > >>>> is
> > >>>>>>>>> any
> > >>>>>>>>>>> issue for some combination of their matrix and eventually
> turn
> > >> the
> > >>>>>>>>> result
> > >>>>>>>>>>> as unstable (this is a very typical use case in Jenkins and
> was
> > >>>> even
> > >>>>>>>> a
> > >>>>>>>>>>> built in feature of the previous Jenkins Maven plugin)
> > >>>>>>>>>>> BUT if for any reasons one of the module do not start his
> tests
> > >>>>>>>> because
> > >>>>>>>>>> the
> > >>>>>>>>>>> jvm cannot be start the users will not see that and will be
> > >> totally
> > >>>>>>>>> blind
> > >>>>>>>>>>> until maybe someone look inside a very very large log file
> > (which
> > >>>>>>>> means
> > >>>>>>>>>>> probably never)
> > >>>>>>>>>>>
> > >>>>>>>>>>> Long story short as my experience as a user facing this
> > >>>> problem/bug:
> > >>>>>>>>>>> I had the case on a very large multi modules (~250 modules)
> > build
> > >>>> of
> > >>>>>>>> a
> > >>>>>>>>>> very
> > >>>>>>>>>>> used open source project.
> > >>>>>>>>>>> I was using this maven.test.failure.ignore property but one
> of
> > >> the
> > >>>>>>>>>> modules
> > >>>>>>>>>>> had a bad jpms configuration for a jdk17 profile on the CI.
> > >>>>>>>>>>> The build has a matrix of 2 os and 4 jdks and different maven
> > run
> > >>>>>>>> which
> > >>>>>>>>>>> means around ~60k tests and a Jenkins log file about 40M
> > >>>>>>>>>>> So because of this property the build was not failing and
> kept
> > >>>> saying
> > >>>>>>>>>> BUILD
> > >>>>>>>>>>> SUCCESS for weeks/months and basically not testing one module
> > >> with
> > >>>>>>>> jdk
> > >>>>>>>>>> 17...
> > >>>>>>>>>>> And frankly you do not dig into a log file of 32M after each
> > run
> > >>>>>>>>>> especially
> > >>>>>>>>>>> when it says success...
> > >>>>>>>>>>> 3 days after the first release claiming supporting jdk 17 we
> > >>>>>>>> received a
> > >>>>>>>>>> bug
> > >>>>>>>>>>> report about a something not working with jdk17....
> > >>>>>>>>>>> and guess what? Where was this feature supposed to be tested?
> > >>>>>>>>>>>
> > >>>>>>>>>>> so I frankly believe we do not need a complex new property,
> in
> > >> this
> > >>>>>>>>> case
> > >>>>>>>>>>> this should fail directly because this will improve user
> > >>>> experience.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I would like to open the discussion on this topic. You're
> > >> welcome!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Cheers
> > >>>>>>>>>>>> Tibor
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > >>>>>>>>>> For additional commands, e-mail: dev-help@maven.apache.org
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>> ------------------------
> > >>>>>>>>> Guillaume Nodet
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > ---------------------------------------------------------------------
> > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > >>>>>> For additional commands, e-mail: dev-help@maven.apache.org
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > >>>> For additional commands, e-mail: dev-help@maven.apache.org
> > >>>>
> > >>>>
> > >>>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > >> For additional commands, e-mail: dev-help@maven.apache.org
> > >>
> > >>
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
> >
>
> --
> Sławomir Jaranowski
>
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Slawomir Jaranowski <s....@gmail.com>.
Hi,
I start to feel that we first try to define technical details but we don't
know for what.
Each business requirement should be resolved as simply as possible from a
technical perspective.
In other ways we will start to build complicated and many features which
probably were not needed at all.
Please first define business requirements with examples, next we can find
technical ways to cover it.
Example scenario:
Businesses requirement:
We want to collect history of test result in order to detect flaky tests,
analize tests execution time, etc
Given:
- build is done by CI system
- there is system which collect reports for executed tests and build
history of it
- there are broken tests in build - we know them and accept (eg, on some os)
When:
- build finished with success on CI
Then:
- all tests result are reported
- failed tests are reported
sob., 19 mar 2022 o 06:27 Christoph Läubrich <ma...@laeubi-soft.de>
napisał(a):
> Hi Tibor,
>
> just to make it clear, I don't talkin about log-levels here but how to
> intepret/handle a failure/failing/crashing test.
>
>
> Am 18.03.22 um 21:54 schrieb Tibor Digaňa:
> > Christoph, just let me briefly explain the log level hierarchy.
> > If you select WARN log level, then ERROR can be printed too.
> > Similar with INFO, means that WARN and ERROR would be printed as well.
> >
> > The real decision making in the plugin is a bit more complicated, see the
> > pull request https://github.com/apache/maven-surefire/pull/478
> > After I have replied to this email, I would verify the behavior of the
> > plugin simulating OOM.
> >
> >
> > T
> >
> >
> > On Fri, Mar 18, 2022 at 5:42 AM Christoph Läubrich <ma...@laeubi-soft.de>
> > wrote:
> >
> >> No I think more about
> >>
> >> if (severity == "WARN") {
> >> log.warn("There is something wrong");
> >> } else if (severity == "ERROR") {
> >> throw new MojoFailureException("...") {
> >> } else {
> >> throw new MojoExecutionException("...") {
> >> }
> >>
> >> That way the plugin can handle any error/failure/... in a way that the
> >> user can "downgrade" a certain problem if desired.
> >>
> >>
> >> Am 18.03.22 um 00:06 schrieb Tibor Digana:
> >>> @Christoph
> >>> FATAL , WARN , ERROR
> >>> They are log levels?
> >>> The plugin does not control the log level after caught exception in
> Maven
> >>> core. The Maven Core does!
> >>> I think it's time to make a demo.
> >>>
> >>> On Thu, Mar 17, 2022 at 6:21 AM Christoph Läubrich <
> mail@laeubi-soft.de>
> >>> wrote:
> >>>
> >>>> Hi Tibor,
> >>>>
> >>>> it shouldn't be to hard to guess another property like
> >>>>
> >>>> maven.test.jvmcrash=FATAL
> >>>> maven.test.failure=WARN
> >>>> maven.test.error=ERROR
> >>>>
> >>>> :-)
> >>>>
> >>>> Am 16.03.22 um 12:25 schrieb Tibor Digana:
> >>>>> Hi Christoph,
> >>>>>
> >>>>> Such a granularity with error/failure might be also an additional
> >>>>> requirement but still you miss the third option to JVM error which is
> >>>>> different from test error/failure - they don;t have the same meaning.
> >>>>>
> >>>>> T
> >>>>>
> >>>>>
> >>>>> On Mon, Mar 14, 2022 at 10:57 AM Christoph Läubrich <
> >> mail@laeubi-soft.de
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Just wondering but maybe it would be better to configure the
> severity
> >>>>>> instead of a true/false, something like:
> >>>>>>
> >>>>>> maven.test.failure=WARN
> >>>>>> maven.test.error=ERROR
> >>>>>>
> >>>>>> would only warn about failing tests but thrw exception if starting
> the
> >>>>>> test fails?
> >>>>>>
> >>>>>> Am 14.03.22 um 10:52 schrieb Tibor Digana:
> >>>>>>> Romain, it is not a bug.
> >>>>>>> Don't consider this as a bug. It was a feature request for change
> by
> >>>>>>> Olivier, and not a bug.
> >>>>>>> I closed both issues years ago but not because of ignorance but
> >> because
> >>>>>> the
> >>>>>>> appearance of the exceptional behavior is a wrong compromise and
> >> which
> >>>>>> does
> >>>>>>> not help anyone and even it makes the situation worse because
> >> typically
> >>>>>>> other group of users would fire a new Jira tickets. You would not
> be
> >>>> able
> >>>>>>> to satisfy two contradictory parties which have completely
> different
> >>>>>>> opinions, and so we use to introduce new params and this way we
> >> satisfy
> >>>>>>> both parties, they may combine the params as they wish, and
> everybody
> >>>>>> would
> >>>>>>> be happy and nobody would claim. Many technical solutions might
> >> exist,
> >>>>>> e.g.
> >>>>>>> param=boolean|string or new param=boolean, whatever.
> >>>>>>>
> >>>>>>> The truth is that this problem with (java --add-reads ...) happened
> >> in
> >>>>>> our
> >>>>>>> ASF environment on Java 8 which in good configuration would not
> >> happen
> >>>>>> and
> >>>>>>> should not.
> >>>>>>> It is not right way that we abuse "maven.test.failure.ignore" which
> >>>> would
> >>>>>>> tell us "Hey, you have illegal configuration in your build system
> and
> >>>> it
> >>>>>>> would not work by JDK design", it is not the goal of the plugin to
> >> tell
> >>>>>> you
> >>>>>>> that you have configured the build wrong because it won't and the
> >> same
> >>>>>>> configuration of the plugin (not the build) says "ignore the
> error".
> >>>>>>> Yesterday I discussed this problem with Herve and we independently
> >>>>>> observed
> >>>>>>> equal opinions and that's not everything because we understood that
> >> the
> >>>>>>> purpose of the config parameter is to not throw MOJO exception
> which
> >> is
> >>>>>>> right, but the exceptional behavior should have an exact new param
> >> for
> >>>>>> the
> >>>>>>> exact use case.
> >>>>>>> SPI for this parameter is too much because no user would implement
> it
> >>>>>> for a
> >>>>>>> trivial parameter;; the SPI is used to be implemented by frameworks
> >> or
> >>>>>>> systems or application servers but this is not our case.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <
> >>>>>> rmannibucau@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> +1
> >>>>>>>> if it is to investigate a CI issue, it is generally easy to add
> >> debug
> >>>>>>>> insights (by code or agent) so a SPI sounds like the sanest for
> the
> >>>>>> plugin
> >>>>>>>> to me.
> >>>>>>>>
> >>>>>>>> Romain Manni-Bucau
> >>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> >>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
> >>>>>>>> <http://rmannibucau.wordpress.com> | Github <
> >>>>>>>> https://github.com/rmannibucau> |
> >>>>>>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> >>>>>>>> <
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <gn...@apache.org>
> a
> >>>>>> écrit
> >>>>>>>> :
> >>>>>>>>
> >>>>>>>>> If that's not currently possible, maybe a SPI should be provided
> so
> >>>>>> that
> >>>>>>>>> people can use plug in extensions to analyze the test result and
> >>>>>> override
> >>>>>>>>> it if necessary (transforming an error into a warning, storing
> >>>> results
> >>>>>>>> in a
> >>>>>>>>> way which is easier to use by other tools later...) ?
> >>>>>>>>>
> >>>>>>>>> Guillaume
> >>>>>>>>>
> >>>>>>>>> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <
> >>>> mail@laeubi-soft.de>
> >>>>>> a
> >>>>>>>>> écrit :
> >>>>>>>>>
> >>>>>>>>>> I also agree that the test at least should run, we use this
> >> property
> >>>>>> to
> >>>>>>>>>> run the test and produce result and later on have a buildstep
> that
> >>>>>>>>>> analyze the results (and probably fail the build job).
> >>>>>>>>>>
> >>>>>>>>>> As it is not recommend, I wonder what is the recommended way to
> >>>>>> archive
> >>>>>>>>>> something similar?
> >>>>>>>>>>
> >>>>>>>>>> Am 14.03.22 um 06:29 schrieb Olivier Lamy:
> >>>>>>>>>>> On Mon, 14 Mar 2022 at 11:55, Tibor Digana <
> >> tibordigana@apache.org
> >>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> In case of the user property *maven.test.failure.ignore* the
> >> MOJO
> >>>>>>>> must
> >>>>>>>>>> not
> >>>>>>>>>>>> throw any exception which is interpreted by the Maven Core as
> >>>> BUILD
> >>>>>>>>>>>> SUCCESS.
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> This is a very simple reduction of the problem description.
> >>>>>>>>>>> The documentation here
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
> >>>>>>>>>>> says
> >>>>>>>>>>> "Set this to "true" to ignore a failure during testing. Its use
> >> is
> >>>>>>>> NOT
> >>>>>>>>>>> RECOMMENDED, but quite convenient on occasion"
> >>>>>>>>>>>
> >>>>>>>>>>> Personally, I understand this to ignore failures in junit
> results
> >>>> BUT
> >>>>>>>>> at
> >>>>>>>>>>> least I want the tests to run.
> >>>>>>>>>>> I guess this is how our users use this feature (feature we do
> not
> >>>>>>>>>> recommend
> >>>>>>>>>>> by the way...)
> >>>>>>>>>>> And this is perfectly explained here
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
> >>>>>>>>>>> there is a difference between ignoring some junit failures and
> >>>>>>>>> ignoring a
> >>>>>>>>>>> configuration error because some jvm args has been
> misconfigured
> >>>> for
> >>>>>>>>> many
> >>>>>>>>>>> reasons and surefire cannot start.
> >>>>>>>>>>>
> >>>>>>>>>>> If I follow your reasoning ("MOJO must not throw any exception
> ")
> >>>> we
> >>>>>>>>>> should
> >>>>>>>>>>> ignore every surefire configuration error and keep running the
> >>>> build
> >>>>>>>>>> until
> >>>>>>>>>>> the end and says BUILD SUCCESS
> >>>>>>>>>>> what about
> >>>>>>>>>>>
> >>>>>>>>>>> mvn test -Dsurefire.rerunFailingTestsCount=notanumber
> >>>>>>>>>>> -Dmaven.test.failure.ignore=true
> >>>>>>>>>>>
> >>>>>>>>>>> we should not throw any exceptions as you said below and keep
> >>>> saying
> >>>>>>>>>> BUILD
> >>>>>>>>>>> SUCCESS?
> >>>>>>>>>>> argLine is a configuration element as any others so it should
> >> fail
> >>>>>>>>>> directly
> >>>>>>>>>>> and not be ignored especially when the surefire plugin cannot
> >> run.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> We have received an internal requirement to change the
> behavior
> >> of
> >>>>>>>> the
> >>>>>>>>>> user
> >>>>>>>>>>>> property *maven.test.failure.ignore* so that the behavior will
> >>>> have
> >>>>>>>>> one
> >>>>>>>>>>>> exception.
> >>>>>>>>>>>
> >>>>>>>>>>> Suppose that you have JDK 1.8 but you use:
> >>>>>>>>>>>> /jre/java --add-reads ...
> >>>>>>>>>>>> The outcome is JVM exit with an error message.
> >>>>>>>>>>>> I agree with Herve who also says that
> >> *maven.test.failure.ignore*
> >>>>>>>>>> should
> >>>>>>>>>>>> not allow the MOJO to throw the exception. It is not a typical
> >> JVM
> >>>>>>>>>>>> segmentation fault or another JVM crash where we cannot do
> >>>> anything
> >>>>>>>>>> about
> >>>>>>>>>>>> it, and where the entire build would crash in the command line
> >>>> which
> >>>>>>>>>>>> of course means that the build cannot normally be interpreted
> as
> >>>>>>>> BUILD
> >>>>>>>>>>>> SUCCESS. So we are still on the same level of failures related
> >> to
> >>>>>>>> the
> >>>>>>>>>> test
> >>>>>>>>>>>> purposes.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On the other hand, Olivier has reopened the issues
> SUREFIRE-1426
> >>>> and
> >>>>>>>>>>>> SUREFIRE-1681 where the exceptional behavior of the feature is
> >>>>>>>>> expected.
> >>>>>>>>>>>> This is exactly the reason why I closed these tickets several
> >>>> years
> >>>>>>>>> ago
> >>>>>>>>>> and
> >>>>>>>>>>>> my proposal was to not to have any exceptions in the feature
> >>>>>>>> behavior
> >>>>>>>>>> and
> >>>>>>>>>>>> the proposal was to introduce a new user property for exact
> use
> >>>>>>>> cases.
> >>>>>>>>>>>> The next idea, which comes from two developers, would provide
> us
> >>>>>>>> with
> >>>>>>>>>> the
> >>>>>>>>>>>> same non exceptional and exact behavior of the user property
> >>>>>>>>>>>> *maven.test.failure.ignore* but it would also provide us with
> >> new
> >>>>>>>> user
> >>>>>>>>>>>> property in the case with fine grade control of the build
> >> errors,
> >>>>>>>> e.g.
> >>>>>>>>>>>> *maven.test.jvm.error.ignore*.
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> with a default of?
> >>>>>>>>>>> honestly I just see this new parameter as introducing more
> >>>> complexity
> >>>>>>>>> in
> >>>>>>>>>> an
> >>>>>>>>>>> already very complex code and not worth it.
> >>>>>>>>>>> again read both issue's comments and my comments.
> >>>>>>>>>>> Please try to have a user POV and think about making our users'
> >>>>>>>>>> experience
> >>>>>>>>>>> more simple.
> >>>>>>>>>>>
> >>>>>>>>>>> This should be very simple:
> >>>>>>>>>>> If surefire forked jvm cannot start it's build error and cannot
> >> run
> >>>>>>>> any
> >>>>>>>>>>> tests, it's a problem users want to know immediately because it
> >> can
> >>>>>>>> be
> >>>>>>>>>> for
> >>>>>>>>>>> a lot of reasons: wrong argLine, not enough memory on the
> system
> >>>>>>>> etc...
> >>>>>>>>>>>
> >>>>>>>>>>> AND AGAIN it is very different from ignoring junit result
> >> failures.
> >>>>>>>>>>>
> >>>>>>>>>>> Try to look at a user point of view and think about what is the
> >> use
> >>>>>>>>> case
> >>>>>>>>>> of
> >>>>>>>>>>> the property maven.test.failure.ignore=true, I guess 99% of the
> >>>> time,
> >>>>>>>>>> users
> >>>>>>>>>>> wants to run all their tests (even on a CI with different
> matrix)
> >>>> so
> >>>>>>>>> they
> >>>>>>>>>>> can collect all the tests results which has runned to see if
> >> there
> >>>> is
> >>>>>>>>> any
> >>>>>>>>>>> issue for some combination of their matrix and eventually turn
> >> the
> >>>>>>>>> result
> >>>>>>>>>>> as unstable (this is a very typical use case in Jenkins and was
> >>>> even
> >>>>>>>> a
> >>>>>>>>>>> built in feature of the previous Jenkins Maven plugin)
> >>>>>>>>>>> BUT if for any reasons one of the module do not start his tests
> >>>>>>>> because
> >>>>>>>>>> the
> >>>>>>>>>>> jvm cannot be start the users will not see that and will be
> >> totally
> >>>>>>>>> blind
> >>>>>>>>>>> until maybe someone look inside a very very large log file
> (which
> >>>>>>>> means
> >>>>>>>>>>> probably never)
> >>>>>>>>>>>
> >>>>>>>>>>> Long story short as my experience as a user facing this
> >>>> problem/bug:
> >>>>>>>>>>> I had the case on a very large multi modules (~250 modules)
> build
> >>>> of
> >>>>>>>> a
> >>>>>>>>>> very
> >>>>>>>>>>> used open source project.
> >>>>>>>>>>> I was using this maven.test.failure.ignore property but one of
> >> the
> >>>>>>>>>> modules
> >>>>>>>>>>> had a bad jpms configuration for a jdk17 profile on the CI.
> >>>>>>>>>>> The build has a matrix of 2 os and 4 jdks and different maven
> run
> >>>>>>>> which
> >>>>>>>>>>> means around ~60k tests and a Jenkins log file about 40M
> >>>>>>>>>>> So because of this property the build was not failing and kept
> >>>> saying
> >>>>>>>>>> BUILD
> >>>>>>>>>>> SUCCESS for weeks/months and basically not testing one module
> >> with
> >>>>>>>> jdk
> >>>>>>>>>> 17...
> >>>>>>>>>>> And frankly you do not dig into a log file of 32M after each
> run
> >>>>>>>>>> especially
> >>>>>>>>>>> when it says success...
> >>>>>>>>>>> 3 days after the first release claiming supporting jdk 17 we
> >>>>>>>> received a
> >>>>>>>>>> bug
> >>>>>>>>>>> report about a something not working with jdk17....
> >>>>>>>>>>> and guess what? Where was this feature supposed to be tested?
> >>>>>>>>>>>
> >>>>>>>>>>> so I frankly believe we do not need a complex new property, in
> >> this
> >>>>>>>>> case
> >>>>>>>>>>> this should fail directly because this will improve user
> >>>> experience.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I would like to open the discussion on this topic. You're
> >> welcome!
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers
> >>>>>>>>>>>> Tibor
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> ---------------------------------------------------------------------
> >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>>>>>>>> For additional commands, e-mail: dev-help@maven.apache.org
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> ------------------------
> >>>>>>>>> Guillaume Nodet
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>>>> For additional commands, e-mail: dev-help@maven.apache.org
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>> For additional commands, e-mail: dev-help@maven.apache.org
> >>>>
> >>>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >> For additional commands, e-mail: dev-help@maven.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
--
Sławomir Jaranowski
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Christoph Läubrich <ma...@laeubi-soft.de>.
Hi Tibor,
just to make it clear, I don't talkin about log-levels here but how to
intepret/handle a failure/failing/crashing test.
Am 18.03.22 um 21:54 schrieb Tibor Digaňa:
> Christoph, just let me briefly explain the log level hierarchy.
> If you select WARN log level, then ERROR can be printed too.
> Similar with INFO, means that WARN and ERROR would be printed as well.
>
> The real decision making in the plugin is a bit more complicated, see the
> pull request https://github.com/apache/maven-surefire/pull/478
> After I have replied to this email, I would verify the behavior of the
> plugin simulating OOM.
>
>
> T
>
>
> On Fri, Mar 18, 2022 at 5:42 AM Christoph Läubrich <ma...@laeubi-soft.de>
> wrote:
>
>> No I think more about
>>
>> if (severity == "WARN") {
>> log.warn("There is something wrong");
>> } else if (severity == "ERROR") {
>> throw new MojoFailureException("...") {
>> } else {
>> throw new MojoExecutionException("...") {
>> }
>>
>> That way the plugin can handle any error/failure/... in a way that the
>> user can "downgrade" a certain problem if desired.
>>
>>
>> Am 18.03.22 um 00:06 schrieb Tibor Digana:
>>> @Christoph
>>> FATAL , WARN , ERROR
>>> They are log levels?
>>> The plugin does not control the log level after caught exception in Maven
>>> core. The Maven Core does!
>>> I think it's time to make a demo.
>>>
>>> On Thu, Mar 17, 2022 at 6:21 AM Christoph Läubrich <ma...@laeubi-soft.de>
>>> wrote:
>>>
>>>> Hi Tibor,
>>>>
>>>> it shouldn't be to hard to guess another property like
>>>>
>>>> maven.test.jvmcrash=FATAL
>>>> maven.test.failure=WARN
>>>> maven.test.error=ERROR
>>>>
>>>> :-)
>>>>
>>>> Am 16.03.22 um 12:25 schrieb Tibor Digana:
>>>>> Hi Christoph,
>>>>>
>>>>> Such a granularity with error/failure might be also an additional
>>>>> requirement but still you miss the third option to JVM error which is
>>>>> different from test error/failure - they don;t have the same meaning.
>>>>>
>>>>> T
>>>>>
>>>>>
>>>>> On Mon, Mar 14, 2022 at 10:57 AM Christoph Läubrich <
>> mail@laeubi-soft.de
>>>>>
>>>>> wrote:
>>>>>
>>>>>> Just wondering but maybe it would be better to configure the severity
>>>>>> instead of a true/false, something like:
>>>>>>
>>>>>> maven.test.failure=WARN
>>>>>> maven.test.error=ERROR
>>>>>>
>>>>>> would only warn about failing tests but thrw exception if starting the
>>>>>> test fails?
>>>>>>
>>>>>> Am 14.03.22 um 10:52 schrieb Tibor Digana:
>>>>>>> Romain, it is not a bug.
>>>>>>> Don't consider this as a bug. It was a feature request for change by
>>>>>>> Olivier, and not a bug.
>>>>>>> I closed both issues years ago but not because of ignorance but
>> because
>>>>>> the
>>>>>>> appearance of the exceptional behavior is a wrong compromise and
>> which
>>>>>> does
>>>>>>> not help anyone and even it makes the situation worse because
>> typically
>>>>>>> other group of users would fire a new Jira tickets. You would not be
>>>> able
>>>>>>> to satisfy two contradictory parties which have completely different
>>>>>>> opinions, and so we use to introduce new params and this way we
>> satisfy
>>>>>>> both parties, they may combine the params as they wish, and everybody
>>>>>> would
>>>>>>> be happy and nobody would claim. Many technical solutions might
>> exist,
>>>>>> e.g.
>>>>>>> param=boolean|string or new param=boolean, whatever.
>>>>>>>
>>>>>>> The truth is that this problem with (java --add-reads ...) happened
>> in
>>>>>> our
>>>>>>> ASF environment on Java 8 which in good configuration would not
>> happen
>>>>>> and
>>>>>>> should not.
>>>>>>> It is not right way that we abuse "maven.test.failure.ignore" which
>>>> would
>>>>>>> tell us "Hey, you have illegal configuration in your build system and
>>>> it
>>>>>>> would not work by JDK design", it is not the goal of the plugin to
>> tell
>>>>>> you
>>>>>>> that you have configured the build wrong because it won't and the
>> same
>>>>>>> configuration of the plugin (not the build) says "ignore the error".
>>>>>>> Yesterday I discussed this problem with Herve and we independently
>>>>>> observed
>>>>>>> equal opinions and that's not everything because we understood that
>> the
>>>>>>> purpose of the config parameter is to not throw MOJO exception which
>> is
>>>>>>> right, but the exceptional behavior should have an exact new param
>> for
>>>>>> the
>>>>>>> exact use case.
>>>>>>> SPI for this parameter is too much because no user would implement it
>>>>>> for a
>>>>>>> trivial parameter;; the SPI is used to be implemented by frameworks
>> or
>>>>>>> systems or application servers but this is not our case.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1
>>>>>>>> if it is to investigate a CI issue, it is generally easy to add
>> debug
>>>>>>>> insights (by code or agent) so a SPI sounds like the sanest for the
>>>>>> plugin
>>>>>>>> to me.
>>>>>>>>
>>>>>>>> Romain Manni-Bucau
>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>> <http://rmannibucau.wordpress.com> | Github <
>>>>>>>> https://github.com/rmannibucau> |
>>>>>>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>> <
>>>>>>>>
>>>>>>
>>>>
>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <gn...@apache.org> a
>>>>>> écrit
>>>>>>>> :
>>>>>>>>
>>>>>>>>> If that's not currently possible, maybe a SPI should be provided so
>>>>>> that
>>>>>>>>> people can use plug in extensions to analyze the test result and
>>>>>> override
>>>>>>>>> it if necessary (transforming an error into a warning, storing
>>>> results
>>>>>>>> in a
>>>>>>>>> way which is easier to use by other tools later...) ?
>>>>>>>>>
>>>>>>>>> Guillaume
>>>>>>>>>
>>>>>>>>> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <
>>>> mail@laeubi-soft.de>
>>>>>> a
>>>>>>>>> écrit :
>>>>>>>>>
>>>>>>>>>> I also agree that the test at least should run, we use this
>> property
>>>>>> to
>>>>>>>>>> run the test and produce result and later on have a buildstep that
>>>>>>>>>> analyze the results (and probably fail the build job).
>>>>>>>>>>
>>>>>>>>>> As it is not recommend, I wonder what is the recommended way to
>>>>>> archive
>>>>>>>>>> something similar?
>>>>>>>>>>
>>>>>>>>>> Am 14.03.22 um 06:29 schrieb Olivier Lamy:
>>>>>>>>>>> On Mon, 14 Mar 2022 at 11:55, Tibor Digana <
>> tibordigana@apache.org
>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> In case of the user property *maven.test.failure.ignore* the
>> MOJO
>>>>>>>> must
>>>>>>>>>> not
>>>>>>>>>>>> throw any exception which is interpreted by the Maven Core as
>>>> BUILD
>>>>>>>>>>>> SUCCESS.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This is a very simple reduction of the problem description.
>>>>>>>>>>> The documentation here
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
>>>>>>>>>>> says
>>>>>>>>>>> "Set this to "true" to ignore a failure during testing. Its use
>> is
>>>>>>>> NOT
>>>>>>>>>>> RECOMMENDED, but quite convenient on occasion"
>>>>>>>>>>>
>>>>>>>>>>> Personally, I understand this to ignore failures in junit results
>>>> BUT
>>>>>>>>> at
>>>>>>>>>>> least I want the tests to run.
>>>>>>>>>>> I guess this is how our users use this feature (feature we do not
>>>>>>>>>> recommend
>>>>>>>>>>> by the way...)
>>>>>>>>>>> And this is perfectly explained here
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
>>>>>>>>>>> there is a difference between ignoring some junit failures and
>>>>>>>>> ignoring a
>>>>>>>>>>> configuration error because some jvm args has been misconfigured
>>>> for
>>>>>>>>> many
>>>>>>>>>>> reasons and surefire cannot start.
>>>>>>>>>>>
>>>>>>>>>>> If I follow your reasoning ("MOJO must not throw any exception ")
>>>> we
>>>>>>>>>> should
>>>>>>>>>>> ignore every surefire configuration error and keep running the
>>>> build
>>>>>>>>>> until
>>>>>>>>>>> the end and says BUILD SUCCESS
>>>>>>>>>>> what about
>>>>>>>>>>>
>>>>>>>>>>> mvn test -Dsurefire.rerunFailingTestsCount=notanumber
>>>>>>>>>>> -Dmaven.test.failure.ignore=true
>>>>>>>>>>>
>>>>>>>>>>> we should not throw any exceptions as you said below and keep
>>>> saying
>>>>>>>>>> BUILD
>>>>>>>>>>> SUCCESS?
>>>>>>>>>>> argLine is a configuration element as any others so it should
>> fail
>>>>>>>>>> directly
>>>>>>>>>>> and not be ignored especially when the surefire plugin cannot
>> run.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> We have received an internal requirement to change the behavior
>> of
>>>>>>>> the
>>>>>>>>>> user
>>>>>>>>>>>> property *maven.test.failure.ignore* so that the behavior will
>>>> have
>>>>>>>>> one
>>>>>>>>>>>> exception.
>>>>>>>>>>>
>>>>>>>>>>> Suppose that you have JDK 1.8 but you use:
>>>>>>>>>>>> /jre/java --add-reads ...
>>>>>>>>>>>> The outcome is JVM exit with an error message.
>>>>>>>>>>>> I agree with Herve who also says that
>> *maven.test.failure.ignore*
>>>>>>>>>> should
>>>>>>>>>>>> not allow the MOJO to throw the exception. It is not a typical
>> JVM
>>>>>>>>>>>> segmentation fault or another JVM crash where we cannot do
>>>> anything
>>>>>>>>>> about
>>>>>>>>>>>> it, and where the entire build would crash in the command line
>>>> which
>>>>>>>>>>>> of course means that the build cannot normally be interpreted as
>>>>>>>> BUILD
>>>>>>>>>>>> SUCCESS. So we are still on the same level of failures related
>> to
>>>>>>>> the
>>>>>>>>>> test
>>>>>>>>>>>> purposes.
>>>>>>>>>>>>
>>>>>>>>>>>> On the other hand, Olivier has reopened the issues SUREFIRE-1426
>>>> and
>>>>>>>>>>>> SUREFIRE-1681 where the exceptional behavior of the feature is
>>>>>>>>> expected.
>>>>>>>>>>>> This is exactly the reason why I closed these tickets several
>>>> years
>>>>>>>>> ago
>>>>>>>>>> and
>>>>>>>>>>>> my proposal was to not to have any exceptions in the feature
>>>>>>>> behavior
>>>>>>>>>> and
>>>>>>>>>>>> the proposal was to introduce a new user property for exact use
>>>>>>>> cases.
>>>>>>>>>>>> The next idea, which comes from two developers, would provide us
>>>>>>>> with
>>>>>>>>>> the
>>>>>>>>>>>> same non exceptional and exact behavior of the user property
>>>>>>>>>>>> *maven.test.failure.ignore* but it would also provide us with
>> new
>>>>>>>> user
>>>>>>>>>>>> property in the case with fine grade control of the build
>> errors,
>>>>>>>> e.g.
>>>>>>>>>>>> *maven.test.jvm.error.ignore*.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> with a default of?
>>>>>>>>>>> honestly I just see this new parameter as introducing more
>>>> complexity
>>>>>>>>> in
>>>>>>>>>> an
>>>>>>>>>>> already very complex code and not worth it.
>>>>>>>>>>> again read both issue's comments and my comments.
>>>>>>>>>>> Please try to have a user POV and think about making our users'
>>>>>>>>>> experience
>>>>>>>>>>> more simple.
>>>>>>>>>>>
>>>>>>>>>>> This should be very simple:
>>>>>>>>>>> If surefire forked jvm cannot start it's build error and cannot
>> run
>>>>>>>> any
>>>>>>>>>>> tests, it's a problem users want to know immediately because it
>> can
>>>>>>>> be
>>>>>>>>>> for
>>>>>>>>>>> a lot of reasons: wrong argLine, not enough memory on the system
>>>>>>>> etc...
>>>>>>>>>>>
>>>>>>>>>>> AND AGAIN it is very different from ignoring junit result
>> failures.
>>>>>>>>>>>
>>>>>>>>>>> Try to look at a user point of view and think about what is the
>> use
>>>>>>>>> case
>>>>>>>>>> of
>>>>>>>>>>> the property maven.test.failure.ignore=true, I guess 99% of the
>>>> time,
>>>>>>>>>> users
>>>>>>>>>>> wants to run all their tests (even on a CI with different matrix)
>>>> so
>>>>>>>>> they
>>>>>>>>>>> can collect all the tests results which has runned to see if
>> there
>>>> is
>>>>>>>>> any
>>>>>>>>>>> issue for some combination of their matrix and eventually turn
>> the
>>>>>>>>> result
>>>>>>>>>>> as unstable (this is a very typical use case in Jenkins and was
>>>> even
>>>>>>>> a
>>>>>>>>>>> built in feature of the previous Jenkins Maven plugin)
>>>>>>>>>>> BUT if for any reasons one of the module do not start his tests
>>>>>>>> because
>>>>>>>>>> the
>>>>>>>>>>> jvm cannot be start the users will not see that and will be
>> totally
>>>>>>>>> blind
>>>>>>>>>>> until maybe someone look inside a very very large log file (which
>>>>>>>> means
>>>>>>>>>>> probably never)
>>>>>>>>>>>
>>>>>>>>>>> Long story short as my experience as a user facing this
>>>> problem/bug:
>>>>>>>>>>> I had the case on a very large multi modules (~250 modules) build
>>>> of
>>>>>>>> a
>>>>>>>>>> very
>>>>>>>>>>> used open source project.
>>>>>>>>>>> I was using this maven.test.failure.ignore property but one of
>> the
>>>>>>>>>> modules
>>>>>>>>>>> had a bad jpms configuration for a jdk17 profile on the CI.
>>>>>>>>>>> The build has a matrix of 2 os and 4 jdks and different maven run
>>>>>>>> which
>>>>>>>>>>> means around ~60k tests and a Jenkins log file about 40M
>>>>>>>>>>> So because of this property the build was not failing and kept
>>>> saying
>>>>>>>>>> BUILD
>>>>>>>>>>> SUCCESS for weeks/months and basically not testing one module
>> with
>>>>>>>> jdk
>>>>>>>>>> 17...
>>>>>>>>>>> And frankly you do not dig into a log file of 32M after each run
>>>>>>>>>> especially
>>>>>>>>>>> when it says success...
>>>>>>>>>>> 3 days after the first release claiming supporting jdk 17 we
>>>>>>>> received a
>>>>>>>>>> bug
>>>>>>>>>>> report about a something not working with jdk17....
>>>>>>>>>>> and guess what? Where was this feature supposed to be tested?
>>>>>>>>>>>
>>>>>>>>>>> so I frankly believe we do not need a complex new property, in
>> this
>>>>>>>>> case
>>>>>>>>>>> this should fail directly because this will improve user
>>>> experience.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I would like to open the discussion on this topic. You're
>> welcome!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers
>>>>>>>>>>>> Tibor
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>>>>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> ------------------------
>>>>>>>>> Guillaume Nodet
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Tibor Digaňa <ti...@googlemail.com.INVALID>.
Christoph, just let me briefly explain the log level hierarchy.
If you select WARN log level, then ERROR can be printed too.
Similar with INFO, means that WARN and ERROR would be printed as well.
The real decision making in the plugin is a bit more complicated, see the
pull request https://github.com/apache/maven-surefire/pull/478
After I have replied to this email, I would verify the behavior of the
plugin simulating OOM.
T
On Fri, Mar 18, 2022 at 5:42 AM Christoph Läubrich <ma...@laeubi-soft.de>
wrote:
> No I think more about
>
> if (severity == "WARN") {
> log.warn("There is something wrong");
> } else if (severity == "ERROR") {
> throw new MojoFailureException("...") {
> } else {
> throw new MojoExecutionException("...") {
> }
>
> That way the plugin can handle any error/failure/... in a way that the
> user can "downgrade" a certain problem if desired.
>
>
> Am 18.03.22 um 00:06 schrieb Tibor Digana:
> > @Christoph
> > FATAL , WARN , ERROR
> > They are log levels?
> > The plugin does not control the log level after caught exception in Maven
> > core. The Maven Core does!
> > I think it's time to make a demo.
> >
> > On Thu, Mar 17, 2022 at 6:21 AM Christoph Läubrich <ma...@laeubi-soft.de>
> > wrote:
> >
> >> Hi Tibor,
> >>
> >> it shouldn't be to hard to guess another property like
> >>
> >> maven.test.jvmcrash=FATAL
> >> maven.test.failure=WARN
> >> maven.test.error=ERROR
> >>
> >> :-)
> >>
> >> Am 16.03.22 um 12:25 schrieb Tibor Digana:
> >>> Hi Christoph,
> >>>
> >>> Such a granularity with error/failure might be also an additional
> >>> requirement but still you miss the third option to JVM error which is
> >>> different from test error/failure - they don;t have the same meaning.
> >>>
> >>> T
> >>>
> >>>
> >>> On Mon, Mar 14, 2022 at 10:57 AM Christoph Läubrich <
> mail@laeubi-soft.de
> >>>
> >>> wrote:
> >>>
> >>>> Just wondering but maybe it would be better to configure the severity
> >>>> instead of a true/false, something like:
> >>>>
> >>>> maven.test.failure=WARN
> >>>> maven.test.error=ERROR
> >>>>
> >>>> would only warn about failing tests but thrw exception if starting the
> >>>> test fails?
> >>>>
> >>>> Am 14.03.22 um 10:52 schrieb Tibor Digana:
> >>>>> Romain, it is not a bug.
> >>>>> Don't consider this as a bug. It was a feature request for change by
> >>>>> Olivier, and not a bug.
> >>>>> I closed both issues years ago but not because of ignorance but
> because
> >>>> the
> >>>>> appearance of the exceptional behavior is a wrong compromise and
> which
> >>>> does
> >>>>> not help anyone and even it makes the situation worse because
> typically
> >>>>> other group of users would fire a new Jira tickets. You would not be
> >> able
> >>>>> to satisfy two contradictory parties which have completely different
> >>>>> opinions, and so we use to introduce new params and this way we
> satisfy
> >>>>> both parties, they may combine the params as they wish, and everybody
> >>>> would
> >>>>> be happy and nobody would claim. Many technical solutions might
> exist,
> >>>> e.g.
> >>>>> param=boolean|string or new param=boolean, whatever.
> >>>>>
> >>>>> The truth is that this problem with (java --add-reads ...) happened
> in
> >>>> our
> >>>>> ASF environment on Java 8 which in good configuration would not
> happen
> >>>> and
> >>>>> should not.
> >>>>> It is not right way that we abuse "maven.test.failure.ignore" which
> >> would
> >>>>> tell us "Hey, you have illegal configuration in your build system and
> >> it
> >>>>> would not work by JDK design", it is not the goal of the plugin to
> tell
> >>>> you
> >>>>> that you have configured the build wrong because it won't and the
> same
> >>>>> configuration of the plugin (not the build) says "ignore the error".
> >>>>> Yesterday I discussed this problem with Herve and we independently
> >>>> observed
> >>>>> equal opinions and that's not everything because we understood that
> the
> >>>>> purpose of the config parameter is to not throw MOJO exception which
> is
> >>>>> right, but the exceptional behavior should have an exact new param
> for
> >>>> the
> >>>>> exact use case.
> >>>>> SPI for this parameter is too much because no user would implement it
> >>>> for a
> >>>>> trivial parameter;; the SPI is used to be implemented by frameworks
> or
> >>>>> systems or application servers but this is not our case.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <
> >>>> rmannibucau@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> +1
> >>>>>> if it is to investigate a CI issue, it is generally easy to add
> debug
> >>>>>> insights (by code or agent) so a SPI sounds like the sanest for the
> >>>> plugin
> >>>>>> to me.
> >>>>>>
> >>>>>> Romain Manni-Bucau
> >>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> >>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
> >>>>>> <http://rmannibucau.wordpress.com> | Github <
> >>>>>> https://github.com/rmannibucau> |
> >>>>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> >>>>>> <
> >>>>>>
> >>>>
> >>
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <gn...@apache.org> a
> >>>> écrit
> >>>>>> :
> >>>>>>
> >>>>>>> If that's not currently possible, maybe a SPI should be provided so
> >>>> that
> >>>>>>> people can use plug in extensions to analyze the test result and
> >>>> override
> >>>>>>> it if necessary (transforming an error into a warning, storing
> >> results
> >>>>>> in a
> >>>>>>> way which is easier to use by other tools later...) ?
> >>>>>>>
> >>>>>>> Guillaume
> >>>>>>>
> >>>>>>> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <
> >> mail@laeubi-soft.de>
> >>>> a
> >>>>>>> écrit :
> >>>>>>>
> >>>>>>>> I also agree that the test at least should run, we use this
> property
> >>>> to
> >>>>>>>> run the test and produce result and later on have a buildstep that
> >>>>>>>> analyze the results (and probably fail the build job).
> >>>>>>>>
> >>>>>>>> As it is not recommend, I wonder what is the recommended way to
> >>>> archive
> >>>>>>>> something similar?
> >>>>>>>>
> >>>>>>>> Am 14.03.22 um 06:29 schrieb Olivier Lamy:
> >>>>>>>>> On Mon, 14 Mar 2022 at 11:55, Tibor Digana <
> tibordigana@apache.org
> >>>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> In case of the user property *maven.test.failure.ignore* the
> MOJO
> >>>>>> must
> >>>>>>>> not
> >>>>>>>>>> throw any exception which is interpreted by the Maven Core as
> >> BUILD
> >>>>>>>>>> SUCCESS.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> This is a very simple reduction of the problem description.
> >>>>>>>>> The documentation here
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
> >>>>>>>>> says
> >>>>>>>>> "Set this to "true" to ignore a failure during testing. Its use
> is
> >>>>>> NOT
> >>>>>>>>> RECOMMENDED, but quite convenient on occasion"
> >>>>>>>>>
> >>>>>>>>> Personally, I understand this to ignore failures in junit results
> >> BUT
> >>>>>>> at
> >>>>>>>>> least I want the tests to run.
> >>>>>>>>> I guess this is how our users use this feature (feature we do not
> >>>>>>>> recommend
> >>>>>>>>> by the way...)
> >>>>>>>>> And this is perfectly explained here
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
> >>>>>>>>> there is a difference between ignoring some junit failures and
> >>>>>>> ignoring a
> >>>>>>>>> configuration error because some jvm args has been misconfigured
> >> for
> >>>>>>> many
> >>>>>>>>> reasons and surefire cannot start.
> >>>>>>>>>
> >>>>>>>>> If I follow your reasoning ("MOJO must not throw any exception ")
> >> we
> >>>>>>>> should
> >>>>>>>>> ignore every surefire configuration error and keep running the
> >> build
> >>>>>>>> until
> >>>>>>>>> the end and says BUILD SUCCESS
> >>>>>>>>> what about
> >>>>>>>>>
> >>>>>>>>> mvn test -Dsurefire.rerunFailingTestsCount=notanumber
> >>>>>>>>> -Dmaven.test.failure.ignore=true
> >>>>>>>>>
> >>>>>>>>> we should not throw any exceptions as you said below and keep
> >> saying
> >>>>>>>> BUILD
> >>>>>>>>> SUCCESS?
> >>>>>>>>> argLine is a configuration element as any others so it should
> fail
> >>>>>>>> directly
> >>>>>>>>> and not be ignored especially when the surefire plugin cannot
> run.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> We have received an internal requirement to change the behavior
> of
> >>>>>> the
> >>>>>>>> user
> >>>>>>>>>> property *maven.test.failure.ignore* so that the behavior will
> >> have
> >>>>>>> one
> >>>>>>>>>> exception.
> >>>>>>>>>
> >>>>>>>>> Suppose that you have JDK 1.8 but you use:
> >>>>>>>>>> /jre/java --add-reads ...
> >>>>>>>>>> The outcome is JVM exit with an error message.
> >>>>>>>>>> I agree with Herve who also says that
> *maven.test.failure.ignore*
> >>>>>>>> should
> >>>>>>>>>> not allow the MOJO to throw the exception. It is not a typical
> JVM
> >>>>>>>>>> segmentation fault or another JVM crash where we cannot do
> >> anything
> >>>>>>>> about
> >>>>>>>>>> it, and where the entire build would crash in the command line
> >> which
> >>>>>>>>>> of course means that the build cannot normally be interpreted as
> >>>>>> BUILD
> >>>>>>>>>> SUCCESS. So we are still on the same level of failures related
> to
> >>>>>> the
> >>>>>>>> test
> >>>>>>>>>> purposes.
> >>>>>>>>>>
> >>>>>>>>>> On the other hand, Olivier has reopened the issues SUREFIRE-1426
> >> and
> >>>>>>>>>> SUREFIRE-1681 where the exceptional behavior of the feature is
> >>>>>>> expected.
> >>>>>>>>>> This is exactly the reason why I closed these tickets several
> >> years
> >>>>>>> ago
> >>>>>>>> and
> >>>>>>>>>> my proposal was to not to have any exceptions in the feature
> >>>>>> behavior
> >>>>>>>> and
> >>>>>>>>>> the proposal was to introduce a new user property for exact use
> >>>>>> cases.
> >>>>>>>>>> The next idea, which comes from two developers, would provide us
> >>>>>> with
> >>>>>>>> the
> >>>>>>>>>> same non exceptional and exact behavior of the user property
> >>>>>>>>>> *maven.test.failure.ignore* but it would also provide us with
> new
> >>>>>> user
> >>>>>>>>>> property in the case with fine grade control of the build
> errors,
> >>>>>> e.g.
> >>>>>>>>>> *maven.test.jvm.error.ignore*.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> with a default of?
> >>>>>>>>> honestly I just see this new parameter as introducing more
> >> complexity
> >>>>>>> in
> >>>>>>>> an
> >>>>>>>>> already very complex code and not worth it.
> >>>>>>>>> again read both issue's comments and my comments.
> >>>>>>>>> Please try to have a user POV and think about making our users'
> >>>>>>>> experience
> >>>>>>>>> more simple.
> >>>>>>>>>
> >>>>>>>>> This should be very simple:
> >>>>>>>>> If surefire forked jvm cannot start it's build error and cannot
> run
> >>>>>> any
> >>>>>>>>> tests, it's a problem users want to know immediately because it
> can
> >>>>>> be
> >>>>>>>> for
> >>>>>>>>> a lot of reasons: wrong argLine, not enough memory on the system
> >>>>>> etc...
> >>>>>>>>>
> >>>>>>>>> AND AGAIN it is very different from ignoring junit result
> failures.
> >>>>>>>>>
> >>>>>>>>> Try to look at a user point of view and think about what is the
> use
> >>>>>>> case
> >>>>>>>> of
> >>>>>>>>> the property maven.test.failure.ignore=true, I guess 99% of the
> >> time,
> >>>>>>>> users
> >>>>>>>>> wants to run all their tests (even on a CI with different matrix)
> >> so
> >>>>>>> they
> >>>>>>>>> can collect all the tests results which has runned to see if
> there
> >> is
> >>>>>>> any
> >>>>>>>>> issue for some combination of their matrix and eventually turn
> the
> >>>>>>> result
> >>>>>>>>> as unstable (this is a very typical use case in Jenkins and was
> >> even
> >>>>>> a
> >>>>>>>>> built in feature of the previous Jenkins Maven plugin)
> >>>>>>>>> BUT if for any reasons one of the module do not start his tests
> >>>>>> because
> >>>>>>>> the
> >>>>>>>>> jvm cannot be start the users will not see that and will be
> totally
> >>>>>>> blind
> >>>>>>>>> until maybe someone look inside a very very large log file (which
> >>>>>> means
> >>>>>>>>> probably never)
> >>>>>>>>>
> >>>>>>>>> Long story short as my experience as a user facing this
> >> problem/bug:
> >>>>>>>>> I had the case on a very large multi modules (~250 modules) build
> >> of
> >>>>>> a
> >>>>>>>> very
> >>>>>>>>> used open source project.
> >>>>>>>>> I was using this maven.test.failure.ignore property but one of
> the
> >>>>>>>> modules
> >>>>>>>>> had a bad jpms configuration for a jdk17 profile on the CI.
> >>>>>>>>> The build has a matrix of 2 os and 4 jdks and different maven run
> >>>>>> which
> >>>>>>>>> means around ~60k tests and a Jenkins log file about 40M
> >>>>>>>>> So because of this property the build was not failing and kept
> >> saying
> >>>>>>>> BUILD
> >>>>>>>>> SUCCESS for weeks/months and basically not testing one module
> with
> >>>>>> jdk
> >>>>>>>> 17...
> >>>>>>>>> And frankly you do not dig into a log file of 32M after each run
> >>>>>>>> especially
> >>>>>>>>> when it says success...
> >>>>>>>>> 3 days after the first release claiming supporting jdk 17 we
> >>>>>> received a
> >>>>>>>> bug
> >>>>>>>>> report about a something not working with jdk17....
> >>>>>>>>> and guess what? Where was this feature supposed to be tested?
> >>>>>>>>>
> >>>>>>>>> so I frankly believe we do not need a complex new property, in
> this
> >>>>>>> case
> >>>>>>>>> this should fail directly because this will improve user
> >> experience.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I would like to open the discussion on this topic. You're
> welcome!
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Cheers
> >>>>>>>>>> Tibor
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >> ---------------------------------------------------------------------
> >>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>>>>>> For additional commands, e-mail: dev-help@maven.apache.org
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> ------------------------
> >>>>>>> Guillaume Nodet
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>> For additional commands, e-mail: dev-help@maven.apache.org
> >>>>
> >>>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >> For additional commands, e-mail: dev-help@maven.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
--
Cheers
Tibor
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Christoph Läubrich <ma...@laeubi-soft.de>.
No I think more about
if (severity == "WARN") {
log.warn("There is something wrong");
} else if (severity == "ERROR") {
throw new MojoFailureException("...") {
} else {
throw new MojoExecutionException("...") {
}
That way the plugin can handle any error/failure/... in a way that the
user can "downgrade" a certain problem if desired.
Am 18.03.22 um 00:06 schrieb Tibor Digana:
> @Christoph
> FATAL , WARN , ERROR
> They are log levels?
> The plugin does not control the log level after caught exception in Maven
> core. The Maven Core does!
> I think it's time to make a demo.
>
> On Thu, Mar 17, 2022 at 6:21 AM Christoph Läubrich <ma...@laeubi-soft.de>
> wrote:
>
>> Hi Tibor,
>>
>> it shouldn't be to hard to guess another property like
>>
>> maven.test.jvmcrash=FATAL
>> maven.test.failure=WARN
>> maven.test.error=ERROR
>>
>> :-)
>>
>> Am 16.03.22 um 12:25 schrieb Tibor Digana:
>>> Hi Christoph,
>>>
>>> Such a granularity with error/failure might be also an additional
>>> requirement but still you miss the third option to JVM error which is
>>> different from test error/failure - they don;t have the same meaning.
>>>
>>> T
>>>
>>>
>>> On Mon, Mar 14, 2022 at 10:57 AM Christoph Läubrich <mail@laeubi-soft.de
>>>
>>> wrote:
>>>
>>>> Just wondering but maybe it would be better to configure the severity
>>>> instead of a true/false, something like:
>>>>
>>>> maven.test.failure=WARN
>>>> maven.test.error=ERROR
>>>>
>>>> would only warn about failing tests but thrw exception if starting the
>>>> test fails?
>>>>
>>>> Am 14.03.22 um 10:52 schrieb Tibor Digana:
>>>>> Romain, it is not a bug.
>>>>> Don't consider this as a bug. It was a feature request for change by
>>>>> Olivier, and not a bug.
>>>>> I closed both issues years ago but not because of ignorance but because
>>>> the
>>>>> appearance of the exceptional behavior is a wrong compromise and which
>>>> does
>>>>> not help anyone and even it makes the situation worse because typically
>>>>> other group of users would fire a new Jira tickets. You would not be
>> able
>>>>> to satisfy two contradictory parties which have completely different
>>>>> opinions, and so we use to introduce new params and this way we satisfy
>>>>> both parties, they may combine the params as they wish, and everybody
>>>> would
>>>>> be happy and nobody would claim. Many technical solutions might exist,
>>>> e.g.
>>>>> param=boolean|string or new param=boolean, whatever.
>>>>>
>>>>> The truth is that this problem with (java --add-reads ...) happened in
>>>> our
>>>>> ASF environment on Java 8 which in good configuration would not happen
>>>> and
>>>>> should not.
>>>>> It is not right way that we abuse "maven.test.failure.ignore" which
>> would
>>>>> tell us "Hey, you have illegal configuration in your build system and
>> it
>>>>> would not work by JDK design", it is not the goal of the plugin to tell
>>>> you
>>>>> that you have configured the build wrong because it won't and the same
>>>>> configuration of the plugin (not the build) says "ignore the error".
>>>>> Yesterday I discussed this problem with Herve and we independently
>>>> observed
>>>>> equal opinions and that's not everything because we understood that the
>>>>> purpose of the config parameter is to not throw MOJO exception which is
>>>>> right, but the exceptional behavior should have an exact new param for
>>>> the
>>>>> exact use case.
>>>>> SPI for this parameter is too much because no user would implement it
>>>> for a
>>>>> trivial parameter;; the SPI is used to be implemented by frameworks or
>>>>> systems or application servers but this is not our case.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <
>>>> rmannibucau@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> +1
>>>>>> if it is to investigate a CI issue, it is generally easy to add debug
>>>>>> insights (by code or agent) so a SPI sounds like the sanest for the
>>>> plugin
>>>>>> to me.
>>>>>>
>>>>>> Romain Manni-Bucau
>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>> <http://rmannibucau.wordpress.com> | Github <
>>>>>> https://github.com/rmannibucau> |
>>>>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>> <
>>>>>>
>>>>
>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <gn...@apache.org> a
>>>> écrit
>>>>>> :
>>>>>>
>>>>>>> If that's not currently possible, maybe a SPI should be provided so
>>>> that
>>>>>>> people can use plug in extensions to analyze the test result and
>>>> override
>>>>>>> it if necessary (transforming an error into a warning, storing
>> results
>>>>>> in a
>>>>>>> way which is easier to use by other tools later...) ?
>>>>>>>
>>>>>>> Guillaume
>>>>>>>
>>>>>>> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <
>> mail@laeubi-soft.de>
>>>> a
>>>>>>> écrit :
>>>>>>>
>>>>>>>> I also agree that the test at least should run, we use this property
>>>> to
>>>>>>>> run the test and produce result and later on have a buildstep that
>>>>>>>> analyze the results (and probably fail the build job).
>>>>>>>>
>>>>>>>> As it is not recommend, I wonder what is the recommended way to
>>>> archive
>>>>>>>> something similar?
>>>>>>>>
>>>>>>>> Am 14.03.22 um 06:29 schrieb Olivier Lamy:
>>>>>>>>> On Mon, 14 Mar 2022 at 11:55, Tibor Digana <tibordigana@apache.org
>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> In case of the user property *maven.test.failure.ignore* the MOJO
>>>>>> must
>>>>>>>> not
>>>>>>>>>> throw any exception which is interpreted by the Maven Core as
>> BUILD
>>>>>>>>>> SUCCESS.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is a very simple reduction of the problem description.
>>>>>>>>> The documentation here
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
>>>>>>>>> says
>>>>>>>>> "Set this to "true" to ignore a failure during testing. Its use is
>>>>>> NOT
>>>>>>>>> RECOMMENDED, but quite convenient on occasion"
>>>>>>>>>
>>>>>>>>> Personally, I understand this to ignore failures in junit results
>> BUT
>>>>>>> at
>>>>>>>>> least I want the tests to run.
>>>>>>>>> I guess this is how our users use this feature (feature we do not
>>>>>>>> recommend
>>>>>>>>> by the way...)
>>>>>>>>> And this is perfectly explained here
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
>>>>>>>>> there is a difference between ignoring some junit failures and
>>>>>>> ignoring a
>>>>>>>>> configuration error because some jvm args has been misconfigured
>> for
>>>>>>> many
>>>>>>>>> reasons and surefire cannot start.
>>>>>>>>>
>>>>>>>>> If I follow your reasoning ("MOJO must not throw any exception ")
>> we
>>>>>>>> should
>>>>>>>>> ignore every surefire configuration error and keep running the
>> build
>>>>>>>> until
>>>>>>>>> the end and says BUILD SUCCESS
>>>>>>>>> what about
>>>>>>>>>
>>>>>>>>> mvn test -Dsurefire.rerunFailingTestsCount=notanumber
>>>>>>>>> -Dmaven.test.failure.ignore=true
>>>>>>>>>
>>>>>>>>> we should not throw any exceptions as you said below and keep
>> saying
>>>>>>>> BUILD
>>>>>>>>> SUCCESS?
>>>>>>>>> argLine is a configuration element as any others so it should fail
>>>>>>>> directly
>>>>>>>>> and not be ignored especially when the surefire plugin cannot run.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> We have received an internal requirement to change the behavior of
>>>>>> the
>>>>>>>> user
>>>>>>>>>> property *maven.test.failure.ignore* so that the behavior will
>> have
>>>>>>> one
>>>>>>>>>> exception.
>>>>>>>>>
>>>>>>>>> Suppose that you have JDK 1.8 but you use:
>>>>>>>>>> /jre/java --add-reads ...
>>>>>>>>>> The outcome is JVM exit with an error message.
>>>>>>>>>> I agree with Herve who also says that *maven.test.failure.ignore*
>>>>>>>> should
>>>>>>>>>> not allow the MOJO to throw the exception. It is not a typical JVM
>>>>>>>>>> segmentation fault or another JVM crash where we cannot do
>> anything
>>>>>>>> about
>>>>>>>>>> it, and where the entire build would crash in the command line
>> which
>>>>>>>>>> of course means that the build cannot normally be interpreted as
>>>>>> BUILD
>>>>>>>>>> SUCCESS. So we are still on the same level of failures related to
>>>>>> the
>>>>>>>> test
>>>>>>>>>> purposes.
>>>>>>>>>>
>>>>>>>>>> On the other hand, Olivier has reopened the issues SUREFIRE-1426
>> and
>>>>>>>>>> SUREFIRE-1681 where the exceptional behavior of the feature is
>>>>>>> expected.
>>>>>>>>>> This is exactly the reason why I closed these tickets several
>> years
>>>>>>> ago
>>>>>>>> and
>>>>>>>>>> my proposal was to not to have any exceptions in the feature
>>>>>> behavior
>>>>>>>> and
>>>>>>>>>> the proposal was to introduce a new user property for exact use
>>>>>> cases.
>>>>>>>>>> The next idea, which comes from two developers, would provide us
>>>>>> with
>>>>>>>> the
>>>>>>>>>> same non exceptional and exact behavior of the user property
>>>>>>>>>> *maven.test.failure.ignore* but it would also provide us with new
>>>>>> user
>>>>>>>>>> property in the case with fine grade control of the build errors,
>>>>>> e.g.
>>>>>>>>>> *maven.test.jvm.error.ignore*.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> with a default of?
>>>>>>>>> honestly I just see this new parameter as introducing more
>> complexity
>>>>>>> in
>>>>>>>> an
>>>>>>>>> already very complex code and not worth it.
>>>>>>>>> again read both issue's comments and my comments.
>>>>>>>>> Please try to have a user POV and think about making our users'
>>>>>>>> experience
>>>>>>>>> more simple.
>>>>>>>>>
>>>>>>>>> This should be very simple:
>>>>>>>>> If surefire forked jvm cannot start it's build error and cannot run
>>>>>> any
>>>>>>>>> tests, it's a problem users want to know immediately because it can
>>>>>> be
>>>>>>>> for
>>>>>>>>> a lot of reasons: wrong argLine, not enough memory on the system
>>>>>> etc...
>>>>>>>>>
>>>>>>>>> AND AGAIN it is very different from ignoring junit result failures.
>>>>>>>>>
>>>>>>>>> Try to look at a user point of view and think about what is the use
>>>>>>> case
>>>>>>>> of
>>>>>>>>> the property maven.test.failure.ignore=true, I guess 99% of the
>> time,
>>>>>>>> users
>>>>>>>>> wants to run all their tests (even on a CI with different matrix)
>> so
>>>>>>> they
>>>>>>>>> can collect all the tests results which has runned to see if there
>> is
>>>>>>> any
>>>>>>>>> issue for some combination of their matrix and eventually turn the
>>>>>>> result
>>>>>>>>> as unstable (this is a very typical use case in Jenkins and was
>> even
>>>>>> a
>>>>>>>>> built in feature of the previous Jenkins Maven plugin)
>>>>>>>>> BUT if for any reasons one of the module do not start his tests
>>>>>> because
>>>>>>>> the
>>>>>>>>> jvm cannot be start the users will not see that and will be totally
>>>>>>> blind
>>>>>>>>> until maybe someone look inside a very very large log file (which
>>>>>> means
>>>>>>>>> probably never)
>>>>>>>>>
>>>>>>>>> Long story short as my experience as a user facing this
>> problem/bug:
>>>>>>>>> I had the case on a very large multi modules (~250 modules) build
>> of
>>>>>> a
>>>>>>>> very
>>>>>>>>> used open source project.
>>>>>>>>> I was using this maven.test.failure.ignore property but one of the
>>>>>>>> modules
>>>>>>>>> had a bad jpms configuration for a jdk17 profile on the CI.
>>>>>>>>> The build has a matrix of 2 os and 4 jdks and different maven run
>>>>>> which
>>>>>>>>> means around ~60k tests and a Jenkins log file about 40M
>>>>>>>>> So because of this property the build was not failing and kept
>> saying
>>>>>>>> BUILD
>>>>>>>>> SUCCESS for weeks/months and basically not testing one module with
>>>>>> jdk
>>>>>>>> 17...
>>>>>>>>> And frankly you do not dig into a log file of 32M after each run
>>>>>>>> especially
>>>>>>>>> when it says success...
>>>>>>>>> 3 days after the first release claiming supporting jdk 17 we
>>>>>> received a
>>>>>>>> bug
>>>>>>>>> report about a something not working with jdk17....
>>>>>>>>> and guess what? Where was this feature supposed to be tested?
>>>>>>>>>
>>>>>>>>> so I frankly believe we do not need a complex new property, in this
>>>>>>> case
>>>>>>>>> this should fail directly because this will improve user
>> experience.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I would like to open the discussion on this topic. You're welcome!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>> Tibor
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> ------------------------
>>>>>>> Guillaume Nodet
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Tibor Digana <ti...@apache.org>.
@Christoph
FATAL , WARN , ERROR
They are log levels?
The plugin does not control the log level after caught exception in Maven
core. The Maven Core does!
I think it's time to make a demo.
On Thu, Mar 17, 2022 at 6:21 AM Christoph Läubrich <ma...@laeubi-soft.de>
wrote:
> Hi Tibor,
>
> it shouldn't be to hard to guess another property like
>
> maven.test.jvmcrash=FATAL
> maven.test.failure=WARN
> maven.test.error=ERROR
>
> :-)
>
> Am 16.03.22 um 12:25 schrieb Tibor Digana:
> > Hi Christoph,
> >
> > Such a granularity with error/failure might be also an additional
> > requirement but still you miss the third option to JVM error which is
> > different from test error/failure - they don;t have the same meaning.
> >
> > T
> >
> >
> > On Mon, Mar 14, 2022 at 10:57 AM Christoph Läubrich <mail@laeubi-soft.de
> >
> > wrote:
> >
> >> Just wondering but maybe it would be better to configure the severity
> >> instead of a true/false, something like:
> >>
> >> maven.test.failure=WARN
> >> maven.test.error=ERROR
> >>
> >> would only warn about failing tests but thrw exception if starting the
> >> test fails?
> >>
> >> Am 14.03.22 um 10:52 schrieb Tibor Digana:
> >>> Romain, it is not a bug.
> >>> Don't consider this as a bug. It was a feature request for change by
> >>> Olivier, and not a bug.
> >>> I closed both issues years ago but not because of ignorance but because
> >> the
> >>> appearance of the exceptional behavior is a wrong compromise and which
> >> does
> >>> not help anyone and even it makes the situation worse because typically
> >>> other group of users would fire a new Jira tickets. You would not be
> able
> >>> to satisfy two contradictory parties which have completely different
> >>> opinions, and so we use to introduce new params and this way we satisfy
> >>> both parties, they may combine the params as they wish, and everybody
> >> would
> >>> be happy and nobody would claim. Many technical solutions might exist,
> >> e.g.
> >>> param=boolean|string or new param=boolean, whatever.
> >>>
> >>> The truth is that this problem with (java --add-reads ...) happened in
> >> our
> >>> ASF environment on Java 8 which in good configuration would not happen
> >> and
> >>> should not.
> >>> It is not right way that we abuse "maven.test.failure.ignore" which
> would
> >>> tell us "Hey, you have illegal configuration in your build system and
> it
> >>> would not work by JDK design", it is not the goal of the plugin to tell
> >> you
> >>> that you have configured the build wrong because it won't and the same
> >>> configuration of the plugin (not the build) says "ignore the error".
> >>> Yesterday I discussed this problem with Herve and we independently
> >> observed
> >>> equal opinions and that's not everything because we understood that the
> >>> purpose of the config parameter is to not throw MOJO exception which is
> >>> right, but the exceptional behavior should have an exact new param for
> >> the
> >>> exact use case.
> >>> SPI for this parameter is too much because no user would implement it
> >> for a
> >>> trivial parameter;; the SPI is used to be implemented by frameworks or
> >>> systems or application servers but this is not our case.
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <
> >> rmannibucau@gmail.com>
> >>> wrote:
> >>>
> >>>> +1
> >>>> if it is to investigate a CI issue, it is generally easy to add debug
> >>>> insights (by code or agent) so a SPI sounds like the sanest for the
> >> plugin
> >>>> to me.
> >>>>
> >>>> Romain Manni-Bucau
> >>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> >>>> <https://rmannibucau.metawerx.net/> | Old Blog
> >>>> <http://rmannibucau.wordpress.com> | Github <
> >>>> https://github.com/rmannibucau> |
> >>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> >>>> <
> >>>>
> >>
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >>>>>
> >>>>
> >>>>
> >>>> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <gn...@apache.org> a
> >> écrit
> >>>> :
> >>>>
> >>>>> If that's not currently possible, maybe a SPI should be provided so
> >> that
> >>>>> people can use plug in extensions to analyze the test result and
> >> override
> >>>>> it if necessary (transforming an error into a warning, storing
> results
> >>>> in a
> >>>>> way which is easier to use by other tools later...) ?
> >>>>>
> >>>>> Guillaume
> >>>>>
> >>>>> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <
> mail@laeubi-soft.de>
> >> a
> >>>>> écrit :
> >>>>>
> >>>>>> I also agree that the test at least should run, we use this property
> >> to
> >>>>>> run the test and produce result and later on have a buildstep that
> >>>>>> analyze the results (and probably fail the build job).
> >>>>>>
> >>>>>> As it is not recommend, I wonder what is the recommended way to
> >> archive
> >>>>>> something similar?
> >>>>>>
> >>>>>> Am 14.03.22 um 06:29 schrieb Olivier Lamy:
> >>>>>>> On Mon, 14 Mar 2022 at 11:55, Tibor Digana <tibordigana@apache.org
> >
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> In case of the user property *maven.test.failure.ignore* the MOJO
> >>>> must
> >>>>>> not
> >>>>>>>> throw any exception which is interpreted by the Maven Core as
> BUILD
> >>>>>>>> SUCCESS.
> >>>>>>>>
> >>>>>>>
> >>>>>>> This is a very simple reduction of the problem description.
> >>>>>>> The documentation here
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
> >>>>>>> says
> >>>>>>> "Set this to "true" to ignore a failure during testing. Its use is
> >>>> NOT
> >>>>>>> RECOMMENDED, but quite convenient on occasion"
> >>>>>>>
> >>>>>>> Personally, I understand this to ignore failures in junit results
> BUT
> >>>>> at
> >>>>>>> least I want the tests to run.
> >>>>>>> I guess this is how our users use this feature (feature we do not
> >>>>>> recommend
> >>>>>>> by the way...)
> >>>>>>> And this is perfectly explained here
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
> >>>>>>> there is a difference between ignoring some junit failures and
> >>>>> ignoring a
> >>>>>>> configuration error because some jvm args has been misconfigured
> for
> >>>>> many
> >>>>>>> reasons and surefire cannot start.
> >>>>>>>
> >>>>>>> If I follow your reasoning ("MOJO must not throw any exception ")
> we
> >>>>>> should
> >>>>>>> ignore every surefire configuration error and keep running the
> build
> >>>>>> until
> >>>>>>> the end and says BUILD SUCCESS
> >>>>>>> what about
> >>>>>>>
> >>>>>>> mvn test -Dsurefire.rerunFailingTestsCount=notanumber
> >>>>>>> -Dmaven.test.failure.ignore=true
> >>>>>>>
> >>>>>>> we should not throw any exceptions as you said below and keep
> saying
> >>>>>> BUILD
> >>>>>>> SUCCESS?
> >>>>>>> argLine is a configuration element as any others so it should fail
> >>>>>> directly
> >>>>>>> and not be ignored especially when the surefire plugin cannot run.
> >>>>>>>
> >>>>>>>
> >>>>>>>> We have received an internal requirement to change the behavior of
> >>>> the
> >>>>>> user
> >>>>>>>> property *maven.test.failure.ignore* so that the behavior will
> have
> >>>>> one
> >>>>>>>> exception.
> >>>>>>>
> >>>>>>> Suppose that you have JDK 1.8 but you use:
> >>>>>>>> /jre/java --add-reads ...
> >>>>>>>> The outcome is JVM exit with an error message.
> >>>>>>>> I agree with Herve who also says that *maven.test.failure.ignore*
> >>>>>> should
> >>>>>>>> not allow the MOJO to throw the exception. It is not a typical JVM
> >>>>>>>> segmentation fault or another JVM crash where we cannot do
> anything
> >>>>>> about
> >>>>>>>> it, and where the entire build would crash in the command line
> which
> >>>>>>>> of course means that the build cannot normally be interpreted as
> >>>> BUILD
> >>>>>>>> SUCCESS. So we are still on the same level of failures related to
> >>>> the
> >>>>>> test
> >>>>>>>> purposes.
> >>>>>>>>
> >>>>>>>> On the other hand, Olivier has reopened the issues SUREFIRE-1426
> and
> >>>>>>>> SUREFIRE-1681 where the exceptional behavior of the feature is
> >>>>> expected.
> >>>>>>>> This is exactly the reason why I closed these tickets several
> years
> >>>>> ago
> >>>>>> and
> >>>>>>>> my proposal was to not to have any exceptions in the feature
> >>>> behavior
> >>>>>> and
> >>>>>>>> the proposal was to introduce a new user property for exact use
> >>>> cases.
> >>>>>>>> The next idea, which comes from two developers, would provide us
> >>>> with
> >>>>>> the
> >>>>>>>> same non exceptional and exact behavior of the user property
> >>>>>>>> *maven.test.failure.ignore* but it would also provide us with new
> >>>> user
> >>>>>>>> property in the case with fine grade control of the build errors,
> >>>> e.g.
> >>>>>>>> *maven.test.jvm.error.ignore*.
> >>>>>>>>
> >>>>>>>
> >>>>>>> with a default of?
> >>>>>>> honestly I just see this new parameter as introducing more
> complexity
> >>>>> in
> >>>>>> an
> >>>>>>> already very complex code and not worth it.
> >>>>>>> again read both issue's comments and my comments.
> >>>>>>> Please try to have a user POV and think about making our users'
> >>>>>> experience
> >>>>>>> more simple.
> >>>>>>>
> >>>>>>> This should be very simple:
> >>>>>>> If surefire forked jvm cannot start it's build error and cannot run
> >>>> any
> >>>>>>> tests, it's a problem users want to know immediately because it can
> >>>> be
> >>>>>> for
> >>>>>>> a lot of reasons: wrong argLine, not enough memory on the system
> >>>> etc...
> >>>>>>>
> >>>>>>> AND AGAIN it is very different from ignoring junit result failures.
> >>>>>>>
> >>>>>>> Try to look at a user point of view and think about what is the use
> >>>>> case
> >>>>>> of
> >>>>>>> the property maven.test.failure.ignore=true, I guess 99% of the
> time,
> >>>>>> users
> >>>>>>> wants to run all their tests (even on a CI with different matrix)
> so
> >>>>> they
> >>>>>>> can collect all the tests results which has runned to see if there
> is
> >>>>> any
> >>>>>>> issue for some combination of their matrix and eventually turn the
> >>>>> result
> >>>>>>> as unstable (this is a very typical use case in Jenkins and was
> even
> >>>> a
> >>>>>>> built in feature of the previous Jenkins Maven plugin)
> >>>>>>> BUT if for any reasons one of the module do not start his tests
> >>>> because
> >>>>>> the
> >>>>>>> jvm cannot be start the users will not see that and will be totally
> >>>>> blind
> >>>>>>> until maybe someone look inside a very very large log file (which
> >>>> means
> >>>>>>> probably never)
> >>>>>>>
> >>>>>>> Long story short as my experience as a user facing this
> problem/bug:
> >>>>>>> I had the case on a very large multi modules (~250 modules) build
> of
> >>>> a
> >>>>>> very
> >>>>>>> used open source project.
> >>>>>>> I was using this maven.test.failure.ignore property but one of the
> >>>>>> modules
> >>>>>>> had a bad jpms configuration for a jdk17 profile on the CI.
> >>>>>>> The build has a matrix of 2 os and 4 jdks and different maven run
> >>>> which
> >>>>>>> means around ~60k tests and a Jenkins log file about 40M
> >>>>>>> So because of this property the build was not failing and kept
> saying
> >>>>>> BUILD
> >>>>>>> SUCCESS for weeks/months and basically not testing one module with
> >>>> jdk
> >>>>>> 17...
> >>>>>>> And frankly you do not dig into a log file of 32M after each run
> >>>>>> especially
> >>>>>>> when it says success...
> >>>>>>> 3 days after the first release claiming supporting jdk 17 we
> >>>> received a
> >>>>>> bug
> >>>>>>> report about a something not working with jdk17....
> >>>>>>> and guess what? Where was this feature supposed to be tested?
> >>>>>>>
> >>>>>>> so I frankly believe we do not need a complex new property, in this
> >>>>> case
> >>>>>>> this should fail directly because this will improve user
> experience.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I would like to open the discussion on this topic. You're welcome!
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Cheers
> >>>>>>>> Tibor
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>>>> For additional commands, e-mail: dev-help@maven.apache.org
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> --
> >>>>> ------------------------
> >>>>> Guillaume Nodet
> >>>>>
> >>>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >> For additional commands, e-mail: dev-help@maven.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Christoph Läubrich <ma...@laeubi-soft.de>.
Hi Tibor,
it shouldn't be to hard to guess another property like
maven.test.jvmcrash=FATAL
maven.test.failure=WARN
maven.test.error=ERROR
:-)
Am 16.03.22 um 12:25 schrieb Tibor Digana:
> Hi Christoph,
>
> Such a granularity with error/failure might be also an additional
> requirement but still you miss the third option to JVM error which is
> different from test error/failure - they don;t have the same meaning.
>
> T
>
>
> On Mon, Mar 14, 2022 at 10:57 AM Christoph Läubrich <ma...@laeubi-soft.de>
> wrote:
>
>> Just wondering but maybe it would be better to configure the severity
>> instead of a true/false, something like:
>>
>> maven.test.failure=WARN
>> maven.test.error=ERROR
>>
>> would only warn about failing tests but thrw exception if starting the
>> test fails?
>>
>> Am 14.03.22 um 10:52 schrieb Tibor Digana:
>>> Romain, it is not a bug.
>>> Don't consider this as a bug. It was a feature request for change by
>>> Olivier, and not a bug.
>>> I closed both issues years ago but not because of ignorance but because
>> the
>>> appearance of the exceptional behavior is a wrong compromise and which
>> does
>>> not help anyone and even it makes the situation worse because typically
>>> other group of users would fire a new Jira tickets. You would not be able
>>> to satisfy two contradictory parties which have completely different
>>> opinions, and so we use to introduce new params and this way we satisfy
>>> both parties, they may combine the params as they wish, and everybody
>> would
>>> be happy and nobody would claim. Many technical solutions might exist,
>> e.g.
>>> param=boolean|string or new param=boolean, whatever.
>>>
>>> The truth is that this problem with (java --add-reads ...) happened in
>> our
>>> ASF environment on Java 8 which in good configuration would not happen
>> and
>>> should not.
>>> It is not right way that we abuse "maven.test.failure.ignore" which would
>>> tell us "Hey, you have illegal configuration in your build system and it
>>> would not work by JDK design", it is not the goal of the plugin to tell
>> you
>>> that you have configured the build wrong because it won't and the same
>>> configuration of the plugin (not the build) says "ignore the error".
>>> Yesterday I discussed this problem with Herve and we independently
>> observed
>>> equal opinions and that's not everything because we understood that the
>>> purpose of the config parameter is to not throw MOJO exception which is
>>> right, but the exceptional behavior should have an exact new param for
>> the
>>> exact use case.
>>> SPI for this parameter is too much because no user would implement it
>> for a
>>> trivial parameter;; the SPI is used to be implemented by frameworks or
>>> systems or application servers but this is not our case.
>>>
>>>
>>>
>>>
>>> On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <
>> rmannibucau@gmail.com>
>>> wrote:
>>>
>>>> +1
>>>> if it is to investigate a CI issue, it is generally easy to add debug
>>>> insights (by code or agent) so a SPI sounds like the sanest for the
>> plugin
>>>> to me.
>>>>
>>>> Romain Manni-Bucau
>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>> <http://rmannibucau.wordpress.com> | Github <
>>>> https://github.com/rmannibucau> |
>>>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
>>>> <
>>>>
>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>>>
>>>>
>>>>
>>>> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <gn...@apache.org> a
>> écrit
>>>> :
>>>>
>>>>> If that's not currently possible, maybe a SPI should be provided so
>> that
>>>>> people can use plug in extensions to analyze the test result and
>> override
>>>>> it if necessary (transforming an error into a warning, storing results
>>>> in a
>>>>> way which is easier to use by other tools later...) ?
>>>>>
>>>>> Guillaume
>>>>>
>>>>> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <ma...@laeubi-soft.de>
>> a
>>>>> écrit :
>>>>>
>>>>>> I also agree that the test at least should run, we use this property
>> to
>>>>>> run the test and produce result and later on have a buildstep that
>>>>>> analyze the results (and probably fail the build job).
>>>>>>
>>>>>> As it is not recommend, I wonder what is the recommended way to
>> archive
>>>>>> something similar?
>>>>>>
>>>>>> Am 14.03.22 um 06:29 schrieb Olivier Lamy:
>>>>>>> On Mon, 14 Mar 2022 at 11:55, Tibor Digana <ti...@apache.org>
>>>>>> wrote:
>>>>>>>
>>>>>>>> In case of the user property *maven.test.failure.ignore* the MOJO
>>>> must
>>>>>> not
>>>>>>>> throw any exception which is interpreted by the Maven Core as BUILD
>>>>>>>> SUCCESS.
>>>>>>>>
>>>>>>>
>>>>>>> This is a very simple reduction of the problem description.
>>>>>>> The documentation here
>>>>>>>
>>>>>>
>>>>>
>>>>
>> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
>>>>>>> says
>>>>>>> "Set this to "true" to ignore a failure during testing. Its use is
>>>> NOT
>>>>>>> RECOMMENDED, but quite convenient on occasion"
>>>>>>>
>>>>>>> Personally, I understand this to ignore failures in junit results BUT
>>>>> at
>>>>>>> least I want the tests to run.
>>>>>>> I guess this is how our users use this feature (feature we do not
>>>>>> recommend
>>>>>>> by the way...)
>>>>>>> And this is perfectly explained here
>>>>>>>
>>>>>>
>>>>>
>>>>
>> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
>>>>>>> there is a difference between ignoring some junit failures and
>>>>> ignoring a
>>>>>>> configuration error because some jvm args has been misconfigured for
>>>>> many
>>>>>>> reasons and surefire cannot start.
>>>>>>>
>>>>>>> If I follow your reasoning ("MOJO must not throw any exception ") we
>>>>>> should
>>>>>>> ignore every surefire configuration error and keep running the build
>>>>>> until
>>>>>>> the end and says BUILD SUCCESS
>>>>>>> what about
>>>>>>>
>>>>>>> mvn test -Dsurefire.rerunFailingTestsCount=notanumber
>>>>>>> -Dmaven.test.failure.ignore=true
>>>>>>>
>>>>>>> we should not throw any exceptions as you said below and keep saying
>>>>>> BUILD
>>>>>>> SUCCESS?
>>>>>>> argLine is a configuration element as any others so it should fail
>>>>>> directly
>>>>>>> and not be ignored especially when the surefire plugin cannot run.
>>>>>>>
>>>>>>>
>>>>>>>> We have received an internal requirement to change the behavior of
>>>> the
>>>>>> user
>>>>>>>> property *maven.test.failure.ignore* so that the behavior will have
>>>>> one
>>>>>>>> exception.
>>>>>>>
>>>>>>> Suppose that you have JDK 1.8 but you use:
>>>>>>>> /jre/java --add-reads ...
>>>>>>>> The outcome is JVM exit with an error message.
>>>>>>>> I agree with Herve who also says that *maven.test.failure.ignore*
>>>>>> should
>>>>>>>> not allow the MOJO to throw the exception. It is not a typical JVM
>>>>>>>> segmentation fault or another JVM crash where we cannot do anything
>>>>>> about
>>>>>>>> it, and where the entire build would crash in the command line which
>>>>>>>> of course means that the build cannot normally be interpreted as
>>>> BUILD
>>>>>>>> SUCCESS. So we are still on the same level of failures related to
>>>> the
>>>>>> test
>>>>>>>> purposes.
>>>>>>>>
>>>>>>>> On the other hand, Olivier has reopened the issues SUREFIRE-1426 and
>>>>>>>> SUREFIRE-1681 where the exceptional behavior of the feature is
>>>>> expected.
>>>>>>>> This is exactly the reason why I closed these tickets several years
>>>>> ago
>>>>>> and
>>>>>>>> my proposal was to not to have any exceptions in the feature
>>>> behavior
>>>>>> and
>>>>>>>> the proposal was to introduce a new user property for exact use
>>>> cases.
>>>>>>>> The next idea, which comes from two developers, would provide us
>>>> with
>>>>>> the
>>>>>>>> same non exceptional and exact behavior of the user property
>>>>>>>> *maven.test.failure.ignore* but it would also provide us with new
>>>> user
>>>>>>>> property in the case with fine grade control of the build errors,
>>>> e.g.
>>>>>>>> *maven.test.jvm.error.ignore*.
>>>>>>>>
>>>>>>>
>>>>>>> with a default of?
>>>>>>> honestly I just see this new parameter as introducing more complexity
>>>>> in
>>>>>> an
>>>>>>> already very complex code and not worth it.
>>>>>>> again read both issue's comments and my comments.
>>>>>>> Please try to have a user POV and think about making our users'
>>>>>> experience
>>>>>>> more simple.
>>>>>>>
>>>>>>> This should be very simple:
>>>>>>> If surefire forked jvm cannot start it's build error and cannot run
>>>> any
>>>>>>> tests, it's a problem users want to know immediately because it can
>>>> be
>>>>>> for
>>>>>>> a lot of reasons: wrong argLine, not enough memory on the system
>>>> etc...
>>>>>>>
>>>>>>> AND AGAIN it is very different from ignoring junit result failures.
>>>>>>>
>>>>>>> Try to look at a user point of view and think about what is the use
>>>>> case
>>>>>> of
>>>>>>> the property maven.test.failure.ignore=true, I guess 99% of the time,
>>>>>> users
>>>>>>> wants to run all their tests (even on a CI with different matrix) so
>>>>> they
>>>>>>> can collect all the tests results which has runned to see if there is
>>>>> any
>>>>>>> issue for some combination of their matrix and eventually turn the
>>>>> result
>>>>>>> as unstable (this is a very typical use case in Jenkins and was even
>>>> a
>>>>>>> built in feature of the previous Jenkins Maven plugin)
>>>>>>> BUT if for any reasons one of the module do not start his tests
>>>> because
>>>>>> the
>>>>>>> jvm cannot be start the users will not see that and will be totally
>>>>> blind
>>>>>>> until maybe someone look inside a very very large log file (which
>>>> means
>>>>>>> probably never)
>>>>>>>
>>>>>>> Long story short as my experience as a user facing this problem/bug:
>>>>>>> I had the case on a very large multi modules (~250 modules) build of
>>>> a
>>>>>> very
>>>>>>> used open source project.
>>>>>>> I was using this maven.test.failure.ignore property but one of the
>>>>>> modules
>>>>>>> had a bad jpms configuration for a jdk17 profile on the CI.
>>>>>>> The build has a matrix of 2 os and 4 jdks and different maven run
>>>> which
>>>>>>> means around ~60k tests and a Jenkins log file about 40M
>>>>>>> So because of this property the build was not failing and kept saying
>>>>>> BUILD
>>>>>>> SUCCESS for weeks/months and basically not testing one module with
>>>> jdk
>>>>>> 17...
>>>>>>> And frankly you do not dig into a log file of 32M after each run
>>>>>> especially
>>>>>>> when it says success...
>>>>>>> 3 days after the first release claiming supporting jdk 17 we
>>>> received a
>>>>>> bug
>>>>>>> report about a something not working with jdk17....
>>>>>>> and guess what? Where was this feature supposed to be tested?
>>>>>>>
>>>>>>> so I frankly believe we do not need a complex new property, in this
>>>>> case
>>>>>>> this should fail directly because this will improve user experience.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I would like to open the discussion on this topic. You're welcome!
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> Tibor
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> ------------------------
>>>>> Guillaume Nodet
>>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Tibor Digana <ti...@apache.org>.
Hi Christoph,
Such a granularity with error/failure might be also an additional
requirement but still you miss the third option to JVM error which is
different from test error/failure - they don;t have the same meaning.
T
On Mon, Mar 14, 2022 at 10:57 AM Christoph Läubrich <ma...@laeubi-soft.de>
wrote:
> Just wondering but maybe it would be better to configure the severity
> instead of a true/false, something like:
>
> maven.test.failure=WARN
> maven.test.error=ERROR
>
> would only warn about failing tests but thrw exception if starting the
> test fails?
>
> Am 14.03.22 um 10:52 schrieb Tibor Digana:
> > Romain, it is not a bug.
> > Don't consider this as a bug. It was a feature request for change by
> > Olivier, and not a bug.
> > I closed both issues years ago but not because of ignorance but because
> the
> > appearance of the exceptional behavior is a wrong compromise and which
> does
> > not help anyone and even it makes the situation worse because typically
> > other group of users would fire a new Jira tickets. You would not be able
> > to satisfy two contradictory parties which have completely different
> > opinions, and so we use to introduce new params and this way we satisfy
> > both parties, they may combine the params as they wish, and everybody
> would
> > be happy and nobody would claim. Many technical solutions might exist,
> e.g.
> > param=boolean|string or new param=boolean, whatever.
> >
> > The truth is that this problem with (java --add-reads ...) happened in
> our
> > ASF environment on Java 8 which in good configuration would not happen
> and
> > should not.
> > It is not right way that we abuse "maven.test.failure.ignore" which would
> > tell us "Hey, you have illegal configuration in your build system and it
> > would not work by JDK design", it is not the goal of the plugin to tell
> you
> > that you have configured the build wrong because it won't and the same
> > configuration of the plugin (not the build) says "ignore the error".
> > Yesterday I discussed this problem with Herve and we independently
> observed
> > equal opinions and that's not everything because we understood that the
> > purpose of the config parameter is to not throw MOJO exception which is
> > right, but the exceptional behavior should have an exact new param for
> the
> > exact use case.
> > SPI for this parameter is too much because no user would implement it
> for a
> > trivial parameter;; the SPI is used to be implemented by frameworks or
> > systems or application servers but this is not our case.
> >
> >
> >
> >
> > On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <
> rmannibucau@gmail.com>
> > wrote:
> >
> >> +1
> >> if it is to investigate a CI issue, it is generally easy to add debug
> >> insights (by code or agent) so a SPI sounds like the sanest for the
> plugin
> >> to me.
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> >> <https://rmannibucau.metawerx.net/> | Old Blog
> >> <http://rmannibucau.wordpress.com> | Github <
> >> https://github.com/rmannibucau> |
> >> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> >> <
> >>
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >>>
> >>
> >>
> >> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <gn...@apache.org> a
> écrit
> >> :
> >>
> >>> If that's not currently possible, maybe a SPI should be provided so
> that
> >>> people can use plug in extensions to analyze the test result and
> override
> >>> it if necessary (transforming an error into a warning, storing results
> >> in a
> >>> way which is easier to use by other tools later...) ?
> >>>
> >>> Guillaume
> >>>
> >>> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <ma...@laeubi-soft.de>
> a
> >>> écrit :
> >>>
> >>>> I also agree that the test at least should run, we use this property
> to
> >>>> run the test and produce result and later on have a buildstep that
> >>>> analyze the results (and probably fail the build job).
> >>>>
> >>>> As it is not recommend, I wonder what is the recommended way to
> archive
> >>>> something similar?
> >>>>
> >>>> Am 14.03.22 um 06:29 schrieb Olivier Lamy:
> >>>>> On Mon, 14 Mar 2022 at 11:55, Tibor Digana <ti...@apache.org>
> >>>> wrote:
> >>>>>
> >>>>>> In case of the user property *maven.test.failure.ignore* the MOJO
> >> must
> >>>> not
> >>>>>> throw any exception which is interpreted by the Maven Core as BUILD
> >>>>>> SUCCESS.
> >>>>>>
> >>>>>
> >>>>> This is a very simple reduction of the problem description.
> >>>>> The documentation here
> >>>>>
> >>>>
> >>>
> >>
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
> >>>>> says
> >>>>> "Set this to "true" to ignore a failure during testing. Its use is
> >> NOT
> >>>>> RECOMMENDED, but quite convenient on occasion"
> >>>>>
> >>>>> Personally, I understand this to ignore failures in junit results BUT
> >>> at
> >>>>> least I want the tests to run.
> >>>>> I guess this is how our users use this feature (feature we do not
> >>>> recommend
> >>>>> by the way...)
> >>>>> And this is perfectly explained here
> >>>>>
> >>>>
> >>>
> >>
> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
> >>>>> there is a difference between ignoring some junit failures and
> >>> ignoring a
> >>>>> configuration error because some jvm args has been misconfigured for
> >>> many
> >>>>> reasons and surefire cannot start.
> >>>>>
> >>>>> If I follow your reasoning ("MOJO must not throw any exception ") we
> >>>> should
> >>>>> ignore every surefire configuration error and keep running the build
> >>>> until
> >>>>> the end and says BUILD SUCCESS
> >>>>> what about
> >>>>>
> >>>>> mvn test -Dsurefire.rerunFailingTestsCount=notanumber
> >>>>> -Dmaven.test.failure.ignore=true
> >>>>>
> >>>>> we should not throw any exceptions as you said below and keep saying
> >>>> BUILD
> >>>>> SUCCESS?
> >>>>> argLine is a configuration element as any others so it should fail
> >>>> directly
> >>>>> and not be ignored especially when the surefire plugin cannot run.
> >>>>>
> >>>>>
> >>>>>> We have received an internal requirement to change the behavior of
> >> the
> >>>> user
> >>>>>> property *maven.test.failure.ignore* so that the behavior will have
> >>> one
> >>>>>> exception.
> >>>>>
> >>>>> Suppose that you have JDK 1.8 but you use:
> >>>>>> /jre/java --add-reads ...
> >>>>>> The outcome is JVM exit with an error message.
> >>>>>> I agree with Herve who also says that *maven.test.failure.ignore*
> >>>> should
> >>>>>> not allow the MOJO to throw the exception. It is not a typical JVM
> >>>>>> segmentation fault or another JVM crash where we cannot do anything
> >>>> about
> >>>>>> it, and where the entire build would crash in the command line which
> >>>>>> of course means that the build cannot normally be interpreted as
> >> BUILD
> >>>>>> SUCCESS. So we are still on the same level of failures related to
> >> the
> >>>> test
> >>>>>> purposes.
> >>>>>>
> >>>>>> On the other hand, Olivier has reopened the issues SUREFIRE-1426 and
> >>>>>> SUREFIRE-1681 where the exceptional behavior of the feature is
> >>> expected.
> >>>>>> This is exactly the reason why I closed these tickets several years
> >>> ago
> >>>> and
> >>>>>> my proposal was to not to have any exceptions in the feature
> >> behavior
> >>>> and
> >>>>>> the proposal was to introduce a new user property for exact use
> >> cases.
> >>>>>> The next idea, which comes from two developers, would provide us
> >> with
> >>>> the
> >>>>>> same non exceptional and exact behavior of the user property
> >>>>>> *maven.test.failure.ignore* but it would also provide us with new
> >> user
> >>>>>> property in the case with fine grade control of the build errors,
> >> e.g.
> >>>>>> *maven.test.jvm.error.ignore*.
> >>>>>>
> >>>>>
> >>>>> with a default of?
> >>>>> honestly I just see this new parameter as introducing more complexity
> >>> in
> >>>> an
> >>>>> already very complex code and not worth it.
> >>>>> again read both issue's comments and my comments.
> >>>>> Please try to have a user POV and think about making our users'
> >>>> experience
> >>>>> more simple.
> >>>>>
> >>>>> This should be very simple:
> >>>>> If surefire forked jvm cannot start it's build error and cannot run
> >> any
> >>>>> tests, it's a problem users want to know immediately because it can
> >> be
> >>>> for
> >>>>> a lot of reasons: wrong argLine, not enough memory on the system
> >> etc...
> >>>>>
> >>>>> AND AGAIN it is very different from ignoring junit result failures.
> >>>>>
> >>>>> Try to look at a user point of view and think about what is the use
> >>> case
> >>>> of
> >>>>> the property maven.test.failure.ignore=true, I guess 99% of the time,
> >>>> users
> >>>>> wants to run all their tests (even on a CI with different matrix) so
> >>> they
> >>>>> can collect all the tests results which has runned to see if there is
> >>> any
> >>>>> issue for some combination of their matrix and eventually turn the
> >>> result
> >>>>> as unstable (this is a very typical use case in Jenkins and was even
> >> a
> >>>>> built in feature of the previous Jenkins Maven plugin)
> >>>>> BUT if for any reasons one of the module do not start his tests
> >> because
> >>>> the
> >>>>> jvm cannot be start the users will not see that and will be totally
> >>> blind
> >>>>> until maybe someone look inside a very very large log file (which
> >> means
> >>>>> probably never)
> >>>>>
> >>>>> Long story short as my experience as a user facing this problem/bug:
> >>>>> I had the case on a very large multi modules (~250 modules) build of
> >> a
> >>>> very
> >>>>> used open source project.
> >>>>> I was using this maven.test.failure.ignore property but one of the
> >>>> modules
> >>>>> had a bad jpms configuration for a jdk17 profile on the CI.
> >>>>> The build has a matrix of 2 os and 4 jdks and different maven run
> >> which
> >>>>> means around ~60k tests and a Jenkins log file about 40M
> >>>>> So because of this property the build was not failing and kept saying
> >>>> BUILD
> >>>>> SUCCESS for weeks/months and basically not testing one module with
> >> jdk
> >>>> 17...
> >>>>> And frankly you do not dig into a log file of 32M after each run
> >>>> especially
> >>>>> when it says success...
> >>>>> 3 days after the first release claiming supporting jdk 17 we
> >> received a
> >>>> bug
> >>>>> report about a something not working with jdk17....
> >>>>> and guess what? Where was this feature supposed to be tested?
> >>>>>
> >>>>> so I frankly believe we do not need a complex new property, in this
> >>> case
> >>>>> this should fail directly because this will improve user experience.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> I would like to open the discussion on this topic. You're welcome!
> >>>>>>
> >>>>>>
> >>>>>> Cheers
> >>>>>> Tibor
> >>>>>>
> >>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>> For additional commands, e-mail: dev-help@maven.apache.org
> >>>>
> >>>>
> >>>
> >>> --
> >>> ------------------------
> >>> Guillaume Nodet
> >>>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Christoph Läubrich <ma...@laeubi-soft.de>.
Just wondering but maybe it would be better to configure the severity
instead of a true/false, something like:
maven.test.failure=WARN
maven.test.error=ERROR
would only warn about failing tests but thrw exception if starting the
test fails?
Am 14.03.22 um 10:52 schrieb Tibor Digana:
> Romain, it is not a bug.
> Don't consider this as a bug. It was a feature request for change by
> Olivier, and not a bug.
> I closed both issues years ago but not because of ignorance but because the
> appearance of the exceptional behavior is a wrong compromise and which does
> not help anyone and even it makes the situation worse because typically
> other group of users would fire a new Jira tickets. You would not be able
> to satisfy two contradictory parties which have completely different
> opinions, and so we use to introduce new params and this way we satisfy
> both parties, they may combine the params as they wish, and everybody would
> be happy and nobody would claim. Many technical solutions might exist, e.g.
> param=boolean|string or new param=boolean, whatever.
>
> The truth is that this problem with (java --add-reads ...) happened in our
> ASF environment on Java 8 which in good configuration would not happen and
> should not.
> It is not right way that we abuse "maven.test.failure.ignore" which would
> tell us "Hey, you have illegal configuration in your build system and it
> would not work by JDK design", it is not the goal of the plugin to tell you
> that you have configured the build wrong because it won't and the same
> configuration of the plugin (not the build) says "ignore the error".
> Yesterday I discussed this problem with Herve and we independently observed
> equal opinions and that's not everything because we understood that the
> purpose of the config parameter is to not throw MOJO exception which is
> right, but the exceptional behavior should have an exact new param for the
> exact use case.
> SPI for this parameter is too much because no user would implement it for a
> trivial parameter;; the SPI is used to be implemented by frameworks or
> systems or application servers but this is not our case.
>
>
>
>
> On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> +1
>> if it is to investigate a CI issue, it is generally easy to add debug
>> insights (by code or agent) so a SPI sounds like the sanest for the plugin
>> to me.
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github <
>> https://github.com/rmannibucau> |
>> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
>> <
>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>
>>
>>
>> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <gn...@apache.org> a écrit
>> :
>>
>>> If that's not currently possible, maybe a SPI should be provided so that
>>> people can use plug in extensions to analyze the test result and override
>>> it if necessary (transforming an error into a warning, storing results
>> in a
>>> way which is easier to use by other tools later...) ?
>>>
>>> Guillaume
>>>
>>> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <ma...@laeubi-soft.de> a
>>> écrit :
>>>
>>>> I also agree that the test at least should run, we use this property to
>>>> run the test and produce result and later on have a buildstep that
>>>> analyze the results (and probably fail the build job).
>>>>
>>>> As it is not recommend, I wonder what is the recommended way to archive
>>>> something similar?
>>>>
>>>> Am 14.03.22 um 06:29 schrieb Olivier Lamy:
>>>>> On Mon, 14 Mar 2022 at 11:55, Tibor Digana <ti...@apache.org>
>>>> wrote:
>>>>>
>>>>>> In case of the user property *maven.test.failure.ignore* the MOJO
>> must
>>>> not
>>>>>> throw any exception which is interpreted by the Maven Core as BUILD
>>>>>> SUCCESS.
>>>>>>
>>>>>
>>>>> This is a very simple reduction of the problem description.
>>>>> The documentation here
>>>>>
>>>>
>>>
>> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
>>>>> says
>>>>> "Set this to "true" to ignore a failure during testing. Its use is
>> NOT
>>>>> RECOMMENDED, but quite convenient on occasion"
>>>>>
>>>>> Personally, I understand this to ignore failures in junit results BUT
>>> at
>>>>> least I want the tests to run.
>>>>> I guess this is how our users use this feature (feature we do not
>>>> recommend
>>>>> by the way...)
>>>>> And this is perfectly explained here
>>>>>
>>>>
>>>
>> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
>>>>> there is a difference between ignoring some junit failures and
>>> ignoring a
>>>>> configuration error because some jvm args has been misconfigured for
>>> many
>>>>> reasons and surefire cannot start.
>>>>>
>>>>> If I follow your reasoning ("MOJO must not throw any exception ") we
>>>> should
>>>>> ignore every surefire configuration error and keep running the build
>>>> until
>>>>> the end and says BUILD SUCCESS
>>>>> what about
>>>>>
>>>>> mvn test -Dsurefire.rerunFailingTestsCount=notanumber
>>>>> -Dmaven.test.failure.ignore=true
>>>>>
>>>>> we should not throw any exceptions as you said below and keep saying
>>>> BUILD
>>>>> SUCCESS?
>>>>> argLine is a configuration element as any others so it should fail
>>>> directly
>>>>> and not be ignored especially when the surefire plugin cannot run.
>>>>>
>>>>>
>>>>>> We have received an internal requirement to change the behavior of
>> the
>>>> user
>>>>>> property *maven.test.failure.ignore* so that the behavior will have
>>> one
>>>>>> exception.
>>>>>
>>>>> Suppose that you have JDK 1.8 but you use:
>>>>>> /jre/java --add-reads ...
>>>>>> The outcome is JVM exit with an error message.
>>>>>> I agree with Herve who also says that *maven.test.failure.ignore*
>>>> should
>>>>>> not allow the MOJO to throw the exception. It is not a typical JVM
>>>>>> segmentation fault or another JVM crash where we cannot do anything
>>>> about
>>>>>> it, and where the entire build would crash in the command line which
>>>>>> of course means that the build cannot normally be interpreted as
>> BUILD
>>>>>> SUCCESS. So we are still on the same level of failures related to
>> the
>>>> test
>>>>>> purposes.
>>>>>>
>>>>>> On the other hand, Olivier has reopened the issues SUREFIRE-1426 and
>>>>>> SUREFIRE-1681 where the exceptional behavior of the feature is
>>> expected.
>>>>>> This is exactly the reason why I closed these tickets several years
>>> ago
>>>> and
>>>>>> my proposal was to not to have any exceptions in the feature
>> behavior
>>>> and
>>>>>> the proposal was to introduce a new user property for exact use
>> cases.
>>>>>> The next idea, which comes from two developers, would provide us
>> with
>>>> the
>>>>>> same non exceptional and exact behavior of the user property
>>>>>> *maven.test.failure.ignore* but it would also provide us with new
>> user
>>>>>> property in the case with fine grade control of the build errors,
>> e.g.
>>>>>> *maven.test.jvm.error.ignore*.
>>>>>>
>>>>>
>>>>> with a default of?
>>>>> honestly I just see this new parameter as introducing more complexity
>>> in
>>>> an
>>>>> already very complex code and not worth it.
>>>>> again read both issue's comments and my comments.
>>>>> Please try to have a user POV and think about making our users'
>>>> experience
>>>>> more simple.
>>>>>
>>>>> This should be very simple:
>>>>> If surefire forked jvm cannot start it's build error and cannot run
>> any
>>>>> tests, it's a problem users want to know immediately because it can
>> be
>>>> for
>>>>> a lot of reasons: wrong argLine, not enough memory on the system
>> etc...
>>>>>
>>>>> AND AGAIN it is very different from ignoring junit result failures.
>>>>>
>>>>> Try to look at a user point of view and think about what is the use
>>> case
>>>> of
>>>>> the property maven.test.failure.ignore=true, I guess 99% of the time,
>>>> users
>>>>> wants to run all their tests (even on a CI with different matrix) so
>>> they
>>>>> can collect all the tests results which has runned to see if there is
>>> any
>>>>> issue for some combination of their matrix and eventually turn the
>>> result
>>>>> as unstable (this is a very typical use case in Jenkins and was even
>> a
>>>>> built in feature of the previous Jenkins Maven plugin)
>>>>> BUT if for any reasons one of the module do not start his tests
>> because
>>>> the
>>>>> jvm cannot be start the users will not see that and will be totally
>>> blind
>>>>> until maybe someone look inside a very very large log file (which
>> means
>>>>> probably never)
>>>>>
>>>>> Long story short as my experience as a user facing this problem/bug:
>>>>> I had the case on a very large multi modules (~250 modules) build of
>> a
>>>> very
>>>>> used open source project.
>>>>> I was using this maven.test.failure.ignore property but one of the
>>>> modules
>>>>> had a bad jpms configuration for a jdk17 profile on the CI.
>>>>> The build has a matrix of 2 os and 4 jdks and different maven run
>> which
>>>>> means around ~60k tests and a Jenkins log file about 40M
>>>>> So because of this property the build was not failing and kept saying
>>>> BUILD
>>>>> SUCCESS for weeks/months and basically not testing one module with
>> jdk
>>>> 17...
>>>>> And frankly you do not dig into a log file of 32M after each run
>>>> especially
>>>>> when it says success...
>>>>> 3 days after the first release claiming supporting jdk 17 we
>> received a
>>>> bug
>>>>> report about a something not working with jdk17....
>>>>> and guess what? Where was this feature supposed to be tested?
>>>>>
>>>>> so I frankly believe we do not need a complex new property, in this
>>> case
>>>>> this should fail directly because this will improve user experience.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>> I would like to open the discussion on this topic. You're welcome!
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>> Tibor
>>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>
>>>>
>>>
>>> --
>>> ------------------------
>>> Guillaume Nodet
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Tibor Digana <ti...@apache.org>.
Hi Slawomir,
Your second second example -DargLine=-Xxx is used in an integration test
in order to force producing exceptions, but IMHO it wants to simulate two
different situations.
In such cases the JVM:
1. fails on JVM startup because of low RAM
(fails the JVM on --add-reads or illegal VM parameter -Xxx and the user
wants to fail the build due to he considers this as a config error)
2. fails on OOM during the test - another CI process concurrently consumes
RAM too and randomly some processes reach OOM
(OOM during the tests - should not fail the build with
-Dmaven.test.failure.ignore=true)
We are not able to distinguish between JVM and ForkedBooter startup errors
due to the ForkedBooter does not notify the plugin with an event "I am
alive".
Currently this config parameter has this feature description in Javadoc:
*Set this to "true" to ignore a failure during testing. Its use is NOT
RECOMMENDED, but quite convenient on occasion.*
So, we have two options to fix it.
1. implement one more config param, or make a string from boolean in the
current one, or
2. distinguish between JVM and ForkedBooter startup by implementing a new
event HELLO, and use SurefireBooterForkException with some
marker forkStarted:boolean.
exit(1) is our custom error code, so we know that the ForkedBooter has
started and the build should not fail because you have
used -Dmaven.test.failure.ignore=true.
Personally, I would vote for impl #2.
Any thoughts?
Tibor
On Mon, Mar 14, 2022 at 6:18 PM Slawomir Jaranowski <s....@gmail.com>
wrote:
> pon., 14 mar 2022 o 10:52 Tibor Digana <ti...@apache.org>
> napisał(a):
>
> > Romain, it is not a bug.
> > Don't consider this as a bug. It was a feature request for change by
> > Olivier, and not a bug.
> > I closed both issues years ago but not because of ignorance but because
> the
> > appearance of the exceptional behavior is a wrong compromise and which
> does
> > not help anyone and even it makes the situation worse because typically
> > other group of users would fire a new Jira tickets. You would not be able
> > to satisfy two contradictory parties which have completely different
> > opinions, and so we use to introduce new params and this way we satisfy
> > both parties, they may combine the params as they wish, and everybody
> would
> > be happy and nobody would claim. Many technical solutions might exist,
> e.g.
> > param=boolean|string or new param=boolean, whatever.
> >
> > The truth is that this problem with (java --add-reads ...) happened in
> our
> > ASF environment on Java 8 which in good configuration would not happen
> and
> > should not.
> > It is not right way that we abuse "maven.test.failure.ignore" which would
> > tell us "Hey, you have illegal configuration in your build system and it
> > would not work by JDK design", it is not the goal of the plugin to tell
> you
> > that you have configured the build wrong because it won't and the same
> > configuration of the plugin (not the build) says "ignore the error".
> >
>
> So what is the difference:
>
> mvn test -Dmaven.test.failure.ignore=true
> -Dsurefire.rerunFailingTestsCount=notanumber
>
> mvn test -Dmaven.test.failure.ignore=true -DargLine=-Xxx
>
> for both I have illegal configuration in my build system, but one brak
> Maven build and one does not ...
>
>
> Yesterday I discussed this problem with Herve and we independently observed
> > equal opinions and that's not everything because we understood that the
> > purpose of the config parameter is to not throw MOJO exception which is
> > right, but the exceptional behavior should have an exact new param for
> the
> > exact use case.
> > SPI for this parameter is too much because no user would implement it
> for a
> > trivial parameter;; the SPI is used to be implemented by frameworks or
> > systems or application servers but this is not our case.
> >
> >
> >
> >
> > On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <
> rmannibucau@gmail.com>
> > wrote:
> >
> > > +1
> > > if it is to investigate a CI issue, it is generally easy to add debug
> > > insights (by code or agent) so a SPI sounds like the sanest for the
> > plugin
> > > to me.
> > >
> > > Romain Manni-Bucau
> > > @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > > <https://rmannibucau.metawerx.net/> | Old Blog
> > > <http://rmannibucau.wordpress.com> | Github <
> > > https://github.com/rmannibucau> |
> > > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> > > <
> > >
> >
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> > > >
> > >
> > >
> > > Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <gn...@apache.org> a
> > écrit
> > > :
> > >
> > > > If that's not currently possible, maybe a SPI should be provided so
> > that
> > > > people can use plug in extensions to analyze the test result and
> > override
> > > > it if necessary (transforming an error into a warning, storing
> results
> > > in a
> > > > way which is easier to use by other tools later...) ?
> > > >
> > > > Guillaume
> > > >
> > > > Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <
> mail@laeubi-soft.de>
> > a
> > > > écrit :
> > > >
> > > > > I also agree that the test at least should run, we use this
> property
> > to
> > > > > run the test and produce result and later on have a buildstep that
> > > > > analyze the results (and probably fail the build job).
> > > > >
> > > > > As it is not recommend, I wonder what is the recommended way to
> > archive
> > > > > something similar?
> > > > >
> > > > > Am 14.03.22 um 06:29 schrieb Olivier Lamy:
> > > > > > On Mon, 14 Mar 2022 at 11:55, Tibor Digana <
> tibordigana@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > >> In case of the user property *maven.test.failure.ignore* the
> MOJO
> > > must
> > > > > not
> > > > > >> throw any exception which is interpreted by the Maven Core as
> > BUILD
> > > > > >> SUCCESS.
> > > > > >>
> > > > > >
> > > > > > This is a very simple reduction of the problem description.
> > > > > > The documentation here
> > > > > >
> > > > >
> > > >
> > >
> >
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
> > > > > > says
> > > > > > "Set this to "true" to ignore a failure during testing. Its use
> is
> > > NOT
> > > > > > RECOMMENDED, but quite convenient on occasion"
> > > > > >
> > > > > > Personally, I understand this to ignore failures in junit results
> > BUT
> > > > at
> > > > > > least I want the tests to run.
> > > > > > I guess this is how our users use this feature (feature we do not
> > > > > recommend
> > > > > > by the way...)
> > > > > > And this is perfectly explained here
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
> > > > > > there is a difference between ignoring some junit failures and
> > > > ignoring a
> > > > > > configuration error because some jvm args has been misconfigured
> > for
> > > > many
> > > > > > reasons and surefire cannot start.
> > > > > >
> > > > > > If I follow your reasoning ("MOJO must not throw any exception ")
> > we
> > > > > should
> > > > > > ignore every surefire configuration error and keep running the
> > build
> > > > > until
> > > > > > the end and says BUILD SUCCESS
> > > > > > what about
> > > > > >
> > > > > > mvn test -Dsurefire.rerunFailingTestsCount=notanumber
> > > > > > -Dmaven.test.failure.ignore=true
> > > > > >
> > > > > > we should not throw any exceptions as you said below and keep
> > saying
> > > > > BUILD
> > > > > > SUCCESS?
> > > > > > argLine is a configuration element as any others so it should
> fail
> > > > > directly
> > > > > > and not be ignored especially when the surefire plugin cannot
> run.
> > > > > >
> > > > > >
> > > > > >> We have received an internal requirement to change the behavior
> of
> > > the
> > > > > user
> > > > > >> property *maven.test.failure.ignore* so that the behavior will
> > have
> > > > one
> > > > > >> exception.
> > > > > >
> > > > > > Suppose that you have JDK 1.8 but you use:
> > > > > >> /jre/java --add-reads ...
> > > > > >> The outcome is JVM exit with an error message.
> > > > > >> I agree with Herve who also says that
> *maven.test.failure.ignore*
> > > > > should
> > > > > >> not allow the MOJO to throw the exception. It is not a typical
> JVM
> > > > > >> segmentation fault or another JVM crash where we cannot do
> > anything
> > > > > about
> > > > > >> it, and where the entire build would crash in the command line
> > which
> > > > > >> of course means that the build cannot normally be interpreted as
> > > BUILD
> > > > > >> SUCCESS. So we are still on the same level of failures related
> to
> > > the
> > > > > test
> > > > > >> purposes.
> > > > > >>
> > > > > >> On the other hand, Olivier has reopened the issues SUREFIRE-1426
> > and
> > > > > >> SUREFIRE-1681 where the exceptional behavior of the feature is
> > > > expected.
> > > > > >> This is exactly the reason why I closed these tickets several
> > years
> > > > ago
> > > > > and
> > > > > >> my proposal was to not to have any exceptions in the feature
> > > behavior
> > > > > and
> > > > > >> the proposal was to introduce a new user property for exact use
> > > cases.
> > > > > >> The next idea, which comes from two developers, would provide us
> > > with
> > > > > the
> > > > > >> same non exceptional and exact behavior of the user property
> > > > > >> *maven.test.failure.ignore* but it would also provide us with
> new
> > > user
> > > > > >> property in the case with fine grade control of the build
> errors,
> > > e.g.
> > > > > >> *maven.test.jvm.error.ignore*.
> > > > > >>
> > > > > >
> > > > > > with a default of?
> > > > > > honestly I just see this new parameter as introducing more
> > complexity
> > > > in
> > > > > an
> > > > > > already very complex code and not worth it.
> > > > > > again read both issue's comments and my comments.
> > > > > > Please try to have a user POV and think about making our users'
> > > > > experience
> > > > > > more simple.
> > > > > >
> > > > > > This should be very simple:
> > > > > > If surefire forked jvm cannot start it's build error and cannot
> run
> > > any
> > > > > > tests, it's a problem users want to know immediately because it
> can
> > > be
> > > > > for
> > > > > > a lot of reasons: wrong argLine, not enough memory on the system
> > > etc...
> > > > > >
> > > > > > AND AGAIN it is very different from ignoring junit result
> failures.
> > > > > >
> > > > > > Try to look at a user point of view and think about what is the
> use
> > > > case
> > > > > of
> > > > > > the property maven.test.failure.ignore=true, I guess 99% of the
> > time,
> > > > > users
> > > > > > wants to run all their tests (even on a CI with different matrix)
> > so
> > > > they
> > > > > > can collect all the tests results which has runned to see if
> there
> > is
> > > > any
> > > > > > issue for some combination of their matrix and eventually turn
> the
> > > > result
> > > > > > as unstable (this is a very typical use case in Jenkins and was
> > even
> > > a
> > > > > > built in feature of the previous Jenkins Maven plugin)
> > > > > > BUT if for any reasons one of the module do not start his tests
> > > because
> > > > > the
> > > > > > jvm cannot be start the users will not see that and will be
> totally
> > > > blind
> > > > > > until maybe someone look inside a very very large log file (which
> > > means
> > > > > > probably never)
> > > > > >
> > > > > > Long story short as my experience as a user facing this
> > problem/bug:
> > > > > > I had the case on a very large multi modules (~250 modules) build
> > of
> > > a
> > > > > very
> > > > > > used open source project.
> > > > > > I was using this maven.test.failure.ignore property but one of
> the
> > > > > modules
> > > > > > had a bad jpms configuration for a jdk17 profile on the CI.
> > > > > > The build has a matrix of 2 os and 4 jdks and different maven run
> > > which
> > > > > > means around ~60k tests and a Jenkins log file about 40M
> > > > > > So because of this property the build was not failing and kept
> > saying
> > > > > BUILD
> > > > > > SUCCESS for weeks/months and basically not testing one module
> with
> > > jdk
> > > > > 17...
> > > > > > And frankly you do not dig into a log file of 32M after each run
> > > > > especially
> > > > > > when it says success...
> > > > > > 3 days after the first release claiming supporting jdk 17 we
> > > received a
> > > > > bug
> > > > > > report about a something not working with jdk17....
> > > > > > and guess what? Where was this feature supposed to be tested?
> > > > > >
> > > > > > so I frankly believe we do not need a complex new property, in
> this
> > > > case
> > > > > > this should fail directly because this will improve user
> > experience.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >>
> > > > > >>
> > > > > >> I would like to open the discussion on this topic. You're
> welcome!
> > > > > >>
> > > > > >>
> > > > > >> Cheers
> > > > > >> Tibor
> > > > > >>
> > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > > > For additional commands, e-mail: dev-help@maven.apache.org
> > > > >
> > > > >
> > > >
> > > > --
> > > > ------------------------
> > > > Guillaume Nodet
> > > >
> > >
> >
>
>
> --
> Sławomir Jaranowski
>
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Slawomir Jaranowski <s....@gmail.com>.
pon., 14 mar 2022 o 10:52 Tibor Digana <ti...@apache.org> napisał(a):
> Romain, it is not a bug.
> Don't consider this as a bug. It was a feature request for change by
> Olivier, and not a bug.
> I closed both issues years ago but not because of ignorance but because the
> appearance of the exceptional behavior is a wrong compromise and which does
> not help anyone and even it makes the situation worse because typically
> other group of users would fire a new Jira tickets. You would not be able
> to satisfy two contradictory parties which have completely different
> opinions, and so we use to introduce new params and this way we satisfy
> both parties, they may combine the params as they wish, and everybody would
> be happy and nobody would claim. Many technical solutions might exist, e.g.
> param=boolean|string or new param=boolean, whatever.
>
> The truth is that this problem with (java --add-reads ...) happened in our
> ASF environment on Java 8 which in good configuration would not happen and
> should not.
> It is not right way that we abuse "maven.test.failure.ignore" which would
> tell us "Hey, you have illegal configuration in your build system and it
> would not work by JDK design", it is not the goal of the plugin to tell you
> that you have configured the build wrong because it won't and the same
> configuration of the plugin (not the build) says "ignore the error".
>
So what is the difference:
mvn test -Dmaven.test.failure.ignore=true
-Dsurefire.rerunFailingTestsCount=notanumber
mvn test -Dmaven.test.failure.ignore=true -DargLine=-Xxx
for both I have illegal configuration in my build system, but one brak
Maven build and one does not ...
Yesterday I discussed this problem with Herve and we independently observed
> equal opinions and that's not everything because we understood that the
> purpose of the config parameter is to not throw MOJO exception which is
> right, but the exceptional behavior should have an exact new param for the
> exact use case.
> SPI for this parameter is too much because no user would implement it for a
> trivial parameter;; the SPI is used to be implemented by frameworks or
> systems or application servers but this is not our case.
>
>
>
>
> On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
> > +1
> > if it is to investigate a CI issue, it is generally easy to add debug
> > insights (by code or agent) so a SPI sounds like the sanest for the
> plugin
> > to me.
> >
> > Romain Manni-Bucau
> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > <https://rmannibucau.metawerx.net/> | Old Blog
> > <http://rmannibucau.wordpress.com> | Github <
> > https://github.com/rmannibucau> |
> > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> > <
> >
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> > >
> >
> >
> > Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <gn...@apache.org> a
> écrit
> > :
> >
> > > If that's not currently possible, maybe a SPI should be provided so
> that
> > > people can use plug in extensions to analyze the test result and
> override
> > > it if necessary (transforming an error into a warning, storing results
> > in a
> > > way which is easier to use by other tools later...) ?
> > >
> > > Guillaume
> > >
> > > Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <ma...@laeubi-soft.de>
> a
> > > écrit :
> > >
> > > > I also agree that the test at least should run, we use this property
> to
> > > > run the test and produce result and later on have a buildstep that
> > > > analyze the results (and probably fail the build job).
> > > >
> > > > As it is not recommend, I wonder what is the recommended way to
> archive
> > > > something similar?
> > > >
> > > > Am 14.03.22 um 06:29 schrieb Olivier Lamy:
> > > > > On Mon, 14 Mar 2022 at 11:55, Tibor Digana <tibordigana@apache.org
> >
> > > > wrote:
> > > > >
> > > > >> In case of the user property *maven.test.failure.ignore* the MOJO
> > must
> > > > not
> > > > >> throw any exception which is interpreted by the Maven Core as
> BUILD
> > > > >> SUCCESS.
> > > > >>
> > > > >
> > > > > This is a very simple reduction of the problem description.
> > > > > The documentation here
> > > > >
> > > >
> > >
> >
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
> > > > > says
> > > > > "Set this to "true" to ignore a failure during testing. Its use is
> > NOT
> > > > > RECOMMENDED, but quite convenient on occasion"
> > > > >
> > > > > Personally, I understand this to ignore failures in junit results
> BUT
> > > at
> > > > > least I want the tests to run.
> > > > > I guess this is how our users use this feature (feature we do not
> > > > recommend
> > > > > by the way...)
> > > > > And this is perfectly explained here
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
> > > > > there is a difference between ignoring some junit failures and
> > > ignoring a
> > > > > configuration error because some jvm args has been misconfigured
> for
> > > many
> > > > > reasons and surefire cannot start.
> > > > >
> > > > > If I follow your reasoning ("MOJO must not throw any exception ")
> we
> > > > should
> > > > > ignore every surefire configuration error and keep running the
> build
> > > > until
> > > > > the end and says BUILD SUCCESS
> > > > > what about
> > > > >
> > > > > mvn test -Dsurefire.rerunFailingTestsCount=notanumber
> > > > > -Dmaven.test.failure.ignore=true
> > > > >
> > > > > we should not throw any exceptions as you said below and keep
> saying
> > > > BUILD
> > > > > SUCCESS?
> > > > > argLine is a configuration element as any others so it should fail
> > > > directly
> > > > > and not be ignored especially when the surefire plugin cannot run.
> > > > >
> > > > >
> > > > >> We have received an internal requirement to change the behavior of
> > the
> > > > user
> > > > >> property *maven.test.failure.ignore* so that the behavior will
> have
> > > one
> > > > >> exception.
> > > > >
> > > > > Suppose that you have JDK 1.8 but you use:
> > > > >> /jre/java --add-reads ...
> > > > >> The outcome is JVM exit with an error message.
> > > > >> I agree with Herve who also says that *maven.test.failure.ignore*
> > > > should
> > > > >> not allow the MOJO to throw the exception. It is not a typical JVM
> > > > >> segmentation fault or another JVM crash where we cannot do
> anything
> > > > about
> > > > >> it, and where the entire build would crash in the command line
> which
> > > > >> of course means that the build cannot normally be interpreted as
> > BUILD
> > > > >> SUCCESS. So we are still on the same level of failures related to
> > the
> > > > test
> > > > >> purposes.
> > > > >>
> > > > >> On the other hand, Olivier has reopened the issues SUREFIRE-1426
> and
> > > > >> SUREFIRE-1681 where the exceptional behavior of the feature is
> > > expected.
> > > > >> This is exactly the reason why I closed these tickets several
> years
> > > ago
> > > > and
> > > > >> my proposal was to not to have any exceptions in the feature
> > behavior
> > > > and
> > > > >> the proposal was to introduce a new user property for exact use
> > cases.
> > > > >> The next idea, which comes from two developers, would provide us
> > with
> > > > the
> > > > >> same non exceptional and exact behavior of the user property
> > > > >> *maven.test.failure.ignore* but it would also provide us with new
> > user
> > > > >> property in the case with fine grade control of the build errors,
> > e.g.
> > > > >> *maven.test.jvm.error.ignore*.
> > > > >>
> > > > >
> > > > > with a default of?
> > > > > honestly I just see this new parameter as introducing more
> complexity
> > > in
> > > > an
> > > > > already very complex code and not worth it.
> > > > > again read both issue's comments and my comments.
> > > > > Please try to have a user POV and think about making our users'
> > > > experience
> > > > > more simple.
> > > > >
> > > > > This should be very simple:
> > > > > If surefire forked jvm cannot start it's build error and cannot run
> > any
> > > > > tests, it's a problem users want to know immediately because it can
> > be
> > > > for
> > > > > a lot of reasons: wrong argLine, not enough memory on the system
> > etc...
> > > > >
> > > > > AND AGAIN it is very different from ignoring junit result failures.
> > > > >
> > > > > Try to look at a user point of view and think about what is the use
> > > case
> > > > of
> > > > > the property maven.test.failure.ignore=true, I guess 99% of the
> time,
> > > > users
> > > > > wants to run all their tests (even on a CI with different matrix)
> so
> > > they
> > > > > can collect all the tests results which has runned to see if there
> is
> > > any
> > > > > issue for some combination of their matrix and eventually turn the
> > > result
> > > > > as unstable (this is a very typical use case in Jenkins and was
> even
> > a
> > > > > built in feature of the previous Jenkins Maven plugin)
> > > > > BUT if for any reasons one of the module do not start his tests
> > because
> > > > the
> > > > > jvm cannot be start the users will not see that and will be totally
> > > blind
> > > > > until maybe someone look inside a very very large log file (which
> > means
> > > > > probably never)
> > > > >
> > > > > Long story short as my experience as a user facing this
> problem/bug:
> > > > > I had the case on a very large multi modules (~250 modules) build
> of
> > a
> > > > very
> > > > > used open source project.
> > > > > I was using this maven.test.failure.ignore property but one of the
> > > > modules
> > > > > had a bad jpms configuration for a jdk17 profile on the CI.
> > > > > The build has a matrix of 2 os and 4 jdks and different maven run
> > which
> > > > > means around ~60k tests and a Jenkins log file about 40M
> > > > > So because of this property the build was not failing and kept
> saying
> > > > BUILD
> > > > > SUCCESS for weeks/months and basically not testing one module with
> > jdk
> > > > 17...
> > > > > And frankly you do not dig into a log file of 32M after each run
> > > > especially
> > > > > when it says success...
> > > > > 3 days after the first release claiming supporting jdk 17 we
> > received a
> > > > bug
> > > > > report about a something not working with jdk17....
> > > > > and guess what? Where was this feature supposed to be tested?
> > > > >
> > > > > so I frankly believe we do not need a complex new property, in this
> > > case
> > > > > this should fail directly because this will improve user
> experience.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >>
> > > > >>
> > > > >> I would like to open the discussion on this topic. You're welcome!
> > > > >>
> > > > >>
> > > > >> Cheers
> > > > >> Tibor
> > > > >>
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > > For additional commands, e-mail: dev-help@maven.apache.org
> > > >
> > > >
> > >
> > > --
> > > ------------------------
> > > Guillaume Nodet
> > >
> >
>
--
Sławomir Jaranowski
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Tibor Digana <ti...@apache.org>.
Romain, it is not a bug.
Don't consider this as a bug. It was a feature request for change by
Olivier, and not a bug.
I closed both issues years ago but not because of ignorance but because the
appearance of the exceptional behavior is a wrong compromise and which does
not help anyone and even it makes the situation worse because typically
other group of users would fire a new Jira tickets. You would not be able
to satisfy two contradictory parties which have completely different
opinions, and so we use to introduce new params and this way we satisfy
both parties, they may combine the params as they wish, and everybody would
be happy and nobody would claim. Many technical solutions might exist, e.g.
param=boolean|string or new param=boolean, whatever.
The truth is that this problem with (java --add-reads ...) happened in our
ASF environment on Java 8 which in good configuration would not happen and
should not.
It is not right way that we abuse "maven.test.failure.ignore" which would
tell us "Hey, you have illegal configuration in your build system and it
would not work by JDK design", it is not the goal of the plugin to tell you
that you have configured the build wrong because it won't and the same
configuration of the plugin (not the build) says "ignore the error".
Yesterday I discussed this problem with Herve and we independently observed
equal opinions and that's not everything because we understood that the
purpose of the config parameter is to not throw MOJO exception which is
right, but the exceptional behavior should have an exact new param for the
exact use case.
SPI for this parameter is too much because no user would implement it for a
trivial parameter;; the SPI is used to be implemented by frameworks or
systems or application servers but this is not our case.
On Mon, Mar 14, 2022 at 9:11 AM Romain Manni-Bucau <rm...@gmail.com>
wrote:
> +1
> if it is to investigate a CI issue, it is generally easy to add debug
> insights (by code or agent) so a SPI sounds like the sanest for the plugin
> to me.
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github <
> https://github.com/rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
>
>
> Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <gn...@apache.org> a écrit
> :
>
> > If that's not currently possible, maybe a SPI should be provided so that
> > people can use plug in extensions to analyze the test result and override
> > it if necessary (transforming an error into a warning, storing results
> in a
> > way which is easier to use by other tools later...) ?
> >
> > Guillaume
> >
> > Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <ma...@laeubi-soft.de> a
> > écrit :
> >
> > > I also agree that the test at least should run, we use this property to
> > > run the test and produce result and later on have a buildstep that
> > > analyze the results (and probably fail the build job).
> > >
> > > As it is not recommend, I wonder what is the recommended way to archive
> > > something similar?
> > >
> > > Am 14.03.22 um 06:29 schrieb Olivier Lamy:
> > > > On Mon, 14 Mar 2022 at 11:55, Tibor Digana <ti...@apache.org>
> > > wrote:
> > > >
> > > >> In case of the user property *maven.test.failure.ignore* the MOJO
> must
> > > not
> > > >> throw any exception which is interpreted by the Maven Core as BUILD
> > > >> SUCCESS.
> > > >>
> > > >
> > > > This is a very simple reduction of the problem description.
> > > > The documentation here
> > > >
> > >
> >
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
> > > > says
> > > > "Set this to "true" to ignore a failure during testing. Its use is
> NOT
> > > > RECOMMENDED, but quite convenient on occasion"
> > > >
> > > > Personally, I understand this to ignore failures in junit results BUT
> > at
> > > > least I want the tests to run.
> > > > I guess this is how our users use this feature (feature we do not
> > > recommend
> > > > by the way...)
> > > > And this is perfectly explained here
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
> > > > there is a difference between ignoring some junit failures and
> > ignoring a
> > > > configuration error because some jvm args has been misconfigured for
> > many
> > > > reasons and surefire cannot start.
> > > >
> > > > If I follow your reasoning ("MOJO must not throw any exception ") we
> > > should
> > > > ignore every surefire configuration error and keep running the build
> > > until
> > > > the end and says BUILD SUCCESS
> > > > what about
> > > >
> > > > mvn test -Dsurefire.rerunFailingTestsCount=notanumber
> > > > -Dmaven.test.failure.ignore=true
> > > >
> > > > we should not throw any exceptions as you said below and keep saying
> > > BUILD
> > > > SUCCESS?
> > > > argLine is a configuration element as any others so it should fail
> > > directly
> > > > and not be ignored especially when the surefire plugin cannot run.
> > > >
> > > >
> > > >> We have received an internal requirement to change the behavior of
> the
> > > user
> > > >> property *maven.test.failure.ignore* so that the behavior will have
> > one
> > > >> exception.
> > > >
> > > > Suppose that you have JDK 1.8 but you use:
> > > >> /jre/java --add-reads ...
> > > >> The outcome is JVM exit with an error message.
> > > >> I agree with Herve who also says that *maven.test.failure.ignore*
> > > should
> > > >> not allow the MOJO to throw the exception. It is not a typical JVM
> > > >> segmentation fault or another JVM crash where we cannot do anything
> > > about
> > > >> it, and where the entire build would crash in the command line which
> > > >> of course means that the build cannot normally be interpreted as
> BUILD
> > > >> SUCCESS. So we are still on the same level of failures related to
> the
> > > test
> > > >> purposes.
> > > >>
> > > >> On the other hand, Olivier has reopened the issues SUREFIRE-1426 and
> > > >> SUREFIRE-1681 where the exceptional behavior of the feature is
> > expected.
> > > >> This is exactly the reason why I closed these tickets several years
> > ago
> > > and
> > > >> my proposal was to not to have any exceptions in the feature
> behavior
> > > and
> > > >> the proposal was to introduce a new user property for exact use
> cases.
> > > >> The next idea, which comes from two developers, would provide us
> with
> > > the
> > > >> same non exceptional and exact behavior of the user property
> > > >> *maven.test.failure.ignore* but it would also provide us with new
> user
> > > >> property in the case with fine grade control of the build errors,
> e.g.
> > > >> *maven.test.jvm.error.ignore*.
> > > >>
> > > >
> > > > with a default of?
> > > > honestly I just see this new parameter as introducing more complexity
> > in
> > > an
> > > > already very complex code and not worth it.
> > > > again read both issue's comments and my comments.
> > > > Please try to have a user POV and think about making our users'
> > > experience
> > > > more simple.
> > > >
> > > > This should be very simple:
> > > > If surefire forked jvm cannot start it's build error and cannot run
> any
> > > > tests, it's a problem users want to know immediately because it can
> be
> > > for
> > > > a lot of reasons: wrong argLine, not enough memory on the system
> etc...
> > > >
> > > > AND AGAIN it is very different from ignoring junit result failures.
> > > >
> > > > Try to look at a user point of view and think about what is the use
> > case
> > > of
> > > > the property maven.test.failure.ignore=true, I guess 99% of the time,
> > > users
> > > > wants to run all their tests (even on a CI with different matrix) so
> > they
> > > > can collect all the tests results which has runned to see if there is
> > any
> > > > issue for some combination of their matrix and eventually turn the
> > result
> > > > as unstable (this is a very typical use case in Jenkins and was even
> a
> > > > built in feature of the previous Jenkins Maven plugin)
> > > > BUT if for any reasons one of the module do not start his tests
> because
> > > the
> > > > jvm cannot be start the users will not see that and will be totally
> > blind
> > > > until maybe someone look inside a very very large log file (which
> means
> > > > probably never)
> > > >
> > > > Long story short as my experience as a user facing this problem/bug:
> > > > I had the case on a very large multi modules (~250 modules) build of
> a
> > > very
> > > > used open source project.
> > > > I was using this maven.test.failure.ignore property but one of the
> > > modules
> > > > had a bad jpms configuration for a jdk17 profile on the CI.
> > > > The build has a matrix of 2 os and 4 jdks and different maven run
> which
> > > > means around ~60k tests and a Jenkins log file about 40M
> > > > So because of this property the build was not failing and kept saying
> > > BUILD
> > > > SUCCESS for weeks/months and basically not testing one module with
> jdk
> > > 17...
> > > > And frankly you do not dig into a log file of 32M after each run
> > > especially
> > > > when it says success...
> > > > 3 days after the first release claiming supporting jdk 17 we
> received a
> > > bug
> > > > report about a something not working with jdk17....
> > > > and guess what? Where was this feature supposed to be tested?
> > > >
> > > > so I frankly believe we do not need a complex new property, in this
> > case
> > > > this should fail directly because this will improve user experience.
> > > >
> > > >
> > > >
> > > >
> > > >>
> > > >>
> > > >> I would like to open the discussion on this topic. You're welcome!
> > > >>
> > > >>
> > > >> Cheers
> > > >> Tibor
> > > >>
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > For additional commands, e-mail: dev-help@maven.apache.org
> > >
> > >
> >
> > --
> > ------------------------
> > Guillaume Nodet
> >
>
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Romain Manni-Bucau <rm...@gmail.com>.
+1
if it is to investigate a CI issue, it is generally easy to add debug
insights (by code or agent) so a SPI sounds like the sanest for the plugin
to me.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le lun. 14 mars 2022 à 09:08, Guillaume Nodet <gn...@apache.org> a écrit :
> If that's not currently possible, maybe a SPI should be provided so that
> people can use plug in extensions to analyze the test result and override
> it if necessary (transforming an error into a warning, storing results in a
> way which is easier to use by other tools later...) ?
>
> Guillaume
>
> Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <ma...@laeubi-soft.de> a
> écrit :
>
> > I also agree that the test at least should run, we use this property to
> > run the test and produce result and later on have a buildstep that
> > analyze the results (and probably fail the build job).
> >
> > As it is not recommend, I wonder what is the recommended way to archive
> > something similar?
> >
> > Am 14.03.22 um 06:29 schrieb Olivier Lamy:
> > > On Mon, 14 Mar 2022 at 11:55, Tibor Digana <ti...@apache.org>
> > wrote:
> > >
> > >> In case of the user property *maven.test.failure.ignore* the MOJO must
> > not
> > >> throw any exception which is interpreted by the Maven Core as BUILD
> > >> SUCCESS.
> > >>
> > >
> > > This is a very simple reduction of the problem description.
> > > The documentation here
> > >
> >
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
> > > says
> > > "Set this to "true" to ignore a failure during testing. Its use is NOT
> > > RECOMMENDED, but quite convenient on occasion"
> > >
> > > Personally, I understand this to ignore failures in junit results BUT
> at
> > > least I want the tests to run.
> > > I guess this is how our users use this feature (feature we do not
> > recommend
> > > by the way...)
> > > And this is perfectly explained here
> > >
> >
> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
> > > there is a difference between ignoring some junit failures and
> ignoring a
> > > configuration error because some jvm args has been misconfigured for
> many
> > > reasons and surefire cannot start.
> > >
> > > If I follow your reasoning ("MOJO must not throw any exception ") we
> > should
> > > ignore every surefire configuration error and keep running the build
> > until
> > > the end and says BUILD SUCCESS
> > > what about
> > >
> > > mvn test -Dsurefire.rerunFailingTestsCount=notanumber
> > > -Dmaven.test.failure.ignore=true
> > >
> > > we should not throw any exceptions as you said below and keep saying
> > BUILD
> > > SUCCESS?
> > > argLine is a configuration element as any others so it should fail
> > directly
> > > and not be ignored especially when the surefire plugin cannot run.
> > >
> > >
> > >> We have received an internal requirement to change the behavior of the
> > user
> > >> property *maven.test.failure.ignore* so that the behavior will have
> one
> > >> exception.
> > >
> > > Suppose that you have JDK 1.8 but you use:
> > >> /jre/java --add-reads ...
> > >> The outcome is JVM exit with an error message.
> > >> I agree with Herve who also says that *maven.test.failure.ignore*
> > should
> > >> not allow the MOJO to throw the exception. It is not a typical JVM
> > >> segmentation fault or another JVM crash where we cannot do anything
> > about
> > >> it, and where the entire build would crash in the command line which
> > >> of course means that the build cannot normally be interpreted as BUILD
> > >> SUCCESS. So we are still on the same level of failures related to the
> > test
> > >> purposes.
> > >>
> > >> On the other hand, Olivier has reopened the issues SUREFIRE-1426 and
> > >> SUREFIRE-1681 where the exceptional behavior of the feature is
> expected.
> > >> This is exactly the reason why I closed these tickets several years
> ago
> > and
> > >> my proposal was to not to have any exceptions in the feature behavior
> > and
> > >> the proposal was to introduce a new user property for exact use cases.
> > >> The next idea, which comes from two developers, would provide us with
> > the
> > >> same non exceptional and exact behavior of the user property
> > >> *maven.test.failure.ignore* but it would also provide us with new user
> > >> property in the case with fine grade control of the build errors, e.g.
> > >> *maven.test.jvm.error.ignore*.
> > >>
> > >
> > > with a default of?
> > > honestly I just see this new parameter as introducing more complexity
> in
> > an
> > > already very complex code and not worth it.
> > > again read both issue's comments and my comments.
> > > Please try to have a user POV and think about making our users'
> > experience
> > > more simple.
> > >
> > > This should be very simple:
> > > If surefire forked jvm cannot start it's build error and cannot run any
> > > tests, it's a problem users want to know immediately because it can be
> > for
> > > a lot of reasons: wrong argLine, not enough memory on the system etc...
> > >
> > > AND AGAIN it is very different from ignoring junit result failures.
> > >
> > > Try to look at a user point of view and think about what is the use
> case
> > of
> > > the property maven.test.failure.ignore=true, I guess 99% of the time,
> > users
> > > wants to run all their tests (even on a CI with different matrix) so
> they
> > > can collect all the tests results which has runned to see if there is
> any
> > > issue for some combination of their matrix and eventually turn the
> result
> > > as unstable (this is a very typical use case in Jenkins and was even a
> > > built in feature of the previous Jenkins Maven plugin)
> > > BUT if for any reasons one of the module do not start his tests because
> > the
> > > jvm cannot be start the users will not see that and will be totally
> blind
> > > until maybe someone look inside a very very large log file (which means
> > > probably never)
> > >
> > > Long story short as my experience as a user facing this problem/bug:
> > > I had the case on a very large multi modules (~250 modules) build of a
> > very
> > > used open source project.
> > > I was using this maven.test.failure.ignore property but one of the
> > modules
> > > had a bad jpms configuration for a jdk17 profile on the CI.
> > > The build has a matrix of 2 os and 4 jdks and different maven run which
> > > means around ~60k tests and a Jenkins log file about 40M
> > > So because of this property the build was not failing and kept saying
> > BUILD
> > > SUCCESS for weeks/months and basically not testing one module with jdk
> > 17...
> > > And frankly you do not dig into a log file of 32M after each run
> > especially
> > > when it says success...
> > > 3 days after the first release claiming supporting jdk 17 we received a
> > bug
> > > report about a something not working with jdk17....
> > > and guess what? Where was this feature supposed to be tested?
> > >
> > > so I frankly believe we do not need a complex new property, in this
> case
> > > this should fail directly because this will improve user experience.
> > >
> > >
> > >
> > >
> > >>
> > >>
> > >> I would like to open the discussion on this topic. You're welcome!
> > >>
> > >>
> > >> Cheers
> > >> Tibor
> > >>
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
> >
>
> --
> ------------------------
> Guillaume Nodet
>
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Guillaume Nodet <gn...@apache.org>.
If that's not currently possible, maybe a SPI should be provided so that
people can use plug in extensions to analyze the test result and override
it if necessary (transforming an error into a warning, storing results in a
way which is easier to use by other tools later...) ?
Guillaume
Le lun. 14 mars 2022 à 07:43, Christoph Läubrich <ma...@laeubi-soft.de> a
écrit :
> I also agree that the test at least should run, we use this property to
> run the test and produce result and later on have a buildstep that
> analyze the results (and probably fail the build job).
>
> As it is not recommend, I wonder what is the recommended way to archive
> something similar?
>
> Am 14.03.22 um 06:29 schrieb Olivier Lamy:
> > On Mon, 14 Mar 2022 at 11:55, Tibor Digana <ti...@apache.org>
> wrote:
> >
> >> In case of the user property *maven.test.failure.ignore* the MOJO must
> not
> >> throw any exception which is interpreted by the Maven Core as BUILD
> >> SUCCESS.
> >>
> >
> > This is a very simple reduction of the problem description.
> > The documentation here
> >
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
> > says
> > "Set this to "true" to ignore a failure during testing. Its use is NOT
> > RECOMMENDED, but quite convenient on occasion"
> >
> > Personally, I understand this to ignore failures in junit results BUT at
> > least I want the tests to run.
> > I guess this is how our users use this feature (feature we do not
> recommend
> > by the way...)
> > And this is perfectly explained here
> >
> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
> > there is a difference between ignoring some junit failures and ignoring a
> > configuration error because some jvm args has been misconfigured for many
> > reasons and surefire cannot start.
> >
> > If I follow your reasoning ("MOJO must not throw any exception ") we
> should
> > ignore every surefire configuration error and keep running the build
> until
> > the end and says BUILD SUCCESS
> > what about
> >
> > mvn test -Dsurefire.rerunFailingTestsCount=notanumber
> > -Dmaven.test.failure.ignore=true
> >
> > we should not throw any exceptions as you said below and keep saying
> BUILD
> > SUCCESS?
> > argLine is a configuration element as any others so it should fail
> directly
> > and not be ignored especially when the surefire plugin cannot run.
> >
> >
> >> We have received an internal requirement to change the behavior of the
> user
> >> property *maven.test.failure.ignore* so that the behavior will have one
> >> exception.
> >
> > Suppose that you have JDK 1.8 but you use:
> >> /jre/java --add-reads ...
> >> The outcome is JVM exit with an error message.
> >> I agree with Herve who also says that *maven.test.failure.ignore*
> should
> >> not allow the MOJO to throw the exception. It is not a typical JVM
> >> segmentation fault or another JVM crash where we cannot do anything
> about
> >> it, and where the entire build would crash in the command line which
> >> of course means that the build cannot normally be interpreted as BUILD
> >> SUCCESS. So we are still on the same level of failures related to the
> test
> >> purposes.
> >>
> >> On the other hand, Olivier has reopened the issues SUREFIRE-1426 and
> >> SUREFIRE-1681 where the exceptional behavior of the feature is expected.
> >> This is exactly the reason why I closed these tickets several years ago
> and
> >> my proposal was to not to have any exceptions in the feature behavior
> and
> >> the proposal was to introduce a new user property for exact use cases.
> >> The next idea, which comes from two developers, would provide us with
> the
> >> same non exceptional and exact behavior of the user property
> >> *maven.test.failure.ignore* but it would also provide us with new user
> >> property in the case with fine grade control of the build errors, e.g.
> >> *maven.test.jvm.error.ignore*.
> >>
> >
> > with a default of?
> > honestly I just see this new parameter as introducing more complexity in
> an
> > already very complex code and not worth it.
> > again read both issue's comments and my comments.
> > Please try to have a user POV and think about making our users'
> experience
> > more simple.
> >
> > This should be very simple:
> > If surefire forked jvm cannot start it's build error and cannot run any
> > tests, it's a problem users want to know immediately because it can be
> for
> > a lot of reasons: wrong argLine, not enough memory on the system etc...
> >
> > AND AGAIN it is very different from ignoring junit result failures.
> >
> > Try to look at a user point of view and think about what is the use case
> of
> > the property maven.test.failure.ignore=true, I guess 99% of the time,
> users
> > wants to run all their tests (even on a CI with different matrix) so they
> > can collect all the tests results which has runned to see if there is any
> > issue for some combination of their matrix and eventually turn the result
> > as unstable (this is a very typical use case in Jenkins and was even a
> > built in feature of the previous Jenkins Maven plugin)
> > BUT if for any reasons one of the module do not start his tests because
> the
> > jvm cannot be start the users will not see that and will be totally blind
> > until maybe someone look inside a very very large log file (which means
> > probably never)
> >
> > Long story short as my experience as a user facing this problem/bug:
> > I had the case on a very large multi modules (~250 modules) build of a
> very
> > used open source project.
> > I was using this maven.test.failure.ignore property but one of the
> modules
> > had a bad jpms configuration for a jdk17 profile on the CI.
> > The build has a matrix of 2 os and 4 jdks and different maven run which
> > means around ~60k tests and a Jenkins log file about 40M
> > So because of this property the build was not failing and kept saying
> BUILD
> > SUCCESS for weeks/months and basically not testing one module with jdk
> 17...
> > And frankly you do not dig into a log file of 32M after each run
> especially
> > when it says success...
> > 3 days after the first release claiming supporting jdk 17 we received a
> bug
> > report about a something not working with jdk17....
> > and guess what? Where was this feature supposed to be tested?
> >
> > so I frankly believe we do not need a complex new property, in this case
> > this should fail directly because this will improve user experience.
> >
> >
> >
> >
> >>
> >>
> >> I would like to open the discussion on this topic. You're welcome!
> >>
> >>
> >> Cheers
> >> Tibor
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
--
------------------------
Guillaume Nodet
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Christoph Läubrich <ma...@laeubi-soft.de>.
I also agree that the test at least should run, we use this property to
run the test and produce result and later on have a buildstep that
analyze the results (and probably fail the build job).
As it is not recommend, I wonder what is the recommended way to archive
something similar?
Am 14.03.22 um 06:29 schrieb Olivier Lamy:
> On Mon, 14 Mar 2022 at 11:55, Tibor Digana <ti...@apache.org> wrote:
>
>> In case of the user property *maven.test.failure.ignore* the MOJO must not
>> throw any exception which is interpreted by the Maven Core as BUILD
>> SUCCESS.
>>
>
> This is a very simple reduction of the problem description.
> The documentation here
> https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
> says
> "Set this to "true" to ignore a failure during testing. Its use is NOT
> RECOMMENDED, but quite convenient on occasion"
>
> Personally, I understand this to ignore failures in junit results BUT at
> least I want the tests to run.
> I guess this is how our users use this feature (feature we do not recommend
> by the way...)
> And this is perfectly explained here
> https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
> there is a difference between ignoring some junit failures and ignoring a
> configuration error because some jvm args has been misconfigured for many
> reasons and surefire cannot start.
>
> If I follow your reasoning ("MOJO must not throw any exception ") we should
> ignore every surefire configuration error and keep running the build until
> the end and says BUILD SUCCESS
> what about
>
> mvn test -Dsurefire.rerunFailingTestsCount=notanumber
> -Dmaven.test.failure.ignore=true
>
> we should not throw any exceptions as you said below and keep saying BUILD
> SUCCESS?
> argLine is a configuration element as any others so it should fail directly
> and not be ignored especially when the surefire plugin cannot run.
>
>
>> We have received an internal requirement to change the behavior of the user
>> property *maven.test.failure.ignore* so that the behavior will have one
>> exception.
>
> Suppose that you have JDK 1.8 but you use:
>> /jre/java --add-reads ...
>> The outcome is JVM exit with an error message.
>> I agree with Herve who also says that *maven.test.failure.ignore* should
>> not allow the MOJO to throw the exception. It is not a typical JVM
>> segmentation fault or another JVM crash where we cannot do anything about
>> it, and where the entire build would crash in the command line which
>> of course means that the build cannot normally be interpreted as BUILD
>> SUCCESS. So we are still on the same level of failures related to the test
>> purposes.
>>
>> On the other hand, Olivier has reopened the issues SUREFIRE-1426 and
>> SUREFIRE-1681 where the exceptional behavior of the feature is expected.
>> This is exactly the reason why I closed these tickets several years ago and
>> my proposal was to not to have any exceptions in the feature behavior and
>> the proposal was to introduce a new user property for exact use cases.
>> The next idea, which comes from two developers, would provide us with the
>> same non exceptional and exact behavior of the user property
>> *maven.test.failure.ignore* but it would also provide us with new user
>> property in the case with fine grade control of the build errors, e.g.
>> *maven.test.jvm.error.ignore*.
>>
>
> with a default of?
> honestly I just see this new parameter as introducing more complexity in an
> already very complex code and not worth it.
> again read both issue's comments and my comments.
> Please try to have a user POV and think about making our users' experience
> more simple.
>
> This should be very simple:
> If surefire forked jvm cannot start it's build error and cannot run any
> tests, it's a problem users want to know immediately because it can be for
> a lot of reasons: wrong argLine, not enough memory on the system etc...
>
> AND AGAIN it is very different from ignoring junit result failures.
>
> Try to look at a user point of view and think about what is the use case of
> the property maven.test.failure.ignore=true, I guess 99% of the time, users
> wants to run all their tests (even on a CI with different matrix) so they
> can collect all the tests results which has runned to see if there is any
> issue for some combination of their matrix and eventually turn the result
> as unstable (this is a very typical use case in Jenkins and was even a
> built in feature of the previous Jenkins Maven plugin)
> BUT if for any reasons one of the module do not start his tests because the
> jvm cannot be start the users will not see that and will be totally blind
> until maybe someone look inside a very very large log file (which means
> probably never)
>
> Long story short as my experience as a user facing this problem/bug:
> I had the case on a very large multi modules (~250 modules) build of a very
> used open source project.
> I was using this maven.test.failure.ignore property but one of the modules
> had a bad jpms configuration for a jdk17 profile on the CI.
> The build has a matrix of 2 os and 4 jdks and different maven run which
> means around ~60k tests and a Jenkins log file about 40M
> So because of this property the build was not failing and kept saying BUILD
> SUCCESS for weeks/months and basically not testing one module with jdk 17...
> And frankly you do not dig into a log file of 32M after each run especially
> when it says success...
> 3 days after the first release claiming supporting jdk 17 we received a bug
> report about a something not working with jdk17....
> and guess what? Where was this feature supposed to be tested?
>
> so I frankly believe we do not need a complex new property, in this case
> this should fail directly because this will improve user experience.
>
>
>
>
>>
>>
>> I would like to open the discussion on this topic. You're welcome!
>>
>>
>> Cheers
>> Tibor
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Olivier Lamy <ol...@apache.org>.
On Mon, 14 Mar 2022 at 11:55, Tibor Digana <ti...@apache.org> wrote:
> In case of the user property *maven.test.failure.ignore* the MOJO must not
> throw any exception which is interpreted by the Maven Core as BUILD
> SUCCESS.
>
This is a very simple reduction of the problem description.
The documentation here
https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore
says
"Set this to "true" to ignore a failure during testing. Its use is NOT
RECOMMENDED, but quite convenient on occasion"
Personally, I understand this to ignore failures in junit results BUT at
least I want the tests to run.
I guess this is how our users use this feature (feature we do not recommend
by the way...)
And this is perfectly explained here
https://issues.apache.org/jira/browse/SUREFIRE-1426?focusedCommentId=16188077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16188077
there is a difference between ignoring some junit failures and ignoring a
configuration error because some jvm args has been misconfigured for many
reasons and surefire cannot start.
If I follow your reasoning ("MOJO must not throw any exception ") we should
ignore every surefire configuration error and keep running the build until
the end and says BUILD SUCCESS
what about
mvn test -Dsurefire.rerunFailingTestsCount=notanumber
-Dmaven.test.failure.ignore=true
we should not throw any exceptions as you said below and keep saying BUILD
SUCCESS?
argLine is a configuration element as any others so it should fail directly
and not be ignored especially when the surefire plugin cannot run.
> We have received an internal requirement to change the behavior of the user
> property *maven.test.failure.ignore* so that the behavior will have one
> exception.
Suppose that you have JDK 1.8 but you use:
> /jre/java --add-reads ...
> The outcome is JVM exit with an error message.
> I agree with Herve who also says that *maven.test.failure.ignore* should
> not allow the MOJO to throw the exception. It is not a typical JVM
> segmentation fault or another JVM crash where we cannot do anything about
> it, and where the entire build would crash in the command line which
> of course means that the build cannot normally be interpreted as BUILD
> SUCCESS. So we are still on the same level of failures related to the test
> purposes.
>
> On the other hand, Olivier has reopened the issues SUREFIRE-1426 and
> SUREFIRE-1681 where the exceptional behavior of the feature is expected.
> This is exactly the reason why I closed these tickets several years ago and
> my proposal was to not to have any exceptions in the feature behavior and
> the proposal was to introduce a new user property for exact use cases.
> The next idea, which comes from two developers, would provide us with the
> same non exceptional and exact behavior of the user property
> *maven.test.failure.ignore* but it would also provide us with new user
> property in the case with fine grade control of the build errors, e.g.
> *maven.test.jvm.error.ignore*.
>
with a default of?
honestly I just see this new parameter as introducing more complexity in an
already very complex code and not worth it.
again read both issue's comments and my comments.
Please try to have a user POV and think about making our users' experience
more simple.
This should be very simple:
If surefire forked jvm cannot start it's build error and cannot run any
tests, it's a problem users want to know immediately because it can be for
a lot of reasons: wrong argLine, not enough memory on the system etc...
AND AGAIN it is very different from ignoring junit result failures.
Try to look at a user point of view and think about what is the use case of
the property maven.test.failure.ignore=true, I guess 99% of the time, users
wants to run all their tests (even on a CI with different matrix) so they
can collect all the tests results which has runned to see if there is any
issue for some combination of their matrix and eventually turn the result
as unstable (this is a very typical use case in Jenkins and was even a
built in feature of the previous Jenkins Maven plugin)
BUT if for any reasons one of the module do not start his tests because the
jvm cannot be start the users will not see that and will be totally blind
until maybe someone look inside a very very large log file (which means
probably never)
Long story short as my experience as a user facing this problem/bug:
I had the case on a very large multi modules (~250 modules) build of a very
used open source project.
I was using this maven.test.failure.ignore property but one of the modules
had a bad jpms configuration for a jdk17 profile on the CI.
The build has a matrix of 2 os and 4 jdks and different maven run which
means around ~60k tests and a Jenkins log file about 40M
So because of this property the build was not failing and kept saying BUILD
SUCCESS for weeks/months and basically not testing one module with jdk 17...
And frankly you do not dig into a log file of 32M after each run especially
when it says success...
3 days after the first release claiming supporting jdk 17 we received a bug
report about a something not working with jdk17....
and guess what? Where was this feature supposed to be tested?
so I frankly believe we do not need a complex new property, in this case
this should fail directly because this will improve user experience.
>
>
> I would like to open the discussion on this topic. You're welcome!
>
>
> Cheers
> Tibor
>
Re: issues with the user property "maven.test.failure.ignore" and new proposals
Posted by Slawomir Jaranowski <s....@gmail.com>.
Hi
First sentence: Set this to "true" to ignore a failure during testing.
It is clear for me - we want to ignore a failure during testing ... So if
the testing process is not starting for any reason or was broken for
reasons other than test failure this property should not be considered.
Most of the case we use it in CI systems for generating test reports to
analyze trends of test stability and so on ...
So if the report of tests is not generated or is truncated property should
not be considered.
And last, surefire has plenty of parameters, introduced next one for cover
1% of cases is form over substance.
Surefire code and configuration is complicated enough. We should find a way
to simplify it - not introduce something more complicated.
Current code of surefire is difficult to understand, we have many corner
cases introduced by plenty of configuration options.
When we have simpler code, bug fix will be easier, more contributors can
help us to improve.
Now we see every bug, new features produce a lot of noise and it is a
problem to merge it.
And in consequence when users wait two years for fixing something ... I
don't think they are happy about it.
pon., 14 mar 2022 o 02:55 Tibor Digana <ti...@apache.org> napisał(a):
> In case of the user property *maven.test.failure.ignore* the MOJO must not
> throw any exception which is interpreted by the Maven Core as BUILD
> SUCCESS.
>
> We have received an internal requirement to change the behavior of the user
> property *maven.test.failure.ignore* so that the behavior will have one
> exception.
>
> Suppose that you have JDK 1.8 but you use:
> /jre/java --add-reads ...
> The outcome is JVM exit with an error message.
> I agree with Herve who also says that *maven.test.failure.ignore* should
> not allow the MOJO to throw the exception. It is not a typical JVM
> segmentation fault or another JVM crash where we cannot do anything about
> it, and where the entire build would crash in the command line which
> of course means that the build cannot normally be interpreted as BUILD
> SUCCESS. So we are still on the same level of failures related to the test
> purposes.
>
> On the other hand, Olivier has reopened the issues SUREFIRE-1426 and
> SUREFIRE-1681 where the exceptional behavior of the feature is expected.
> This is exactly the reason why I closed these tickets several years ago and
> my proposal was to not to have any exceptions in the feature behavior and
> the proposal was to introduce a new user property for exact use cases.
> The next idea, which comes from two developers, would provide us with the
> same non exceptional and exact behavior of the user property
> *maven.test.failure.ignore* but it would also provide us with new user
> property in the case with fine grade control of the build errors, e.g.
> *maven.test.jvm.error.ignore*.
>
>
> I would like to open the discussion on this topic. You're welcome!
>
>
> Cheers
> Tibor
>
--
Sławomir Jaranowski