[jira] [Commented] (YETUS-587) Report if junit drops a significant amount of tests

["Allen Wittenauer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YETUS-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277536#comment-16277536 ] 

Allen Wittenauer commented on YETUS-587:
----------------------------------------

So some background on this:

During The Great HDFS Causing Build Node Crashes of 2017, it was discovered that surefire doesn't report any data on any tests that it doesn't run.  For example, if surefire crashes before the mvn test is complete, then all of the tests that were to be run post-crash don't effectively disappear.

From a UI perspective, this is disastrous if any of the previous tests have failed.  Users are led to believe that the only tests that failed are those that were reported.

FWIW, I filed SUREFIRE-1447 as to get a means to ask surefire how many tests were expected to run.  

Playing around with the idea presented above, I had a lot of trouble matching the actual totals.   However, it does provide a pretty good estimate for a floor value.  If we don't reach that floor value, we should throw out all of the results and declare the whole test failed rather than reporting any other results.  This will cause end users to actually look at the log to see how bad things are.

> Report if junit drops a significant amount of tests
> ---------------------------------------------------
>
>                 Key: YETUS-587
>                 URL: https://issues.apache.org/jira/browse/YETUS-587
>             Project: Yetus
>          Issue Type: New Feature
>          Components: Test Patch
>            Reporter: Allen Wittenauer
>            Priority: Blocker
>             Fix For: 0.7.0
>
>
> I believe precommit should be able to guesstimate how many tests results are expected from junit by taking the number @test lines subtracted by the number of @ignore lines. Precommit can then compare that to the actual numbers received.  If that number is below some configurable percentage, precommit should throw a very significant warning probably in lieu of reporting the actual failed tests since things almost certainly went very very wrong.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)