Worrying JUnit5 results

[Christofer Dutz <ch...@c-ware.de>]

Hi all,

I just noticed something.

A while ago I added a test in the S7 module, which tested all sorts of combinations of types.
This test alone added more than 7000 tests to the test-suite. We could see that in the trend for tests on Jenkins.
Today I noticed a huge drop in the number of executed tests and it currently seems as if Junit5 tests are no longer being executed correctly.

I’ll investigate the issue, but I have to admit that my trust in Junit 5 is again beginning to sink dramatically.

Chris

AW: Worrying JUnit5 results

[Julian Feinauer <j....@pragmaticminds.de>]

Hey Chris,


I'm no Jenkins and no groovy expert, but shouldn't this be doable in the Jenkinsfile... there should be some way to get the number of executed tests from the context and simply fail the build (or set it to yellow) if number of tests is lower than last run (should be doable via Jenkins REST API from build number -1).


But.. sounds like a typical task which is "easy" and "can be done fast" and takes 2 days in the end 😃


Julian


PS.: Added my comment also th the Jira.

________________________________
Von: Christofer Dutz <ch...@c-ware.de>
Gesendet: Donnerstag, 25. Oktober 2018 12:21:17
An: dev@plc4x.apache.org
Betreff: Re: Worrying JUnit5 results

Ok,

so I went through the builds, searching for the one, where the number of tests dropped.
Turned  out to be the one I added the groovy support.

Did a while of searching for a solution and it seems that following the tutorials and importing "groovy-all" imported an insane number of libraries.
So I tried out reducing that to only the test-related ones, and the Junit5 tests came back.

But we should think of a way to be able to detect that in the future. Cause we never know if someone
adds a dependency and a lot of our tests silently just stop running without any complaint.

Think I'll create a Jira for coming up with something ... done PLC4X-68

Chris


Am 25.10.18, 00:41 schrieb "Christofer Dutz" <ch...@c-ware.de>:

    Hi all,

    I just noticed something.

    A while ago I added a test in the S7 module, which tested all sorts of combinations of types.
    This test alone added more than 7000 tests to the test-suite. We could see that in the trend for tests on Jenkins.
    Today I noticed a huge drop in the number of executed tests and it currently seems as if Junit5 tests are no longer being executed correctly.

    I’ll investigate the issue, but I have to admit that my trust in Junit 5 is again beginning to sink dramatically.

    Chris

Re: Worrying JUnit5 results

[Christofer Dutz <ch...@c-ware.de>]

Ok,

so I went through the builds, searching for the one, where the number of tests dropped.
Turned  out to be the one I added the groovy support. 

Did a while of searching for a solution and it seems that following the tutorials and importing "groovy-all" imported an insane number of libraries. 
So I tried out reducing that to only the test-related ones, and the Junit5 tests came back.

But we should think of a way to be able to detect that in the future. Cause we never know if someone 
adds a dependency and a lot of our tests silently just stop running without any complaint.

Think I'll create a Jira for coming up with something ... done PLC4X-68

Chris


Am 25.10.18, 00:41 schrieb "Christofer Dutz" <ch...@c-ware.de>:

    Hi all,
    
    I just noticed something.
    
    A while ago I added a test in the S7 module, which tested all sorts of combinations of types.
    This test alone added more than 7000 tests to the test-suite. We could see that in the trend for tests on Jenkins.
    Today I noticed a huge drop in the number of executed tests and it currently seems as if Junit5 tests are no longer being executed correctly.
    
    I’ll investigate the issue, but I have to admit that my trust in Junit 5 is again beginning to sink dramatically.
    
    Chris