Running tests after gradle update

[Felix Schumacher <fe...@internetallee.de>]

Hi all,

I tried to rebase pr 723 to the current main trunk and it seems to me, 
that most tests are not run anymore. Maybe I am doing something wrong.

I usually do

 > ./gradlew test

inside the checked out directory and it runs a lot of tests for a few 
minutes.

Now it is ready after 6s and it seems to run only a few of our tests. It 
reports the following at the end:

BUILD SUCCESSFUL in 6s
205 actionable tasks: 62 executed, 82 from cache, 61 up-to-date

Is this expected?

Felix

Re: Running tests after gradle update

[Felix Schumacher <fe...@internetallee.de>]

Am 29.04.23 um 15:44 schrieb Vladimir Sitnikov:
> Felix> That is what I thought, too. But even changing the source of the
> test (with something, I know will break it), didn't work. Spock tests in
> the same package (src/components) will run just fine.
>
> It turns out that junit5 tests were not executed since there was no
> junit-jupiter engine on the test classpath.
> junit4 worked because the vintage engine was there.
>
> I've fixed the issue in
> https://github.com/apache/jmeter/commit/e52305d6a4a867ad96e374fadeffe0b67e514894

Yes, works again.

Thanks for fixing this.

Felix

>
> Vladimir
>

Re: Running tests after gradle update

[Vladimir Sitnikov <si...@gmail.com>]

Felix> That is what I thought, too. But even changing the source of the
test (with something, I know will break it), didn't work. Spock tests in
the same package (src/components) will run just fine.

It turns out that junit5 tests were not executed since there was no
junit-jupiter engine on the test classpath.
junit4 worked because the vintage engine was there.

I've fixed the issue in
https://github.com/apache/jmeter/commit/e52305d6a4a867ad96e374fadeffe0b67e514894

Vladimir

Re: Running tests after gradle update

[Felix Schumacher <fe...@internetallee.de>]

Am 29.04.23 um 10:59 schrieb Vladimir Sitnikov:
> I guess the reason is that execute tests several times. Gradle caches
> test results.
> In other words, if you execute the same tests with the same source,
> then the build does not execute tests again.
> However, if you modify test code and/or change dependency, then tests
> would be re-executed.

That is what I thought, too. But even changing the source of the test 
(with something, I know will break it), didn't work. Spock tests in the 
same package (src/components) will run just fine.

Another strange thing is, that intellij shows the test as runnable 
(green triangle) and starts running tests, when I press that button, but 
afterwards it reports, that no tests are found:

  Execution failed for task ':src:components:test'.
  > No tests found for given includes: 
[org.apache.jmeter.assertions.TestJSONPathAssertion.testGetExpectedValue](--tests 
filter)

  * Try:
  > Run with --stacktrace option to get the stack trace.
  > Run with --info or --debug option to get more log output.

>
> batchTests are not cached though since they depend on external systems,
> so if you execute ./gradlew batchTests, then it will take some time
> (batch tests are executed sequentially for now).
>
> I think it is expected that we cache test results.
> If we have randomized tests and or tests that depend on external
> systems, we should configure them accordingly
> so Gradle does not cache them.
>
> WDYT?

I have nothing against caching tests, as long as it works :)

   There are only two hard problems in computer science:

    * Naming things
    * Cache invalidation
    * One off problems

Felix

>
> Vladimir

Re: Running tests after gradle update

[Vladimir Sitnikov <si...@gmail.com>]

I guess the reason is that execute tests several times. Gradle caches
test results.
In other words, if you execute the same tests with the same source,
then the build does not execute tests again.
However, if you modify test code and/or change dependency, then tests
would be re-executed.

batchTests are not cached though since they depend on external systems,
so if you execute ./gradlew batchTests, then it will take some time
(batch tests are executed sequentially for now).

I think it is expected that we cache test results.
If we have randomized tests and or tests that depend on external
systems, we should configure them accordingly
so Gradle does not cache them.

WDYT?

Vladimir