You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Jan Høydahl <ja...@cominvent.com> on 2022/01/14 12:53:21 UTC
Tests: Flag to run only unit tests
Tests take forever to run, and we have totally intermingled unit tests (testing one class or with mocks) with integration tests (spinning up solr clusters, indexing etc), which is not good project hygiene imo.
Can we start tagging integration tests in a way so we can choose to leave them out for the quick dev iterations? We already have these properties for test filtering:
// filtering
[propName: 'tests.filter', value: null, description: "Applies a test filter (see :helpTests)."],
[propName: 'tests.slow', value: true, description: "Enables or disables @Slow tests."],
[propName: 'tests.nightly', value: false, description: "Enables or disables @Nightly tests."],
[propName: 'tests.weekly', value: false, description: "Enables or disables @Weekly tests."],
[propName: 'tests.monster', value: false, description: "Enables or disables @Monster tests."],
[propName: 'tests.awaitsfix', value: null, description: "Enables or disables @AwaitsFix tests."],
[propName: 'tests.badapples', value: null, description: "Enables or disables @BadApple tests."],
So perhaps add an @Ingtegrationtest annotation and a -Ptests.integrationtest=true/false flag. Then we don't need to move test files to other folder, enough to annotate them. For a start, all SolrCloudTestCase tests could be annotated (are annotations inherited?)
Then I imagine I'd run unit tests frequently and the whole suite occationally right before merging a large feature.
It would also be interesting to run only unit tests and then have a look at the JaCoco test coverage stats. I have a suspicion we have not been good enough writing basic unit tests, including for failure and corner cases.
Don't get me wrong. I think it is crucial to test real, live clusters for a project like Solr, and we should keep the tests (and stabilize them). But we cannot focus only on integration. It is developer hostile :)
I'd like a "gradlew test" to take 3-5 minutes, at most. Perhaps "gradlew check" could include integration tests while "test" don't?
I also know that Mark Miller could write 24 e-mails on this topic and how fast our tests could be. In fact, lightning fast I'm told. But let's not go there in this thread :) Progress over perfection.
Jan
Re: Tests: Flag to run only unit tests
Posted by David Smiley <ds...@apache.org>.
Something is needed.
I think unit tests should allow use of a single embedded Solr server (not
Jetty and thus not SolrCloud either). Whatever we choose, we should
document this in Javadocs on the annotation so we can point ourselves &
contributors to correct use of these annotations.
We could even have MiniSolrCloudCluster or Jetty Solr runner thing do a
test runtime look to see if the current test is classified as an
integration test, and fail otherwise. Just an idea to help us enforce use
of this correctly.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Fri, Jan 14, 2022 at 10:47 AM Jan Høydahl <ja...@cominvent.com> wrote:
> Created https://issues.apache.org/jira/browse/SOLR-15925 for this
> Feel free to continue the concept discussion here and tech discussion in
> the JIRA.
> PS: I'm not planning to assign myself to this JIRA, so up for grabs.
>
> Jan
>
> 14. jan. 2022 kl. 15:06 skrev Uwe Schindler <uw...@thetaphi.de>:
>
> +1
>
> On Jenkins we would enable those.
>
> Am 14. Januar 2022 12:53:21 UTC schrieb "Jan Høydahl" <
> jan.asf@cominvent.com>:
>>
>> Tests take forever to run, and we have totally intermingled unit tests
>> (testing one class or with mocks) with integration tests (spinning up solr
>> clusters, indexing etc), which is not good project hygiene imo.
>> Can we start tagging integration tests in a way so we can choose to leave
>> them out for the quick dev iterations? We already have these properties for
>> test filtering:
>>
>> // filtering
>> [propName: 'tests.filter', value: null, description: "Applies a test filter (see :helpTests)."],
>> [propName: 'tests.slow', value: true, description: "Enables or disables @Slow tests."],
>> [propName: 'tests.nightly', value: false, description: "Enables or disables @Nightly tests."],
>> [propName: 'tests.weekly', value: false, description: "Enables or disables @Weekly tests."],
>> [propName: 'tests.monster', value: false, description: "Enables or disables @Monster tests."],
>> [propName: 'tests.awaitsfix', value: null, description: "Enables or disables @AwaitsFix tests."],
>> [propName: 'tests.badapples', value: null, description: "Enables or disables @BadApple tests."],
>>
>> So perhaps add an @Ingtegrationtest annotation and a
>> -Ptests.integrationtest=true/false flag. Then we don't need to move test
>> files to other folder, enough to annotate them. For a start,
>> all SolrCloudTestCase tests could be annotated (are annotations inherited?)
>> Then I imagine I'd run unit tests frequently and the whole suite
>> occationally right before merging a large feature.
>> It would also be interesting to run only unit tests and then have a look
>> at the JaCoco test coverage stats. I have a suspicion we have not been good
>> enough writing basic unit tests, including for failure and corner cases.
>>
>> Don't get me wrong. I think it is crucial to test real, live clusters for
>> a project like Solr, and we should keep the tests (and stabilize them). But
>> we cannot focus only on integration. It is developer hostile :)
>> I'd like a "gradlew test" to take 3-5 minutes, at most. Perhaps "gradlew
>> check" could include integration tests while "test" don't?
>> I also know that Mark Miller could write 24 e-mails on this topic and how
>> fast our tests could be. In fact, lightning fast I'm told. But let's not go
>> there in this thread :) Progress over perfection.
>>
>> Jan
>>
>> --
> Uwe Schindler
> Achterdiek 19, 28357 Bremen
> https://www.thetaphi.de
>
>
>
Re: Tests: Flag to run only unit tests
Posted by Jan Høydahl <ja...@cominvent.com>.
Created https://issues.apache.org/jira/browse/SOLR-15925 <https://issues.apache.org/jira/browse/SOLR-15925> for this
Feel free to continue the concept discussion here and tech discussion in the JIRA.
PS: I'm not planning to assign myself to this JIRA, so up for grabs.
Jan
> 14. jan. 2022 kl. 15:06 skrev Uwe Schindler <uwe@thetaphi.de <ma...@thetaphi.de>>:
>
> +1
>
> On Jenkins we would enable those.
>
> Am 14. Januar 2022 12:53:21 UTC schrieb "Jan Høydahl" <jan.asf@cominvent.com <ma...@cominvent.com>>:
> Tests take forever to run, and we have totally intermingled unit tests (testing one class or with mocks) with integration tests (spinning up solr clusters, indexing etc), which is not good project hygiene imo.
> Can we start tagging integration tests in a way so we can choose to leave them out for the quick dev iterations? We already have these properties for test filtering:
> // filtering
> [propName: 'tests.filter', value: null, description: "Applies a test filter (see :helpTests)."],
> [propName: 'tests.slow', value: true, description: "Enables or disables @Slow tests."],
> [propName: 'tests.nightly', value: false, description: "Enables or disables @Nightly tests."],
> [propName: 'tests.weekly', value: false, description: "Enables or disables @Weekly tests."],
> [propName: 'tests.monster', value: false, description: "Enables or disables @Monster tests."],
> [propName: 'tests.awaitsfix', value: null, description: "Enables or disables @AwaitsFix tests."],
> [propName: 'tests.badapples', value: null, description: "Enables or disables @BadApple tests."],
> So perhaps add an @Ingtegrationtest annotation and a -Ptests.integrationtest=true/false flag. Then we don't need to move test files to other folder, enough to annotate them. For a start, all SolrCloudTestCase tests could be annotated (are annotations inherited?)
> Then I imagine I'd run unit tests frequently and the whole suite occationally right before merging a large feature.
> It would also be interesting to run only unit tests and then have a look at the JaCoco test coverage stats. I have a suspicion we have not been good enough writing basic unit tests, including for failure and corner cases.
>
> Don't get me wrong. I think it is crucial to test real, live clusters for a project like Solr, and we should keep the tests (and stabilize them). But we cannot focus only on integration. It is developer hostile :)
> I'd like a "gradlew test" to take 3-5 minutes, at most. Perhaps "gradlew check" could include integration tests while "test" don't?
> I also know that Mark Miller could write 24 e-mails on this topic and how fast our tests could be. In fact, lightning fast I'm told. But let's not go there in this thread :) Progress over perfection.
>
> Jan
>
> --
> Uwe Schindler
> Achterdiek 19, 28357 Bremen
> https://www.thetaphi.de <https://www.thetaphi.de/>
Re: Tests: Flag to run only unit tests
Posted by Uwe Schindler <uw...@thetaphi.de>.
+1
On Jenkins we would enable those.
Am 14. Januar 2022 12:53:21 UTC schrieb "Jan Høydahl" <ja...@cominvent.com>:
>Tests take forever to run, and we have totally intermingled unit tests (testing one class or with mocks) with integration tests (spinning up solr clusters, indexing etc), which is not good project hygiene imo.
>Can we start tagging integration tests in a way so we can choose to leave them out for the quick dev iterations? We already have these properties for test filtering:
>// filtering
>[propName: 'tests.filter', value: null, description: "Applies a test filter (see :helpTests)."],
>[propName: 'tests.slow', value: true, description: "Enables or disables @Slow tests."],
>[propName: 'tests.nightly', value: false, description: "Enables or disables @Nightly tests."],
>[propName: 'tests.weekly', value: false, description: "Enables or disables @Weekly tests."],
>[propName: 'tests.monster', value: false, description: "Enables or disables @Monster tests."],
>[propName: 'tests.awaitsfix', value: null, description: "Enables or disables @AwaitsFix tests."],
>[propName: 'tests.badapples', value: null, description: "Enables or disables @BadApple tests."],
>So perhaps add an @Ingtegrationtest annotation and a -Ptests.integrationtest=true/false flag. Then we don't need to move test files to other folder, enough to annotate them. For a start, all SolrCloudTestCase tests could be annotated (are annotations inherited?)
>Then I imagine I'd run unit tests frequently and the whole suite occationally right before merging a large feature.
>It would also be interesting to run only unit tests and then have a look at the JaCoco test coverage stats. I have a suspicion we have not been good enough writing basic unit tests, including for failure and corner cases.
>
>Don't get me wrong. I think it is crucial to test real, live clusters for a project like Solr, and we should keep the tests (and stabilize them). But we cannot focus only on integration. It is developer hostile :)
>I'd like a "gradlew test" to take 3-5 minutes, at most. Perhaps "gradlew check" could include integration tests while "test" don't?
>I also know that Mark Miller could write 24 e-mails on this topic and how fast our tests could be. In fact, lightning fast I'm told. But let's not go there in this thread :) Progress over perfection.
>
>Jan
>
--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de