You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by Philipp Zehnder <ph...@bytefabrik.ai> on 2023/01/03 09:49:38 UTC
[DISCUSS] When to run the integration tests?
Hi all,
in PR [1] we fixed the unit tests and added them to the GitHub actions build pipeline.
When I ran the unit tests locally, I saw that the new module "streampipes-integration-test" was also executed.
I wanted to discuss when to run these tests. When I ran `mvn clean test` locally, it started downloading Docker containers and the integration tests took over two minutes.
I think this is too long for unit tests. So, my suggestion would be to run them only in the CI/CD pipeline and skip when a developer "only" runs the unit test.
What do you think and how could we configure such behavior?
Cheers,
Philipp
Re: AW: [DISCUSS] When to run the integration tests?
Posted by Dominik Riemer <ri...@apache.org>.
Hi,
yes, we could use some flag to control execution of certain flags.
I would be in favor of having a default which runs all tests (e.g., as a user that downloads our release and performs an mvn package, I would expect that all Junit and integration tests are running).
So similar to the -DskipTests flag that disables Junit flags in Maven, we could have something like -DskipIntegrationTests which skips integration tests only, but in my opinion this should be explicitly called and not be the default.
Best
Dominik
On 2023/01/06 10:10:04 Tim wrote:
> Hi Philipp,
>
> I think JUnit's categories could be a solution for this idea.
> Base on this stack overflow reference [1] and a blog post [3], I have
> created a minimal example (see here [2]).
> With this setup you could run unit tests ("mvn clean test -P UnitTests")
> apart from integration tests ("mvn clean test -P IntegrationTests").
> The downside of the current setup is that we would need to categorize
> every existing test but maybe there is also another way how to define
> the maven profiles resp. test groups.
>
> Unfortunately, this runs not out of the box due to some dependecy issues
> which I couldn't resolve immediately.
>
> In case we want to follow up on that idea, we can investigate further.
>
> Best
> Tim
>
>
> [1] https://stackoverflow.com/a/18297553
> [2]
> https://github.com/apache/streampipes/tree/experimental/test-categories
> [3]
> https://technicaltesting.wordpress.com/2012/09/24/using-junit-category-and-maven-profiles-to-get-stable-test-suites/
>
>
>
> Am 04.01.2023 17:21 schrieb Philipp Zehnder:
> > Hi Tim,
> >
> > I agree. I think it makes sense to run them once a day. We could
> > integrate them into our nightly test pipeline [1].
> > However, I am not sure how to configure maven accordingly. Our goal
> > would be to exclude the integration tests from the regular unit tests
> > and “only” run them during the nightly build.
> >
> > Is this a problem that could be solved with prefixes or suffixes of the
> > tests?
> >
> > Cheers,
> > Philipp
> >
> > [1]
> > https://github.com/apache/streampipes/blob/dev/.github/workflows/cypress-test.yml
> >
> >
> > Von: Tim <bo...@posteo.de>
> > Datum: Mittwoch, 4. Januar 2023 um 17:08
> > An: dev@streampipes.apache.org <de...@streampipes.apache.org>
> > Betreff: Re: [DISCUSS] When to run the integration tests?
> > Hi Philipp,
> >
> > I like the idea of having a (dedicated?) CI pipeline for our
> > integration
> > tests.
> > How do you think about running them once per day (probably somewhen in
> > the night) or once per week?
> >
> > Is there a possibility to work with contexts or flags in Maven?
> > Maybe something towards this direction could allow us to carve out the
> > integration tests.
> >
> > Best
> > Tim
> >
> > Am 03.01.2023 10:49 schrieb Philipp Zehnder:
> >> Hi all,
> >>
> >> in PR [1] we fixed the unit tests and added them to the GitHub actions
> >> build pipeline.
> >> When I ran the unit tests locally, I saw that the new module
> >> "streampipes-integration-test" was also executed.
> >>
> >> I wanted to discuss when to run these tests. When I ran `mvn clean
> >> test` locally, it started downloading Docker containers and the
> >> integration tests took over two minutes.
> >> I think this is too long for unit tests. So, my suggestion would be to
> >> run them only in the CI/CD pipeline and skip when a developer "only"
> >> runs the unit test.
> >>
> >> What do you think and how could we configure such behavior?
> >>
> >> Cheers,
> >> Philipp
>
AW: AW: [DISCUSS] When to run the integration tests?
Posted by Philipp Zehnder <ph...@bytefabrik.ai>.
Hi,
I agree, a flag would be good to activate and deactivate the integrations tests. Another solution might be to execute them in different cycles of the maven build process.
I looked it up and I guess we can use a combination of the "maven-surefire- plugin" to run the unit tests and the "maven-failsafe-plugin" for the integration tests. But I'm not quite sure yet which is the best solution.
For the e2e tests we use different suffixes for different test types, maybe we can do something similar for these tests?
Regarding the default settings, I would suggest that we look at the actual run time and then decide when to run them.
Cheers,
Philipp
Von: Dominik Riemer <do...@bytefabrik.ai>
Datum: Mittwoch, 11. Januar 2023 um 08:50
An: dev@streampipes.apache.org <de...@streampipes.apache.org>
Betreff: RE: AW: [DISCUSS] When to run the integration tests?
Hi,
just answering again as it seems the Ponymail web interface has somehow eaten my last message.
I like the idea to add more control over which tests are running when. As a user, when downloading a release, I would expect that all tests are running (Junit + Integration Tests) by default when doing an mvn package.
So I would prefer to add a skip flag which skips certain tests by explicit command.
E.g., as -DskipTests is the standard command to run the build without any tests, we could have some flag such as -DskipIntegrationTests which skips the integration tests, but I'm not in favor of having this as default or require a specific profile.
Cheers
Dominik
-----Original Message-----
From: Tim <bo...@posteo.de>
Sent: Friday, January 6, 2023 11:10 AM
To: dev@streampipes.apache.org
Subject: Re: AW: [DISCUSS] When to run the integration tests?
Hi Philipp,
I think JUnit's categories could be a solution for this idea.
Base on this stack overflow reference [1] and a blog post [3], I have created a minimal example (see here [2]).
With this setup you could run unit tests ("mvn clean test -P UnitTests") apart from integration tests ("mvn clean test -P IntegrationTests").
The downside of the current setup is that we would need to categorize every existing test but maybe there is also another way how to define the maven profiles resp. test groups.
Unfortunately, this runs not out of the box due to some dependecy issues which I couldn't resolve immediately.
In case we want to follow up on that idea, we can investigate further.
Best
Tim
[1] https://stackoverflow.com/a/18297553
[2]
https://github.com/apache/streampipes/tree/experimental/test-categories
[3]
https://technicaltesting.wordpress.com/2012/09/24/using-junit-category-and-maven-profiles-to-get-stable-test-suites/
Am 04.01.2023 17:21 schrieb Philipp Zehnder:
> Hi Tim,
>
> I agree. I think it makes sense to run them once a day. We could
> integrate them into our nightly test pipeline [1].
> However, I am not sure how to configure maven accordingly. Our goal
> would be to exclude the integration tests from the regular unit tests
> and “only” run them during the nightly build.
>
> Is this a problem that could be solved with prefixes or suffixes of
> the tests?
>
> Cheers,
> Philipp
>
> [1]
> https://github.com/apache/streampipes/blob/dev/.github/workflows/cypre
> ss-test.yml
>
>
> Von: Tim <bo...@posteo.de>
> Datum: Mittwoch, 4. Januar 2023 um 17:08
> An: dev@streampipes.apache.org <de...@streampipes.apache.org>
> Betreff: Re: [DISCUSS] When to run the integration tests?
> Hi Philipp,
>
> I like the idea of having a (dedicated?) CI pipeline for our
> integration tests.
> How do you think about running them once per day (probably somewhen in
> the night) or once per week?
>
> Is there a possibility to work with contexts or flags in Maven?
> Maybe something towards this direction could allow us to carve out the
> integration tests.
>
> Best
> Tim
>
> Am 03.01.2023 10:49 schrieb Philipp Zehnder:
>> Hi all,
>>
>> in PR [1] we fixed the unit tests and added them to the GitHub
>> actions build pipeline.
>> When I ran the unit tests locally, I saw that the new module
>> "streampipes-integration-test" was also executed.
>>
>> I wanted to discuss when to run these tests. When I ran `mvn clean
>> test` locally, it started downloading Docker containers and the
>> integration tests took over two minutes.
>> I think this is too long for unit tests. So, my suggestion would be
>> to run them only in the CI/CD pipeline and skip when a developer "only"
>> runs the unit test.
>>
>> What do you think and how could we configure such behavior?
>>
>> Cheers,
>> Philipp
RE: AW: [DISCUSS] When to run the integration tests?
Posted by Dominik Riemer <do...@bytefabrik.ai>.
Hi,
just answering again as it seems the Ponymail web interface has somehow eaten my last message.
I like the idea to add more control over which tests are running when. As a user, when downloading a release, I would expect that all tests are running (Junit + Integration Tests) by default when doing an mvn package.
So I would prefer to add a skip flag which skips certain tests by explicit command.
E.g., as -DskipTests is the standard command to run the build without any tests, we could have some flag such as -DskipIntegrationTests which skips the integration tests, but I'm not in favor of having this as default or require a specific profile.
Cheers
Dominik
-----Original Message-----
From: Tim <bo...@posteo.de>
Sent: Friday, January 6, 2023 11:10 AM
To: dev@streampipes.apache.org
Subject: Re: AW: [DISCUSS] When to run the integration tests?
Hi Philipp,
I think JUnit's categories could be a solution for this idea.
Base on this stack overflow reference [1] and a blog post [3], I have created a minimal example (see here [2]).
With this setup you could run unit tests ("mvn clean test -P UnitTests") apart from integration tests ("mvn clean test -P IntegrationTests").
The downside of the current setup is that we would need to categorize every existing test but maybe there is also another way how to define the maven profiles resp. test groups.
Unfortunately, this runs not out of the box due to some dependecy issues which I couldn't resolve immediately.
In case we want to follow up on that idea, we can investigate further.
Best
Tim
[1] https://stackoverflow.com/a/18297553
[2]
https://github.com/apache/streampipes/tree/experimental/test-categories
[3]
https://technicaltesting.wordpress.com/2012/09/24/using-junit-category-and-maven-profiles-to-get-stable-test-suites/
Am 04.01.2023 17:21 schrieb Philipp Zehnder:
> Hi Tim,
>
> I agree. I think it makes sense to run them once a day. We could
> integrate them into our nightly test pipeline [1].
> However, I am not sure how to configure maven accordingly. Our goal
> would be to exclude the integration tests from the regular unit tests
> and “only” run them during the nightly build.
>
> Is this a problem that could be solved with prefixes or suffixes of
> the tests?
>
> Cheers,
> Philipp
>
> [1]
> https://github.com/apache/streampipes/blob/dev/.github/workflows/cypre
> ss-test.yml
>
>
> Von: Tim <bo...@posteo.de>
> Datum: Mittwoch, 4. Januar 2023 um 17:08
> An: dev@streampipes.apache.org <de...@streampipes.apache.org>
> Betreff: Re: [DISCUSS] When to run the integration tests?
> Hi Philipp,
>
> I like the idea of having a (dedicated?) CI pipeline for our
> integration tests.
> How do you think about running them once per day (probably somewhen in
> the night) or once per week?
>
> Is there a possibility to work with contexts or flags in Maven?
> Maybe something towards this direction could allow us to carve out the
> integration tests.
>
> Best
> Tim
>
> Am 03.01.2023 10:49 schrieb Philipp Zehnder:
>> Hi all,
>>
>> in PR [1] we fixed the unit tests and added them to the GitHub
>> actions build pipeline.
>> When I ran the unit tests locally, I saw that the new module
>> "streampipes-integration-test" was also executed.
>>
>> I wanted to discuss when to run these tests. When I ran `mvn clean
>> test` locally, it started downloading Docker containers and the
>> integration tests took over two minutes.
>> I think this is too long for unit tests. So, my suggestion would be
>> to run them only in the CI/CD pipeline and skip when a developer "only"
>> runs the unit test.
>>
>> What do you think and how could we configure such behavior?
>>
>> Cheers,
>> Philipp
Re: AW: [DISCUSS] When to run the integration tests?
Posted by Tim <bo...@posteo.de>.
Hi Philipp,
I think JUnit's categories could be a solution for this idea.
Base on this stack overflow reference [1] and a blog post [3], I have
created a minimal example (see here [2]).
With this setup you could run unit tests ("mvn clean test -P UnitTests")
apart from integration tests ("mvn clean test -P IntegrationTests").
The downside of the current setup is that we would need to categorize
every existing test but maybe there is also another way how to define
the maven profiles resp. test groups.
Unfortunately, this runs not out of the box due to some dependecy issues
which I couldn't resolve immediately.
In case we want to follow up on that idea, we can investigate further.
Best
Tim
[1] https://stackoverflow.com/a/18297553
[2]
https://github.com/apache/streampipes/tree/experimental/test-categories
[3]
https://technicaltesting.wordpress.com/2012/09/24/using-junit-category-and-maven-profiles-to-get-stable-test-suites/
Am 04.01.2023 17:21 schrieb Philipp Zehnder:
> Hi Tim,
>
> I agree. I think it makes sense to run them once a day. We could
> integrate them into our nightly test pipeline [1].
> However, I am not sure how to configure maven accordingly. Our goal
> would be to exclude the integration tests from the regular unit tests
> and “only” run them during the nightly build.
>
> Is this a problem that could be solved with prefixes or suffixes of the
> tests?
>
> Cheers,
> Philipp
>
> [1]
> https://github.com/apache/streampipes/blob/dev/.github/workflows/cypress-test.yml
>
>
> Von: Tim <bo...@posteo.de>
> Datum: Mittwoch, 4. Januar 2023 um 17:08
> An: dev@streampipes.apache.org <de...@streampipes.apache.org>
> Betreff: Re: [DISCUSS] When to run the integration tests?
> Hi Philipp,
>
> I like the idea of having a (dedicated?) CI pipeline for our
> integration
> tests.
> How do you think about running them once per day (probably somewhen in
> the night) or once per week?
>
> Is there a possibility to work with contexts or flags in Maven?
> Maybe something towards this direction could allow us to carve out the
> integration tests.
>
> Best
> Tim
>
> Am 03.01.2023 10:49 schrieb Philipp Zehnder:
>> Hi all,
>>
>> in PR [1] we fixed the unit tests and added them to the GitHub actions
>> build pipeline.
>> When I ran the unit tests locally, I saw that the new module
>> "streampipes-integration-test" was also executed.
>>
>> I wanted to discuss when to run these tests. When I ran `mvn clean
>> test` locally, it started downloading Docker containers and the
>> integration tests took over two minutes.
>> I think this is too long for unit tests. So, my suggestion would be to
>> run them only in the CI/CD pipeline and skip when a developer "only"
>> runs the unit test.
>>
>> What do you think and how could we configure such behavior?
>>
>> Cheers,
>> Philipp
AW: [DISCUSS] When to run the integration tests?
Posted by Philipp Zehnder <ph...@bytefabrik.ai>.
Hi Tim,
I agree. I think it makes sense to run them once a day. We could integrate them into our nightly test pipeline [1].
However, I am not sure how to configure maven accordingly. Our goal would be to exclude the integration tests from the regular unit tests and “only” run them during the nightly build.
Is this a problem that could be solved with prefixes or suffixes of the tests?
Cheers,
Philipp
[1] https://github.com/apache/streampipes/blob/dev/.github/workflows/cypress-test.yml
Von: Tim <bo...@posteo.de>
Datum: Mittwoch, 4. Januar 2023 um 17:08
An: dev@streampipes.apache.org <de...@streampipes.apache.org>
Betreff: Re: [DISCUSS] When to run the integration tests?
Hi Philipp,
I like the idea of having a (dedicated?) CI pipeline for our integration
tests.
How do you think about running them once per day (probably somewhen in
the night) or once per week?
Is there a possibility to work with contexts or flags in Maven?
Maybe something towards this direction could allow us to carve out the
integration tests.
Best
Tim
Am 03.01.2023 10:49 schrieb Philipp Zehnder:
> Hi all,
>
> in PR [1] we fixed the unit tests and added them to the GitHub actions
> build pipeline.
> When I ran the unit tests locally, I saw that the new module
> "streampipes-integration-test" was also executed.
>
> I wanted to discuss when to run these tests. When I ran `mvn clean
> test` locally, it started downloading Docker containers and the
> integration tests took over two minutes.
> I think this is too long for unit tests. So, my suggestion would be to
> run them only in the CI/CD pipeline and skip when a developer "only"
> runs the unit test.
>
> What do you think and how could we configure such behavior?
>
> Cheers,
> Philipp
Re: [DISCUSS] When to run the integration tests?
Posted by Tim <bo...@posteo.de>.
Hi Philipp,
I like the idea of having a (dedicated?) CI pipeline for our integration
tests.
How do you think about running them once per day (probably somewhen in
the night) or once per week?
Is there a possibility to work with contexts or flags in Maven?
Maybe something towards this direction could allow us to carve out the
integration tests.
Best
Tim
Am 03.01.2023 10:49 schrieb Philipp Zehnder:
> Hi all,
>
> in PR [1] we fixed the unit tests and added them to the GitHub actions
> build pipeline.
> When I ran the unit tests locally, I saw that the new module
> "streampipes-integration-test" was also executed.
>
> I wanted to discuss when to run these tests. When I ran `mvn clean
> test` locally, it started downloading Docker containers and the
> integration tests took over two minutes.
> I think this is too long for unit tests. So, my suggestion would be to
> run them only in the CI/CD pipeline and skip when a developer "only"
> runs the unit test.
>
> What do you think and how could we configure such behavior?
>
> Cheers,
> Philipp