You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sling.apache.org by Robert Munteanu <ro...@apache.org> on 2018/11/07 16:29:01 UTC
[PROPOSAL] Moving integration tests closer to the code they test
Hi,
We discussed a couple of time that it would be desireable to move tests
closer to the code they're testing.
I wrote a proposal about how this could look like in terms of results
https://cwiki.apache.org/confluence/display/SLING/Moving+integration+tests+closer+to+the+code+they+test
I purposefully avoided looking into implementation details to avoid
doing too much work for nothing.
What do you think about this possibility?
Thanks,
Robert
Re: [PROPOSAL] Moving integration tests closer to the code they test
Posted by Robert Munteanu <ro...@apache.org>.
On Thu, 2018-11-08 at 14:37 +0100, Bertrand Delacretaz wrote:
> Hi,
>
> On Wed, Nov 7, 2018 at 5:29 PM Robert Munteanu <ro...@apache.org>
> wrote:
> > ...
> > https://cwiki.apache.org/confluence/display/SLING/Moving+integration+tests+closer+to+the+code+they+test
> > ...
>
> I think it's a great idea to have tests closer to their code, while
> also running a large test suite against a "complete" Sling instance.
>
> Not all tests might make sense in the global test suite however, but
> that's easily fixable using test annotations or tags I suppose.
+1 on that.
FWIW, my plan would be to first move some/most of the ITs from the
launchpad.integration-testing module to specific modules - e.g.
servlets.get, servlets.post, resourceresolver.
To keep things simple, we could create an annotation or a naming
convention that would exclude certain tests that don't make sense for
the launchpad IT run.
Robert
Re: [PROPOSAL] Moving integration tests closer to the code they test
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Wed, Nov 7, 2018 at 5:29 PM Robert Munteanu <ro...@apache.org> wrote:
> ... https://cwiki.apache.org/confluence/display/SLING/Moving+integration+tests+closer+to+the+code+they+test ...
I think it's a great idea to have tests closer to their code, while
also running a large test suite against a "complete" Sling instance.
Not all tests might make sense in the global test suite however, but
that's easily fixable using test annotations or tags I suppose.
-Bertrand
Re: [PROPOSAL] Moving integration tests closer to the code they test
Posted by Robert Munteanu <ro...@apache.org>.
Hi Valentin,
On Wed, 2018-11-07 at 21:05 +0100, Valentin Olteanu wrote:
> I just wanted to drop [1] which can give you some guidelines on where
> to
> put the tests. It shows the drawbacks of having the tests in
> src/test/ and
> it states that the preferred way is to create a separate project.
>
> [1]
> https://maven.apache.org/plugins/maven-jar-plugin/examples/create-test-jar.html
Thanks for that link. I see that the intended drawback is
Note: The downside of this solution is that you don't get the
transitive test-scoped dependencies automatically.
That's definitely something to keep in mind. I don't think it's a huge
drawback for us since we will be reassembling multiple test artifacts
with similar requirements into a single module.
Actually, I would say that it's a plus since we get to control the
dependencies rather than get a mess from potentially different
dependencies from the test modules.
Robert
Re: [PROPOSAL] Moving integration tests closer to the code they test
Posted by Valentin Olteanu <va...@gmail.com>.
On Wed, Nov 7, 2018 at 5:29 PM Robert Munteanu <ro...@apache.org> wrote:
> Hi,
>
> We discussed a couple of time that it would be desireable to move tests
> closer to the code they're testing.
>
> I wrote a proposal about how this could look like in terms of results
>
>
> https://cwiki.apache.org/confluence/display/SLING/Moving+integration+tests+closer+to+the+code+they+test
>
> I purposefully avoided looking into implementation details to avoid
> doing too much work for nothing.
>
> What do you think about this possibility?
>
> Thanks,
>
> Robert
>
>
Hi Robert,
I just wanted to drop [1] which can give you some guidelines on where to
put the tests. It shows the drawbacks of having the tests in src/test/ and
it states that the preferred way is to create a separate project.
[1]
https://maven.apache.org/plugins/maven-jar-plugin/examples/create-test-jar.html
Valentin
Re: [PROPOSAL] Moving integration tests closer to the code they test
Posted by Robert Munteanu <ro...@apache.org>.
On Thu, 2018-11-08 at 17:07 +0100, Oliver Lietz wrote:
> I wrote a proposal about how this could look like in terms of
> > results
> >
> > https://cwiki.apache.org/confluence/display/SLING/Moving+integration+tests+c
> > loser+to+the+code+they+test
> >
> > I purposefully avoided looking into implementation details to avoid
> > doing too much work for nothing.
> >
> > What do you think about this possibility?
>
> did I get your point - we keep free-style and "managed" (using Sling
> Testing
> PaxExam) Pax Exam tests (which may also use Sling's HTTP API) as is?
Yes, my suggestion is to keep pax-exam tests separate and not run with
the larger set of ITs. This is mainly because I think it's simpler to
not tackle them right now. The Pax-Exam ITs manage their own Sling
runtime rather than run against a given Sling release. This in turn
makes it harder for me to propose a unified way of launching them as
ITs running against the latest Sling starter.
Thanks,
Robert
Re: [PROPOSAL] Moving integration tests closer to the code they test
Posted by Oliver Lietz <ap...@oliverlietz.de>.
On Wednesday 07 November 2018 17:29:01 Robert Munteanu wrote:
> Hi,
Hi Robert,
> We discussed a couple of time that it would be desireable to move tests
> closer to the code they're testing.
>
> I wrote a proposal about how this could look like in terms of results
>
> https://cwiki.apache.org/confluence/display/SLING/Moving+integration+tests+c
> loser+to+the+code+they+test
>
> I purposefully avoided looking into implementation details to avoid
> doing too much work for nothing.
>
> What do you think about this possibility?
did I get your point - we keep free-style and "managed" (using Sling Testing
PaxExam) Pax Exam tests (which may also use Sling's HTTP API) as is?
Thanks,
O.
> Thanks,
>
> Robert
Re: [PROPOSAL] Moving integration tests closer to the code they test
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Tue, Nov 20, 2018 at 4:56 PM Robert Munteanu <ro...@apache.org> wrote:
> ...If I get my ITs executed on `mvn install` it's perfect. If I need to run
> something else, well, ... not so much :-)..
+1 to keeping ITs in the modules that they test, with about 467 Sling
modules at last count, I think it's important for them to be as
self-contained as possible.
-Bertrand
Re: [PROPOSAL] Moving integration tests closer to the code they test
Posted by Robert Munteanu <ro...@apache.org>.
Hi Andrei,
Thanks for commenting and sorry for taking so long to reply.
On Fri, 2018-11-09 at 15:22 +0100, Andrei Dulvac wrote:
> +1 for moving the tests to the code as well.
>
> However... I would be careful to check what is packaged in the
> `tests`
> artifact produced by the build.
> Should we package the test transient dependencies? Otherwise it will
> be
> complicated to run multiple test artifacts under one build, with
> different
> tests having different dependencies. e.g. tests with dep to a
> different
> version of the sling-testing-rules. And speaking of running multiple
> tests
> together in a different job, one big concern (if we decide to do some
> magic) is classpath shadowing.
Well, now having spent enough time with OSGi I'd say we should stay
away from transitive (or any other implicit) dependencies. The
integration tests that that will be part of the 'main' IT run should
have a small, fixed, set of dependencies.
> I've seen the following happen:
>
> A maven project that
> 1. Collected multiple tests modules
> 2. Collected their dependencies
> 3. Ran failsafe on that dir, sharing the classpath
>
> Sounds ok, but it isn't. because of classpath shadowing.
... which will not happen if we only use the single set of dependencies
from the 'main' IT module.
>
> So because of complications like these, I'd second Valentin's slight
> concern and make them a different maven module and put the tests in
> src/main. Arguably makes it a bit easier to manage what's packaged
> inside
> and how they're ran together?
The main downside that I see with different Maven modules is that the
tests no longer live in the same project as the code being tested,
which is basically the reason for kicking off this discussion. If I get
my ITs executed on `mvn install` it's perfect. If I need to run
something else, well, ... not so much :-)
Thanks,
Robert
>
> WDYT?
> - Andrei
>
> On Fri, Nov 9, 2018 at 2:28 PM Robert Munteanu <ro...@apache.org>
> wrote:
>
> > On Fri, 2018-11-09 at 13:50 +0100, Georg Henzler wrote:
> > > +1 to move the tests closer to the code.
> > >
> > > Regarding the path src/test/it (mentioned in the wiki page):
> > > Shouldn't
> > > we rather stick with src/test/java for both ITs and regular tests
> > > and
> > > just use the standard name conventions as configured by default
> > > for
> > > plugins surefire/failsafe maven plugins, that is *Test.java [1]
> > > for
> > > unit
> > > tests and *IT.java [2] for integration tests?
> >
> > Yes, you're right.
> >
> > I'll update the wiki page. And also this is what I'm doing in my
> > POC,
> > see [3]
> >
> > Robert
> >
> > []3;
> > https://github.com/apache/sling-org-apache-sling-servlets-get/compare/feature/SLING-7936
> >
> >
Re: [PROPOSAL] Moving integration tests closer to the code they test
Posted by Andrei Dulvac <du...@apache.org>.
+1 for moving the tests to the code as well.
However... I would be careful to check what is packaged in the `tests`
artifact produced by the build.
Should we package the test transient dependencies? Otherwise it will be
complicated to run multiple test artifacts under one build, with different
tests having different dependencies. e.g. tests with dep to a different
version of the sling-testing-rules. And speaking of running multiple tests
together in a different job, one big concern (if we decide to do some
magic) is classpath shadowing. I've seen the following happen:
A maven project that
1. Collected multiple tests modules
2. Collected their dependencies
3. Ran failsafe on that dir, sharing the classpath
Sounds ok, but it isn't. because of classpath shadowing.
So because of complications like these, I'd second Valentin's slight
concern and make them a different maven module and put the tests in
src/main. Arguably makes it a bit easier to manage what's packaged inside
and how they're ran together?
WDYT?
- Andrei
On Fri, Nov 9, 2018 at 2:28 PM Robert Munteanu <ro...@apache.org> wrote:
> On Fri, 2018-11-09 at 13:50 +0100, Georg Henzler wrote:
> > +1 to move the tests closer to the code.
> >
> > Regarding the path src/test/it (mentioned in the wiki page):
> > Shouldn't
> > we rather stick with src/test/java for both ITs and regular tests
> > and
> > just use the standard name conventions as configured by default for
> > plugins surefire/failsafe maven plugins, that is *Test.java [1] for
> > unit
> > tests and *IT.java [2] for integration tests?
>
>
> Yes, you're right.
>
> I'll update the wiki page. And also this is what I'm doing in my POC,
> see [3]
>
> Robert
>
> []3;
> https://github.com/apache/sling-org-apache-sling-servlets-get/compare/feature/SLING-7936
>
>
Re: [PROPOSAL] Moving integration tests closer to the code they test
Posted by Robert Munteanu <ro...@apache.org>.
On Fri, 2018-11-09 at 13:50 +0100, Georg Henzler wrote:
> +1 to move the tests closer to the code.
>
> Regarding the path src/test/it (mentioned in the wiki page):
> Shouldn't
> we rather stick with src/test/java for both ITs and regular tests
> and
> just use the standard name conventions as configured by default for
> plugins surefire/failsafe maven plugins, that is *Test.java [1] for
> unit
> tests and *IT.java [2] for integration tests?
Yes, you're right.
I'll update the wiki page. And also this is what I'm doing in my POC,
see [3]
Robert
[]3; https://github.com/apache/sling-org-apache-sling-servlets-get/compare/feature/SLING-7936
Re: [PROPOSAL] Moving integration tests closer to the code they test
Posted by Georg Henzler <sl...@ghenzler.de>.
+1 to move the tests closer to the code.
Regarding the path src/test/it (mentioned in the wiki page): Shouldn't
we rather stick with src/test/java for both ITs and regular tests and
just use the standard name conventions as configured by default for
plugins surefire/failsafe maven plugins, that is *Test.java [1] for unit
tests and *IT.java [2] for integration tests?
- Georg
[1]
https://maven.apache.org/surefire/maven-surefire-plugin/examples/inclusion-exclusion.html
[2]
https://maven.apache.org/surefire/maven-failsafe-plugin/examples/inclusion-exclusion.html
On 2018-11-07 17:29, Robert Munteanu wrote:
> Hi,
>
> We discussed a couple of time that it would be desireable to move tests
> closer to the code they're testing.
>
> I wrote a proposal about how this could look like in terms of results
>
> https://cwiki.apache.org/confluence/display/SLING/Moving+integration+tests+closer+to+the+code+they+test
>
> I purposefully avoided looking into implementation details to avoid
> doing too much work for nothing.
>
> What do you think about this possibility?
>
> Thanks,
>
> Robert