You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sis.apache.org by Martin Desruisseaux <ma...@geomatys.fr> on 2012/09/20 11:10:07 UTC
Testing framework: JUnit vs TestNG
Hello all
SIS currently uses JUnit for testing. Myself I have always used JUnit
too for all my projects. However during the last OGC meeting, I had a
talk with some peoples involved in the OGC CITE tests and we considered
the possibility to share some effort between CITE tests and GeoAPI
conformance module. However the CITE team has chosen TestNG, and they
suggested me to switch to TestNG too for easier integration.
Switching to TestNG may be a significant effort since we would need to
port not only the existing SIS tests, but also the GeoAPI conformance
module and the Geotk tests ported to SIS. As far as GeoAPI-conformance
is concerned, we may also lost some potential users since TestNG can run
JUnit tests through an adapter, but I have not hear about the converse.
However the advantage would be potentially easier cooperation with CITE
tests, and TestNG is said to be a more powerful framework (I didn't
evaluated myself). The capability to specify tests dependencies seems
attractive, and the fact that TestNG doesn't recreate new "FooTest"
instance for every tests may speed-up when the constructor performs
heavy initialization.
What do peoples think?
Martin
Re: Testing framework: JUnit vs TestNG
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Martin,
+1 to leveraging assumes which is under ALv2 -- the JExample thing seems like it's inactive, so
I'm not sure we should depend on it unless we want to take on the burden of maintaining it.
+1 to stick with JUnit too...
Cheers,
Chris
On Sep 28, 2012, at 10:35 PM, Martin Desruisseaux wrote:
> I still have a lot of hesitations about whatever we should migrate to TestNG or not... I had a look at the TestNG API and got the (admittedly superficial) impression of an API slightly less structured than the JUnit one. Keeping in mind that the choice could impact the GeoAPI conformance module, staying with JUnit may increase the users base since TestNG can execute JUnit tests, while the converse is not true. It would also avoid disturbing geoapi-conformance users (I have no idea if there is many). Furthermore JUnit is bundled in the Android API (but only JUnit 3, not 4), which may be a platform to explore.
>
> Some arguments in the past in favour of TestNG was parametrized tests. But JUnit 4 now have them, and indeed GeoAPI-conformance uses them a lot. An other argument was test grouping, but JUnit 4.10 has an experimental "Category" feature which seems similar (I didn't tried it yet).
>
> My main grief against the current JUnit is the lack of test dependencies, which seems to be considered against the JUnit principle of test isolation. However I found this project which seems to add exactly that:
>
> https://github.com/junit-team/junit.contrib/tree/master/assumes
>
> This is only 2 small classes under Apache 2 license, so we could easily copy them in the SIS test classes.
>
> I also found this lovely JUnit extension, which seem yet more attractive than TestNG to me :-).
>
> http://scg.unibe.ch/research/jexample
> http://chem-bla-ics.blogspot.jp/2010/08/specifying-unit-test-dependencies-with.html
>
> JExample is under Common Public License Version 1.0 (CPL), which I think is the same one than JUnit itself. However this project seems inactive for about 3 years. For example it still built on top of JUnit 4.3, while the current version is 4.10.
>
> Any though?
>
> Martin
>
>
> Le 20/09/12 23:38, Mattmann, Chris A (388J) a écrit :
>> I'd say I'm more familiar with Junit ATM, so if we switched to TestNG I'd have to learn it.
>> Maybe we can do it, slowly, as it makes sense on a class by class basis, over time? Or
>> even a SIS module by module basis?
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Testing framework: JUnit vs TestNG
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
+1 this approach works for me!
Cheers,
Chris
On Sep 29, 2012, at 2:55 AM, Martin Desruisseaux wrote:
> Hello Peter
>
> Le 29/09/12 17:50, Peter K a écrit :
>> For tests I would go mainstream, so pure junit or pure testng. One
>> approach would be using testng for some modules and testng for others
>> like Chris suggested.
>> So e.g. in SIS use testng only where it is really necessary. I for
>> myself cannot live without unit tests but I'm comfortable with junit as
>> well as testng.
>
> Thanks for the feedback. So I'm tempted to stick with JUnit (without external extensions) and copies (with adaptations) the 2 Apache2-licensed classes from "junit-team/junit.contrib" under the sis-utility/src/test of Apache SIS. Those classes are about 200 lines of code all together, which seems reasonable to me. With the functionalities of a @Dependency annotation, I would have no more technical reasons to go for TestNG.
>
> Any objections?
>
> Martin
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Testing framework: JUnit vs TestNG
Posted by Martin Desruisseaux <ma...@geomatys.fr>.
Hello Peter
Le 29/09/12 17:50, Peter K a écrit :
> For tests I would go mainstream, so pure junit or pure testng. One
> approach would be using testng for some modules and testng for others
> like Chris suggested.
> So e.g. in SIS use testng only where it is really necessary. I for
> myself cannot live without unit tests but I'm comfortable with junit as
> well as testng.
Thanks for the feedback. So I'm tempted to stick with JUnit (without
external extensions) and copies (with adaptations) the 2
Apache2-licensed classes from "junit-team/junit.contrib" under the
sis-utility/src/test of Apache SIS. Those classes are about 200 lines of
code all together, which seems reasonable to me. With the
functionalities of a @Dependency annotation, I would have no more
technical reasons to go for TestNG.
Any objections?
Martin
Re: Testing framework: JUnit vs TestNG
Posted by Peter K <pe...@yahoo.de>.
Hi Martin,
> I still have a lot of hesitations about whatever we should migrate to
> TestNG or not... I had a look at the TestNG API and got the
> (admittedly superficial) impression of an API slightly less structured
> than the JUnit one. Keeping in mind that the choice could impact the
> GeoAPI conformance module, staying with JUnit may increase the users
> base since TestNG can execute JUnit tests, while the converse is not
> true. It would also avoid disturbing geoapi-conformance users (I have
> no idea if there is many). Furthermore JUnit is bundled in the Android
> API (but only JUnit 3, not 4), which may be a platform to explore.
mavenized android projects do not have this problem, so no argument ;)
> Any though?
I definitely wouldn't go with additional dependencies! Because I doubt that
1. the development is active (or stays active as one wants it) -> what
if they have a bug?
2. there are jars in some maven repository -> very important for easy
development setup!!
For tests I would go mainstream, so pure junit or pure testng. One
approach would be using testng for some modules and testng for others
like Chris suggested.
So e.g. in SIS use testng only where it is really necessary. I for
myself cannot live without unit tests but I'm comfortable with junit as
well as testng.
And at the end the most important point is not the test framework itself
or its features it is the presence and simplicity/readability of tests :) !
Regards,
Peter.
Re: Testing framework: JUnit vs TestNG
Posted by Martin Desruisseaux <ma...@geomatys.fr>.
I still have a lot of hesitations about whatever we should migrate to
TestNG or not... I had a look at the TestNG API and got the (admittedly
superficial) impression of an API slightly less structured than the
JUnit one. Keeping in mind that the choice could impact the GeoAPI
conformance module, staying with JUnit may increase the users base since
TestNG can execute JUnit tests, while the converse is not true. It would
also avoid disturbing geoapi-conformance users (I have no idea if there
is many). Furthermore JUnit is bundled in the Android API (but only
JUnit 3, not 4), which may be a platform to explore.
Some arguments in the past in favour of TestNG was parametrized tests.
But JUnit 4 now have them, and indeed GeoAPI-conformance uses them a
lot. An other argument was test grouping, but JUnit 4.10 has an
experimental "Category" feature which seems similar (I didn't tried it yet).
My main grief against the current JUnit is the lack of test
dependencies, which seems to be considered against the JUnit principle
of test isolation. However I found this project which seems to add
exactly that:
https://github.com/junit-team/junit.contrib/tree/master/assumes
This is only 2 small classes under Apache 2 license, so we could easily
copy them in the SIS test classes.
I also found this lovely JUnit extension, which seem yet more attractive
than TestNG to me :-).
http://scg.unibe.ch/research/jexample
http://chem-bla-ics.blogspot.jp/2010/08/specifying-unit-test-dependencies-with.html
JExample is under Common Public License Version 1.0 (CPL), which I think
is the same one than JUnit itself. However this project seems inactive
for about 3 years. For example it still built on top of JUnit 4.3, while
the current version is 4.10.
Any though?
Martin
Le 20/09/12 23:38, Mattmann, Chris A (388J) a écrit :
> I'd say I'm more familiar with Junit ATM, so if we switched to TestNG I'd have to learn it.
> Maybe we can do it, slowly, as it makes sense on a class by class basis, over time? Or
> even a SIS module by module basis?
Re: Testing framework: JUnit vs TestNG
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Guys,
I'd say I'm more familiar with Junit ATM, so if we switched to TestNG I'd have to learn it.
Maybe we can do it, slowly, as it makes sense on a class by class basis, over time? Or
even a SIS module by module basis?
Thanks!
Cheers,
Chris
On Sep 20, 2012, at 4:10 AM, Peter K wrote:
> Hi Martin,
>
>>> BTW: Switching can be done nearly automatically via an eclipse plugin:
>>> http://testng.org/doc/migrating.html
>>
>> That would probably be useful for a large amount of tests. But
>> GeoAPI-conformance makes extensive use of parametrized tests (since
>> the implementation must be specified) and listeners; I would be
>> impressed if the plugin was sophisticated enough for converting such
>> tests automatically.
>
> ok, I understand
>
>>>> The capability to specify tests dependencies seems attractive
>>> you can do so for junit too or what do you mean here?
>>
>> JUnit provides TestSuite, but I was hopping for a simpler annotation
>> like:
>>
>> class TestA {...}
>>
>> @Depends(TestA.class)
>> class TestB {...}
>
> ah, I see. I don't think there is something similar (only a limited
> functionality via subclassing tests)
>
>>>> , and the fact that TestNG doesn't recreate new "FooTest" instance for
>>>> every tests may speed-up when the constructor performs heavy
>>>> initialization.
>>> Constructors shouldn't do that ;)
>>
>> I should have mentioned methods annotated by @Before instead. JUnit
>> has @BeforeClass for initializing resources only once, but it force us
>> to store the resources in static fields, which may cause the JVM to
>> retain them longer then needed (while I admit we can set the static
>> fields to null explicitly in an @AfterClass method).
>
> Yes, static is ugly. Also one wouldn't be able to parallelize the tests
> at some point in the future
> ... now I'm unsure what to choose ;) !
>
> Regards,
> Peter.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Testing framework: JUnit vs TestNG
Posted by Peter K <pe...@yahoo.de>.
Hi Martin,
>> BTW: Switching can be done nearly automatically via an eclipse plugin:
>> http://testng.org/doc/migrating.html
>
> That would probably be useful for a large amount of tests. But
> GeoAPI-conformance makes extensive use of parametrized tests (since
> the implementation must be specified) and listeners; I would be
> impressed if the plugin was sophisticated enough for converting such
> tests automatically.
ok, I understand
>>> The capability to specify tests dependencies seems attractive
>> you can do so for junit too or what do you mean here?
>
> JUnit provides TestSuite, but I was hopping for a simpler annotation
> like:
>
> class TestA {...}
>
> @Depends(TestA.class)
> class TestB {...}
ah, I see. I don't think there is something similar (only a limited
functionality via subclassing tests)
>>> , and the fact that TestNG doesn't recreate new "FooTest" instance for
>>> every tests may speed-up when the constructor performs heavy
>>> initialization.
>> Constructors shouldn't do that ;)
>
> I should have mentioned methods annotated by @Before instead. JUnit
> has @BeforeClass for initializing resources only once, but it force us
> to store the resources in static fields, which may cause the JVM to
> retain them longer then needed (while I admit we can set the static
> fields to null explicitly in an @AfterClass method).
Yes, static is ugly. Also one wouldn't be able to parallelize the tests
at some point in the future
... now I'm unsure what to choose ;) !
Regards,
Peter.
Re: Testing framework: JUnit vs TestNG
Posted by Martin Desruisseaux <ma...@geomatys.fr>.
Hello Peter
Le 20/09/12 18:32, Peter K a écrit :
> although I find TestNG to be a better tool I don't think that switching
> makes sense (now).
> I would go making the Geo* integration, adding tests (! which is more
> important then everything else IMO :)),
> making a first release and then probably switch if it is really worth
> the effort
> (e.g. junit is still more wide spread yet and has better tool support
> yet. Probably in some months this gets better for TestNG **).
This is fine for me, thanks for the feedback.
> BTW: Switching can be done nearly automatically via an eclipse plugin:
> http://testng.org/doc/migrating.html
That would probably be useful for a large amount of tests. But
GeoAPI-conformance makes extensive use of parametrized tests (since the
implementation must be specified) and listeners; I would be impressed if
the plugin was sophisticated enough for converting such tests automatically.
>> The capability to specify tests dependencies seems attractive
> you can do so for junit too or what do you mean here?
JUnit provides TestSuite, but I was hopping for a simpler annotation like:
class TestA {...}
@Depends(TestA.class)
class TestB {...}
or maybe I missed an equivalent functionality in JUnit?
>> , and the fact that TestNG doesn't recreate new "FooTest" instance for
>> every tests may speed-up when the constructor performs heavy
>> initialization.
> Constructors shouldn't do that ;)
I should have mentioned methods annotated by @Before instead. JUnit has
@BeforeClass for initializing resources only once, but it force us to
store the resources in static fields, which may cause the JVM to retain
them longer then needed (while I admit we can set the static fields to
null explicitly in an @AfterClass method).
Regards,
Martin
Re: Testing framework: JUnit vs TestNG
Posted by Peter K <pe...@yahoo.de>.
Hi Martin,
although I find TestNG to be a better tool I don't think that switching
makes sense (now).
I would go making the Geo* integration, adding tests (! which is more
important then everything else IMO :)),
making a first release and then probably switch if it is really worth
the effort
(e.g. junit is still more wide spread yet and has better tool support
yet. Probably in some months this gets better for TestNG **).
BTW: Switching can be done nearly automatically via an eclipse plugin:
http://testng.org/doc/migrating.html
> The capability to specify tests dependencies seems attractive
you can do so for junit too or what do you mean here?
> , and the fact that TestNG doesn't recreate new "FooTest" instance for
every tests may speed-up when the constructor performs heavy
initialization.
Constructors shouldn't do that ;)
Regards,
Peter.
**
e.g. there was no TestNG integration in Netbeans which is now there, but
still misses some important features
http://wiki.netbeans.org/TestNG
> Hello all
>
> SIS currently uses JUnit for testing. Myself I have always used JUnit
> too for all my projects. However during the last OGC meeting, I had a
> talk with some peoples involved in the OGC CITE tests and we
> considered the possibility to share some effort between CITE tests and
> GeoAPI conformance module. However the CITE team has chosen TestNG,
> and they suggested me to switch to TestNG too for easier integration.
>
> Switching to TestNG may be a significant effort since we would need to
> port not only the existing SIS tests, but also the GeoAPI conformance
> module and the Geotk tests ported to SIS. As far as GeoAPI-conformance
> is concerned, we may also lost some potential users since TestNG can
> run JUnit tests through an adapter, but I have not hear about the
> converse.
>
> However the advantage would be potentially easier cooperation with
> CITE tests, and TestNG is said to be a more powerful framework (I
> didn't evaluated myself). The capability to specify tests dependencies
> seems attractive, and the fact that TestNG doesn't recreate new
> "FooTest" instance for every tests may speed-up when the constructor
> performs heavy initialization.
>
> What do peoples think?
>
> Martin