You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@river.apache.org by Zsolt Kúti <la...@gmail.com> on 2010/11/07 11:54:12 UTC
test framework migration - was: Re: Hudson build is back to normal
On Sat, 6 Nov 2010 17:40:19 +0100
Jonathan Costers <jo...@googlemail.com> wrote:
...
> Not all of them use/need a multi VM setup. Those are candidates for
> JUnit. The others would be QA candidates.
> I'm not saying it is easy to migrate any of these though, doing so
> requires knowledge of how the jtreg framework operates, as well as
> the proposed target framework (JUnit, QA).
>
>
> >
> > JUnit's good when we're only testing a single object
> > implementation, we can document and expect people to utilse the qa
> > suite for more complex tests.
>
> Agreed.
Hello hard workers,
It would be worth considering the use of TestNG instead of JUnit.
I have no experience in their comparison, so relied on other
sources when I was to decid what framework to use (like this:
http://www.mkyong.com/unittest/junit-4-vs-testng-comparison/).
TestNG features that are missing from JUnit can be useful in a complex
test environment like that of River.
Zsolt
Re: test framework migration
Posted by Zsolt Kúti <la...@gmail.com>.
On Mon, 15 Nov 2010 12:38:06 -0500
"Christopher Dolan" <ch...@avid.com> wrote:
> While useful, I recommend against that feature in general.
> Philosophically, it's bad practice to connect the unit tests -- they
> should be as simple as possible and stand alone. Technologically,
> when the first test fails, then the second one fails with a really
> obscure error. I can't remember the specifics, but it was bad enough
> to convince me to rewrite all of my "dependsOnMethod" tests.
I agree in general :-)
In some cases test setup can take long or a resource cannot be reused
quickly enough and (test) class level setup is in order. For the former
doing an exhaustive testing of database functionality can be an
example. I also had a case involving extended XML trees for which
initializations would have been cumbersome and would have made more
hardly followable code. (Admittedly it is still a trade-off considering
the brittleness you mention.) For the latter I have only a gut feeling
that some use of the resources by River might fall into that category.
Zsolt
Ps.: It should be noted that TestNG runs test cases parallely by
default.
RE: test framework migration
Posted by Christopher Dolan <ch...@avid.com>.
While useful, I recommend against that feature in general. Philosophically, it's bad practice to connect the unit tests -- they should be as simple as possible and stand alone. Technologically, when the first test fails, then the second one fails with a really obscure error. I can't remember the specifics, but it was bad enough to convince me to rewrite all of my "dependsOnMethod" tests.
Chris
-----Original Message-----
From: Zsolt Kúti [mailto:la.tinca@gmail.com]
Sent: Saturday, November 13, 2010 7:29 AM
To: river-dev@incubator.apache.org
Subject: Re: test framework migration
On Fri, 12 Nov 2010 13:16:58 -0500
"Christopher Dolan" <ch...@avid.com> wrote:
> One more thing: the annotations are named similarly, but they are NOT
> shared. It's @org.testng.Test vs. @org.junit.Test, for example. The
> similarity makes it easier to change the source code via
> s/junit/testng/g but it's not zero-effort.
> Chris
A further thing I found useful: test methods can be
ordered/made dependent.
@Test
public void m1() {...}
@Test(dependsOnMethods="m1")
public void m2() {...}
Zsolt
Re: test framework migration
Posted by Zsolt Kúti <la...@gmail.com>.
On Fri, 12 Nov 2010 13:16:58 -0500
"Christopher Dolan" <ch...@avid.com> wrote:
> One more thing: the annotations are named similarly, but they are NOT
> shared. It's @org.testng.Test vs. @org.junit.Test, for example. The
> similarity makes it easier to change the source code via
> s/junit/testng/g but it's not zero-effort.
> Chris
A further thing I found useful: test methods can be
ordered/made dependent.
@Test
public void m1() {...}
@Test(dependsOnMethods="m1")
public void m2() {...}
Zsolt
RE: test framework migration - was: Re: Hudson build is back to normal
Posted by Christopher Dolan <ch...@avid.com>.
One more thing: the annotations are named similarly, but they are NOT
shared. It's @org.testng.Test vs. @org.junit.Test, for example. The
similarity makes it easier to change the source code via
s/junit/testng/g but it's not zero-effort.
Chris
-----Original Message-----
From: Peter Firmstone [mailto:jini@zeus.net.au]
Sent: Thursday, November 11, 2010 7:04 PM
To: river-dev@incubator.apache.org
Subject: Re: test framework migration - was: Re: Hudson build is back to
normal
[...snip...]
Has anyone else written any junit tests other than myself? If TestNG is
justifiably better than Junit, I'd be prepared to convert my tests, I
believe many of the annotations are common?
[...snip...]
RE: test framework migration - was: Re: Hudson build is back to normal
Posted by Christopher Dolan <ch...@avid.com>.
Last year my project switched all of our tests from Junit 3 to TestNG. My big win with TestNG vs. Junit 4 was the ability to classify the tests and decide at runtime which to run. Most of my tests are annotated with @Test(groups="build") which are the tests executed by my continuous integration process. Only a subset of my code is Mac-specific, so those tests are marked as @Test(groups={"build","macos"}). My really slow tests are marked as @Test(groups="all") and are only executed on specific machines.
>From Ant, I choose which groups to run via <testng groups="build" ... /> or <testng groups="macos" ... /> etc.
TestNG also has an @DataProvider notation that dramatically simplifies groups of tests that have the exact same code but different initial conditions. Here's a too-simple example just to show the technique:
@DataProvider(name="testcases")
public Object[][] getTestcases() {
return new Object[][] {
{ 1, "1" },
{ -0, "0" },
};
}
@Test(groups="build", dataProvider="testcases")
public void testNumification(int num, String expected) {
Assert.assertEquals("" + num, expected);
}
Another useful annotation regards exceptions, which says to fail the test unless an exception class (or subclass) is thrown
@Test(groups="build", expectedException=IllegalArgumentException.class)
public void testParseIntFailure() {
Integer.parseInt("not a number");
}
TestNG also lets you throw org.testng.SkipException at runtime to neither pass nor fail the test. For example, if I want to run a test on Win7 but skip it on WinXP.
But, I agree with Peter that it has to be all TestNG or all JUnit. You can make them play nice, but it's not worth the hassle. The conversion was a little tedious (do NOT trust the auto-convert tools! Convert by hand instead!) but bearable. TestNG has their own org.testng.Assert that has a different API from org.junit.Assert, but both throw AssertionError in the end so it doesn't actually matter which you use. I like TestNG's API a little better (methods arguments are ordered "got, expected, message" vs. Junit's "message, expected, got").
Chris
-----Original Message-----
From: Peter Firmstone [mailto:jini@zeus.net.au]
Sent: Friday, November 12, 2010 4:06 AM
To: river-dev@incubator.apache.org
Subject: Re: test framework migration - was: Re: Hudson build is back to normal
Thanks Tom, I agree, if someone has a good reason...
Cheers,
Peter.
Tom Hobbs wrote:
> I've written a couple of JUnit tests.
>
> I've not used TestNG for many years so can't really comment on it's
> comparison with JUnit. I don't object to converting my tests, however
> I'd rather convert them because "TestNG does X which we need to do Y"
> rather than "We should swap to TestNG because it's better."
>
> I agree, this should be an either/or decision. We should definitely
> *not* be using both.
>
> On Fri, Nov 12, 2010 at 1:04 AM, Peter Firmstone <ji...@zeus.net.au> wrote:
>
>> Patricia Shanahan wrote:
>>
>>> Zsolt Kúti wrote:
>>>
>>>> On Sat, 6 Nov 2010 17:40:19 +0100
>>>> Jonathan Costers <jo...@googlemail.com> wrote:
>>>>
>>>> ...
>>>>
>>>>> Not all of them use/need a multi VM setup. Those are candidates for
>>>>> JUnit. The others would be QA candidates.
>>>>> I'm not saying it is easy to migrate any of these though, doing so
>>>>> requires knowledge of how the jtreg framework operates, as well as
>>>>> the proposed target framework (JUnit, QA).
>>>>>
>>>>>
>>>>>
>>>>>> JUnit's good when we're only testing a single object
>>>>>> implementation, we can document and expect people to utilse the qa
>>>>>> suite for more complex tests.
>>>>>>
>>>>> Agreed.
>>>>>
>>>> Hello hard workers,
>>>>
>>>> It would be worth considering the use of TestNG instead of JUnit.
>>>> I have no experience in their comparison, so relied on other
>>>> sources when I was to decid what framework to use (like this:
>>>> http://www.mkyong.com/unittest/junit-4-vs-testng-comparison/).
>>>> TestNG features that are missing from JUnit can be useful in a complex
>>>> test environment like that of River.
>>>>
>>> If we were starting cold, with no existing tests, I might be open to this
>>> suggestion. As it is, we already have a QA framework that can do all the
>>> complex, multi-JVM tests, and we have over 1000 existing tests using it.
>>>
>>> I think the objective in converting jtreg tests would be to reduce the
>>> number of frameworks, and the amount of software we need installed, in order
>>> to run a full test. Switching them to TestNG, or anything else other than
>>> JUnit or the River QA framework, would not achieve that.
>>>
>>> Patricia
>>>
>>>
>> Has anyone else written any junit tests other than myself? If TestNG is
>> justifiably better than Junit, I'd be prepared to convert my tests, I
>> believe many of the annotations are common?
>>
>> But it would have to be Junit OR TestNG, not both.
>>
>> Cheers,
>>
>> Peter.
>>
>>
>
>
Re: test framework migration - was: Re: Hudson build is back to normal
Posted by Peter Firmstone <ji...@zeus.net.au>.
Thanks Tom, I agree, if someone has a good reason...
Cheers,
Peter.
Tom Hobbs wrote:
> I've written a couple of JUnit tests.
>
> I've not used TestNG for many years so can't really comment on it's
> comparison with JUnit. I don't object to converting my tests, however
> I'd rather convert them because "TestNG does X which we need to do Y"
> rather than "We should swap to TestNG because it's better."
>
> I agree, this should be an either/or decision. We should definitely
> *not* be using both.
>
> On Fri, Nov 12, 2010 at 1:04 AM, Peter Firmstone <ji...@zeus.net.au> wrote:
>
>> Patricia Shanahan wrote:
>>
>>> Zsolt Kúti wrote:
>>>
>>>> On Sat, 6 Nov 2010 17:40:19 +0100
>>>> Jonathan Costers <jo...@googlemail.com> wrote:
>>>>
>>>> ...
>>>>
>>>>> Not all of them use/need a multi VM setup. Those are candidates for
>>>>> JUnit. The others would be QA candidates.
>>>>> I'm not saying it is easy to migrate any of these though, doing so
>>>>> requires knowledge of how the jtreg framework operates, as well as
>>>>> the proposed target framework (JUnit, QA).
>>>>>
>>>>>
>>>>>
>>>>>> JUnit's good when we're only testing a single object
>>>>>> implementation, we can document and expect people to utilse the qa
>>>>>> suite for more complex tests.
>>>>>>
>>>>> Agreed.
>>>>>
>>>> Hello hard workers,
>>>>
>>>> It would be worth considering the use of TestNG instead of JUnit.
>>>> I have no experience in their comparison, so relied on other
>>>> sources when I was to decid what framework to use (like this:
>>>> http://www.mkyong.com/unittest/junit-4-vs-testng-comparison/).
>>>> TestNG features that are missing from JUnit can be useful in a complex
>>>> test environment like that of River.
>>>>
>>> If we were starting cold, with no existing tests, I might be open to this
>>> suggestion. As it is, we already have a QA framework that can do all the
>>> complex, multi-JVM tests, and we have over 1000 existing tests using it.
>>>
>>> I think the objective in converting jtreg tests would be to reduce the
>>> number of frameworks, and the amount of software we need installed, in order
>>> to run a full test. Switching them to TestNG, or anything else other than
>>> JUnit or the River QA framework, would not achieve that.
>>>
>>> Patricia
>>>
>>>
>> Has anyone else written any junit tests other than myself? If TestNG is
>> justifiably better than Junit, I'd be prepared to convert my tests, I
>> believe many of the annotations are common?
>>
>> But it would have to be Junit OR TestNG, not both.
>>
>> Cheers,
>>
>> Peter.
>>
>>
>
>
Re: test framework migration - was: Re: Hudson build is back to normal
Posted by Tom Hobbs <tv...@googlemail.com>.
I've written a couple of JUnit tests.
I've not used TestNG for many years so can't really comment on it's
comparison with JUnit. I don't object to converting my tests, however
I'd rather convert them because "TestNG does X which we need to do Y"
rather than "We should swap to TestNG because it's better."
I agree, this should be an either/or decision. We should definitely
*not* be using both.
On Fri, Nov 12, 2010 at 1:04 AM, Peter Firmstone <ji...@zeus.net.au> wrote:
> Patricia Shanahan wrote:
>>
>> Zsolt Kúti wrote:
>>>
>>> On Sat, 6 Nov 2010 17:40:19 +0100
>>> Jonathan Costers <jo...@googlemail.com> wrote:
>>>
>>> ...
>>>>
>>>> Not all of them use/need a multi VM setup. Those are candidates for
>>>> JUnit. The others would be QA candidates.
>>>> I'm not saying it is easy to migrate any of these though, doing so
>>>> requires knowledge of how the jtreg framework operates, as well as
>>>> the proposed target framework (JUnit, QA).
>>>>
>>>>
>>>>> JUnit's good when we're only testing a single object
>>>>> implementation, we can document and expect people to utilse the qa
>>>>> suite for more complex tests.
>>>>
>>>> Agreed.
>>>
>>> Hello hard workers,
>>>
>>> It would be worth considering the use of TestNG instead of JUnit.
>>> I have no experience in their comparison, so relied on other
>>> sources when I was to decid what framework to use (like this:
>>> http://www.mkyong.com/unittest/junit-4-vs-testng-comparison/).
>>> TestNG features that are missing from JUnit can be useful in a complex
>>> test environment like that of River.
>>
>> If we were starting cold, with no existing tests, I might be open to this
>> suggestion. As it is, we already have a QA framework that can do all the
>> complex, multi-JVM tests, and we have over 1000 existing tests using it.
>>
>> I think the objective in converting jtreg tests would be to reduce the
>> number of frameworks, and the amount of software we need installed, in order
>> to run a full test. Switching them to TestNG, or anything else other than
>> JUnit or the River QA framework, would not achieve that.
>>
>> Patricia
>>
> Has anyone else written any junit tests other than myself? If TestNG is
> justifiably better than Junit, I'd be prepared to convert my tests, I
> believe many of the annotations are common?
>
> But it would have to be Junit OR TestNG, not both.
>
> Cheers,
>
> Peter.
>
Re: test framework migration - was: Re: Hudson build is back to normal
Posted by Peter Firmstone <ji...@zeus.net.au>.
Patricia Shanahan wrote:
> Zsolt Kúti wrote:
>> On Sat, 6 Nov 2010 17:40:19 +0100
>> Jonathan Costers <jo...@googlemail.com> wrote:
>>
>> ...
>>> Not all of them use/need a multi VM setup. Those are candidates for
>>> JUnit. The others would be QA candidates.
>>> I'm not saying it is easy to migrate any of these though, doing so
>>> requires knowledge of how the jtreg framework operates, as well as
>>> the proposed target framework (JUnit, QA).
>>>
>>>
>>>> JUnit's good when we're only testing a single object
>>>> implementation, we can document and expect people to utilse the qa
>>>> suite for more complex tests.
>>> Agreed.
>>
>> Hello hard workers,
>>
>> It would be worth considering the use of TestNG instead of JUnit.
>> I have no experience in their comparison, so relied on other
>> sources when I was to decid what framework to use (like this:
>> http://www.mkyong.com/unittest/junit-4-vs-testng-comparison/).
>> TestNG features that are missing from JUnit can be useful in a complex
>> test environment like that of River.
>
> If we were starting cold, with no existing tests, I might be open to
> this suggestion. As it is, we already have a QA framework that can do
> all the complex, multi-JVM tests, and we have over 1000 existing tests
> using it.
>
> I think the objective in converting jtreg tests would be to reduce the
> number of frameworks, and the amount of software we need installed, in
> order to run a full test. Switching them to TestNG, or anything else
> other than JUnit or the River QA framework, would not achieve that.
>
> Patricia
>
Has anyone else written any junit tests other than myself? If TestNG is
justifiably better than Junit, I'd be prepared to convert my tests, I
believe many of the annotations are common?
But it would have to be Junit OR TestNG, not both.
Cheers,
Peter.
Re: test framework migration - was: Re: Hudson build is back to normal
Posted by Patricia Shanahan <pa...@acm.org>.
Zsolt Kúti wrote:
> On Sat, 6 Nov 2010 17:40:19 +0100
> Jonathan Costers <jo...@googlemail.com> wrote:
>
> ...
>> Not all of them use/need a multi VM setup. Those are candidates for
>> JUnit. The others would be QA candidates.
>> I'm not saying it is easy to migrate any of these though, doing so
>> requires knowledge of how the jtreg framework operates, as well as
>> the proposed target framework (JUnit, QA).
>>
>>
>>> JUnit's good when we're only testing a single object
>>> implementation, we can document and expect people to utilse the qa
>>> suite for more complex tests.
>> Agreed.
>
> Hello hard workers,
>
> It would be worth considering the use of TestNG instead of JUnit.
> I have no experience in their comparison, so relied on other
> sources when I was to decid what framework to use (like this:
> http://www.mkyong.com/unittest/junit-4-vs-testng-comparison/).
> TestNG features that are missing from JUnit can be useful in a complex
> test environment like that of River.
If we were starting cold, with no existing tests, I might be open to
this suggestion. As it is, we already have a QA framework that can do
all the complex, multi-JVM tests, and we have over 1000 existing tests
using it.
I think the objective in converting jtreg tests would be to reduce the
number of frameworks, and the amount of software we need installed, in
order to run a full test. Switching them to TestNG, or anything else
other than JUnit or the River QA framework, would not achieve that.
Patricia