You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Sean Qiu <se...@gmail.com> on 2008/06/06 05:21:40 UTC
[classlib][test] Migration to testNG?
Hi, all.
We had discussed the migration to testNG before and got some conclusions for
grouping[1]
including how to deal with boot path test[2]. Am i missing something?
Is it still in our schedule? I think it's valueable to Harmony.
I volunteer to carry out this job if no one objects. Any other volunteers?
IMHO, we can only add some ant tasks to integrate testng at the beginning.
So our original junit tests can still work at the mean time when migrating.
When one module's migration task is finished, we can judge the result
to dertermine whether we should go on for other modules.
Maybe we can create a branch for luni to start this work, shall we?
therefore there won't be any impact on other's development.
Once it is completed in the branch, we could merge it back to our trunk.
Does it make sense?
Any sugestions or comments are welcomed. Thanks very much.
[1] http://wiki.apache.org/harmony/Testing_Convention
[2]
http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
[3] http://testng.org/doc/documentation-main.html#annotations
--
Best Regards
Sean, Xiao Xia Qiu
China Software Development Lab, IBM
Re: [classlib][test] Migration to testNG?
Posted by Regis <xu...@gmail.com>.
Hi,
It's interesting, I'm in.
I agree that we need a new branch to do this, and luni module may be the
most
complex module, i think, if we migrate luni successfully, other modules
will become
easier, but is it too hard to startup? I prefer to pick up a small and
simple module
at first step, for example prefs, if it's ok, we can go further and faster.
Best Regards,
Regis.
Sean Qiu wrote:
> Hi, all.
>
> We had discussed the migration to testNG before and got some conclusions for
> grouping[1]
> including how to deal with boot path test[2]. Am i missing something?
> Is it still in our schedule? I think it's valueable to Harmony.
> I volunteer to carry out this job if no one objects. Any other volunteers?
>
> IMHO, we can only add some ant tasks to integrate testng at the beginning.
> So our original junit tests can still work at the mean time when migrating.
> When one module's migration task is finished, we can judge the result
> to dertermine whether we should go on for other modules.
>
> Maybe we can create a branch for luni to start this work, shall we?
> therefore there won't be any impact on other's development.
> Once it is completed in the branch, we could merge it back to our trunk.
> Does it make sense?
>
> Any sugestions or comments are welcomed. Thanks very much.
>
> [1] http://wiki.apache.org/harmony/Testing_Convention
> [2]
> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
> [3] http://testng.org/doc/documentation-main.html#annotations
> --
> Best Regards
> Sean, Xiao Xia Qiu
> China Software Development Lab, IBM
>
Re: [classlib][test] Migration to testNG?
Posted by Sean Qiu <se...@gmail.com>.
By groups annotation, we can group them by arch.<arch id>, type.impl,
os.<os id>, vm.<vm id>.
These groups indicate that we have different behaviors but it is allowed.
Again, we can keep track of them without moving them to other
directories by TestNG.
2008/6/12 Nathan Beyer <nd...@apache.org>:
> I think having a mechanism for exclusions of failing tests is bad,
> especially any mechanism that allows excluding tests to sit hidden within
> working tests. I think tests that are excluded because they are failing and
> aren't going to be fixed need to be moved out of the running tests and
> dropped in an separate folder like "src/test/failing" so that they stand
> out.
>
> -Nathan
>
> On Wed, Jun 11, 2008 at 6:46 AM, Regis <xu...@gmail.com> wrote:
>
>>
>> Junit 4 has @ignore annotation which of target is method, which can ignore
>> failing methods.
>> But some modules, such as luni, have different exclude lists on different
>> os or vm, I haven't
>> found any ways to do this using @ignore annotation. While TestNG's @group
>> annotation
>> can do this easily, just define groups for different exclude lists.
>>
>> Best Regards,
>> Regis.
>>
>>
>> Alexei Zakharov wrote:
>>
>>> As far as I understand in spite of the fact there were no
>>> TestNG-related discussions since 2006 the problem is still relevant.
>>> There are big exclude lists in some classlib modules still, and many
>>> tests are excluded only because of a couple of failing methods.
>>> Frankly speaking I'm not familiar with new feature introduced in Junit
>>> 4.4. Are there any enhancements that can help to resolve this
>>> exclude-whole-class-because-of-one-bad-method issue?
>>>
>>> Thanks,
>>> Alexei
>>>
>>> 2008/6/11, Regis <xu...@gmail.com>:
>>>
>>>> Nathan Beyer wrote:
>>>>
>>>>> On Fri, Jun 6, 2008 at 1:08 AM, Regis <xu...@gmail.com> wrote:
>>>>>
>>>>>
>>>>> Hi,
>>>>>>
>>>>>> Matcher and Assumptions are great ideas! Thanks Nathan.
>>>>>> They would be very helpful for our new test cases. But I notice that
>>>>>> Junit 4.4 doesn't support group which is very important feature for
>>>>>> both old tests and new tests. We can partition our test suite and run
>>>>>> them separately. It's make our tests more flexible and configurable,
>>>>>> and it's the main reason we discuss to migrate to TestNG long time ago.
>>>>>>
>>>>>>
>>>>> Don't we partition our tests already? Isn't that what the 'api' and
>>>>> 'impl'
>>>>> folders are about?
>>>>>
>>>>> Yes, but it's not enough. We have discussed and created a wiki page[1]
>>>> about
>>>> how
>>>> to configuration and group harmony tests. The page is a little old, but I
>>>> think the problems
>>>> it tried to resolve still exist now. The partitions are not only include
>>>> 'api' and 'impl', but also
>>>> include partition of different os, architecture, partition of broken
>>>> tests
>>>> and level of tests.
>>>> folder structure or exclude files can't help in this complex situation,
>>>> so
>>>> we need some tools
>>>> to help us to deal with this, i think TestNG is suitable. If JUnit 4.4
>>>> can
>>>> do it, i will vote to JUnit,
>>>> update to a new version is always easier than switch to a new tool after
>>>> all.
>>>>
>>>> [1] http://wiki.apache.org/harmony/Testing_Convention
>>>>
>>>> Best Regards,
>>>> Regis.
>>>>
>>>>
>>>> -Nathan
>>>>>
>>>>>
>>>>>
>>>>> Best Regards,
>>>>>> Regis.
>>>>>>
>>>>>>
>>>>>> Nathan Beyer wrote:
>>>>>>
>>>>>>
>>>>>> That discussion was a very long time ago. Is there still value in
>>>>>>>
>>>>>> TestNG?
>>>>
>>>>> I'd prefer to move to JUnit 4.4. All of our current tests will
>>>>>>>
>>>>>> continue to
>>>>
>>>>> work and new tests can be implemented using the latest conventions and
>>>>>>> older
>>>>>>> tests can be updated as we get to them. JUnit 4.4 is a far cry from
>>>>>>>
>>>>>> 4.0.
>>>>
>>>>> Here's the things I think would be create for our use and testing in
>>>>>>> general
>>>>>>> - Matchers and the 'assertThat' - much more readable code and readable
>>>>>>> failure messages
>>>>>>> - Assumptions and the 'assumeThat' - allows methods to add statements
>>>>>>>
>>>>>> that
>>>>
>>>>> guarantee that preconditions for the test are correct; this allows
>>>>>>>
>>>>>> tests
>>>>
>>>>> to
>>>>>>> fail such that you know it's an environment issue and not an actual
>>>>>>>
>>>>>> test
>>>>
>>>>> failure
>>>>>>>
>>>>>>> If you're not familiar with matchers, check out this quick tutorial -
>>>>>>> http://code.google.com/p/hamcrest/wiki/Tutorial.
>>>>>>>
>>>>>>> -Nathan
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <se...@gmail.com>
>>>>>>>
>>>>>> wrote:
>>>>
>>>>> Hi, all.
>>>>>>>
>>>>>>> We had discussed the migration to testNG before and got some
>>>>>>>>
>>>>>>> conclusions
>>>>
>>>>> for
>>>>>>>> grouping[1]
>>>>>>>> including how to deal with boot path test[2]. Am i missing
>>>>>>>>
>>>>>>> something?
>>>>
>>>>> Is it still in our schedule? I think it's valueable to Harmony.
>>>>>>>> I volunteer to carry out this job if no one objects. Any other
>>>>>>>> volunteers?
>>>>>>>>
>>>>>>>> IMHO, we can only add some ant tasks to integrate testng at the
>>>>>>>> beginning.
>>>>>>>> So our original junit tests can still work at the mean time when
>>>>>>>> migrating.
>>>>>>>> When one module's migration task is finished, we can judge the
>>>>>>>>
>>>>>>> result
>>>>
>>>>> to dertermine whether we should go on for other modules.
>>>>>>>>
>>>>>>>> Maybe we can create a branch for luni to start this work, shall we?
>>>>>>>> therefore there won't be any impact on other's development.
>>>>>>>> Once it is completed in the branch, we could merge it back to our
>>>>>>>>
>>>>>>> trunk.
>>>>
>>>>> Does it make sense?
>>>>>>>>
>>>>>>>> Any sugestions or comments are welcomed. Thanks very much.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>>
>>>>>>> http://wiki.apache.org/harmony/Testing_Convention
>>>>
>>>>> [2]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
>>>>
>>>>> [3]
>>>>>>>>
>>>>>>> http://testng.org/doc/documentation-main.html#annotations
>>>>
>>>>> --
>>>>>>>> Best Regards
>>>>>>>> Sean, Xiao Xia Qiu
>>>>>>>> China Software Development Lab, IBM
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>
>>>
>
--
Best Regards
Sean, Xiao Xia Qiu
China Software Development Lab, IBM
Re: [classlib][test] Migration to testNG?
Posted by Nathan Beyer <nd...@apache.org>.
I think having a mechanism for exclusions of failing tests is bad,
especially any mechanism that allows excluding tests to sit hidden within
working tests. I think tests that are excluded because they are failing and
aren't going to be fixed need to be moved out of the running tests and
dropped in an separate folder like "src/test/failing" so that they stand
out.
-Nathan
On Wed, Jun 11, 2008 at 6:46 AM, Regis <xu...@gmail.com> wrote:
>
> Junit 4 has @ignore annotation which of target is method, which can ignore
> failing methods.
> But some modules, such as luni, have different exclude lists on different
> os or vm, I haven't
> found any ways to do this using @ignore annotation. While TestNG's @group
> annotation
> can do this easily, just define groups for different exclude lists.
>
> Best Regards,
> Regis.
>
>
> Alexei Zakharov wrote:
>
>> As far as I understand in spite of the fact there were no
>> TestNG-related discussions since 2006 the problem is still relevant.
>> There are big exclude lists in some classlib modules still, and many
>> tests are excluded only because of a couple of failing methods.
>> Frankly speaking I'm not familiar with new feature introduced in Junit
>> 4.4. Are there any enhancements that can help to resolve this
>> exclude-whole-class-because-of-one-bad-method issue?
>>
>> Thanks,
>> Alexei
>>
>> 2008/6/11, Regis <xu...@gmail.com>:
>>
>>> Nathan Beyer wrote:
>>>
>>>> On Fri, Jun 6, 2008 at 1:08 AM, Regis <xu...@gmail.com> wrote:
>>>>
>>>>
>>>> Hi,
>>>>>
>>>>> Matcher and Assumptions are great ideas! Thanks Nathan.
>>>>> They would be very helpful for our new test cases. But I notice that
>>>>> Junit 4.4 doesn't support group which is very important feature for
>>>>> both old tests and new tests. We can partition our test suite and run
>>>>> them separately. It's make our tests more flexible and configurable,
>>>>> and it's the main reason we discuss to migrate to TestNG long time ago.
>>>>>
>>>>>
>>>> Don't we partition our tests already? Isn't that what the 'api' and
>>>> 'impl'
>>>> folders are about?
>>>>
>>>> Yes, but it's not enough. We have discussed and created a wiki page[1]
>>> about
>>> how
>>> to configuration and group harmony tests. The page is a little old, but I
>>> think the problems
>>> it tried to resolve still exist now. The partitions are not only include
>>> 'api' and 'impl', but also
>>> include partition of different os, architecture, partition of broken
>>> tests
>>> and level of tests.
>>> folder structure or exclude files can't help in this complex situation,
>>> so
>>> we need some tools
>>> to help us to deal with this, i think TestNG is suitable. If JUnit 4.4
>>> can
>>> do it, i will vote to JUnit,
>>> update to a new version is always easier than switch to a new tool after
>>> all.
>>>
>>> [1] http://wiki.apache.org/harmony/Testing_Convention
>>>
>>> Best Regards,
>>> Regis.
>>>
>>>
>>> -Nathan
>>>>
>>>>
>>>>
>>>> Best Regards,
>>>>> Regis.
>>>>>
>>>>>
>>>>> Nathan Beyer wrote:
>>>>>
>>>>>
>>>>> That discussion was a very long time ago. Is there still value in
>>>>>>
>>>>> TestNG?
>>>
>>>> I'd prefer to move to JUnit 4.4. All of our current tests will
>>>>>>
>>>>> continue to
>>>
>>>> work and new tests can be implemented using the latest conventions and
>>>>>> older
>>>>>> tests can be updated as we get to them. JUnit 4.4 is a far cry from
>>>>>>
>>>>> 4.0.
>>>
>>>> Here's the things I think would be create for our use and testing in
>>>>>> general
>>>>>> - Matchers and the 'assertThat' - much more readable code and readable
>>>>>> failure messages
>>>>>> - Assumptions and the 'assumeThat' - allows methods to add statements
>>>>>>
>>>>> that
>>>
>>>> guarantee that preconditions for the test are correct; this allows
>>>>>>
>>>>> tests
>>>
>>>> to
>>>>>> fail such that you know it's an environment issue and not an actual
>>>>>>
>>>>> test
>>>
>>>> failure
>>>>>>
>>>>>> If you're not familiar with matchers, check out this quick tutorial -
>>>>>> http://code.google.com/p/hamcrest/wiki/Tutorial.
>>>>>>
>>>>>> -Nathan
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <se...@gmail.com>
>>>>>>
>>>>> wrote:
>>>
>>>> Hi, all.
>>>>>>
>>>>>> We had discussed the migration to testNG before and got some
>>>>>>>
>>>>>> conclusions
>>>
>>>> for
>>>>>>> grouping[1]
>>>>>>> including how to deal with boot path test[2]. Am i missing
>>>>>>>
>>>>>> something?
>>>
>>>> Is it still in our schedule? I think it's valueable to Harmony.
>>>>>>> I volunteer to carry out this job if no one objects. Any other
>>>>>>> volunteers?
>>>>>>>
>>>>>>> IMHO, we can only add some ant tasks to integrate testng at the
>>>>>>> beginning.
>>>>>>> So our original junit tests can still work at the mean time when
>>>>>>> migrating.
>>>>>>> When one module's migration task is finished, we can judge the
>>>>>>>
>>>>>> result
>>>
>>>> to dertermine whether we should go on for other modules.
>>>>>>>
>>>>>>> Maybe we can create a branch for luni to start this work, shall we?
>>>>>>> therefore there won't be any impact on other's development.
>>>>>>> Once it is completed in the branch, we could merge it back to our
>>>>>>>
>>>>>> trunk.
>>>
>>>> Does it make sense?
>>>>>>>
>>>>>>> Any sugestions or comments are welcomed. Thanks very much.
>>>>>>>
>>>>>>> [1]
>>>>>>>
>>>>>> http://wiki.apache.org/harmony/Testing_Convention
>>>
>>>> [2]
>>>>>>>
>>>>>>>
>>>>>>>
>>> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
>>>
>>>> [3]
>>>>>>>
>>>>>> http://testng.org/doc/documentation-main.html#annotations
>>>
>>>> --
>>>>>>> Best Regards
>>>>>>> Sean, Xiao Xia Qiu
>>>>>>> China Software Development Lab, IBM
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>
Re: [classlib][test] Migration to testNG?
Posted by Regis <xu...@gmail.com>.
Junit 4 has @ignore annotation which of target is method, which can
ignore failing methods.
But some modules, such as luni, have different exclude lists on
different os or vm, I haven't
found any ways to do this using @ignore annotation. While TestNG's
@group annotation
can do this easily, just define groups for different exclude lists.
Best Regards,
Regis.
Alexei Zakharov wrote:
> As far as I understand in spite of the fact there were no
> TestNG-related discussions since 2006 the problem is still relevant.
> There are big exclude lists in some classlib modules still, and many
> tests are excluded only because of a couple of failing methods.
> Frankly speaking I'm not familiar with new feature introduced in Junit
> 4.4. Are there any enhancements that can help to resolve this
> exclude-whole-class-because-of-one-bad-method issue?
>
> Thanks,
> Alexei
>
> 2008/6/11, Regis <xu...@gmail.com>:
>> Nathan Beyer wrote:
>>> On Fri, Jun 6, 2008 at 1:08 AM, Regis <xu...@gmail.com> wrote:
>>>
>>>
>>>> Hi,
>>>>
>>>> Matcher and Assumptions are great ideas! Thanks Nathan.
>>>> They would be very helpful for our new test cases. But I notice that
>>>> Junit 4.4 doesn't support group which is very important feature for
>>>> both old tests and new tests. We can partition our test suite and run
>>>> them separately. It's make our tests more flexible and configurable,
>>>> and it's the main reason we discuss to migrate to TestNG long time ago.
>>>>
>>>
>>> Don't we partition our tests already? Isn't that what the 'api' and 'impl'
>>> folders are about?
>>>
>> Yes, but it's not enough. We have discussed and created a wiki page[1] about
>> how
>> to configuration and group harmony tests. The page is a little old, but I
>> think the problems
>> it tried to resolve still exist now. The partitions are not only include
>> 'api' and 'impl', but also
>> include partition of different os, architecture, partition of broken tests
>> and level of tests.
>> folder structure or exclude files can't help in this complex situation, so
>> we need some tools
>> to help us to deal with this, i think TestNG is suitable. If JUnit 4.4 can
>> do it, i will vote to JUnit,
>> update to a new version is always easier than switch to a new tool after
>> all.
>>
>> [1] http://wiki.apache.org/harmony/Testing_Convention
>>
>> Best Regards,
>> Regis.
>>
>>
>>> -Nathan
>>>
>>>
>>>
>>>> Best Regards,
>>>> Regis.
>>>>
>>>>
>>>> Nathan Beyer wrote:
>>>>
>>>>
>>>>> That discussion was a very long time ago. Is there still value in
>> TestNG?
>>>>> I'd prefer to move to JUnit 4.4. All of our current tests will
>> continue to
>>>>> work and new tests can be implemented using the latest conventions and
>>>>> older
>>>>> tests can be updated as we get to them. JUnit 4.4 is a far cry from
>> 4.0.
>>>>> Here's the things I think would be create for our use and testing in
>>>>> general
>>>>> - Matchers and the 'assertThat' - much more readable code and readable
>>>>> failure messages
>>>>> - Assumptions and the 'assumeThat' - allows methods to add statements
>> that
>>>>> guarantee that preconditions for the test are correct; this allows
>> tests
>>>>> to
>>>>> fail such that you know it's an environment issue and not an actual
>> test
>>>>> failure
>>>>>
>>>>> If you're not familiar with matchers, check out this quick tutorial -
>>>>> http://code.google.com/p/hamcrest/wiki/Tutorial.
>>>>>
>>>>> -Nathan
>>>>>
>>>>>
>>>>> On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <se...@gmail.com>
>> wrote:
>>>>> Hi, all.
>>>>>
>>>>>> We had discussed the migration to testNG before and got some
>> conclusions
>>>>>> for
>>>>>> grouping[1]
>>>>>> including how to deal with boot path test[2]. Am i missing
>> something?
>>>>>> Is it still in our schedule? I think it's valueable to Harmony.
>>>>>> I volunteer to carry out this job if no one objects. Any other
>>>>>> volunteers?
>>>>>>
>>>>>> IMHO, we can only add some ant tasks to integrate testng at the
>>>>>> beginning.
>>>>>> So our original junit tests can still work at the mean time when
>>>>>> migrating.
>>>>>> When one module's migration task is finished, we can judge the
>> result
>>>>>> to dertermine whether we should go on for other modules.
>>>>>>
>>>>>> Maybe we can create a branch for luni to start this work, shall we?
>>>>>> therefore there won't be any impact on other's development.
>>>>>> Once it is completed in the branch, we could merge it back to our
>> trunk.
>>>>>> Does it make sense?
>>>>>>
>>>>>> Any sugestions or comments are welcomed. Thanks very much.
>>>>>>
>>>>>> [1]
>> http://wiki.apache.org/harmony/Testing_Convention
>>>>>> [2]
>>>>>>
>>>>>>
>> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
>>>>>> [3]
>> http://testng.org/doc/documentation-main.html#annotations
>>>>>> --
>>>>>> Best Regards
>>>>>> Sean, Xiao Xia Qiu
>>>>>> China Software Development Lab, IBM
>>>>>>
>>>>>>
>>>>>>
>>>
>
Re: [classlib][test] Migration to testNG?
Posted by Sean Qiu <se...@gmail.com>.
Group failed test by annotation doesn't mean we want to keep the issues hidden.
On the contrary, we want to keep track of them.
We can run all failed test to verify if it is fixed.
2008/6/12 Nathan Beyer <nb...@gmail.com>:
> With annotation-based tests, just remove the annotation and that method is
> no longer treated as a test. However, the same thing could be done today by
> just changing the name of the method to NOT start with 'test'. I don't see
> any approach being good though as it keeps the issues hidden; nothing makes
> them stand out.
>
> -Nathan
>
> On Wed, Jun 11, 2008 at 5:59 AM, Alexei Zakharov <al...@gmail.com>
> wrote:
>
>> As far as I understand in spite of the fact there were no
>> TestNG-related discussions since 2006 the problem is still relevant.
>> There are big exclude lists in some classlib modules still, and many
>> tests are excluded only because of a couple of failing methods.
>> Frankly speaking I'm not familiar with new feature introduced in Junit
>> 4.4. Are there any enhancements that can help to resolve this
>> exclude-whole-class-because-of-one-bad-method issue?
>>
>> Thanks,
>> Alexei
>>
>> 2008/6/11, Regis <xu...@gmail.com>:
>> >
>> > Nathan Beyer wrote:
>> > > On Fri, Jun 6, 2008 at 1:08 AM, Regis <xu...@gmail.com> wrote:
>> > >
>> > >
>> > > > Hi,
>> > > >
>> > > > Matcher and Assumptions are great ideas! Thanks Nathan.
>> > > > They would be very helpful for our new test cases. But I notice that
>> > > > Junit 4.4 doesn't support group which is very important feature for
>> > > > both old tests and new tests. We can partition our test suite and run
>> > > > them separately. It's make our tests more flexible and configurable,
>> > > > and it's the main reason we discuss to migrate to TestNG long time
>> ago.
>> > > >
>> > >
>> > >
>> > > Don't we partition our tests already? Isn't that what the 'api' and
>> 'impl'
>> > > folders are about?
>> > >
>> > Yes, but it's not enough. We have discussed and created a wiki page[1]
>> about
>> > how
>> > to configuration and group harmony tests. The page is a little old, but I
>> > think the problems
>> > it tried to resolve still exist now. The partitions are not only include
>> > 'api' and 'impl', but also
>> > include partition of different os, architecture, partition of broken
>> tests
>> > and level of tests.
>> > folder structure or exclude files can't help in this complex situation,
>> so
>> > we need some tools
>> > to help us to deal with this, i think TestNG is suitable. If JUnit 4.4
>> can
>> > do it, i will vote to JUnit,
>> > update to a new version is always easier than switch to a new tool after
>> > all.
>> >
>> > [1] http://wiki.apache.org/harmony/Testing_Convention
>> >
>> > Best Regards,
>> > Regis.
>> >
>> >
>> > >
>> > > -Nathan
>> > >
>> > >
>> > >
>> > > >
>> > > > Best Regards,
>> > > > Regis.
>> > > >
>> > > >
>> > > > Nathan Beyer wrote:
>> > > >
>> > > >
>> > > > > That discussion was a very long time ago. Is there still value in
>> > TestNG?
>> > > > > I'd prefer to move to JUnit 4.4. All of our current tests will
>> > continue to
>> > > > > work and new tests can be implemented using the latest conventions
>> and
>> > > > > older
>> > > > > tests can be updated as we get to them. JUnit 4.4 is a far cry from
>> > 4.0.
>> > > > >
>> > > > > Here's the things I think would be create for our use and testing
>> in
>> > > > > general
>> > > > > - Matchers and the 'assertThat' - much more readable code and
>> readable
>> > > > > failure messages
>> > > > > - Assumptions and the 'assumeThat' - allows methods to add
>> statements
>> > that
>> > > > > guarantee that preconditions for the test are correct; this allows
>> > tests
>> > > > > to
>> > > > > fail such that you know it's an environment issue and not an actual
>> > test
>> > > > > failure
>> > > > >
>> > > > > If you're not familiar with matchers, check out this quick tutorial
>> -
>> > > > > http://code.google.com/p/hamcrest/wiki/Tutorial.
>> > > > >
>> > > > > -Nathan
>> > > > >
>> > > > >
>> > > > > On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <se...@gmail.com>
>> > wrote:
>> > > > >
>> > > > > Hi, all.
>> > > > >
>> > > > > > We had discussed the migration to testNG before and got some
>> > conclusions
>> > > > > > for
>> > > > > > grouping[1]
>> > > > > > including how to deal with boot path test[2]. Am i missing
>> > something?
>> > > > > > Is it still in our schedule? I think it's valueable to Harmony.
>> > > > > > I volunteer to carry out this job if no one objects. Any other
>> > > > > > volunteers?
>> > > > > >
>> > > > > > IMHO, we can only add some ant tasks to integrate testng at the
>> > > > > > beginning.
>> > > > > > So our original junit tests can still work at the mean time when
>> > > > > > migrating.
>> > > > > > When one module's migration task is finished, we can judge the
>> > result
>> > > > > > to dertermine whether we should go on for other modules.
>> > > > > >
>> > > > > > Maybe we can create a branch for luni to start this work, shall
>> we?
>> > > > > > therefore there won't be any impact on other's development.
>> > > > > > Once it is completed in the branch, we could merge it back to our
>> > trunk.
>> > > > > > Does it make sense?
>> > > > > >
>> > > > > > Any sugestions or comments are welcomed. Thanks very much.
>> > > > > >
>> > > > > > [1]
>> > http://wiki.apache.org/harmony/Testing_Convention
>> > > > > > [2]
>> > > > > >
>> > > > > >
>> >
>> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
>> > > > > > [3]
>> > http://testng.org/doc/documentation-main.html#annotations
>> > > > > > --
>> > > > > > Best Regards
>> > > > > > Sean, Xiao Xia Qiu
>> > > > > > China Software Development Lab, IBM
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> >
>>
>
--
Best Regards
Sean, Xiao Xia Qiu
China Software Development Lab, IBM
Re: [classlib][test] Migration to testNG?
Posted by Nathan Beyer <nb...@gmail.com>.
With annotation-based tests, just remove the annotation and that method is
no longer treated as a test. However, the same thing could be done today by
just changing the name of the method to NOT start with 'test'. I don't see
any approach being good though as it keeps the issues hidden; nothing makes
them stand out.
-Nathan
On Wed, Jun 11, 2008 at 5:59 AM, Alexei Zakharov <al...@gmail.com>
wrote:
> As far as I understand in spite of the fact there were no
> TestNG-related discussions since 2006 the problem is still relevant.
> There are big exclude lists in some classlib modules still, and many
> tests are excluded only because of a couple of failing methods.
> Frankly speaking I'm not familiar with new feature introduced in Junit
> 4.4. Are there any enhancements that can help to resolve this
> exclude-whole-class-because-of-one-bad-method issue?
>
> Thanks,
> Alexei
>
> 2008/6/11, Regis <xu...@gmail.com>:
> >
> > Nathan Beyer wrote:
> > > On Fri, Jun 6, 2008 at 1:08 AM, Regis <xu...@gmail.com> wrote:
> > >
> > >
> > > > Hi,
> > > >
> > > > Matcher and Assumptions are great ideas! Thanks Nathan.
> > > > They would be very helpful for our new test cases. But I notice that
> > > > Junit 4.4 doesn't support group which is very important feature for
> > > > both old tests and new tests. We can partition our test suite and run
> > > > them separately. It's make our tests more flexible and configurable,
> > > > and it's the main reason we discuss to migrate to TestNG long time
> ago.
> > > >
> > >
> > >
> > > Don't we partition our tests already? Isn't that what the 'api' and
> 'impl'
> > > folders are about?
> > >
> > Yes, but it's not enough. We have discussed and created a wiki page[1]
> about
> > how
> > to configuration and group harmony tests. The page is a little old, but I
> > think the problems
> > it tried to resolve still exist now. The partitions are not only include
> > 'api' and 'impl', but also
> > include partition of different os, architecture, partition of broken
> tests
> > and level of tests.
> > folder structure or exclude files can't help in this complex situation,
> so
> > we need some tools
> > to help us to deal with this, i think TestNG is suitable. If JUnit 4.4
> can
> > do it, i will vote to JUnit,
> > update to a new version is always easier than switch to a new tool after
> > all.
> >
> > [1] http://wiki.apache.org/harmony/Testing_Convention
> >
> > Best Regards,
> > Regis.
> >
> >
> > >
> > > -Nathan
> > >
> > >
> > >
> > > >
> > > > Best Regards,
> > > > Regis.
> > > >
> > > >
> > > > Nathan Beyer wrote:
> > > >
> > > >
> > > > > That discussion was a very long time ago. Is there still value in
> > TestNG?
> > > > > I'd prefer to move to JUnit 4.4. All of our current tests will
> > continue to
> > > > > work and new tests can be implemented using the latest conventions
> and
> > > > > older
> > > > > tests can be updated as we get to them. JUnit 4.4 is a far cry from
> > 4.0.
> > > > >
> > > > > Here's the things I think would be create for our use and testing
> in
> > > > > general
> > > > > - Matchers and the 'assertThat' - much more readable code and
> readable
> > > > > failure messages
> > > > > - Assumptions and the 'assumeThat' - allows methods to add
> statements
> > that
> > > > > guarantee that preconditions for the test are correct; this allows
> > tests
> > > > > to
> > > > > fail such that you know it's an environment issue and not an actual
> > test
> > > > > failure
> > > > >
> > > > > If you're not familiar with matchers, check out this quick tutorial
> -
> > > > > http://code.google.com/p/hamcrest/wiki/Tutorial.
> > > > >
> > > > > -Nathan
> > > > >
> > > > >
> > > > > On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <se...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi, all.
> > > > >
> > > > > > We had discussed the migration to testNG before and got some
> > conclusions
> > > > > > for
> > > > > > grouping[1]
> > > > > > including how to deal with boot path test[2]. Am i missing
> > something?
> > > > > > Is it still in our schedule? I think it's valueable to Harmony.
> > > > > > I volunteer to carry out this job if no one objects. Any other
> > > > > > volunteers?
> > > > > >
> > > > > > IMHO, we can only add some ant tasks to integrate testng at the
> > > > > > beginning.
> > > > > > So our original junit tests can still work at the mean time when
> > > > > > migrating.
> > > > > > When one module's migration task is finished, we can judge the
> > result
> > > > > > to dertermine whether we should go on for other modules.
> > > > > >
> > > > > > Maybe we can create a branch for luni to start this work, shall
> we?
> > > > > > therefore there won't be any impact on other's development.
> > > > > > Once it is completed in the branch, we could merge it back to our
> > trunk.
> > > > > > Does it make sense?
> > > > > >
> > > > > > Any sugestions or comments are welcomed. Thanks very much.
> > > > > >
> > > > > > [1]
> > http://wiki.apache.org/harmony/Testing_Convention
> > > > > > [2]
> > > > > >
> > > > > >
> >
> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
> > > > > > [3]
> > http://testng.org/doc/documentation-main.html#annotations
> > > > > > --
> > > > > > Best Regards
> > > > > > Sean, Xiao Xia Qiu
> > > > > > China Software Development Lab, IBM
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> >
>
Re: [classlib][test] Migration to testNG?
Posted by Sean Qiu <se...@gmail.com>.
Sorry for late response.
For your question, I am not familiar with Junit 4.4 too, but i know
TestNG could figure it out indeed.
"groups" annotation of TestNG should be a great candidate to solve
this problem.
We defined two groups of state.broken.<os> and state.broken<vm> in
our wiki page[1] in this case.
We can group all these failed methods by these annotations to exclude
them when running all tests.
Once they are fixed , just remove them from state.broken group to other groups.
We are not required to exclude whole test class for one failure of its method.
The complex excluding files are not desired any longer, just "groups"
annotations instead.
By means of "groups" annotation, we can replace our current approach
which classify our test by directories[2].
It is complex and hard to maintain because we don't only have api and
impl tests,
but also platform-specific tests, run on classpath tests,
run on bootclasspath, behaves different between Harmony and RI tests
and so on...
We can find that tests in different modules may divided in different
directory structures currently.
If we adopt TestNG, this nighmare might be gone. All the directories
could be replaced by "groups" annotation.
They can be placed in the same directory as Maven's convention.
So we can archive them into a single jar.( the bootclass path test may
be a exception).
We can just tell the groups parameter to run our desired tests.
IMHO, for these two reasons , TestNG is totally worthy to be considered.
BTW, i have successfully run our luni tests after converting from
Junit to TestNG.
I am trying to replace these excluding files by TestNG "groups"
annotation locally now.
[1] http://wiki.apache.org/harmony/Testing_Convention
[2] http://harmony.apache.org/subcomponents/classlibrary/testing.html
2008/6/11 Alexei Zakharov <al...@gmail.com>:
> As far as I understand in spite of the fact there were no
> TestNG-related discussions since 2006 the problem is still relevant.
> There are big exclude lists in some classlib modules still, and many
> tests are excluded only because of a couple of failing methods.
> Frankly speaking I'm not familiar with new feature introduced in Junit
> 4.4. Are there any enhancements that can help to resolve this
> exclude-whole-class-because-of-one-bad-method issue?
>
> Thanks,
> Alexei
>
> 2008/6/11, Regis <xu...@gmail.com>:
>>
>> Nathan Beyer wrote:
>> > On Fri, Jun 6, 2008 at 1:08 AM, Regis <xu...@gmail.com> wrote:
>> >
>> >
>> > > Hi,
>> > >
>> > > Matcher and Assumptions are great ideas! Thanks Nathan.
>> > > They would be very helpful for our new test cases. But I notice that
>> > > Junit 4.4 doesn't support group which is very important feature for
>> > > both old tests and new tests. We can partition our test suite and run
>> > > them separately. It's make our tests more flexible and configurable,
>> > > and it's the main reason we discuss to migrate to TestNG long time ago.
>> > >
>> >
>> >
>> > Don't we partition our tests already? Isn't that what the 'api' and 'impl'
>> > folders are about?
>> >
>> Yes, but it's not enough. We have discussed and created a wiki page[1] about
>> how
>> to configuration and group harmony tests. The page is a little old, but I
>> think the problems
>> it tried to resolve still exist now. The partitions are not only include
>> 'api' and 'impl', but also
>> include partition of different os, architecture, partition of broken tests
>> and level of tests.
>> folder structure or exclude files can't help in this complex situation, so
>> we need some tools
>> to help us to deal with this, i think TestNG is suitable. If JUnit 4.4 can
>> do it, i will vote to JUnit,
>> update to a new version is always easier than switch to a new tool after
>> all.
>>
>> [1] http://wiki.apache.org/harmony/Testing_Convention
>>
>> Best Regards,
>> Regis.
>>
>>
>> >
>> > -Nathan
>> >
>> >
>> >
>> > >
>> > > Best Regards,
>> > > Regis.
>> > >
>> > >
>> > > Nathan Beyer wrote:
>> > >
>> > >
>> > > > That discussion was a very long time ago. Is there still value in
>> TestNG?
>> > > > I'd prefer to move to JUnit 4.4. All of our current tests will
>> continue to
>> > > > work and new tests can be implemented using the latest conventions and
>> > > > older
>> > > > tests can be updated as we get to them. JUnit 4.4 is a far cry from
>> 4.0.
>> > > >
>> > > > Here's the things I think would be create for our use and testing in
>> > > > general
>> > > > - Matchers and the 'assertThat' - much more readable code and readable
>> > > > failure messages
>> > > > - Assumptions and the 'assumeThat' - allows methods to add statements
>> that
>> > > > guarantee that preconditions for the test are correct; this allows
>> tests
>> > > > to
>> > > > fail such that you know it's an environment issue and not an actual
>> test
>> > > > failure
>> > > >
>> > > > If you're not familiar with matchers, check out this quick tutorial -
>> > > > http://code.google.com/p/hamcrest/wiki/Tutorial.
>> > > >
>> > > > -Nathan
>> > > >
>> > > >
>> > > > On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <se...@gmail.com>
>> wrote:
>> > > >
>> > > > Hi, all.
>> > > >
>> > > > > We had discussed the migration to testNG before and got some
>> conclusions
>> > > > > for
>> > > > > grouping[1]
>> > > > > including how to deal with boot path test[2]. Am i missing
>> something?
>> > > > > Is it still in our schedule? I think it's valueable to Harmony.
>> > > > > I volunteer to carry out this job if no one objects. Any other
>> > > > > volunteers?
>> > > > >
>> > > > > IMHO, we can only add some ant tasks to integrate testng at the
>> > > > > beginning.
>> > > > > So our original junit tests can still work at the mean time when
>> > > > > migrating.
>> > > > > When one module's migration task is finished, we can judge the
>> result
>> > > > > to dertermine whether we should go on for other modules.
>> > > > >
>> > > > > Maybe we can create a branch for luni to start this work, shall we?
>> > > > > therefore there won't be any impact on other's development.
>> > > > > Once it is completed in the branch, we could merge it back to our
>> trunk.
>> > > > > Does it make sense?
>> > > > >
>> > > > > Any sugestions or comments are welcomed. Thanks very much.
>> > > > >
>> > > > > [1]
>> http://wiki.apache.org/harmony/Testing_Convention
>> > > > > [2]
>> > > > >
>> > > > >
>> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
>> > > > > [3]
>> http://testng.org/doc/documentation-main.html#annotations
>> > > > > --
>> > > > > Best Regards
>> > > > > Sean, Xiao Xia Qiu
>> > > > > China Software Development Lab, IBM
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>>
>
--
Best Regards
Sean, Xiao Xia Qiu
China Software Development Lab, IBM
Re: [classlib][test] Migration to testNG?
Posted by Alexei Zakharov <al...@gmail.com>.
As far as I understand in spite of the fact there were no
TestNG-related discussions since 2006 the problem is still relevant.
There are big exclude lists in some classlib modules still, and many
tests are excluded only because of a couple of failing methods.
Frankly speaking I'm not familiar with new feature introduced in Junit
4.4. Are there any enhancements that can help to resolve this
exclude-whole-class-because-of-one-bad-method issue?
Thanks,
Alexei
2008/6/11, Regis <xu...@gmail.com>:
>
> Nathan Beyer wrote:
> > On Fri, Jun 6, 2008 at 1:08 AM, Regis <xu...@gmail.com> wrote:
> >
> >
> > > Hi,
> > >
> > > Matcher and Assumptions are great ideas! Thanks Nathan.
> > > They would be very helpful for our new test cases. But I notice that
> > > Junit 4.4 doesn't support group which is very important feature for
> > > both old tests and new tests. We can partition our test suite and run
> > > them separately. It's make our tests more flexible and configurable,
> > > and it's the main reason we discuss to migrate to TestNG long time ago.
> > >
> >
> >
> > Don't we partition our tests already? Isn't that what the 'api' and 'impl'
> > folders are about?
> >
> Yes, but it's not enough. We have discussed and created a wiki page[1] about
> how
> to configuration and group harmony tests. The page is a little old, but I
> think the problems
> it tried to resolve still exist now. The partitions are not only include
> 'api' and 'impl', but also
> include partition of different os, architecture, partition of broken tests
> and level of tests.
> folder structure or exclude files can't help in this complex situation, so
> we need some tools
> to help us to deal with this, i think TestNG is suitable. If JUnit 4.4 can
> do it, i will vote to JUnit,
> update to a new version is always easier than switch to a new tool after
> all.
>
> [1] http://wiki.apache.org/harmony/Testing_Convention
>
> Best Regards,
> Regis.
>
>
> >
> > -Nathan
> >
> >
> >
> > >
> > > Best Regards,
> > > Regis.
> > >
> > >
> > > Nathan Beyer wrote:
> > >
> > >
> > > > That discussion was a very long time ago. Is there still value in
> TestNG?
> > > > I'd prefer to move to JUnit 4.4. All of our current tests will
> continue to
> > > > work and new tests can be implemented using the latest conventions and
> > > > older
> > > > tests can be updated as we get to them. JUnit 4.4 is a far cry from
> 4.0.
> > > >
> > > > Here's the things I think would be create for our use and testing in
> > > > general
> > > > - Matchers and the 'assertThat' - much more readable code and readable
> > > > failure messages
> > > > - Assumptions and the 'assumeThat' - allows methods to add statements
> that
> > > > guarantee that preconditions for the test are correct; this allows
> tests
> > > > to
> > > > fail such that you know it's an environment issue and not an actual
> test
> > > > failure
> > > >
> > > > If you're not familiar with matchers, check out this quick tutorial -
> > > > http://code.google.com/p/hamcrest/wiki/Tutorial.
> > > >
> > > > -Nathan
> > > >
> > > >
> > > > On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <se...@gmail.com>
> wrote:
> > > >
> > > > Hi, all.
> > > >
> > > > > We had discussed the migration to testNG before and got some
> conclusions
> > > > > for
> > > > > grouping[1]
> > > > > including how to deal with boot path test[2]. Am i missing
> something?
> > > > > Is it still in our schedule? I think it's valueable to Harmony.
> > > > > I volunteer to carry out this job if no one objects. Any other
> > > > > volunteers?
> > > > >
> > > > > IMHO, we can only add some ant tasks to integrate testng at the
> > > > > beginning.
> > > > > So our original junit tests can still work at the mean time when
> > > > > migrating.
> > > > > When one module's migration task is finished, we can judge the
> result
> > > > > to dertermine whether we should go on for other modules.
> > > > >
> > > > > Maybe we can create a branch for luni to start this work, shall we?
> > > > > therefore there won't be any impact on other's development.
> > > > > Once it is completed in the branch, we could merge it back to our
> trunk.
> > > > > Does it make sense?
> > > > >
> > > > > Any sugestions or comments are welcomed. Thanks very much.
> > > > >
> > > > > [1]
> http://wiki.apache.org/harmony/Testing_Convention
> > > > > [2]
> > > > >
> > > > >
> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
> > > > > [3]
> http://testng.org/doc/documentation-main.html#annotations
> > > > > --
> > > > > Best Regards
> > > > > Sean, Xiao Xia Qiu
> > > > > China Software Development Lab, IBM
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> >
>
Re: [classlib][test] Migration to testNG?
Posted by Nathan Beyer <nd...@apache.org>.
Apparently this has been around for awhile, but I've never had a need for it
- @Ignore [1]. An annotation to ignore a test (at the method level) with an
optional message. Supposedly, ignored tests are to be reported in the
analysis report. I'll have to test that.
-Nathan
[1] http://junit.sourceforge.net/javadoc_40/org/junit/Ignore.html
On Mon, Jun 16, 2008 at 8:31 PM, Nathan Beyer <nd...@apache.org> wrote:
> If anyone's curious, here are the JUnit 4.4 release notes that have a good
> summary of the new stuff - assertThat/Matchers, assumptions and theories.
>
> http://junit.sourceforge.net/doc/ReleaseNotes4.4.html
>
>
> On Mon, Jun 16, 2008 at 5:32 AM, Alexei Zakharov <
> alexei.zakharov@gmail.com> wrote:
>
>> As far as I remember my two years old experiments with TestNG - it
>> groups all Junit tests (say from all classlib modules) into the one
>> group while running in JUnit compatibility mode. So the reports
>> produced by JuntReport didn't look very nice. By the way, I remember
>> there was some old contribution about xml-based exclude lists for
>> JUnit that allowed fine-grained (method-based) control over the test
>> execution. I don't remember exact JIRA number. However, we may search
>> for it. It was from George Harley.
>>
>> Regards,
>> Alexei
>>
>> 2008/6/16, Sian January <si...@googlemail.com>:
>> > On 13/06/2008, Sean Qiu <se...@gmail.com> wrote:
>> > >
>> > > 2008/6/12 Sian January <si...@googlemail.com>:
>> > > > This is slightly off track, but my concern about TestNG is that
>> there are
>> > > > already quite a lot of steps for someone new to Harmony to get the
>> source
>> > > > code and run the tests (both for the whole of Harmony and for a
>> single
>> > > class
>> > > > library module). Certainly for Eclipse users if we switched to
>> TestNG
>> > > this
>> > > > would add another step, although I'm not sure about command-line
>> users.
>> > > >
>> > >
>> > > The cost to make use of current tests is low, IMHO.
>> > > I have mentioned that we can add TestNG target to replace Junit
>> target.
>> > > From the command-line users' perspective, they still use the same
>> > > command to run the test.
>> > > The only difference would be the command will invoke TestNG to run
>> our
>> > > test.
>> > > For Eclipse users, they can adopt plug-in for TestNG, it is quite the
>> > > same as plug-in for junit.
>> > > So I think we can migration to TestNG smoothly for the user.
>> > > What's more, they can pick up their desired tests more conveniently.
>> >
>> >
>> >
>> > My point was that for Eclipse users it would be an extra step to
>> download
>> > the TestNG plug-in, where as the JUnit plugin already comes with
>> Eclipse. I
>> > realise that this isn't the biggest consideration, but I feel that our
>> setup
>> > is already quite complex so I thought it would be worth mentioning. I
>> would
>> > assume that switching to JUnit 4 would not require any extra steps
>> because
>> > Eclipse already has support for it embedded.
>> >
>> >
>> > For test developers, on the other hand, it is a little complex.
>> > > But we want to upgrade to a new tool, no matter Junit 4 or TestNG,
>> we
>> > > need to pay for it.Their efforts are similar.
>> > > For TestNG, we need add annotations for each testcase, classify them
>> > > to right groups.
>> > > The good news is we still can use original assert* as before since
>> > > TestNG include JUnit's Assert class.
>> > > As Nathan mentioned , we can use hamcrest as well if we want.
>> > >
>> > >
>> > >
>> > > > Of course if switching to TestNG solves some really major problems
>> then
>> > > it
>> > > > would probably be worth it, but the only thing I can see that it
>> gives us
>> > > > over JUnit 4 is being able to run different sets of tests on
>> different
>> > > OS's
>> > > > and to me that doesn't seem as important as having a project that's
>> easy
>> > > to
>> > > > access. Just my 2p worth...
>> > > >
>> > >
>> > > We can replace the exclude files to get a more accurate tracking for
>> > > failed test.
>> > > We can tell difference between public API tests and harmony specific
>> > > implementation tests .
>> > > We can archive all tests together into a jar.
>> > > I think there should be other benefits as well :)
>> >
>> >
>> >
>> > I think it would be good to be able to reduce the exclude list to just
>> > contain individual methods and keep track of it a bit better, although
>> I
>> > think some of this can be achieved with JUnit and using different
>> folders.
>> > I agree the TestNG solution seems neater and more flexible, I'm just
>> not
>> > convinced yet that the benefit we would be getting is worth the cost to
>> > upgrade. I'm also wondering about the overhead for developers. E.g am
>> I
>> > going to have to add four different annotations to every new test I
>> write in
>> > the TestNG system?
>> >
>> >
>> >
>> > > --
>> >
>> > > Best Regards
>> > > Sean, Xiao Xia Qiu
>> > >
>> > > China Software Development Lab, IBM
>> > >
>> >
>> >
>> > Thanks,
>> >
>> > Sian
>> >
>> >
>> > --
>> > Unless stated otherwise above:
>> > IBM United Kingdom Limited - Registered in England and Wales with
>> number
>> > 741598.
>> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
>> 3AU
>> >
>>
>
>
Re: [classlib][test] Migration to testNG?
Posted by Nathan Beyer <nd...@apache.org>.
If anyone's curious, here are the JUnit 4.4 release notes that have a good
summary of the new stuff - assertThat/Matchers, assumptions and theories.
http://junit.sourceforge.net/doc/ReleaseNotes4.4.html
On Mon, Jun 16, 2008 at 5:32 AM, Alexei Zakharov <al...@gmail.com>
wrote:
> As far as I remember my two years old experiments with TestNG - it
> groups all Junit tests (say from all classlib modules) into the one
> group while running in JUnit compatibility mode. So the reports
> produced by JuntReport didn't look very nice. By the way, I remember
> there was some old contribution about xml-based exclude lists for
> JUnit that allowed fine-grained (method-based) control over the test
> execution. I don't remember exact JIRA number. However, we may search
> for it. It was from George Harley.
>
> Regards,
> Alexei
>
> 2008/6/16, Sian January <si...@googlemail.com>:
> > On 13/06/2008, Sean Qiu <se...@gmail.com> wrote:
> > >
> > > 2008/6/12 Sian January <si...@googlemail.com>:
> > > > This is slightly off track, but my concern about TestNG is that
> there are
> > > > already quite a lot of steps for someone new to Harmony to get the
> source
> > > > code and run the tests (both for the whole of Harmony and for a
> single
> > > class
> > > > library module). Certainly for Eclipse users if we switched to
> TestNG
> > > this
> > > > would add another step, although I'm not sure about command-line
> users.
> > > >
> > >
> > > The cost to make use of current tests is low, IMHO.
> > > I have mentioned that we can add TestNG target to replace Junit
> target.
> > > From the command-line users' perspective, they still use the same
> > > command to run the test.
> > > The only difference would be the command will invoke TestNG to run our
> > > test.
> > > For Eclipse users, they can adopt plug-in for TestNG, it is quite the
> > > same as plug-in for junit.
> > > So I think we can migration to TestNG smoothly for the user.
> > > What's more, they can pick up their desired tests more conveniently.
> >
> >
> >
> > My point was that for Eclipse users it would be an extra step to download
> > the TestNG plug-in, where as the JUnit plugin already comes with
> Eclipse. I
> > realise that this isn't the biggest consideration, but I feel that our
> setup
> > is already quite complex so I thought it would be worth mentioning. I
> would
> > assume that switching to JUnit 4 would not require any extra steps
> because
> > Eclipse already has support for it embedded.
> >
> >
> > For test developers, on the other hand, it is a little complex.
> > > But we want to upgrade to a new tool, no matter Junit 4 or TestNG, we
> > > need to pay for it.Their efforts are similar.
> > > For TestNG, we need add annotations for each testcase, classify them
> > > to right groups.
> > > The good news is we still can use original assert* as before since
> > > TestNG include JUnit's Assert class.
> > > As Nathan mentioned , we can use hamcrest as well if we want.
> > >
> > >
> > >
> > > > Of course if switching to TestNG solves some really major problems
> then
> > > it
> > > > would probably be worth it, but the only thing I can see that it
> gives us
> > > > over JUnit 4 is being able to run different sets of tests on
> different
> > > OS's
> > > > and to me that doesn't seem as important as having a project that's
> easy
> > > to
> > > > access. Just my 2p worth...
> > > >
> > >
> > > We can replace the exclude files to get a more accurate tracking for
> > > failed test.
> > > We can tell difference between public API tests and harmony specific
> > > implementation tests .
> > > We can archive all tests together into a jar.
> > > I think there should be other benefits as well :)
> >
> >
> >
> > I think it would be good to be able to reduce the exclude list to just
> > contain individual methods and keep track of it a bit better, although I
> > think some of this can be achieved with JUnit and using different
> folders.
> > I agree the TestNG solution seems neater and more flexible, I'm just not
> > convinced yet that the benefit we would be getting is worth the cost to
> > upgrade. I'm also wondering about the overhead for developers. E.g am
> I
> > going to have to add four different annotations to every new test I
> write in
> > the TestNG system?
> >
> >
> >
> > > --
> >
> > > Best Regards
> > > Sean, Xiao Xia Qiu
> > >
> > > China Software Development Lab, IBM
> > >
> >
> >
> > Thanks,
> >
> > Sian
> >
> >
> > --
> > Unless stated otherwise above:
> > IBM United Kingdom Limited - Registered in England and Wales with number
> > 741598.
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> 3AU
> >
>
Re: [classlib][test] Migration to testNG?
Posted by Alexei Zakharov <al...@gmail.com>.
As far as I remember my two years old experiments with TestNG - it
groups all Junit tests (say from all classlib modules) into the one
group while running in JUnit compatibility mode. So the reports
produced by JuntReport didn't look very nice. By the way, I remember
there was some old contribution about xml-based exclude lists for
JUnit that allowed fine-grained (method-based) control over the test
execution. I don't remember exact JIRA number. However, we may search
for it. It was from George Harley.
Regards,
Alexei
2008/6/16, Sian January <si...@googlemail.com>:
> On 13/06/2008, Sean Qiu <se...@gmail.com> wrote:
> >
> > 2008/6/12 Sian January <si...@googlemail.com>:
> > > This is slightly off track, but my concern about TestNG is that there are
> > > already quite a lot of steps for someone new to Harmony to get the source
> > > code and run the tests (both for the whole of Harmony and for a single
> > class
> > > library module). Certainly for Eclipse users if we switched to TestNG
> > this
> > > would add another step, although I'm not sure about command-line users.
> > >
> >
> > The cost to make use of current tests is low, IMHO.
> > I have mentioned that we can add TestNG target to replace Junit target.
> > From the command-line users' perspective, they still use the same
> > command to run the test.
> > The only difference would be the command will invoke TestNG to run our
> > test.
> > For Eclipse users, they can adopt plug-in for TestNG, it is quite the
> > same as plug-in for junit.
> > So I think we can migration to TestNG smoothly for the user.
> > What's more, they can pick up their desired tests more conveniently.
>
>
>
> My point was that for Eclipse users it would be an extra step to download
> the TestNG plug-in, where as the JUnit plugin already comes with Eclipse. I
> realise that this isn't the biggest consideration, but I feel that our setup
> is already quite complex so I thought it would be worth mentioning. I would
> assume that switching to JUnit 4 would not require any extra steps because
> Eclipse already has support for it embedded.
>
>
> For test developers, on the other hand, it is a little complex.
> > But we want to upgrade to a new tool, no matter Junit 4 or TestNG, we
> > need to pay for it.Their efforts are similar.
> > For TestNG, we need add annotations for each testcase, classify them
> > to right groups.
> > The good news is we still can use original assert* as before since
> > TestNG include JUnit's Assert class.
> > As Nathan mentioned , we can use hamcrest as well if we want.
> >
> >
> >
> > > Of course if switching to TestNG solves some really major problems then
> > it
> > > would probably be worth it, but the only thing I can see that it gives us
> > > over JUnit 4 is being able to run different sets of tests on different
> > OS's
> > > and to me that doesn't seem as important as having a project that's easy
> > to
> > > access. Just my 2p worth...
> > >
> >
> > We can replace the exclude files to get a more accurate tracking for
> > failed test.
> > We can tell difference between public API tests and harmony specific
> > implementation tests .
> > We can archive all tests together into a jar.
> > I think there should be other benefits as well :)
>
>
>
> I think it would be good to be able to reduce the exclude list to just
> contain individual methods and keep track of it a bit better, although I
> think some of this can be achieved with JUnit and using different folders.
> I agree the TestNG solution seems neater and more flexible, I'm just not
> convinced yet that the benefit we would be getting is worth the cost to
> upgrade. I'm also wondering about the overhead for developers. E.g am I
> going to have to add four different annotations to every new test I write in
> the TestNG system?
>
>
>
> > --
>
> > Best Regards
> > Sean, Xiao Xia Qiu
> >
> > China Software Development Lab, IBM
> >
>
>
> Thanks,
>
> Sian
>
>
> --
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
Re: [classlib][test] Migration to testNG?
Posted by Sian January <si...@googlemail.com>.
On 13/06/2008, Sean Qiu <se...@gmail.com> wrote:
>
> 2008/6/12 Sian January <si...@googlemail.com>:
> > This is slightly off track, but my concern about TestNG is that there are
> > already quite a lot of steps for someone new to Harmony to get the source
> > code and run the tests (both for the whole of Harmony and for a single
> class
> > library module). Certainly for Eclipse users if we switched to TestNG
> this
> > would add another step, although I'm not sure about command-line users.
> >
>
> The cost to make use of current tests is low, IMHO.
> I have mentioned that we can add TestNG target to replace Junit target.
> From the command-line users' perspective, they still use the same
> command to run the test.
> The only difference would be the command will invoke TestNG to run our
> test.
> For Eclipse users, they can adopt plug-in for TestNG, it is quite the
> same as plug-in for junit.
> So I think we can migration to TestNG smoothly for the user.
> What's more, they can pick up their desired tests more conveniently.
My point was that for Eclipse users it would be an extra step to download
the TestNG plug-in, where as the JUnit plugin already comes with Eclipse. I
realise that this isn't the biggest consideration, but I feel that our setup
is already quite complex so I thought it would be worth mentioning. I would
assume that switching to JUnit 4 would not require any extra steps because
Eclipse already has support for it embedded.
For test developers, on the other hand, it is a little complex.
> But we want to upgrade to a new tool, no matter Junit 4 or TestNG, we
> need to pay for it.Their efforts are similar.
> For TestNG, we need add annotations for each testcase, classify them
> to right groups.
> The good news is we still can use original assert* as before since
> TestNG include JUnit's Assert class.
> As Nathan mentioned , we can use hamcrest as well if we want.
>
>
>
> > Of course if switching to TestNG solves some really major problems then
> it
> > would probably be worth it, but the only thing I can see that it gives us
> > over JUnit 4 is being able to run different sets of tests on different
> OS's
> > and to me that doesn't seem as important as having a project that's easy
> to
> > access. Just my 2p worth...
> >
>
> We can replace the exclude files to get a more accurate tracking for
> failed test.
> We can tell difference between public API tests and harmony specific
> implementation tests .
> We can archive all tests together into a jar.
> I think there should be other benefits as well :)
I think it would be good to be able to reduce the exclude list to just
contain individual methods and keep track of it a bit better, although I
think some of this can be achieved with JUnit and using different folders.
I agree the TestNG solution seems neater and more flexible, I'm just not
convinced yet that the benefit we would be getting is worth the cost to
upgrade. I'm also wondering about the overhead for developers. E.g am I
going to have to add four different annotations to every new test I write in
the TestNG system?
> --
> Best Regards
> Sean, Xiao Xia Qiu
>
> China Software Development Lab, IBM
>
Thanks,
Sian
--
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [classlib][test] Migration to testNG?
Posted by Regis <xu...@gmail.com>.
Sure, We could define different @group for particular excluded list on
different platform,
and run them separately when we needed.
Best Regards,
Regis.
Tim Ellison wrote:
> I haven't looked at TestNG, so I don;t have a strong opinion either way.
>
> Does TestNG allow you to produce a report of all tests tagged as
> @excluded (or whatever)? so we can easily see which tests are excluded |
> run on a particular platform | whatever? I would think that'd be useful.
>
> Regards,
> Tim
>
> Sean Qiu wrote:
>> Interesting. So we can upgrade to Junit 4.4 more smoothly or without
>> any pains.
>> But what can we benefit from the upgrading?
>> Get 6 annotations to run tests and hamcrest assertions.
>> IMHO, this can not solve any of our problems that TestNG can.
>>
>> Or we wait until Junit 4 include groups annotations?
>>
>> 2008/6/13 Nathan Beyer <nd...@apache.org>:
>>> On Thu, Jun 12, 2008 at 10:13 PM, Sean Qiu <se...@gmail.com>
>>> wrote:
>>>
>>>> 2008/6/12 Sian January <si...@googlemail.com>:
>>>>> This is slightly off track, but my concern about TestNG is that
>>>>> there are
>>>>> already quite a lot of steps for someone new to Harmony to get the
>>>>> source
>>>>> code and run the tests (both for the whole of Harmony and for a single
>>>> class
>>>>> library module). Certainly for Eclipse users if we switched to TestNG
>>>> this
>>>>> would add another step, although I'm not sure about command-line
>>>>> users.
>>>>>
>>>> The cost to make use of current tests is low, IMHO.
>>>> I have mentioned that we can add TestNG target to replace Junit target.
>>>> From the command-line users' perspective, they still use the same
>>>> command to run the test.
>>>> The only difference would be the command will invoke TestNG to run our
>>>> test.
>>>> For Eclipse users, they can adopt plug-in for TestNG, it is quite the
>>>> same as plug-in for junit.
>>>> So I think we can migration to TestNG smoothly for the user.
>>>> What's more, they can pick up their desired tests more conveniently.
>>>>
>>>> For test developers, on the other hand, it is a little complex.
>>>> But we want to upgrade to a new tool, no matter Junit 4 or TestNG, we
>>>> need to pay for it.Their efforts are similar.
>>>
>>> Actually, we can use JUnit 4 now. If you run with Ant 1.7 and put the
>>> JUnit
>>> 4 library that's in the dependency download already into your Ant
>>> classpath,
>>> JUnit 4 tests will run. You can even mix JUnit 3 and 4 tests in the
>>> same run
>>> and it works fine, so new test classes can be built at JUnit 4 tests
>>> without
>>> having to update any existing test code. You can even use some of the
>>> minimal built-in hamcrest matchers, as they are part of the JUnit 4.4
>>> JAR.
>>>
>>> -Nathan
>>>
>>>
>>>> For TestNG, we need add annotations for each testcase, classify them
>>>> to right groups.
>>>> The good news is we still can use original assert* as before since
>>>> TestNG include JUnit's Assert class.
>>>> As Nathan mentioned , we can use hamcrest as well if we want.
>>>>
>>>>
>>>>
>>>>> Of course if switching to TestNG solves some really major problems
>>>>> then
>>>> it
>>>>> would probably be worth it, but the only thing I can see that it
>>>>> gives us
>>>>> over JUnit 4 is being able to run different sets of tests on different
>>>> OS's
>>>>> and to me that doesn't seem as important as having a project that's
>>>>> easy
>>>> to
>>>>> access. Just my 2p worth...
>>>>>
>>>> We can replace the exclude files to get a more accurate tracking for
>>>> failed test.
>>>> We can tell difference between public API tests and harmony specific
>>>> implementation tests .
>>>> We can archive all tests together into a jar.
>>>> I think there should be other benefits as well :)
>>>>
>>>>
>>>> --
>>>> Best Regards
>>>> Sean, Xiao Xia Qiu
>>>>
>>>> China Software Development Lab, IBM
>>>>
>>
>>
>>
>
Re: [classlib][test] Migration to testNG?
Posted by Tim Ellison <t....@gmail.com>.
I haven't looked at TestNG, so I don;t have a strong opinion either way.
Does TestNG allow you to produce a report of all tests tagged as
@excluded (or whatever)? so we can easily see which tests are excluded |
run on a particular platform | whatever? I would think that'd be useful.
Regards,
Tim
Sean Qiu wrote:
> Interesting. So we can upgrade to Junit 4.4 more smoothly or without any pains.
> But what can we benefit from the upgrading?
> Get 6 annotations to run tests and hamcrest assertions.
> IMHO, this can not solve any of our problems that TestNG can.
>
> Or we wait until Junit 4 include groups annotations?
>
> 2008/6/13 Nathan Beyer <nd...@apache.org>:
>> On Thu, Jun 12, 2008 at 10:13 PM, Sean Qiu <se...@gmail.com> wrote:
>>
>>> 2008/6/12 Sian January <si...@googlemail.com>:
>>>> This is slightly off track, but my concern about TestNG is that there are
>>>> already quite a lot of steps for someone new to Harmony to get the source
>>>> code and run the tests (both for the whole of Harmony and for a single
>>> class
>>>> library module). Certainly for Eclipse users if we switched to TestNG
>>> this
>>>> would add another step, although I'm not sure about command-line users.
>>>>
>>> The cost to make use of current tests is low, IMHO.
>>> I have mentioned that we can add TestNG target to replace Junit target.
>>> From the command-line users' perspective, they still use the same
>>> command to run the test.
>>> The only difference would be the command will invoke TestNG to run our
>>> test.
>>> For Eclipse users, they can adopt plug-in for TestNG, it is quite the
>>> same as plug-in for junit.
>>> So I think we can migration to TestNG smoothly for the user.
>>> What's more, they can pick up their desired tests more conveniently.
>>>
>>> For test developers, on the other hand, it is a little complex.
>>> But we want to upgrade to a new tool, no matter Junit 4 or TestNG, we
>>> need to pay for it.Their efforts are similar.
>>
>> Actually, we can use JUnit 4 now. If you run with Ant 1.7 and put the JUnit
>> 4 library that's in the dependency download already into your Ant classpath,
>> JUnit 4 tests will run. You can even mix JUnit 3 and 4 tests in the same run
>> and it works fine, so new test classes can be built at JUnit 4 tests without
>> having to update any existing test code. You can even use some of the
>> minimal built-in hamcrest matchers, as they are part of the JUnit 4.4 JAR.
>>
>> -Nathan
>>
>>
>>> For TestNG, we need add annotations for each testcase, classify them
>>> to right groups.
>>> The good news is we still can use original assert* as before since
>>> TestNG include JUnit's Assert class.
>>> As Nathan mentioned , we can use hamcrest as well if we want.
>>>
>>>
>>>
>>>> Of course if switching to TestNG solves some really major problems then
>>> it
>>>> would probably be worth it, but the only thing I can see that it gives us
>>>> over JUnit 4 is being able to run different sets of tests on different
>>> OS's
>>>> and to me that doesn't seem as important as having a project that's easy
>>> to
>>>> access. Just my 2p worth...
>>>>
>>> We can replace the exclude files to get a more accurate tracking for
>>> failed test.
>>> We can tell difference between public API tests and harmony specific
>>> implementation tests .
>>> We can archive all tests together into a jar.
>>> I think there should be other benefits as well :)
>>>
>>>
>>> --
>>> Best Regards
>>> Sean, Xiao Xia Qiu
>>>
>>> China Software Development Lab, IBM
>>>
>
>
>
Re: [classlib][test] Migration to testNG?
Posted by Sean Qiu <se...@gmail.com>.
Interesting. So we can upgrade to Junit 4.4 more smoothly or without any pains.
But what can we benefit from the upgrading?
Get 6 annotations to run tests and hamcrest assertions.
IMHO, this can not solve any of our problems that TestNG can.
Or we wait until Junit 4 include groups annotations?
2008/6/13 Nathan Beyer <nd...@apache.org>:
> On Thu, Jun 12, 2008 at 10:13 PM, Sean Qiu <se...@gmail.com> wrote:
>
>> 2008/6/12 Sian January <si...@googlemail.com>:
>> > This is slightly off track, but my concern about TestNG is that there are
>> > already quite a lot of steps for someone new to Harmony to get the source
>> > code and run the tests (both for the whole of Harmony and for a single
>> class
>> > library module). Certainly for Eclipse users if we switched to TestNG
>> this
>> > would add another step, although I'm not sure about command-line users.
>> >
>>
>> The cost to make use of current tests is low, IMHO.
>> I have mentioned that we can add TestNG target to replace Junit target.
>> From the command-line users' perspective, they still use the same
>> command to run the test.
>> The only difference would be the command will invoke TestNG to run our
>> test.
>> For Eclipse users, they can adopt plug-in for TestNG, it is quite the
>> same as plug-in for junit.
>> So I think we can migration to TestNG smoothly for the user.
>> What's more, they can pick up their desired tests more conveniently.
>>
>> For test developers, on the other hand, it is a little complex.
>> But we want to upgrade to a new tool, no matter Junit 4 or TestNG, we
>> need to pay for it.Their efforts are similar.
>
>
> Actually, we can use JUnit 4 now. If you run with Ant 1.7 and put the JUnit
> 4 library that's in the dependency download already into your Ant classpath,
> JUnit 4 tests will run. You can even mix JUnit 3 and 4 tests in the same run
> and it works fine, so new test classes can be built at JUnit 4 tests without
> having to update any existing test code. You can even use some of the
> minimal built-in hamcrest matchers, as they are part of the JUnit 4.4 JAR.
>
> -Nathan
>
>
>>
>> For TestNG, we need add annotations for each testcase, classify them
>> to right groups.
>> The good news is we still can use original assert* as before since
>> TestNG include JUnit's Assert class.
>> As Nathan mentioned , we can use hamcrest as well if we want.
>>
>>
>>
>> > Of course if switching to TestNG solves some really major problems then
>> it
>> > would probably be worth it, but the only thing I can see that it gives us
>> > over JUnit 4 is being able to run different sets of tests on different
>> OS's
>> > and to me that doesn't seem as important as having a project that's easy
>> to
>> > access. Just my 2p worth...
>> >
>>
>> We can replace the exclude files to get a more accurate tracking for
>> failed test.
>> We can tell difference between public API tests and harmony specific
>> implementation tests .
>> We can archive all tests together into a jar.
>> I think there should be other benefits as well :)
>>
>>
>> --
>> Best Regards
>> Sean, Xiao Xia Qiu
>>
>> China Software Development Lab, IBM
>>
>
--
Best Regards
Sean, Xiao Xia Qiu
China Software Development Lab, IBM
Re: [classlib][test] Migration to testNG?
Posted by Nathan Beyer <nd...@apache.org>.
On Thu, Jun 12, 2008 at 10:13 PM, Sean Qiu <se...@gmail.com> wrote:
> 2008/6/12 Sian January <si...@googlemail.com>:
> > This is slightly off track, but my concern about TestNG is that there are
> > already quite a lot of steps for someone new to Harmony to get the source
> > code and run the tests (both for the whole of Harmony and for a single
> class
> > library module). Certainly for Eclipse users if we switched to TestNG
> this
> > would add another step, although I'm not sure about command-line users.
> >
>
> The cost to make use of current tests is low, IMHO.
> I have mentioned that we can add TestNG target to replace Junit target.
> From the command-line users' perspective, they still use the same
> command to run the test.
> The only difference would be the command will invoke TestNG to run our
> test.
> For Eclipse users, they can adopt plug-in for TestNG, it is quite the
> same as plug-in for junit.
> So I think we can migration to TestNG smoothly for the user.
> What's more, they can pick up their desired tests more conveniently.
>
> For test developers, on the other hand, it is a little complex.
> But we want to upgrade to a new tool, no matter Junit 4 or TestNG, we
> need to pay for it.Their efforts are similar.
Actually, we can use JUnit 4 now. If you run with Ant 1.7 and put the JUnit
4 library that's in the dependency download already into your Ant classpath,
JUnit 4 tests will run. You can even mix JUnit 3 and 4 tests in the same run
and it works fine, so new test classes can be built at JUnit 4 tests without
having to update any existing test code. You can even use some of the
minimal built-in hamcrest matchers, as they are part of the JUnit 4.4 JAR.
-Nathan
>
> For TestNG, we need add annotations for each testcase, classify them
> to right groups.
> The good news is we still can use original assert* as before since
> TestNG include JUnit's Assert class.
> As Nathan mentioned , we can use hamcrest as well if we want.
>
>
>
> > Of course if switching to TestNG solves some really major problems then
> it
> > would probably be worth it, but the only thing I can see that it gives us
> > over JUnit 4 is being able to run different sets of tests on different
> OS's
> > and to me that doesn't seem as important as having a project that's easy
> to
> > access. Just my 2p worth...
> >
>
> We can replace the exclude files to get a more accurate tracking for
> failed test.
> We can tell difference between public API tests and harmony specific
> implementation tests .
> We can archive all tests together into a jar.
> I think there should be other benefits as well :)
>
>
> --
> Best Regards
> Sean, Xiao Xia Qiu
>
> China Software Development Lab, IBM
>
Re: [classlib][test] Migration to testNG?
Posted by Sean Qiu <se...@gmail.com>.
2008/6/12 Sian January <si...@googlemail.com>:
> This is slightly off track, but my concern about TestNG is that there are
> already quite a lot of steps for someone new to Harmony to get the source
> code and run the tests (both for the whole of Harmony and for a single class
> library module). Certainly for Eclipse users if we switched to TestNG this
> would add another step, although I'm not sure about command-line users.
>
The cost to make use of current tests is low, IMHO.
I have mentioned that we can add TestNG target to replace Junit target.
>From the command-line users' perspective, they still use the same
command to run the test.
The only difference would be the command will invoke TestNG to run our test.
For Eclipse users, they can adopt plug-in for TestNG, it is quite the
same as plug-in for junit.
So I think we can migration to TestNG smoothly for the user.
What's more, they can pick up their desired tests more conveniently.
For test developers, on the other hand, it is a little complex.
But we want to upgrade to a new tool, no matter Junit 4 or TestNG, we
need to pay for it.Their efforts are similar.
For TestNG, we need add annotations for each testcase, classify them
to right groups.
The good news is we still can use original assert* as before since
TestNG include JUnit's Assert class.
As Nathan mentioned , we can use hamcrest as well if we want.
> Of course if switching to TestNG solves some really major problems then it
> would probably be worth it, but the only thing I can see that it gives us
> over JUnit 4 is being able to run different sets of tests on different OS's
> and to me that doesn't seem as important as having a project that's easy to
> access. Just my 2p worth...
>
We can replace the exclude files to get a more accurate tracking for
failed test.
We can tell difference between public API tests and harmony specific
implementation tests .
We can archive all tests together into a jar.
I think there should be other benefits as well :)
--
Best Regards
Sean, Xiao Xia Qiu
China Software Development Lab, IBM
Re: [classlib][test] Migration to testNG?
Posted by Sian January <si...@googlemail.com>.
This is slightly off track, but my concern about TestNG is that there are
already quite a lot of steps for someone new to Harmony to get the source
code and run the tests (both for the whole of Harmony and for a single class
library module). Certainly for Eclipse users if we switched to TestNG this
would add another step, although I'm not sure about command-line users.
Of course if switching to TestNG solves some really major problems then it
would probably be worth it, but the only thing I can see that it gives us
over JUnit 4 is being able to run different sets of tests on different OS's
and to me that doesn't seem as important as having a project that's easy to
access. Just my 2p worth...
Re: [classlib][test] Migration to testNG?
Posted by Sean Qiu <se...@gmail.com>.
2008/6/12 Nathan Beyer <nd...@apache.org>:
> Like I mentioned in another reply; exclusions for failing tests are bad. The
> only real requirement here is for separate invocation for 'bootclasspath
> tests (api)', 'impl tests' and 'os-specific' tests.
IMHO, TestNG is quite worthy to be considered if it can solve these issues. :)
>
> As for that wiki page you mentioned - Are there really issues, beyond the
> failed exclusions, that is. I contend that if it didn't resolve the issues,
> that's because it was fully implemented. Not every project needs to follow
> the conventions, but it seems like only one or two of the class library
> modules do use it; 'luni' and ???.
>
Yes, some of the modules are simple and and this convention may seem overkill.
But it may be a benefit, since there is no need to pay additional
attention for them.
What about we need to use it in the future when this module may
involve some more compelx tests?
In that case, we need to add impl, api, common, unix, windows
directories and move tests to right place.
With TestNG, we just need add annotations.
> -Nathan
>
> On Wed, Jun 11, 2008 at 4:59 AM, Regis <xu...@gmail.com> wrote:
>
>>
>> Nathan Beyer wrote:
>>
>>> On Fri, Jun 6, 2008 at 1:08 AM, Regis <xu...@gmail.com> wrote:
>>>
>>> Hi,
>>>>
>>>> Matcher and Assumptions are great ideas! Thanks Nathan.
>>>> They would be very helpful for our new test cases. But I notice that
>>>> Junit 4.4 doesn't support group which is very important feature for
>>>> both old tests and new tests. We can partition our test suite and run
>>>> them separately. It's make our tests more flexible and configurable,
>>>> and it's the main reason we discuss to migrate to TestNG long time ago.
>>>>
>>>
>>>
>>> Don't we partition our tests already? Isn't that what the 'api' and 'impl'
>>> folders are about?
>>>
>> Yes, but it's not enough. We have discussed and created a wiki page[1]
>> about how
>> to configuration and group harmony tests. The page is a little old, but I
>> think the problems
>> it tried to resolve still exist now. The partitions are not only include
>> 'api' and 'impl', but also
>> include partition of different os, architecture, partition of broken tests
>> and level of tests.
>> folder structure or exclude files can't help in this complex situation, so
>> we need some tools
>> to help us to deal with this, i think TestNG is suitable. If JUnit 4.4 can
>> do it, i will vote to JUnit,
>> update to a new version is always easier than switch to a new tool after
>> all.
>>
>> [1] http://wiki.apache.org/harmony/Testing_Convention
>>
>> Best Regards,
>> Regis.
>>
>>
>>
>>> -Nathan
>>>
>>>
>>>
>>>> Best Regards,
>>>> Regis.
>>>>
>>>>
>>>> Nathan Beyer wrote:
>>>>
>>>> That discussion was a very long time ago. Is there still value in
>>>>> TestNG?
>>>>> I'd prefer to move to JUnit 4.4. All of our current tests will continue
>>>>> to
>>>>> work and new tests can be implemented using the latest conventions and
>>>>> older
>>>>> tests can be updated as we get to them. JUnit 4.4 is a far cry from 4.0.
>>>>>
>>>>> Here's the things I think would be create for our use and testing in
>>>>> general
>>>>> - Matchers and the 'assertThat' - much more readable code and readable
>>>>> failure messages
>>>>> - Assumptions and the 'assumeThat' - allows methods to add statements
>>>>> that
>>>>> guarantee that preconditions for the test are correct; this allows tests
>>>>> to
>>>>> fail such that you know it's an environment issue and not an actual test
>>>>> failure
>>>>>
>>>>> If you're not familiar with matchers, check out this quick tutorial -
>>>>> http://code.google.com/p/hamcrest/wiki/Tutorial.
>>>>>
>>>>> -Nathan
>>>>>
>>>>>
>>>>> On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <se...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi, all.
>>>>>
>>>>>> We had discussed the migration to testNG before and got some
>>>>>> conclusions
>>>>>> for
>>>>>> grouping[1]
>>>>>> including how to deal with boot path test[2]. Am i missing something?
>>>>>> Is it still in our schedule? I think it's valueable to Harmony.
>>>>>> I volunteer to carry out this job if no one objects. Any other
>>>>>> volunteers?
>>>>>>
>>>>>> IMHO, we can only add some ant tasks to integrate testng at the
>>>>>> beginning.
>>>>>> So our original junit tests can still work at the mean time when
>>>>>> migrating.
>>>>>> When one module's migration task is finished, we can judge the result
>>>>>> to dertermine whether we should go on for other modules.
>>>>>>
>>>>>> Maybe we can create a branch for luni to start this work, shall we?
>>>>>> therefore there won't be any impact on other's development.
>>>>>> Once it is completed in the branch, we could merge it back to our
>>>>>> trunk.
>>>>>> Does it make sense?
>>>>>>
>>>>>> Any sugestions or comments are welcomed. Thanks very much.
>>>>>>
>>>>>> [1] http://wiki.apache.org/harmony/Testing_Convention
>>>>>> [2]
>>>>>>
>>>>>>
>>>>>> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
>>>>>> [3] http://testng.org/doc/documentation-main.html#annotations
>>>>>> --
>>>>>> Best Regards
>>>>>> Sean, Xiao Xia Qiu
>>>>>> China Software Development Lab, IBM
>>>>>>
>>>>>>
>>>>>>
>>>
>
--
Best Regards
Sean, Xiao Xia Qiu
China Software Development Lab, IBM
Re: [classlib][test] Migration to testNG?
Posted by Nathan Beyer <nd...@apache.org>.
Like I mentioned in another reply; exclusions for failing tests are bad. The
only real requirement here is for separate invocation for 'bootclasspath
tests (api)', 'impl tests' and 'os-specific' tests.
As for that wiki page you mentioned - Are there really issues, beyond the
failed exclusions, that is. I contend that if it didn't resolve the issues,
that's because it was fully implemented. Not every project needs to follow
the conventions, but it seems like only one or two of the class library
modules do use it; 'luni' and ???.
-Nathan
On Wed, Jun 11, 2008 at 4:59 AM, Regis <xu...@gmail.com> wrote:
>
> Nathan Beyer wrote:
>
>> On Fri, Jun 6, 2008 at 1:08 AM, Regis <xu...@gmail.com> wrote:
>>
>> Hi,
>>>
>>> Matcher and Assumptions are great ideas! Thanks Nathan.
>>> They would be very helpful for our new test cases. But I notice that
>>> Junit 4.4 doesn't support group which is very important feature for
>>> both old tests and new tests. We can partition our test suite and run
>>> them separately. It's make our tests more flexible and configurable,
>>> and it's the main reason we discuss to migrate to TestNG long time ago.
>>>
>>
>>
>> Don't we partition our tests already? Isn't that what the 'api' and 'impl'
>> folders are about?
>>
> Yes, but it's not enough. We have discussed and created a wiki page[1]
> about how
> to configuration and group harmony tests. The page is a little old, but I
> think the problems
> it tried to resolve still exist now. The partitions are not only include
> 'api' and 'impl', but also
> include partition of different os, architecture, partition of broken tests
> and level of tests.
> folder structure or exclude files can't help in this complex situation, so
> we need some tools
> to help us to deal with this, i think TestNG is suitable. If JUnit 4.4 can
> do it, i will vote to JUnit,
> update to a new version is always easier than switch to a new tool after
> all.
>
> [1] http://wiki.apache.org/harmony/Testing_Convention
>
> Best Regards,
> Regis.
>
>
>
>> -Nathan
>>
>>
>>
>>> Best Regards,
>>> Regis.
>>>
>>>
>>> Nathan Beyer wrote:
>>>
>>> That discussion was a very long time ago. Is there still value in
>>>> TestNG?
>>>> I'd prefer to move to JUnit 4.4. All of our current tests will continue
>>>> to
>>>> work and new tests can be implemented using the latest conventions and
>>>> older
>>>> tests can be updated as we get to them. JUnit 4.4 is a far cry from 4.0.
>>>>
>>>> Here's the things I think would be create for our use and testing in
>>>> general
>>>> - Matchers and the 'assertThat' - much more readable code and readable
>>>> failure messages
>>>> - Assumptions and the 'assumeThat' - allows methods to add statements
>>>> that
>>>> guarantee that preconditions for the test are correct; this allows tests
>>>> to
>>>> fail such that you know it's an environment issue and not an actual test
>>>> failure
>>>>
>>>> If you're not familiar with matchers, check out this quick tutorial -
>>>> http://code.google.com/p/hamcrest/wiki/Tutorial.
>>>>
>>>> -Nathan
>>>>
>>>>
>>>> On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <se...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi, all.
>>>>
>>>>> We had discussed the migration to testNG before and got some
>>>>> conclusions
>>>>> for
>>>>> grouping[1]
>>>>> including how to deal with boot path test[2]. Am i missing something?
>>>>> Is it still in our schedule? I think it's valueable to Harmony.
>>>>> I volunteer to carry out this job if no one objects. Any other
>>>>> volunteers?
>>>>>
>>>>> IMHO, we can only add some ant tasks to integrate testng at the
>>>>> beginning.
>>>>> So our original junit tests can still work at the mean time when
>>>>> migrating.
>>>>> When one module's migration task is finished, we can judge the result
>>>>> to dertermine whether we should go on for other modules.
>>>>>
>>>>> Maybe we can create a branch for luni to start this work, shall we?
>>>>> therefore there won't be any impact on other's development.
>>>>> Once it is completed in the branch, we could merge it back to our
>>>>> trunk.
>>>>> Does it make sense?
>>>>>
>>>>> Any sugestions or comments are welcomed. Thanks very much.
>>>>>
>>>>> [1] http://wiki.apache.org/harmony/Testing_Convention
>>>>> [2]
>>>>>
>>>>>
>>>>> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
>>>>> [3] http://testng.org/doc/documentation-main.html#annotations
>>>>> --
>>>>> Best Regards
>>>>> Sean, Xiao Xia Qiu
>>>>> China Software Development Lab, IBM
>>>>>
>>>>>
>>>>>
>>
Re: [classlib][test] Migration to testNG?
Posted by Regis <xu...@gmail.com>.
Nathan Beyer wrote:
> On Fri, Jun 6, 2008 at 1:08 AM, Regis <xu...@gmail.com> wrote:
>
>> Hi,
>>
>> Matcher and Assumptions are great ideas! Thanks Nathan.
>> They would be very helpful for our new test cases. But I notice that
>> Junit 4.4 doesn't support group which is very important feature for
>> both old tests and new tests. We can partition our test suite and run
>> them separately. It's make our tests more flexible and configurable,
>> and it's the main reason we discuss to migrate to TestNG long time ago.
>
>
> Don't we partition our tests already? Isn't that what the 'api' and 'impl'
> folders are about?
Yes, but it's not enough. We have discussed and created a wiki page[1]
about how
to configuration and group harmony tests. The page is a little old, but
I think the problems
it tried to resolve still exist now. The partitions are not only include
'api' and 'impl', but also
include partition of different os, architecture, partition of broken
tests and level of tests.
folder structure or exclude files can't help in this complex situation,
so we need some tools
to help us to deal with this, i think TestNG is suitable. If JUnit 4.4
can do it, i will vote to JUnit,
update to a new version is always easier than switch to a new tool after
all.
[1] http://wiki.apache.org/harmony/Testing_Convention
Best Regards,
Regis.
>
> -Nathan
>
>
>>
>> Best Regards,
>> Regis.
>>
>>
>> Nathan Beyer wrote:
>>
>>> That discussion was a very long time ago. Is there still value in TestNG?
>>> I'd prefer to move to JUnit 4.4. All of our current tests will continue to
>>> work and new tests can be implemented using the latest conventions and
>>> older
>>> tests can be updated as we get to them. JUnit 4.4 is a far cry from 4.0.
>>>
>>> Here's the things I think would be create for our use and testing in
>>> general
>>> - Matchers and the 'assertThat' - much more readable code and readable
>>> failure messages
>>> - Assumptions and the 'assumeThat' - allows methods to add statements that
>>> guarantee that preconditions for the test are correct; this allows tests
>>> to
>>> fail such that you know it's an environment issue and not an actual test
>>> failure
>>>
>>> If you're not familiar with matchers, check out this quick tutorial -
>>> http://code.google.com/p/hamcrest/wiki/Tutorial.
>>>
>>> -Nathan
>>>
>>>
>>> On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <se...@gmail.com> wrote:
>>>
>>> Hi, all.
>>>> We had discussed the migration to testNG before and got some conclusions
>>>> for
>>>> grouping[1]
>>>> including how to deal with boot path test[2]. Am i missing something?
>>>> Is it still in our schedule? I think it's valueable to Harmony.
>>>> I volunteer to carry out this job if no one objects. Any other
>>>> volunteers?
>>>>
>>>> IMHO, we can only add some ant tasks to integrate testng at the
>>>> beginning.
>>>> So our original junit tests can still work at the mean time when
>>>> migrating.
>>>> When one module's migration task is finished, we can judge the result
>>>> to dertermine whether we should go on for other modules.
>>>>
>>>> Maybe we can create a branch for luni to start this work, shall we?
>>>> therefore there won't be any impact on other's development.
>>>> Once it is completed in the branch, we could merge it back to our trunk.
>>>> Does it make sense?
>>>>
>>>> Any sugestions or comments are welcomed. Thanks very much.
>>>>
>>>> [1] http://wiki.apache.org/harmony/Testing_Convention
>>>> [2]
>>>>
>>>> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
>>>> [3] http://testng.org/doc/documentation-main.html#annotations
>>>> --
>>>> Best Regards
>>>> Sean, Xiao Xia Qiu
>>>> China Software Development Lab, IBM
>>>>
>>>>
>
Re: [classlib][test] Migration to testNG?
Posted by Nathan Beyer <nb...@gmail.com>.
On Fri, Jun 6, 2008 at 1:08 AM, Regis <xu...@gmail.com> wrote:
> Hi,
>
> Matcher and Assumptions are great ideas! Thanks Nathan.
> They would be very helpful for our new test cases. But I notice that
> Junit 4.4 doesn't support group which is very important feature for
> both old tests and new tests. We can partition our test suite and run
> them separately. It's make our tests more flexible and configurable,
> and it's the main reason we discuss to migrate to TestNG long time ago.
Don't we partition our tests already? Isn't that what the 'api' and 'impl'
folders are about?
-Nathan
>
>
> Best Regards,
> Regis.
>
>
> Nathan Beyer wrote:
>
>> That discussion was a very long time ago. Is there still value in TestNG?
>> I'd prefer to move to JUnit 4.4. All of our current tests will continue to
>> work and new tests can be implemented using the latest conventions and
>> older
>> tests can be updated as we get to them. JUnit 4.4 is a far cry from 4.0.
>>
>> Here's the things I think would be create for our use and testing in
>> general
>> - Matchers and the 'assertThat' - much more readable code and readable
>> failure messages
>> - Assumptions and the 'assumeThat' - allows methods to add statements that
>> guarantee that preconditions for the test are correct; this allows tests
>> to
>> fail such that you know it's an environment issue and not an actual test
>> failure
>>
>> If you're not familiar with matchers, check out this quick tutorial -
>> http://code.google.com/p/hamcrest/wiki/Tutorial.
>>
>> -Nathan
>>
>>
>> On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <se...@gmail.com> wrote:
>>
>> Hi, all.
>>>
>>> We had discussed the migration to testNG before and got some conclusions
>>> for
>>> grouping[1]
>>> including how to deal with boot path test[2]. Am i missing something?
>>> Is it still in our schedule? I think it's valueable to Harmony.
>>> I volunteer to carry out this job if no one objects. Any other
>>> volunteers?
>>>
>>> IMHO, we can only add some ant tasks to integrate testng at the
>>> beginning.
>>> So our original junit tests can still work at the mean time when
>>> migrating.
>>> When one module's migration task is finished, we can judge the result
>>> to dertermine whether we should go on for other modules.
>>>
>>> Maybe we can create a branch for luni to start this work, shall we?
>>> therefore there won't be any impact on other's development.
>>> Once it is completed in the branch, we could merge it back to our trunk.
>>> Does it make sense?
>>>
>>> Any sugestions or comments are welcomed. Thanks very much.
>>>
>>> [1] http://wiki.apache.org/harmony/Testing_Convention
>>> [2]
>>>
>>> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
>>> [3] http://testng.org/doc/documentation-main.html#annotations
>>> --
>>> Best Regards
>>> Sean, Xiao Xia Qiu
>>> China Software Development Lab, IBM
>>>
>>>
>>
Re: [classlib][test] Migration to testNG?
Posted by Regis <xu...@gmail.com>.
Hi,
Matcher and Assumptions are great ideas! Thanks Nathan.
They would be very helpful for our new test cases. But I notice that
Junit 4.4 doesn't support group which is very important feature for
both old tests and new tests. We can partition our test suite and run
them separately. It's make our tests more flexible and configurable,
and it's the main reason we discuss to migrate to TestNG long time ago.
Best Regards,
Regis.
Nathan Beyer wrote:
> That discussion was a very long time ago. Is there still value in TestNG?
> I'd prefer to move to JUnit 4.4. All of our current tests will continue to
> work and new tests can be implemented using the latest conventions and older
> tests can be updated as we get to them. JUnit 4.4 is a far cry from 4.0.
>
> Here's the things I think would be create for our use and testing in general
> - Matchers and the 'assertThat' - much more readable code and readable
> failure messages
> - Assumptions and the 'assumeThat' - allows methods to add statements that
> guarantee that preconditions for the test are correct; this allows tests to
> fail such that you know it's an environment issue and not an actual test
> failure
>
> If you're not familiar with matchers, check out this quick tutorial -
> http://code.google.com/p/hamcrest/wiki/Tutorial.
>
> -Nathan
>
>
> On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <se...@gmail.com> wrote:
>
>> Hi, all.
>>
>> We had discussed the migration to testNG before and got some conclusions
>> for
>> grouping[1]
>> including how to deal with boot path test[2]. Am i missing something?
>> Is it still in our schedule? I think it's valueable to Harmony.
>> I volunteer to carry out this job if no one objects. Any other volunteers?
>>
>> IMHO, we can only add some ant tasks to integrate testng at the beginning.
>> So our original junit tests can still work at the mean time when migrating.
>> When one module's migration task is finished, we can judge the result
>> to dertermine whether we should go on for other modules.
>>
>> Maybe we can create a branch for luni to start this work, shall we?
>> therefore there won't be any impact on other's development.
>> Once it is completed in the branch, we could merge it back to our trunk.
>> Does it make sense?
>>
>> Any sugestions or comments are welcomed. Thanks very much.
>>
>> [1] http://wiki.apache.org/harmony/Testing_Convention
>> [2]
>> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
>> [3] http://testng.org/doc/documentation-main.html#annotations
>> --
>> Best Regards
>> Sean, Xiao Xia Qiu
>> China Software Development Lab, IBM
>>
>
Re: [classlib][test] Migration to testNG?
Posted by Nathan Beyer <nd...@apache.org>.
Again, I'll mention that it was a long time ago -- Jul 2006; two years ago.
There has been two years of work on building tests and TestNG wasn't that
compelling then and still isn't that compelling now. I've read many articles
and blog posts like the one you mentioned, but I've yet to actually seen any
project actually using TestNG. I certainly don't survey all open source
projects, but of the dozen or so that I actively observe they all use JUnit.
We're invested in JUnit and it seems the most pragmatic to just upgrade. In
fact, I think I'll try that; there shouldn't be anything holding us back
from upgrading the library and letting people try it out.
As for Hamcrest, yes it is an independent library, but JUnit 4.4 includes
direct support for it in the org.junit.Assert.assertThat and assumeThat
methods.
-Nathan
On Fri, Jun 6, 2008 at 1:04 AM, Sean Qiu <se...@gmail.com> wrote:
> We already have a impassioned debate for these two cadidate[1].
> IMHO, TestNG is more rounded and powful.
> It seems that TestNG has more features that Junit don't[2].
>
> For Hamcrest assert, it is interesting and should valueable to us.
> I think there is no doubt that we could include them with TestNG.
> It should be a independent library. (Corrent me if i was wrong)
>
> [1] http://markmail.org/message/w25yywlbjeicvhhe
> [2]
> http://lijinjoseji.wordpress.com/2008/02/29/testng-56-and-junit-44-which-framework-you-will-choose-for-unit-testing/
>
> 2008/6/6 Nathan Beyer <nd...@apache.org>:
> > That discussion was a very long time ago. Is there still value in TestNG?
> > I'd prefer to move to JUnit 4.4. All of our current tests will continue
> to
> > work and new tests can be implemented using the latest conventions and
> older
> > tests can be updated as we get to them. JUnit 4.4 is a far cry from 4.0.
> >
> > Here's the things I think would be create for our use and testing in
> general
> > - Matchers and the 'assertThat' - much more readable code and readable
> > failure messages
> > - Assumptions and the 'assumeThat' - allows methods to add statements
> that
> > guarantee that preconditions for the test are correct; this allows tests
> to
> > fail such that you know it's an environment issue and not an actual test
> > failure
> >
> > If you're not familiar with matchers, check out this quick tutorial -
> > http://code.google.com/p/hamcrest/wiki/Tutorial.
> >
> > -Nathan
> >
> >
> > On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <se...@gmail.com> wrote:
> >
> >> Hi, all.
> >>
> >> We had discussed the migration to testNG before and got some conclusions
> >> for
> >> grouping[1]
> >> including how to deal with boot path test[2]. Am i missing something?
> >> Is it still in our schedule? I think it's valueable to Harmony.
> >> I volunteer to carry out this job if no one objects. Any other
> volunteers?
> >>
> >> IMHO, we can only add some ant tasks to integrate testng at the
> beginning.
> >> So our original junit tests can still work at the mean time when
> migrating.
> >> When one module's migration task is finished, we can judge the result
> >> to dertermine whether we should go on for other modules.
> >>
> >> Maybe we can create a branch for luni to start this work, shall we?
> >> therefore there won't be any impact on other's development.
> >> Once it is completed in the branch, we could merge it back to our trunk.
> >> Does it make sense?
> >>
> >> Any sugestions or comments are welcomed. Thanks very much.
> >>
> >> [1] http://wiki.apache.org/harmony/Testing_Convention
> >> [2]
> >>
> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
> >> [3] http://testng.org/doc/documentation-main.html#annotations
> >> --
> >> Best Regards
> >> Sean, Xiao Xia Qiu
> >> China Software Development Lab, IBM
> >>
> >
>
>
>
> --
> Best Regards
> Sean, Xiao Xia Qiu
>
> China Software Development Lab, IBM
>
Re: [classlib][test] Migration to testNG?
Posted by Sean Qiu <se...@gmail.com>.
We already have a impassioned debate for these two cadidate[1].
IMHO, TestNG is more rounded and powful.
It seems that TestNG has more features that Junit don't[2].
For Hamcrest assert, it is interesting and should valueable to us.
I think there is no doubt that we could include them with TestNG.
It should be a independent library. (Corrent me if i was wrong)
[1] http://markmail.org/message/w25yywlbjeicvhhe
[2] http://lijinjoseji.wordpress.com/2008/02/29/testng-56-and-junit-44-which-framework-you-will-choose-for-unit-testing/
2008/6/6 Nathan Beyer <nd...@apache.org>:
> That discussion was a very long time ago. Is there still value in TestNG?
> I'd prefer to move to JUnit 4.4. All of our current tests will continue to
> work and new tests can be implemented using the latest conventions and older
> tests can be updated as we get to them. JUnit 4.4 is a far cry from 4.0.
>
> Here's the things I think would be create for our use and testing in general
> - Matchers and the 'assertThat' - much more readable code and readable
> failure messages
> - Assumptions and the 'assumeThat' - allows methods to add statements that
> guarantee that preconditions for the test are correct; this allows tests to
> fail such that you know it's an environment issue and not an actual test
> failure
>
> If you're not familiar with matchers, check out this quick tutorial -
> http://code.google.com/p/hamcrest/wiki/Tutorial.
>
> -Nathan
>
>
> On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <se...@gmail.com> wrote:
>
>> Hi, all.
>>
>> We had discussed the migration to testNG before and got some conclusions
>> for
>> grouping[1]
>> including how to deal with boot path test[2]. Am i missing something?
>> Is it still in our schedule? I think it's valueable to Harmony.
>> I volunteer to carry out this job if no one objects. Any other volunteers?
>>
>> IMHO, we can only add some ant tasks to integrate testng at the beginning.
>> So our original junit tests can still work at the mean time when migrating.
>> When one module's migration task is finished, we can judge the result
>> to dertermine whether we should go on for other modules.
>>
>> Maybe we can create a branch for luni to start this work, shall we?
>> therefore there won't be any impact on other's development.
>> Once it is completed in the branch, we could merge it back to our trunk.
>> Does it make sense?
>>
>> Any sugestions or comments are welcomed. Thanks very much.
>>
>> [1] http://wiki.apache.org/harmony/Testing_Convention
>> [2]
>> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
>> [3] http://testng.org/doc/documentation-main.html#annotations
>> --
>> Best Regards
>> Sean, Xiao Xia Qiu
>> China Software Development Lab, IBM
>>
>
--
Best Regards
Sean, Xiao Xia Qiu
China Software Development Lab, IBM
Re: [classlib][test] Migration to testNG?
Posted by Nathan Beyer <nd...@apache.org>.
That discussion was a very long time ago. Is there still value in TestNG?
I'd prefer to move to JUnit 4.4. All of our current tests will continue to
work and new tests can be implemented using the latest conventions and older
tests can be updated as we get to them. JUnit 4.4 is a far cry from 4.0.
Here's the things I think would be create for our use and testing in general
- Matchers and the 'assertThat' - much more readable code and readable
failure messages
- Assumptions and the 'assumeThat' - allows methods to add statements that
guarantee that preconditions for the test are correct; this allows tests to
fail such that you know it's an environment issue and not an actual test
failure
If you're not familiar with matchers, check out this quick tutorial -
http://code.google.com/p/hamcrest/wiki/Tutorial.
-Nathan
On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <se...@gmail.com> wrote:
> Hi, all.
>
> We had discussed the migration to testNG before and got some conclusions
> for
> grouping[1]
> including how to deal with boot path test[2]. Am i missing something?
> Is it still in our schedule? I think it's valueable to Harmony.
> I volunteer to carry out this job if no one objects. Any other volunteers?
>
> IMHO, we can only add some ant tasks to integrate testng at the beginning.
> So our original junit tests can still work at the mean time when migrating.
> When one module's migration task is finished, we can judge the result
> to dertermine whether we should go on for other modules.
>
> Maybe we can create a branch for luni to start this work, shall we?
> therefore there won't be any impact on other's development.
> Once it is completed in the branch, we could merge it back to our trunk.
> Does it make sense?
>
> Any sugestions or comments are welcomed. Thanks very much.
>
> [1] http://wiki.apache.org/harmony/Testing_Convention
> [2]
> http://www.mail-archive.com/harmony-dev@incubator.apache.org/msg12413.html
> [3] http://testng.org/doc/documentation-main.html#annotations
> --
> Best Regards
> Sean, Xiao Xia Qiu
> China Software Development Lab, IBM
>