You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Richard Liang <ri...@gmail.com> on 2006/08/25 17:17:49 UTC
[classlib][TestNG] groups of Harmony test
Hello All,
Now let's talk about the TestNG groups. I have read the related threads
which posted by George, Vladimir Ivanov and Alexei Zakharov. All of them
are good discussion about TestNG groups.
IMHO, we may define Harmony test groups according the following 4
dimensions:
1) [Platform] os.any, os.<platform id>
*os.any* - group of tests which pass on any platform. IMHO, most of our
tests should be in this group.
*os.<platform id>* - group of tests which are designed for one specific
platform. A test may be in more than one of the groups. e.g.,
@Test(groups={"os.win.IA32", "os.linux.IA32"})
** os.any and os.<platform id> are mutually exclusive, that is,
tests in os.any group should not be in os.win.IA32.
2) [Test state] state.broken, state.broken.<platform id>
*state.broken* - group of tests which fail on every platform, because
of bugs of tests or implementation. We need to fix the bugs of tests or
implementation to make them pass.
*state.broken.<platform id>* - groups of test which only fail on one
specific platform. A test may be in more than one of the groups. e.g.,
@Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
**state.broken.<platform id> group may be used as a convenient way
to indicate that a test is platform-specific. e.g., If we support 10
platforms, and one test are designed for 9 platforms except for MacOS,
instead of list 9 os.<platform id>, we can just use state.broken.MacOS
3) [Test type] type.api, type.impl
*type.api* - group of tests which are tests for APIs in the Java
Specification
*type.impl* - groups of tests which are tests for Harmony-specific
implementation
** type.api and type.impl are also mutually exclusive.
4) [Test Level] level.unit, level.integration, level.system,
level.stress, etc. (Levels of Test refer to the increase in complexity
as moving through test cycle. )
** A test may be in more than one of the groups.
** In fact, some tests such as System tests are the verification of
the entire system. Maybe we'll put them into a separate project. e.g.,
harmony/enhanced/SVT (System Verification Test).
If we want to run all the unit test for APIs on windows, we may use
TestNG groups to select the tests:
<groups>
<run>
<include name="os.any" />
<include name="type.api" />
<include name="os.win.IA32" />
<exclude name="state.broken" />
<exclude name="state.broken.win.IA32" />
</run>
</groups>
Well, I think our most of existing tests are in the groups of {"os.any",
"type.api", "level.unit"}, and I have asked TestNG to add a new option
"-groups" for its JUnitConverter which allow us to specify the test
groups when migrate from JUnit test to TestNG test.
Thanks for reading so far, and I will highly appreciate your comments or
suggestion. ;-)
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
Stepan Mishura wrote:
> On 8/25/06, Richard Liang wrote:
>>
>> Hello All,
>>
>> Now let's talk about the TestNG groups. I have read the related threads
>> which posted by George, Vladimir Ivanov and Alexei Zakharov. All of them
>> are good discussion about TestNG groups.
>>
>> IMHO, we may define Harmony test groups according the following 4
>> dimensions:
>>
>> 1) [Platform] os.any, os.<platform id>
>> *os.any* - group of tests which pass on any platform. IMHO, most of our
>> tests should be in this group.
>> *os.<platform id>* - group of tests which are designed for one specific
>> platform. A test may be in more than one of the groups. e.g.,
>> @Test(groups={"os.win.IA32", "os.linux.IA32"})
>>
>> ** os.any and os.<platform id> are mutually exclusive, that is,
>> tests in os.any group should not be in os.win.IA32.
>>
>> 2) [Test state] state.broken, state.broken.<platform id>
>> *state.broken* - group of tests which fail on every platform, because
>> of bugs of tests or implementation. We need to fix the bugs of tests or
>> implementation to make them pass.
>> *state.broken.<platform id>* - groups of test which only fail on one
>> specific platform. A test may be in more than one of the groups. e.g.,
>> @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>>
>> **state.broken.<platform id> group may be used as a convenient way
>> to indicate that a test is platform-specific. e.g., If we support 10
>> platforms, and one test are designed for 9 platforms except for MacOS,
>> instead of list 9 os.<platform id>, we can just use state.broken.MacOS
>
>
> If a test is marked as *state.broken.MacOS* then it sounds like the
> test/implementation should be fixed. IMO we should use tag
> os.<platform id>
> to define explicitly valid platforms for the test so in this
> particular case
> we should use 9 os.<platform id>s
>
Thank you, Stepan. This sounds reasonable, we should explicitly list
the valid platforms.
Best regards,
Richard
>
> 3) [Test type] type.api, type.impl
>> *type.api* - group of tests which are tests for APIs in the Java
>> Specification
>> *type.impl* - groups of tests which are tests for Harmony-specific
>> implementation
>>
>> ** type.api and type.impl are also mutually exclusive.
>>
>> 4) [Test Level] level.unit, level.integration, level.system,
>> level.stress, etc. (Levels of Test refer to the increase in complexity
>> as moving through test cycle. )
>> ** A test may be in more than one of the groups.
>> ** In fact, some tests such as System tests are the verification of
>> the entire system. Maybe we'll put them into a separate project. e.g.,
>> harmony/enhanced/SVT (System Verification Test).
>
>
> Mixing different types of testing into one test-file doesn't look good
> for
> me. I'd separate such tests by placing into different
> directories/packages.
>
> Thanks,
> Stepan.
>
> If we want to run all the unit test for APIs on windows, we may use
>> TestNG groups to select the tests:
>> <groups>
>> <run>
>> <include name="os.any" />
>> <include name="type.api" />
>> <include name="os.win.IA32" />
>> <exclude name="state.broken" />
>> <exclude name="state.broken.win.IA32" />
>> </run>
>> </groups>
>>
>>
>> Well, I think our most of existing tests are in the groups of {"os.any",
>> "type.api", "level.unit"}, and I have asked TestNG to add a new option
>> "-groups" for its JUnitConverter which allow us to specify the test
>> groups when migrate from JUnit test to TestNG test.
>>
>> Thanks for reading so far, and I will highly appreciate your comments or
>> suggestion. ;-)
>>
>> --
>> Richard Liang
>> China Software Development Lab, IBM
>>
>>
>>
>
> ------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Stepan Mishura <st...@gmail.com>.
On 8/25/06, Richard Liang wrote:
>
> Hello All,
>
> Now let's talk about the TestNG groups. I have read the related threads
> which posted by George, Vladimir Ivanov and Alexei Zakharov. All of them
> are good discussion about TestNG groups.
>
> IMHO, we may define Harmony test groups according the following 4
> dimensions:
>
> 1) [Platform] os.any, os.<platform id>
> *os.any* - group of tests which pass on any platform. IMHO, most of our
> tests should be in this group.
> *os.<platform id>* - group of tests which are designed for one specific
> platform. A test may be in more than one of the groups. e.g.,
> @Test(groups={"os.win.IA32", "os.linux.IA32"})
>
> ** os.any and os.<platform id> are mutually exclusive, that is,
> tests in os.any group should not be in os.win.IA32.
>
> 2) [Test state] state.broken, state.broken.<platform id>
> *state.broken* - group of tests which fail on every platform, because
> of bugs of tests or implementation. We need to fix the bugs of tests or
> implementation to make them pass.
> *state.broken.<platform id>* - groups of test which only fail on one
> specific platform. A test may be in more than one of the groups. e.g.,
> @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>
> **state.broken.<platform id> group may be used as a convenient way
> to indicate that a test is platform-specific. e.g., If we support 10
> platforms, and one test are designed for 9 platforms except for MacOS,
> instead of list 9 os.<platform id>, we can just use state.broken.MacOS
If a test is marked as *state.broken.MacOS* then it sounds like the
test/implementation should be fixed. IMO we should use tag os.<platform id>
to define explicitly valid platforms for the test so in this particular case
we should use 9 os.<platform id>s
3) [Test type] type.api, type.impl
> *type.api* - group of tests which are tests for APIs in the Java
> Specification
> *type.impl* - groups of tests which are tests for Harmony-specific
> implementation
>
> ** type.api and type.impl are also mutually exclusive.
>
> 4) [Test Level] level.unit, level.integration, level.system,
> level.stress, etc. (Levels of Test refer to the increase in complexity
> as moving through test cycle. )
> ** A test may be in more than one of the groups.
> ** In fact, some tests such as System tests are the verification of
> the entire system. Maybe we'll put them into a separate project. e.g.,
> harmony/enhanced/SVT (System Verification Test).
Mixing different types of testing into one test-file doesn't look good for
me. I'd separate such tests by placing into different directories/packages.
Thanks,
Stepan.
If we want to run all the unit test for APIs on windows, we may use
> TestNG groups to select the tests:
> <groups>
> <run>
> <include name="os.any" />
> <include name="type.api" />
> <include name="os.win.IA32" />
> <exclude name="state.broken" />
> <exclude name="state.broken.win.IA32" />
> </run>
> </groups>
>
>
> Well, I think our most of existing tests are in the groups of {"os.any",
> "type.api", "level.unit"}, and I have asked TestNG to add a new option
> "-groups" for its JUnitConverter which allow us to specify the test
> groups when migrate from JUnit test to TestNG test.
>
> Thanks for reading so far, and I will highly appreciate your comments or
> suggestion. ;-)
>
> --
> Richard Liang
> China Software Development Lab, IBM
>
>
>
------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
Paulex Yang wrote:
> Richard Liang wrote:
>> Hello All,
>>
>> Now let's talk about the TestNG groups. I have read the related
>> threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
>> All of them are good discussion about TestNG groups.
>>
>> IMHO, we may define Harmony test groups according the following 4
>> dimensions:
>>
>> 1) [Platform] os.any, os.<platform id>
>> *os.any* - group of tests which pass on any platform. IMHO, most of
>> our tests should be in this group.
>> *os.<platform id>* - group of tests which are designed for one
>> specific platform. A test may be in more than one of the groups.
>> e.g., @Test(groups={"os.win.IA32", "os.linux.IA32"})
>>
>> ** os.any and os.<platform id> are mutually exclusive, that is,
>> tests in os.any group should not be in os.win.IA32.
> I suggest to separate the CPU arch with OS, they are actually
> different things, and we probably will have tests written for all
> 64bits arch, or for all Linux OS.
Thanks a lot, Paulex. I agree. Separating os and arch make our groups
more flexible. I will update the grouping proposal soon to integrate
your and others' suggestion.
Best regards,
Richard
>>
>> 2) [Test state] state.broken, state.broken.<platform id>
>> *state.broken* - group of tests which fail on every platform, because
>> of bugs of tests or implementation. We need to fix the bugs of tests
>> or implementation to make them pass.
>> *state.broken.<platform id>* - groups of test which only fail on one
>> specific platform. A test may be in more than one of the groups.
>> e.g., @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>>
>> **state.broken.<platform id> group may be used as a convenient
>> way to indicate that a test is platform-specific. e.g., If we support
>> 10 platforms, and one test are designed for 9 platforms except for
>> MacOS, instead of list 9 os.<platform id>, we can just use
>> state.broken.MacOS
>>
>> 3) [Test type] type.api, type.impl
>> *type.api* - group of tests which are tests for APIs in the Java
>> Specification
>> *type.impl* - groups of tests which are tests for Harmony-specific
>> implementation
>>
>> ** type.api and type.impl are also mutually exclusive.
>>
>> 4) [Test Level] level.unit, level.integration, level.system,
>> level.stress, etc. (Levels of Test refer to the increase in
>> complexity as moving through test cycle. )
>> ** A test may be in more than one of the groups.
>> ** In fact, some tests such as System tests are the verification
>> of the entire system. Maybe we'll put them into a separate project.
>> e.g., harmony/enhanced/SVT (System Verification Test).
>>
>> If we want to run all the unit test for APIs on windows, we may use
>> TestNG groups to select the tests:
>> <groups>
>> <run>
>> <include name="os.any" />
>> <include name="type.api" />
>> <include name="os.win.IA32" />
>> <exclude name="state.broken" />
>> <exclude name="state.broken.win.IA32" />
>> </run>
>> </groups>
>>
>>
>> Well, I think our most of existing tests are in the groups of
>> {"os.any", "type.api", "level.unit"}, and I have asked TestNG to add
>> a new option "-groups" for its JUnitConverter which allow us to
>> specify the test groups when migrate from JUnit test to TestNG test.
>>
>> Thanks for reading so far, and I will highly appreciate your comments
>> or suggestion. ;-)
>>
>
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Paulex Yang <pa...@gmail.com>.
Richard Liang wrote:
> Hello All,
>
> Now let's talk about the TestNG groups. I have read the related
> threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
> All of them are good discussion about TestNG groups.
>
> IMHO, we may define Harmony test groups according the following 4
> dimensions:
>
> 1) [Platform] os.any, os.<platform id>
> *os.any* - group of tests which pass on any platform. IMHO, most of
> our tests should be in this group.
> *os.<platform id>* - group of tests which are designed for one
> specific platform. A test may be in more than one of the groups. e.g.,
> @Test(groups={"os.win.IA32", "os.linux.IA32"})
>
> ** os.any and os.<platform id> are mutually exclusive, that is,
> tests in os.any group should not be in os.win.IA32.
I suggest to separate the CPU arch with OS, they are actually different
things, and we probably will have tests written for all 64bits arch, or
for all Linux OS.
>
> 2) [Test state] state.broken, state.broken.<platform id>
> *state.broken* - group of tests which fail on every platform, because
> of bugs of tests or implementation. We need to fix the bugs of tests
> or implementation to make them pass.
> *state.broken.<platform id>* - groups of test which only fail on one
> specific platform. A test may be in more than one of the groups. e.g.,
> @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>
> **state.broken.<platform id> group may be used as a convenient way
> to indicate that a test is platform-specific. e.g., If we support 10
> platforms, and one test are designed for 9 platforms except for MacOS,
> instead of list 9 os.<platform id>, we can just use state.broken.MacOS
>
> 3) [Test type] type.api, type.impl
> *type.api* - group of tests which are tests for APIs in the Java
> Specification
> *type.impl* - groups of tests which are tests for Harmony-specific
> implementation
>
> ** type.api and type.impl are also mutually exclusive.
>
> 4) [Test Level] level.unit, level.integration, level.system,
> level.stress, etc. (Levels of Test refer to the increase in complexity
> as moving through test cycle. )
> ** A test may be in more than one of the groups.
> ** In fact, some tests such as System tests are the verification of
> the entire system. Maybe we'll put them into a separate project.
> e.g., harmony/enhanced/SVT (System Verification Test).
>
> If we want to run all the unit test for APIs on windows, we may use
> TestNG groups to select the tests:
> <groups>
> <run>
> <include name="os.any" />
> <include name="type.api" />
> <include name="os.win.IA32" />
> <exclude name="state.broken" />
> <exclude name="state.broken.win.IA32" />
> </run>
> </groups>
>
>
> Well, I think our most of existing tests are in the groups of
> {"os.any", "type.api", "level.unit"}, and I have asked TestNG to add a
> new option "-groups" for its JUnitConverter which allow us to specify
> the test groups when migrate from JUnit test to TestNG test.
>
> Thanks for reading so far, and I will highly appreciate your comments
> or suggestion. ;-)
>
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
Richard Liang wrote:
>
>
> Tony Wu wrote:
>> I think we can treat something like *os.any*,*type.api* as default
>> groups.Itwill make our logic more clear and bring convenience to
>> testcase writing :)
>> for example, an os.<platformid> group indicates that it is designed for
>> <paltformid> only. If there is no group like os.<platformid> here,
>> this is a
>> *os.any* test.
>>
>> so if we want to run all *api* tests on *win32* platform which is *not
>> broken*, we could write the testng.xml like this:
>> <groups>
>> <run>
>> <include name=".*" />
>> <exclude name="type.impl" />
>> <exclude name="os\.(?!win\.IA32).*" />
>> <exclude name="state.broken" />
>> <exclude name="state.broken.win.IA32" />
>> </run>
>> </groups>
>>
> Hello Tony,
>
> It's a good idea. However, we shall define a "default" group which
> means "os.any, type.impl, level.unit". Thanks a lot.
>
And even the "default" group is not necessary by removing the "<include
name=".*" /> "
<groups>
<run>
<exclude name="type.impl" />
<exclude name="os\.(?!win\.IA32).*" />
<exclude name="state.broken" />
<exclude name="state.broken.win.IA32" />
</run>
</groups>
Richard
> Best regards,
> Richard
>
>>
>> 2006/8/29, Vladimir Ivanov <iv...@gmail.com>:
>>>
>>> Also some tag for regression tests should be added.
>>> thanks, Vladimir
>>>
>>>
>>> On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
>>> >
>>> >
>>> >
>>> > Richard Liang wrote:
>>> > > Hello All,
>>> > >
>>> > > Now let's talk about the TestNG groups. I have read the related
>>> > > threads which posted by George, Vladimir Ivanov and Alexei
>>> Zakharov.
>>> > > All of them are good discussion about TestNG groups.
>>> > >
>>> > > IMHO, we may define Harmony test groups according the following 4
>>> > > dimensions:
>>> > >
>>> > > 1) [Platform] os.any, os.<platform id>
>>> > > *os.any* - group of tests which pass on any platform. IMHO, most of
>>> > > our tests should be in this group.
>>> > > *os.<platform id>* - group of tests which are designed for one
>>> > > specific platform. A test may be in more than one of the groups.
>>> e.g.,
>>> > > @Test(groups={"os.win.IA32", "os.linux.IA32"})
>>> > >
>>> > > ** os.any and os.<platform id> are mutually exclusive, that is,
>>> > > tests in os.any group should not be in os.win.IA32.
>>> > >
>>> > > 2) [Test state] state.broken, state.broken.<platform id>
>>> > > *state.broken* - group of tests which fail on every platform,
>>> because
>>> > > of bugs of tests or implementation. We need to fix the bugs of
>>> tests
>>> > > or implementation to make them pass.
>>> > > *state.broken.<platform id>* - groups of test which only fail on
>>> one
>>> > > specific platform. A test may be in more than one of the groups.
>>> e.g.,
>>> > > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>>> > >
>>> > > **state.broken.<platform id> group may be used as a
>>> convenient way
>>> > > to indicate that a test is platform-specific. e.g., If we
>>> support 10
>>> > > platforms, and one test are designed for 9 platforms except for
>>> MacOS,
>>> > > instead of list 9 os.<platform id>, we can just use
>>> state.broken.MacOS
>>> > >
>>> > > 3) [Test type] type.api , type.impl
>>> > > *type.api* - group of tests which are tests for APIs in the Java
>>> > > Specification
>>> > > *type.impl* - groups of tests which are tests for Harmony-specific
>>> > > implementation
>>> > >
>>> > > ** type.api and type.impl are also mutually exclusive.
>>> > >
>>> > > 4) [Test Level] level.unit, level.integration, level.system,
>>> > > level.stress, etc. (Levels of Test refer to the increase in
>>> complexity
>>> > > as moving through test cycle. )
>>> > > ** A test may be in more than one of the groups.
>>> > > ** In fact, some tests such as System tests are the
>>> verification of
>>> > > the entire system. Maybe we'll put them into a separate project.
>>> > > e.g., harmony/enhanced/SVT (System Verification Test).
>>> > >
>>> > > If we want to run all the unit test for APIs on windows, we may use
>>> > > TestNG groups to select the tests:
>>> > > <groups>
>>> > > <run>
>>> > > <include name="os.any" />
>>> > > <include name="type.api" />
>>> > > <include name="os.win.IA32" />
>>> > > <exclude name=" state.broken" />
>>> > > <exclude name="state.broken.win.IA32" />
>>> > > </run>
>>> > > </groups>
>>> > >
>>> > Hello All,
>>> >
>>> > I'm sorry. It seems that the example does not work. I will try to
>>> figure
>>> > another example soon. ;-)
>>> >
>>> > Best regards,
>>> > Richard
>>> > >
>>> > > Well, I think our most of existing tests are in the groups of
>>> > > {"os.any", "type.api", "level.unit"}, and I have asked TestNG to
>>> add a
>>> > > new option "-groups" for its JUnitConverter which allow us to
>>> specify
>>> > > the test groups when migrate from JUnit test to TestNG test.
>>> > >
>>> > > Thanks for reading so far, and I will highly appreciate your
>>> comments
>>> > > or suggestion. ;-)
>>> > >
>>> >
>>> > --
>>> > Richard Liang
>>> > China Software Development Lab, IBM
>>> >
>>> >
>>> >
>>> > ---------------------------------------------------------------------
>>> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>>> > For additional commands, e-mail:
>>> harmony-dev-help@incubator.apache.org
>>> >
>>> >
>>>
>>>
>>
>>
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
Tony Wu wrote:
> I think we can treat something like *os.any*,*type.api* as default
> groups.Itwill make our logic more clear and bring convenience to
> testcase writing :)
> for example, an os.<platformid> group indicates that it is designed for
> <paltformid> only. If there is no group like os.<platformid> here,
> this is a
> *os.any* test.
>
> so if we want to run all *api* tests on *win32* platform which is *not
> broken*, we could write the testng.xml like this:
> <groups>
> <run>
> <include name=".*" />
> <exclude name="type.impl" />
> <exclude name="os\.(?!win\.IA32).*" />
> <exclude name="state.broken" />
> <exclude name="state.broken.win.IA32" />
> </run>
> </groups>
>
Hello Tony,
It's a good idea. However, we shall define a "default" group which means
"os.any, type.impl, level.unit". Thanks a lot.
Best regards,
Richard
>
> 2006/8/29, Vladimir Ivanov <iv...@gmail.com>:
>>
>> Also some tag for regression tests should be added.
>> thanks, Vladimir
>>
>>
>> On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
>> >
>> >
>> >
>> > Richard Liang wrote:
>> > > Hello All,
>> > >
>> > > Now let's talk about the TestNG groups. I have read the related
>> > > threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
>> > > All of them are good discussion about TestNG groups.
>> > >
>> > > IMHO, we may define Harmony test groups according the following 4
>> > > dimensions:
>> > >
>> > > 1) [Platform] os.any, os.<platform id>
>> > > *os.any* - group of tests which pass on any platform. IMHO, most of
>> > > our tests should be in this group.
>> > > *os.<platform id>* - group of tests which are designed for one
>> > > specific platform. A test may be in more than one of the groups.
>> e.g.,
>> > > @Test(groups={"os.win.IA32", "os.linux.IA32"})
>> > >
>> > > ** os.any and os.<platform id> are mutually exclusive, that is,
>> > > tests in os.any group should not be in os.win.IA32.
>> > >
>> > > 2) [Test state] state.broken, state.broken.<platform id>
>> > > *state.broken* - group of tests which fail on every platform,
>> because
>> > > of bugs of tests or implementation. We need to fix the bugs of tests
>> > > or implementation to make them pass.
>> > > *state.broken.<platform id>* - groups of test which only fail on one
>> > > specific platform. A test may be in more than one of the groups.
>> e.g.,
>> > > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>> > >
>> > > **state.broken.<platform id> group may be used as a
>> convenient way
>> > > to indicate that a test is platform-specific. e.g., If we support 10
>> > > platforms, and one test are designed for 9 platforms except for
>> MacOS,
>> > > instead of list 9 os.<platform id>, we can just use
>> state.broken.MacOS
>> > >
>> > > 3) [Test type] type.api , type.impl
>> > > *type.api* - group of tests which are tests for APIs in the Java
>> > > Specification
>> > > *type.impl* - groups of tests which are tests for Harmony-specific
>> > > implementation
>> > >
>> > > ** type.api and type.impl are also mutually exclusive.
>> > >
>> > > 4) [Test Level] level.unit, level.integration, level.system,
>> > > level.stress, etc. (Levels of Test refer to the increase in
>> complexity
>> > > as moving through test cycle. )
>> > > ** A test may be in more than one of the groups.
>> > > ** In fact, some tests such as System tests are the
>> verification of
>> > > the entire system. Maybe we'll put them into a separate project.
>> > > e.g., harmony/enhanced/SVT (System Verification Test).
>> > >
>> > > If we want to run all the unit test for APIs on windows, we may use
>> > > TestNG groups to select the tests:
>> > > <groups>
>> > > <run>
>> > > <include name="os.any" />
>> > > <include name="type.api" />
>> > > <include name="os.win.IA32" />
>> > > <exclude name=" state.broken" />
>> > > <exclude name="state.broken.win.IA32" />
>> > > </run>
>> > > </groups>
>> > >
>> > Hello All,
>> >
>> > I'm sorry. It seems that the example does not work. I will try to
>> figure
>> > another example soon. ;-)
>> >
>> > Best regards,
>> > Richard
>> > >
>> > > Well, I think our most of existing tests are in the groups of
>> > > {"os.any", "type.api", "level.unit"}, and I have asked TestNG to
>> add a
>> > > new option "-groups" for its JUnitConverter which allow us to
>> specify
>> > > the test groups when migrate from JUnit test to TestNG test.
>> > >
>> > > Thanks for reading so far, and I will highly appreciate your
>> comments
>> > > or suggestion. ;-)
>> > >
>> >
>> > --
>> > Richard Liang
>> > China Software Development Lab, IBM
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>> >
>> >
>>
>>
>
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Tony Wu <wu...@gmail.com>.
I think we can treat something like *os.any*,*type.api* as default
groups.Itwill make our logic more clear and bring convenience to
testcase writing :)
for example, an os.<platformid> group indicates that it is designed for
<paltformid> only. If there is no group like os.<platformid> here, this is a
*os.any* test.
so if we want to run all *api* tests on *win32* platform which is *not
broken*, we could write the testng.xml like this:
<groups>
<run>
<include name=".*" />
<exclude name="type.impl" />
<exclude name="os\.(?!win\.IA32).*" />
<exclude name="state.broken" />
<exclude name="state.broken.win.IA32" />
</run>
</groups>
2006/8/29, Vladimir Ivanov <iv...@gmail.com>:
>
> Also some tag for regression tests should be added.
> thanks, Vladimir
>
>
> On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
> >
> >
> >
> > Richard Liang wrote:
> > > Hello All,
> > >
> > > Now let's talk about the TestNG groups. I have read the related
> > > threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
> > > All of them are good discussion about TestNG groups.
> > >
> > > IMHO, we may define Harmony test groups according the following 4
> > > dimensions:
> > >
> > > 1) [Platform] os.any, os.<platform id>
> > > *os.any* - group of tests which pass on any platform. IMHO, most of
> > > our tests should be in this group.
> > > *os.<platform id>* - group of tests which are designed for one
> > > specific platform. A test may be in more than one of the groups. e.g.,
> > > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> > >
> > > ** os.any and os.<platform id> are mutually exclusive, that is,
> > > tests in os.any group should not be in os.win.IA32.
> > >
> > > 2) [Test state] state.broken, state.broken.<platform id>
> > > *state.broken* - group of tests which fail on every platform, because
> > > of bugs of tests or implementation. We need to fix the bugs of tests
> > > or implementation to make them pass.
> > > *state.broken.<platform id>* - groups of test which only fail on one
> > > specific platform. A test may be in more than one of the groups. e.g.,
> > > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
> > >
> > > **state.broken.<platform id> group may be used as a convenient way
> > > to indicate that a test is platform-specific. e.g., If we support 10
> > > platforms, and one test are designed for 9 platforms except for MacOS,
> > > instead of list 9 os.<platform id>, we can just use state.broken.MacOS
> > >
> > > 3) [Test type] type.api , type.impl
> > > *type.api* - group of tests which are tests for APIs in the Java
> > > Specification
> > > *type.impl* - groups of tests which are tests for Harmony-specific
> > > implementation
> > >
> > > ** type.api and type.impl are also mutually exclusive.
> > >
> > > 4) [Test Level] level.unit, level.integration, level.system,
> > > level.stress, etc. (Levels of Test refer to the increase in complexity
> > > as moving through test cycle. )
> > > ** A test may be in more than one of the groups.
> > > ** In fact, some tests such as System tests are the verification of
> > > the entire system. Maybe we'll put them into a separate project.
> > > e.g., harmony/enhanced/SVT (System Verification Test).
> > >
> > > If we want to run all the unit test for APIs on windows, we may use
> > > TestNG groups to select the tests:
> > > <groups>
> > > <run>
> > > <include name="os.any" />
> > > <include name="type.api" />
> > > <include name="os.win.IA32" />
> > > <exclude name=" state.broken" />
> > > <exclude name="state.broken.win.IA32" />
> > > </run>
> > > </groups>
> > >
> > Hello All,
> >
> > I'm sorry. It seems that the example does not work. I will try to figure
> > another example soon. ;-)
> >
> > Best regards,
> > Richard
> > >
> > > Well, I think our most of existing tests are in the groups of
> > > {"os.any", "type.api", "level.unit"}, and I have asked TestNG to add a
> > > new option "-groups" for its JUnitConverter which allow us to specify
> > > the test groups when migrate from JUnit test to TestNG test.
> > >
> > > Thanks for reading so far, and I will highly appreciate your comments
> > > or suggestion. ;-)
> > >
> >
> > --
> > Richard Liang
> > China Software Development Lab, IBM
> >
> >
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>
--
Tony Wu
China Software Development Lab, IBM
Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Mikhail Loenko <ml...@gmail.com>.
+1
2006/9/4, Alex Blewitt <al...@gmail.com>:
> IMNSO it doesn't make sense to arbitrarily partition the tests based
> on a moniker, such as 'integration test', 'unit test', 'regression
> test' etc. For one thing, developers are generally not good at
> agreeing on the difference between them :-)
>
> If you've got fast and slow tests, then have a group for fast and slow
> tests. Then you can choose to just run the fast tests, and any
> automated build system can handle running the slow tests.
>
> Alex.
>
> On 04/09/06, Vladimir Ivanov <iv...@gmail.com> wrote:
> > On 9/4/06, Richard Liang <ri...@gmail.com> wrote:
> > >
> > >
> > >
> > > Mikhail Loenko wrote:
> > > > Hi Vladimir
> > > >
> > > > Could you please decribe for what purpose it will be used?
> > > >
> > > > I mean why one might have to either exclude or run only regression
> > > tests?
> > >
> > > If running all tests takes up much time, running all regression test
> > > (for one particular version) may be a convenient way to verify the
> > > correctness of bug-fixing.
> >
> >
> > Thanks Richard. It is exactly what I want to say :)
> > On the other hand may be all proposed groups need similar explanation (smt.
> > like use-case for group)?
> >
> > thanks, Vladimir
> >
> > Best regards,
> > > Richard.
> > >
> > > >
> > > > Thanks,
> > > > Mikhail
> > > >
> > > > 2006/8/30, Vladimir Ivanov <iv...@gmail.com>:
> > > >> On 8/30/06, Richard Liang <ri...@gmail.com> wrote:
> > > >> >
> > > >> >
> > > >> >
> > > >> > Vladimir Ivanov wrote:
> > > >> > > Also some tag for regression tests should be added.
> > > >> > Yes. Do you think we could annotate regression test as
> > > >> > *level.regression*? Thanks a lot.
> > > >>
> > > >>
> > > >> Yes, I do. While tests can have more than one group it will enough.
> > > >> thanks, Vladimir
> > > >>
> > > >>
> > > >> Richard
> > > >> > > thanks, Vladimir
> > > >> > >
> > > >> > >
> > > >> > > On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >> Richard Liang wrote:
> > > >> > >> > Hello All,
> > > >> > >> >
> > > >> > >> > Now let's talk about the TestNG groups. I have read the related
> > > >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
> > > >> Zakharov.
> > > >> > >> > All of them are good discussion about TestNG groups.
> > > >> > >> >
> > > >> > >> > IMHO, we may define Harmony test groups according the following
> > > 4
> > > >> > >> > dimensions:
> > > >> > >> >
> > > >> > >> > 1) [Platform] os.any, os.<platform id>
> > > >> > >> > *os.any* - group of tests which pass on any platform. IMHO,
> > > >> most of
> > > >> > >> > our tests should be in this group.
> > > >> > >> > *os.<platform id>* - group of tests which are designed for one
> > > >> > >> > specific platform. A test may be in more than one of the
> > > >> groups. e.g
> > > >> > .,
> > > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> > > >> > >> >
> > > >> > >> > ** os.any and os.<platform id> are mutually exclusive, that
> > > >> is,
> > > >> > >> > tests in os.any group should not be in os.win.IA32.
> > > >> > >> >
> > > >> > >> > 2) [Test state] state.broken, state.broken.<platform id>
> > > >> > >> > *state.broken* - group of tests which fail on every platform,
> > > >> because
> > > >> > >> > of bugs of tests or implementation. We need to fix the bugs of
> > > >> tests
> > > >> > >> > or implementation to make them pass.
> > > >> > >> > *state.broken.<platform id>* - groups of test which only fail
> > > >> on one
> > > >> > >> > specific platform. A test may be in more than one of the
> > > >> groups. e.g
> > > >> > .,
> > > >> > >> > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64
> > > "})
> > > >> > >> >
> > > >> > >> > **state.broken.<platform id> group may be used as a
> > > >> convenient
> > > >> > way
> > > >> > >> > to indicate that a test is platform-specific. e.g., If we
> > > >> support 10
> > > >> > >> > platforms, and one test are designed for 9 platforms except for
> > > >> > MacOS,
> > > >> > >> > instead of list 9 os.<platform id>, we can just use
> > > >> > state.broken.MacOS
> > > >> > >> >
> > > >> > >> > 3) [Test type] type.api , type.impl
> > > >> > >> > *type.api* - group of tests which are tests for APIs in the Java
> > > >> > >> > Specification
> > > >> > >> > *type.impl* - groups of tests which are tests for
> > > >> Harmony-specific
> > > >> > >> > implementation
> > > >> > >> >
> > > >> > >> > ** type.api and type.impl are also mutually exclusive.
> > > >> > >> >
> > > >> > >> > 4) [Test Level] level.unit, level.integration, level.system,
> > > >> > >> > level.stress, etc. (Levels of Test refer to the increase in
> > > >> > complexity
> > > >> > >> > as moving through test cycle. )
> > > >> > >> > ** A test may be in more than one of the groups.
> > > >> > >> > ** In fact, some tests such as System tests are the
> > > >> verification
> > > >> > of
> > > >> > >> > the entire system. Maybe we'll put them into a separate
> > > project.
> > > >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
> > > >> > >> >
> > > >> > >> > If we want to run all the unit test for APIs on windows, we
> > > >> may use
> > > >> > >> > TestNG groups to select the tests:
> > > >> > >> > <groups>
> > > >> > >> > <run>
> > > >> > >> > <include name="os.any" />
> > > >> > >> > <include name="type.api" />
> > > >> > >> > <include name="os.win.IA32" />
> > > >> > >> > <exclude name=" state.broken" />
> > > >> > >> > <exclude name="state.broken.win.IA32" />
> > > >> > >> > </run>
> > > >> > >> > </groups>
> > > >> > >> >
> > > >> > >> Hello All,
> > > >> > >>
> > > >> > >> I'm sorry. It seems that the example does not work. I will try to
> > > >> > figure
> > > >> > >> another example soon. ;-)
> > > >> > >>
> > > >> > >> Best regards,
> > > >> > >> Richard
> > > >> > >> >
> > > >> > >> > Well, I think our most of existing tests are in the groups of
> > > >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked TestNG
> > > >> to add
> > > >> > a
> > > >> > >> > new option "-groups" for its JUnitConverter which allow us to
> > > >> specify
> > > >> > >> > the test groups when migrate from JUnit test to TestNG test.
> > > >> > >> >
> > > >> > >> > Thanks for reading so far, and I will highly appreciate your
> > > >> comments
> > > >> > >> > or suggestion. ;-)
> > > >> > >> >
> > > >> > >>
> > > >> > >> --
> > > >> > >> Richard Liang
> > > >> > >> China Software Development Lab, IBM
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> ---------------------------------------------------------------------
> > > >> > >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > >> > >> To unsubscribe, e-mail:
> > > >> harmony-dev-unsubscribe@incubator.apache.org
> > > >> > >> For additional commands, e-mail:
> > > >> harmony-dev-help@incubator.apache.org
> > > >> > >>
> > > >> > >>
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > Richard Liang
> > > >> > China Software Development Lab, IBM
> > > >> >
> > > >> >
> > > >> >
> > > >> > ---------------------------------------------------------------------
> > > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > >> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > >> > For additional commands, e-mail:
> > > harmony-dev-help@incubator.apache.org
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >
> > > > ---------------------------------------------------------------------
> > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > > >
> > > >
> > >
> > > --
> > > Richard Liang
> > > China Software Development Lab, IBM
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >
> > >
> >
> >
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Andrew Zhang <zh...@gmail.com>.
On 9/11/06, Richard Liang <ri...@gmail.com> wrote:
>
> On 9/11/06, Alexei Zakharov <al...@gmail.com> wrote:
> > Hi all,
> >
> > > One more note (seems it already was said sorry if I repeat): the test
> > > without any marks should be run in all configurations (i.e. we have
> > > 'default' group but declaration of this group may be missed).
> >
> > I'd like to point your attention on the previous discussion about
> > "default groups" :
> >
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44C7457B.7080107@googlemail.com%3e
> >
> > I am still for using "os.any" since it is more self-descriptive and
> > the build script will be simpler with "os.any". It will be nice to
> > hear more arguments for using defaults because it seems the arguments
> > that were gathered in that old thread hasn't been taken into account
> > by participants of this thread.
>
> I have not any strong objection about "os.any". And actually I had
> ever proposed to define the "default" group because we could not
> include tests with annotation @Test which belong to no groups. Now it
> isn't a problem as we already have a solution for them. To facilitate
> writing test cases, we annotate the unit tests which are designed to
> pass on all platforms (os + arch) with @Test.
>
> If we use "os.any" and "arch.any", then the default annotation would
> be @Test(groups={"os.any", "arch.any"})
>
> Could any other give more comments? Thanks a lot.
Either is ok. One is more descriptive while the other is more convenient.
If we have no problem to write test.xml with default group(@Test), then I
prefer this option a little. Thanks!
Best regards,
> Richard
>
>
>
> >
> > Thanks,
> >
> > 2006/9/5, Vladimir Ivanov <iv...@gmail.com>:
> > > One more note (seems it already was said sorry if I repeat): the test
> > > without any marks should be run in all configurations (i.e. we have
> > > 'default' group but declaration of this group may be missed).
> > >
> > > thanks, Vladimir
> > >
> > >
> > > On 9/5/06, Vladimir Ivanov <iv...@gmail.com> wrote:
> > > >
> > > > OK, let's return back to the usage model.
> > > > If I understood it correctly, before the commit of any changes each
> > > > developer run *all* tests (at least all which we have now) on all
> available
> > > > to him platforms. In this context seems we don't need in any 'level'
> group
> > > > (while 'stress' tests require reasonable time to pass).
> > > > Seems, that "platform" group also can be deleted (at present time we
> have
> > > > <10 platform-dependent tests and this amount should not increase
> > > > dramatically so the platform-detection can be included to the each
> such
> > > > test).
> > > > Also "cpu" groups can be deleted (while we have not cpu-dependent
> test).
> > > > At the end we need only "state" groups to support test exclusion on
> the
> > > > 'one-element' level (while we have unresolved entries in the current
> exclude
> > > > list).
> > > >
> > > > So, after small update of unit (aka integration, aka regression etc)
> tests
> > > > and resolution of all entries in the exclude list we don't need any
> groups
> > > > and pure JUnit covers all our needs :)
> > > >
> > > > On the other side, if we define some groups it will nice to define
> *all*
> > > > reasonable groups at the begin of the process.
> > > >
> > > > thanks, Vladimir
> > > >
> > > > By the way, our regression tests are 'classic' regression tests that
> > > > demonstrate some issues which were not resolved by initial code. But
> it
> > > > provides less coverage than 'regression tests' + unit tests, of
> cause.
> > > >
> > > > On 9/5/06, Richard Liang <richard.liangyx@gmail.com > wrote:
> > > > >
> > > > > On 9/5/06, Alex Blewitt < alex.blewitt@gmail.com> wrote:
> > > > > > On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> > > > > > > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com > wrote:
> > > > > > > >
> > > > > > > > If you've got fast and slow tests, then have a group for
> fast and
> > > > > slow
> > > > > > > > tests. Then you can choose to just run the fast tests, and
> any
> > > > > > > > automated build system can handle running the slow tests.
> > > > > > >
> > > > > > > IMHO, "fast or slow" may not be the key point. The question is
> > > > > whether we
> > > > > > > have any requirements to run only the regression tests.
> > > > > >
> > > > > > No, probably not the key point, but (a) groups don't have to be
> > > > > > mutually exclusive (so you can decorate it with whatever groups
> you
> > > > > > want)
> > > > >
> > > > > I agree. For example, os.win and os.linux are not mutually
> exclusive.
> > > > >
> > > > > Thanks a lot.
> > > > >
> > > > > and (b) it might be useful for an automated build system to run
> > > > > > fast tests first, followed by slow (or non-fast) tests.
> > > > >
> > > > > That makes sense through we have not clear requirement currently.
> > > > >
> > > > > > Mind you, I don't know what's going to happen with an automated
> > > > > test'n'build
> > > > > > system; so it might not make sense to do it at this point.
> > > > >
> > > > > Really? ;-) We could also discuss whether it's feasible to move to
> > > > > TestNG. As you may know, there are already several threads about
> > > > > TestNG & JUnit. Here I just review the open questions one by one
> so
> > > > > that we have sufficient preparation.
> > > > >
> > > > >
> > > > > [1]http://mail-
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
> > > > >
> > > > > [2]http://mail-
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
> > > > >
> > > > > [3]http://mail-
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
> > > > >
> > > > >
> > > > > Best regards,
> > > > > Richard
> > > > >
> > > > > >
> > > > > > Alex.
> > > > > >
> >
> >
> >
> > --
> > Alexei Zakharov,
> > Intel Middleware Product Division
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>
> --
> Richard Liang
> China Software Development Lab, IBM
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Andrew Zhang
China Software Development Lab, IBM
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
On 9/11/06, Alexei Zakharov <al...@gmail.com> wrote:
> Hi all,
>
> > One more note (seems it already was said sorry if I repeat): the test
> > without any marks should be run in all configurations (i.e. we have
> > 'default' group but declaration of this group may be missed).
>
> I'd like to point your attention on the previous discussion about
> "default groups" :
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44C7457B.7080107@googlemail.com%3e
>
> I am still for using "os.any" since it is more self-descriptive and
> the build script will be simpler with "os.any". It will be nice to
> hear more arguments for using defaults because it seems the arguments
> that were gathered in that old thread hasn't been taken into account
> by participants of this thread.
I have not any strong objection about "os.any". And actually I had
ever proposed to define the "default" group because we could not
include tests with annotation @Test which belong to no groups. Now it
isn't a problem as we already have a solution for them. To facilitate
writing test cases, we annotate the unit tests which are designed to
pass on all platforms (os + arch) with @Test.
If we use "os.any" and "arch.any", then the default annotation would
be @Test(groups={"os.any", "arch.any"})
Could any other give more comments? Thanks a lot.
Best regards,
Richard
>
> Thanks,
>
> 2006/9/5, Vladimir Ivanov <iv...@gmail.com>:
> > One more note (seems it already was said sorry if I repeat): the test
> > without any marks should be run in all configurations (i.e. we have
> > 'default' group but declaration of this group may be missed).
> >
> > thanks, Vladimir
> >
> >
> > On 9/5/06, Vladimir Ivanov <iv...@gmail.com> wrote:
> > >
> > > OK, let's return back to the usage model.
> > > If I understood it correctly, before the commit of any changes each
> > > developer run *all* tests (at least all which we have now) on all available
> > > to him platforms. In this context seems we don't need in any 'level' group
> > > (while 'stress' tests require reasonable time to pass).
> > > Seems, that "platform" group also can be deleted (at present time we have
> > > <10 platform-dependent tests and this amount should not increase
> > > dramatically so the platform-detection can be included to the each such
> > > test).
> > > Also "cpu" groups can be deleted (while we have not cpu-dependent test).
> > > At the end we need only "state" groups to support test exclusion on the
> > > 'one-element' level (while we have unresolved entries in the current exclude
> > > list).
> > >
> > > So, after small update of unit (aka integration, aka regression etc) tests
> > > and resolution of all entries in the exclude list we don't need any groups
> > > and pure JUnit covers all our needs :)
> > >
> > > On the other side, if we define some groups it will nice to define *all*
> > > reasonable groups at the begin of the process.
> > >
> > > thanks, Vladimir
> > >
> > > By the way, our regression tests are 'classic' regression tests that
> > > demonstrate some issues which were not resolved by initial code. But it
> > > provides less coverage than 'regression tests' + unit tests, of cause.
> > >
> > > On 9/5/06, Richard Liang <richard.liangyx@gmail.com > wrote:
> > > >
> > > > On 9/5/06, Alex Blewitt < alex.blewitt@gmail.com> wrote:
> > > > > On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> > > > > > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com > wrote:
> > > > > > >
> > > > > > > If you've got fast and slow tests, then have a group for fast and
> > > > slow
> > > > > > > tests. Then you can choose to just run the fast tests, and any
> > > > > > > automated build system can handle running the slow tests.
> > > > > >
> > > > > > IMHO, "fast or slow" may not be the key point. The question is
> > > > whether we
> > > > > > have any requirements to run only the regression tests.
> > > > >
> > > > > No, probably not the key point, but (a) groups don't have to be
> > > > > mutually exclusive (so you can decorate it with whatever groups you
> > > > > want)
> > > >
> > > > I agree. For example, os.win and os.linux are not mutually exclusive.
> > > >
> > > > Thanks a lot.
> > > >
> > > > and (b) it might be useful for an automated build system to run
> > > > > fast tests first, followed by slow (or non-fast) tests.
> > > >
> > > > That makes sense through we have not clear requirement currently.
> > > >
> > > > > Mind you, I don't know what's going to happen with an automated
> > > > test'n'build
> > > > > system; so it might not make sense to do it at this point.
> > > >
> > > > Really? ;-) We could also discuss whether it's feasible to move to
> > > > TestNG. As you may know, there are already several threads about
> > > > TestNG & JUnit. Here I just review the open questions one by one so
> > > > that we have sufficient preparation.
> > > >
> > > >
> > > > [1]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
> > > >
> > > > [2]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
> > > >
> > > > [3]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
> > > >
> > > >
> > > > Best regards,
> > > > Richard
> > > >
> > > > >
> > > > > Alex.
> > > > >
>
>
>
> --
> Alexei Zakharov,
> Intel Middleware Product Division
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Alexei Zakharov <al...@gmail.com>.
Stepan, All,
> This is not a good argument for using "os.any". Even more it may sound like
> we are going to choose "wrong tool" - why we have to add "os.any" to 99% of
> classlib tests? just because the harness can not do without it?
Well, we have to add the new stuff to these 99% anyway, I mean the
annotation itself. And the copy-pasting of longer block doesn't seem
to be harder than copy-pasting of short block to me.
> As I understand "arguments in the old thread" - TestNG makes us to use "
> os.any" annotation otherwise we have to do a lot of tricks to run tests,
> right?
This was the one of the arguments.
> So a test's annotation should point out that the test is a special one.
I was thinking about a little bit different approach. I will try to
explain. The idea was to use annotations as a *single* place for the
complete information about test: its type, target OS(es) and etc.
Something like inlined test descriptor. Not a red flag. In case
someone is looking at the test at the first time he will get a
complete information just by reading the annotation located at the
same place with the java code. If the annotation itself is not obvious
then the developer is obliged to go to some page with annotation
descriptions, read it carefully and etc. And it seems we get two
places with information about single test: the test's source file and
the page with decoding rules for annotations. This IMHO reduces the
value of the main TestNG benefit - to have all information about the
test in single place. But if we are ok to have several places - we may
use Junit TestSuites instead of TestNG. No refactoring is needed at
all.
To put this another way, these efforts doesn't seem to be so
"unnecessary" to me. Efforts are required to make the life better
sometimes IMHO. Does this make sense to anyone?
Thanks for the reading,
2006/9/12, Stepan Mishura <st...@gmail.com>:
> On 9/11/06, Alexei Zakharov wrote:
> >
> > Hi all,
> >
> > > One more note (seems it already was said sorry if I repeat): the test
> > > without any marks should be run in all configurations (i.e. we have
> > > 'default' group but declaration of this group may be missed).
> >
> > I'd like to point your attention on the previous discussion about
> > "default groups" :
> >
> > http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44C7457B.7080107@googlemail.com%3e
> >
> > I am still for using "os.any" since it is more self-descriptive and
> > the build script will be simpler with "os.any".
>
>
>
> This is not a good argument for using "os.any". Even more it may sound like
> we are going to choose "wrong tool" - why we have to add "os.any" to 99% of
> classlib tests? just because the harness can not do without it?
>
>
> It will be nice to
> > hear more arguments for using defaults because it seems the arguments
> > that were gathered in that old thread hasn't been taken into account
> > by participants of this thread.
>
>
>
> As I understand "arguments in the old thread" - TestNG makes us to use "
> os.any" annotation otherwise we have to do a lot of tricks to run tests,
> right?
>
> IMO a test annotation should be used as "red flag" for developer, for
> example
> * "state.broken" - hey, i'm broken fix me please
> * "os.win" - i'm valid only for Windows, don't try to run me on Linux
>
> So a test's annotation should point out that the test is a special one. But
> if most of tests will contain a similar block of annotations then nobody
> will looked at them. Does this make sense?
>
> Thanks,
> Stepan.
>
> Thanks,
> >
> > 2006/9/5, Vladimir Ivanov <iv...@gmail.com>:
> > > One more note (seems it already was said sorry if I repeat): the test
> > > without any marks should be run in all configurations (i.e. we have
> > > 'default' group but declaration of this group may be missed).
> > >
> > > thanks, Vladimir
> > >
> > >
> > > On 9/5/06, Vladimir Ivanov <iv...@gmail.com> wrote:
> > > >
> > > > OK, let's return back to the usage model.
> > > > If I understood it correctly, before the commit of any changes each
> > > > developer run *all* tests (at least all which we have now) on all
> > available
> > > > to him platforms. In this context seems we don't need in any 'level'
> > group
> > > > (while 'stress' tests require reasonable time to pass).
> > > > Seems, that "platform" group also can be deleted (at present time we
> > have
> > > > <10 platform-dependent tests and this amount should not increase
> > > > dramatically so the platform-detection can be included to the each
> > such
> > > > test).
> > > > Also "cpu" groups can be deleted (while we have not cpu-dependent
> > test).
> > > > At the end we need only "state" groups to support test exclusion on
> > the
> > > > 'one-element' level (while we have unresolved entries in the current
> > exclude
> > > > list).
> > > >
> > > > So, after small update of unit (aka integration, aka regression etc)
> > tests
> > > > and resolution of all entries in the exclude list we don't need any
> > groups
> > > > and pure JUnit covers all our needs :)
> > > >
> > > > On the other side, if we define some groups it will nice to define
> > *all*
> > > > reasonable groups at the begin of the process.
> > > >
> > > > thanks, Vladimir
> > > >
> > > > By the way, our regression tests are 'classic' regression tests that
> > > > demonstrate some issues which were not resolved by initial code. But
> > it
> > > > provides less coverage than 'regression tests' + unit tests, of cause.
> > > >
> > > > On 9/5/06, Richard Liang <richard.liangyx@gmail.com > wrote:
> > > > >
> > > > > On 9/5/06, Alex Blewitt < alex.blewitt@gmail.com> wrote:
> > > > > > On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> > > > > > > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com > wrote:
> > > > > > > >
> > > > > > > > If you've got fast and slow tests, then have a group for fast
> > and
> > > > > slow
> > > > > > > > tests. Then you can choose to just run the fast tests, and any
> > > > > > > > automated build system can handle running the slow tests.
> > > > > > >
> > > > > > > IMHO, "fast or slow" may not be the key point. The question is
> > > > > whether we
> > > > > > > have any requirements to run only the regression tests.
> > > > > >
> > > > > > No, probably not the key point, but (a) groups don't have to be
> > > > > > mutually exclusive (so you can decorate it with whatever groups
> > you
> > > > > > want)
> > > > >
> > > > > I agree. For example, os.win and os.linux are not mutually
> > exclusive.
> > > > >
> > > > > Thanks a lot.
> > > > >
> > > > > and (b) it might be useful for an automated build system to run
> > > > > > fast tests first, followed by slow (or non-fast) tests.
> > > > >
> > > > > That makes sense through we have not clear requirement currently.
> > > > >
> > > > > > Mind you, I don't know what's going to happen with an automated
> > > > > test'n'build
> > > > > > system; so it might not make sense to do it at this point.
> > > > >
> > > > > Really? ;-) We could also discuss whether it's feasible to move to
> > > > > TestNG. As you may know, there are already several threads about
> > > > > TestNG & JUnit. Here I just review the open questions one by one so
> > > > > that we have sufficient preparation.
> > > > >
> > > > >
> > > > > [1]http://mail-
> > archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
> > > > >
> > > > > [2]http://mail-
> > archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
> > > > >
> > > > > [3]http://mail-
> > archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
> > > > >
> > > > >
> > > > > Best regards,
> > > > > Richard
> > > > >
> > > > > >
> > > > > > Alex.
> > > > > >
> >
> >
> >
> >
>
>
> --
> Thanks,
> Stepan Mishura
> Intel Middleware Products Division
--
Alexei Zakharov,
Intel Middleware Product Division
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
On 9/12/06, Stepan Mishura <st...@gmail.com> wrote:
> On 9/11/06, Alexei Zakharov wrote:
> >
> > Hi all,
> >
> > > One more note (seems it already was said sorry if I repeat): the test
> > > without any marks should be run in all configurations (i.e. we have
> > > 'default' group but declaration of this group may be missed).
> >
> > I'd like to point your attention on the previous discussion about
> > "default groups" :
> >
> > http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44C7457B.7080107@googlemail.com%3e
> >
> > I am still for using "os.any" since it is more self-descriptive and
> > the build script will be simpler with "os.any".
>
>
>
> This is not a good argument for using "os.any". Even more it may sound like
> we are going to choose "wrong tool" - why we have to add "os.any" to 99% of
> classlib tests? just because the harness can not do without it?
>
Yes. And we have found a way to avoid "os.any" ;-)
>
> It will be nice to
> > hear more arguments for using defaults because it seems the arguments
> > that were gathered in that old thread hasn't been taken into account
> > by participants of this thread.
>
>
>
> As I understand "arguments in the old thread" - TestNG makes us to use "
> os.any" annotation otherwise we have to do a lot of tricks to run tests,
> right?
>
> IMO a test annotation should be used as "red flag" for developer, for
> example
> * "state.broken" - hey, i'm broken fix me please
> * "os.win" - i'm valid only for Windows, don't try to run me on Linux
>
> So a test's annotation should point out that the test is a special one. But
> if most of tests will contain a similar block of annotations then nobody
> will looked at them. Does this make sense?
Yes. It really makes sense. We do not want to involve developers in
unnecessary effort.
>
> Thanks,
> Stepan.
>
> Thanks,
> >
> > 2006/9/5, Vladimir Ivanov <iv...@gmail.com>:
> > > One more note (seems it already was said sorry if I repeat): the test
> > > without any marks should be run in all configurations (i.e. we have
> > > 'default' group but declaration of this group may be missed).
> > >
> > > thanks, Vladimir
> > >
> > >
> > > On 9/5/06, Vladimir Ivanov <iv...@gmail.com> wrote:
> > > >
> > > > OK, let's return back to the usage model.
> > > > If I understood it correctly, before the commit of any changes each
> > > > developer run *all* tests (at least all which we have now) on all
> > available
> > > > to him platforms. In this context seems we don't need in any 'level'
> > group
> > > > (while 'stress' tests require reasonable time to pass).
> > > > Seems, that "platform" group also can be deleted (at present time we
> > have
> > > > <10 platform-dependent tests and this amount should not increase
> > > > dramatically so the platform-detection can be included to the each
> > such
> > > > test).
> > > > Also "cpu" groups can be deleted (while we have not cpu-dependent
> > test).
> > > > At the end we need only "state" groups to support test exclusion on
> > the
> > > > 'one-element' level (while we have unresolved entries in the current
> > exclude
> > > > list).
> > > >
> > > > So, after small update of unit (aka integration, aka regression etc)
> > tests
> > > > and resolution of all entries in the exclude list we don't need any
> > groups
> > > > and pure JUnit covers all our needs :)
> > > >
> > > > On the other side, if we define some groups it will nice to define
> > *all*
> > > > reasonable groups at the begin of the process.
> > > >
> > > > thanks, Vladimir
> > > >
> > > > By the way, our regression tests are 'classic' regression tests that
> > > > demonstrate some issues which were not resolved by initial code. But
> > it
> > > > provides less coverage than 'regression tests' + unit tests, of cause.
> > > >
> > > > On 9/5/06, Richard Liang <richard.liangyx@gmail.com > wrote:
> > > > >
> > > > > On 9/5/06, Alex Blewitt < alex.blewitt@gmail.com> wrote:
> > > > > > On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> > > > > > > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com > wrote:
> > > > > > > >
> > > > > > > > If you've got fast and slow tests, then have a group for fast
> > and
> > > > > slow
> > > > > > > > tests. Then you can choose to just run the fast tests, and any
> > > > > > > > automated build system can handle running the slow tests.
> > > > > > >
> > > > > > > IMHO, "fast or slow" may not be the key point. The question is
> > > > > whether we
> > > > > > > have any requirements to run only the regression tests.
> > > > > >
> > > > > > No, probably not the key point, but (a) groups don't have to be
> > > > > > mutually exclusive (so you can decorate it with whatever groups
> > you
> > > > > > want)
> > > > >
> > > > > I agree. For example, os.win and os.linux are not mutually
> > exclusive.
> > > > >
> > > > > Thanks a lot.
> > > > >
> > > > > and (b) it might be useful for an automated build system to run
> > > > > > fast tests first, followed by slow (or non-fast) tests.
> > > > >
> > > > > That makes sense through we have not clear requirement currently.
> > > > >
> > > > > > Mind you, I don't know what's going to happen with an automated
> > > > > test'n'build
> > > > > > system; so it might not make sense to do it at this point.
> > > > >
> > > > > Really? ;-) We could also discuss whether it's feasible to move to
> > > > > TestNG. As you may know, there are already several threads about
> > > > > TestNG & JUnit. Here I just review the open questions one by one so
> > > > > that we have sufficient preparation.
> > > > >
> > > > >
> > > > > [1]http://mail-
> > archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
> > > > >
> > > > > [2]http://mail-
> > archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
> > > > >
> > > > > [3]http://mail-
> > archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
> > > > >
> > > > >
> > > > > Best regards,
> > > > > Richard
> > > > >
> > > > > >
> > > > > > Alex.
> > > > > >
> >
> >
> >
> >
>
>
> --
> Thanks,
> Stepan Mishura
> Intel Middleware Products Division
>
> ------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Richard Liang
China Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Stepan Mishura <st...@gmail.com>.
On 9/11/06, Alexei Zakharov wrote:
>
> Hi all,
>
> > One more note (seems it already was said sorry if I repeat): the test
> > without any marks should be run in all configurations (i.e. we have
> > 'default' group but declaration of this group may be missed).
>
> I'd like to point your attention on the previous discussion about
> "default groups" :
>
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44C7457B.7080107@googlemail.com%3e
>
> I am still for using "os.any" since it is more self-descriptive and
> the build script will be simpler with "os.any".
This is not a good argument for using "os.any". Even more it may sound like
we are going to choose "wrong tool" - why we have to add "os.any" to 99% of
classlib tests? just because the harness can not do without it?
It will be nice to
> hear more arguments for using defaults because it seems the arguments
> that were gathered in that old thread hasn't been taken into account
> by participants of this thread.
As I understand "arguments in the old thread" - TestNG makes us to use "
os.any" annotation otherwise we have to do a lot of tricks to run tests,
right?
IMO a test annotation should be used as "red flag" for developer, for
example
* "state.broken" - hey, i'm broken fix me please
* "os.win" - i'm valid only for Windows, don't try to run me on Linux
So a test's annotation should point out that the test is a special one. But
if most of tests will contain a similar block of annotations then nobody
will looked at them. Does this make sense?
Thanks,
Stepan.
Thanks,
>
> 2006/9/5, Vladimir Ivanov <iv...@gmail.com>:
> > One more note (seems it already was said sorry if I repeat): the test
> > without any marks should be run in all configurations (i.e. we have
> > 'default' group but declaration of this group may be missed).
> >
> > thanks, Vladimir
> >
> >
> > On 9/5/06, Vladimir Ivanov <iv...@gmail.com> wrote:
> > >
> > > OK, let's return back to the usage model.
> > > If I understood it correctly, before the commit of any changes each
> > > developer run *all* tests (at least all which we have now) on all
> available
> > > to him platforms. In this context seems we don't need in any 'level'
> group
> > > (while 'stress' tests require reasonable time to pass).
> > > Seems, that "platform" group also can be deleted (at present time we
> have
> > > <10 platform-dependent tests and this amount should not increase
> > > dramatically so the platform-detection can be included to the each
> such
> > > test).
> > > Also "cpu" groups can be deleted (while we have not cpu-dependent
> test).
> > > At the end we need only "state" groups to support test exclusion on
> the
> > > 'one-element' level (while we have unresolved entries in the current
> exclude
> > > list).
> > >
> > > So, after small update of unit (aka integration, aka regression etc)
> tests
> > > and resolution of all entries in the exclude list we don't need any
> groups
> > > and pure JUnit covers all our needs :)
> > >
> > > On the other side, if we define some groups it will nice to define
> *all*
> > > reasonable groups at the begin of the process.
> > >
> > > thanks, Vladimir
> > >
> > > By the way, our regression tests are 'classic' regression tests that
> > > demonstrate some issues which were not resolved by initial code. But
> it
> > > provides less coverage than 'regression tests' + unit tests, of cause.
> > >
> > > On 9/5/06, Richard Liang <richard.liangyx@gmail.com > wrote:
> > > >
> > > > On 9/5/06, Alex Blewitt < alex.blewitt@gmail.com> wrote:
> > > > > On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> > > > > > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com > wrote:
> > > > > > >
> > > > > > > If you've got fast and slow tests, then have a group for fast
> and
> > > > slow
> > > > > > > tests. Then you can choose to just run the fast tests, and any
> > > > > > > automated build system can handle running the slow tests.
> > > > > >
> > > > > > IMHO, "fast or slow" may not be the key point. The question is
> > > > whether we
> > > > > > have any requirements to run only the regression tests.
> > > > >
> > > > > No, probably not the key point, but (a) groups don't have to be
> > > > > mutually exclusive (so you can decorate it with whatever groups
> you
> > > > > want)
> > > >
> > > > I agree. For example, os.win and os.linux are not mutually
> exclusive.
> > > >
> > > > Thanks a lot.
> > > >
> > > > and (b) it might be useful for an automated build system to run
> > > > > fast tests first, followed by slow (or non-fast) tests.
> > > >
> > > > That makes sense through we have not clear requirement currently.
> > > >
> > > > > Mind you, I don't know what's going to happen with an automated
> > > > test'n'build
> > > > > system; so it might not make sense to do it at this point.
> > > >
> > > > Really? ;-) We could also discuss whether it's feasible to move to
> > > > TestNG. As you may know, there are already several threads about
> > > > TestNG & JUnit. Here I just review the open questions one by one so
> > > > that we have sufficient preparation.
> > > >
> > > >
> > > > [1]http://mail-
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
> > > >
> > > > [2]http://mail-
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
> > > >
> > > > [3]http://mail-
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
> > > >
> > > >
> > > > Best regards,
> > > > Richard
> > > >
> > > > >
> > > > > Alex.
> > > > >
>
>
>
>
--
Thanks,
Stepan Mishura
Intel Middleware Products Division
------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Alexei Zakharov <al...@gmail.com>.
Hi all,
> One more note (seems it already was said sorry if I repeat): the test
> without any marks should be run in all configurations (i.e. we have
> 'default' group but declaration of this group may be missed).
I'd like to point your attention on the previous discussion about
"default groups" :
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44C7457B.7080107@googlemail.com%3e
I am still for using "os.any" since it is more self-descriptive and
the build script will be simpler with "os.any". It will be nice to
hear more arguments for using defaults because it seems the arguments
that were gathered in that old thread hasn't been taken into account
by participants of this thread.
Thanks,
2006/9/5, Vladimir Ivanov <iv...@gmail.com>:
> One more note (seems it already was said sorry if I repeat): the test
> without any marks should be run in all configurations (i.e. we have
> 'default' group but declaration of this group may be missed).
>
> thanks, Vladimir
>
>
> On 9/5/06, Vladimir Ivanov <iv...@gmail.com> wrote:
> >
> > OK, let's return back to the usage model.
> > If I understood it correctly, before the commit of any changes each
> > developer run *all* tests (at least all which we have now) on all available
> > to him platforms. In this context seems we don't need in any 'level' group
> > (while 'stress' tests require reasonable time to pass).
> > Seems, that "platform" group also can be deleted (at present time we have
> > <10 platform-dependent tests and this amount should not increase
> > dramatically so the platform-detection can be included to the each such
> > test).
> > Also "cpu" groups can be deleted (while we have not cpu-dependent test).
> > At the end we need only "state" groups to support test exclusion on the
> > 'one-element' level (while we have unresolved entries in the current exclude
> > list).
> >
> > So, after small update of unit (aka integration, aka regression etc) tests
> > and resolution of all entries in the exclude list we don't need any groups
> > and pure JUnit covers all our needs :)
> >
> > On the other side, if we define some groups it will nice to define *all*
> > reasonable groups at the begin of the process.
> >
> > thanks, Vladimir
> >
> > By the way, our regression tests are 'classic' regression tests that
> > demonstrate some issues which were not resolved by initial code. But it
> > provides less coverage than 'regression tests' + unit tests, of cause.
> >
> > On 9/5/06, Richard Liang <richard.liangyx@gmail.com > wrote:
> > >
> > > On 9/5/06, Alex Blewitt < alex.blewitt@gmail.com> wrote:
> > > > On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> > > > > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com > wrote:
> > > > > >
> > > > > > If you've got fast and slow tests, then have a group for fast and
> > > slow
> > > > > > tests. Then you can choose to just run the fast tests, and any
> > > > > > automated build system can handle running the slow tests.
> > > > >
> > > > > IMHO, "fast or slow" may not be the key point. The question is
> > > whether we
> > > > > have any requirements to run only the regression tests.
> > > >
> > > > No, probably not the key point, but (a) groups don't have to be
> > > > mutually exclusive (so you can decorate it with whatever groups you
> > > > want)
> > >
> > > I agree. For example, os.win and os.linux are not mutually exclusive.
> > >
> > > Thanks a lot.
> > >
> > > and (b) it might be useful for an automated build system to run
> > > > fast tests first, followed by slow (or non-fast) tests.
> > >
> > > That makes sense through we have not clear requirement currently.
> > >
> > > > Mind you, I don't know what's going to happen with an automated
> > > test'n'build
> > > > system; so it might not make sense to do it at this point.
> > >
> > > Really? ;-) We could also discuss whether it's feasible to move to
> > > TestNG. As you may know, there are already several threads about
> > > TestNG & JUnit. Here I just review the open questions one by one so
> > > that we have sufficient preparation.
> > >
> > >
> > > [1]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
> > >
> > > [2]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
> > >
> > > [3]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
> > >
> > >
> > > Best regards,
> > > Richard
> > >
> > > >
> > > > Alex.
> > > >
--
Alexei Zakharov,
Intel Middleware Product Division
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Vladimir Ivanov <iv...@gmail.com>.
One more note (seems it already was said sorry if I repeat): the test
without any marks should be run in all configurations (i.e. we have
'default' group but declaration of this group may be missed).
thanks, Vladimir
On 9/5/06, Vladimir Ivanov <iv...@gmail.com> wrote:
>
> OK, let's return back to the usage model.
> If I understood it correctly, before the commit of any changes each
> developer run *all* tests (at least all which we have now) on all available
> to him platforms. In this context seems we don't need in any 'level' group
> (while 'stress' tests require reasonable time to pass).
> Seems, that "platform" group also can be deleted (at present time we have
> <10 platform-dependent tests and this amount should not increase
> dramatically so the platform-detection can be included to the each such
> test).
> Also "cpu" groups can be deleted (while we have not cpu-dependent test).
> At the end we need only "state" groups to support test exclusion on the
> 'one-element' level (while we have unresolved entries in the current exclude
> list).
>
> So, after small update of unit (aka integration, aka regression etc) tests
> and resolution of all entries in the exclude list we don't need any groups
> and pure JUnit covers all our needs :)
>
> On the other side, if we define some groups it will nice to define *all*
> reasonable groups at the begin of the process.
>
> thanks, Vladimir
>
> By the way, our regression tests are 'classic' regression tests that
> demonstrate some issues which were not resolved by initial code. But it
> provides less coverage than 'regression tests' + unit tests, of cause.
>
> On 9/5/06, Richard Liang <richard.liangyx@gmail.com > wrote:
> >
> > On 9/5/06, Alex Blewitt < alex.blewitt@gmail.com> wrote:
> > > On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> > > > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com > wrote:
> > > > >
> > > > > If you've got fast and slow tests, then have a group for fast and
> > slow
> > > > > tests. Then you can choose to just run the fast tests, and any
> > > > > automated build system can handle running the slow tests.
> > > >
> > > > IMHO, "fast or slow" may not be the key point. The question is
> > whether we
> > > > have any requirements to run only the regression tests.
> > >
> > > No, probably not the key point, but (a) groups don't have to be
> > > mutually exclusive (so you can decorate it with whatever groups you
> > > want)
> >
> > I agree. For example, os.win and os.linux are not mutually exclusive.
> >
> > Thanks a lot.
> >
> > and (b) it might be useful for an automated build system to run
> > > fast tests first, followed by slow (or non-fast) tests.
> >
> > That makes sense through we have not clear requirement currently.
> >
> > > Mind you, I don't know what's going to happen with an automated
> > test'n'build
> > > system; so it might not make sense to do it at this point.
> >
> > Really? ;-) We could also discuss whether it's feasible to move to
> > TestNG. As you may know, there are already several threads about
> > TestNG & JUnit. Here I just review the open questions one by one so
> > that we have sufficient preparation.
> >
> >
> > [1]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
> >
> > [2]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
> >
> > [3]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
> >
> >
> > Best regards,
> > Richard
> >
> > >
> > > Alex.
> > >
> > > ---------------------------------------------------------------------
> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >
> > >
> >
> >
> > --
> > Richard Liang
> > China Software Development Lab, IBM
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
On 9/6/06, Vladimir Ivanov <iv...@gmail.com> wrote:
> On 9/6/06, Paulex Yang <pa...@gmail.com> wrote:
> >
> > Vladimir Ivanov wrote:
> > > Do we really need in it? At present time tests were designed for
> > win/unix
> > > only.
> > >
> > I thought at least we have interest to port Harmony to these platforms[1].
> >
> > [1] http://incubator.apache.org/harmony/roadmap.html#Porting%20Matrix
>
>
> Yes, but I don't sure that we will have unique java API tests for all of
> them. In any case, if we define groups set it should include names for all
> defined platforms.
>
We proposed to use os+arch to define platform.
Best regards,
Richard
> thanks, Vladimir
>
>
> --
> > Paulex Yang
> > China Software Development Lab
> > IBM
> >
> >
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Vladimir Ivanov <iv...@gmail.com>.
On 9/6/06, Paulex Yang <pa...@gmail.com> wrote:
>
> Vladimir Ivanov wrote:
> > Do we really need in it? At present time tests were designed for
> win/unix
> > only.
> >
> I thought at least we have interest to port Harmony to these platforms[1].
>
> [1] http://incubator.apache.org/harmony/roadmap.html#Porting%20Matrix
Yes, but I don't sure that we will have unique java API tests for all of
them. In any case, if we define groups set it should include names for all
defined platforms.
thanks, Vladimir
--
> Paulex Yang
> China Software Development Lab
> IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
Re: [classlib][TestNG] groups of Harmony test
Posted by Paulex Yang <pa...@gmail.com>.
Vladimir Ivanov wrote:
> Do we really need in it? At present time tests were designed for win/unix
> only.
>
I thought at least we have interest to port Harmony to these platforms[1].
[1] http://incubator.apache.org/harmony/roadmap.html#Porting%20Matrix
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
On 9/5/06, Vladimir Ivanov <iv...@gmail.com> wrote:
> On 9/5/06, Andrew Zhang <zh...@gmail.com> wrote:
> ...
>
> > > > How to define platform? Get platform information in test code?
> > >
> > >
> > > Yes. It is just one line of code. For example, from Support_Exec.java
> > > class:
> > > boolean onUnix = File.separatorChar == '/';
> >
> >
> > Yes, it does work sometimes, but ...
> >
> > How to differentiate AIX, solaris, linux, unix, windows and etc...
>
>
> Do we really need in it? At present time tests were designed for win/unix
> only.
Yes. In the future, Harmony will support more platforms. We may have
more platform-dependent test cases.
> In any case, I don't against the groups but if we define 'general' groups
> set it should include 'regression' group too.
>
>
>
> > If there's a public API or can be retrieved from system property, it may
> > work.
>
>
> The public API does not specify exact names for platforms (java is platform
> independent :) ) so these ways are equals (but for impl tests it will works
> fine).
>
> thanks, Vladimir
>
>
> > thanks, Vladimir
> > >
> > > In this context seems we don't need in any 'level' group
> > > > > > > (while 'stress' tests require reasonable time to pass).
> > > > > > > Seems, that "platform" group also can be deleted (at present
> > time
> > > we
> > > > > > have
> > > > > > > <10 platform-dependent tests and this amount should not increase
> > > > > > > dramatically so the platform-detection can be included to the
> > each
> > > > > such
> > > > > > > test).
> > > > > > > Also "cpu" groups can be deleted (while we have not
> > cpu-dependent
> > > > > test).
> > > > > > > At the end we need only "state" groups to support test exclusion
> > > on
> > > > > the
> > > > > > > 'one-element' level (while we have unresolved entries in the
> > > current
> > > > > > > exclude
> > > > > > > list).
> > > > > > >
> > > > > > > So, after small update of unit (aka integration, aka regression
> > > etc)
> > > > > > tests
> > > > > > > and resolution of all entries in the exclude list we don't need
> > > any
> > > > > > groups
> > > > > > > and pure JUnit covers all our needs :)
> > > > > > >
> > > > > > > On the other side, if we define some groups it will nice to
> > define
> > > > > *all*
> > > > > > > reasonable groups at the begin of the process.
> > > > > >
> > > > > >
> > > > > > Yes. We should figure out all possible groups. And it can be
> > > > > consolidated
> > > > > > during applying TestNG.
> > > > >
> > > > >
> > > > > Agree.
> > > > > thanks, Vladimir
> > > > >
> > > > > thanks, Vladimir
> > > > > > >
> > > > > > > By the way, our regression tests are 'classic' regression tests
> > > that
> > > > > > > demonstrate some issues which were not resolved by initial code.
> > > But
> > > > > it
> > > > > > > provides less coverage than 'regression tests' + unit tests, of
> > > > cause.
> > > > > > >
> > > > > > > On 9/5/06, Richard Liang <ri...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > On 9/5/06, Alex Blewitt <al...@gmail.com> wrote:
> > > > > > > > > On 04/09/06, Richard Liang <ri...@gmail.com>
> > wrote:
> > > > > > > > > > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com > wrote:
> > > > > > > > > > >
> > > > > > > > > > > If you've got fast and slow tests, then have a group for
> > > > fast
> > > > > > and
> > > > > > > > slow
> > > > > > > > > > > tests. Then you can choose to just run the fast tests,
> > and
> > > > any
> > > > > > > > > > > automated build system can handle running the slow
> > tests.
> > > > > > > > > >
> > > > > > > > > > IMHO, "fast or slow" may not be the key point. The
> > question
> > > is
> > > > > > > whether
> > > > > > > > we
> > > > > > > > > > have any requirements to run only the regression tests.
> > > > > > > > >
> > > > > > > > > No, probably not the key point, but (a) groups don't have to
> > > be
> > > > > > > > > mutually exclusive (so you can decorate it with whatever
> > > groups
> > > > > you
> > > > > > > > > want)
> > > > > > > >
> > > > > > > > I agree. For example, os.win and os.linux are not mutually
> > > > > exclusive.
> > > > > > > >
> > > > > > > > Thanks a lot.
> > > > > > > >
> > > > > > > > and (b) it might be useful for an automated build system to
> > run
> > > > > > > > > fast tests first, followed by slow (or non-fast) tests.
> > > > > > > >
> > > > > > > > That makes sense through we have not clear requirement
> > > currently.
> > > > > > > >
> > > > > > > > > Mind you, I don't know what's going to happen with an
> > > automated
> > > > > > > > test'n'build
> > > > > > > > > system; so it might not make sense to do it at this point.
> > > > > > > >
> > > > > > > > Really? ;-) We could also discuss whether it's feasible to
> > move
> > > to
> > > > > > > > TestNG. As you may know, there are already several threads
> > about
> > > > > > > > TestNG & JUnit. Here I just review the open questions one by
> > one
> > > > so
> > > > > > > > that we have sufficient preparation.
> > > > > > > >
> > > > > > > >
> > > > > > > > [1]http://mail-
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
> > > > > > > >
> > > > > > > > [2]http://mail-
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
> > > > > > > >
> > > > > > > > [3]http://mail-
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
> > > > > > > >
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Richard
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Alex.
> > > > > > > > >
> > > > > > > > >
> > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > Terms of use :
> > > http://incubator.apache.org/harmony/mailing.html
> > > > > > > > > To unsubscribe, e-mail:
> > > > > harmony-dev-unsubscribe@incubator.apache.org
> > > > > > > > > For additional commands, e-mail:
> > > > > > harmony-dev-help@incubator.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Richard Liang
> > > > > > > > China Software Development Lab, IBM
> > > > > > > >
> > > > > > > >
> > > > >
> > ---------------------------------------------------------------------
> > > > > > > > Terms of use :
> > http://incubator.apache.org/harmony/mailing.html
> > > > > > > > To unsubscribe, e-mail:
> > > > harmony-dev-unsubscribe@incubator.apache.org
> > > > > > > > For additional commands, e-mail:
> > > > > harmony-dev-help@incubator.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Andrew Zhang
> > > > > > China Software Development Lab, IBM
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Andrew Zhang
> > > > China Software Development Lab, IBM
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Andrew Zhang
> > China Software Development Lab, IBM
> >
> >
>
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Vladimir Ivanov <iv...@gmail.com>.
On 9/5/06, Andrew Zhang <zh...@gmail.com> wrote:
...
> > > How to define platform? Get platform information in test code?
> >
> >
> > Yes. It is just one line of code. For example, from Support_Exec.java
> > class:
> > boolean onUnix = File.separatorChar == '/';
>
>
> Yes, it does work sometimes, but ...
>
> How to differentiate AIX, solaris, linux, unix, windows and etc...
Do we really need in it? At present time tests were designed for win/unix
only.
In any case, I don't against the groups but if we define 'general' groups
set it should include 'regression' group too.
> If there's a public API or can be retrieved from system property, it may
> work.
The public API does not specify exact names for platforms (java is platform
independent :) ) so these ways are equals (but for impl tests it will works
fine).
thanks, Vladimir
> thanks, Vladimir
> >
> > In this context seems we don't need in any 'level' group
> > > > > > (while 'stress' tests require reasonable time to pass).
> > > > > > Seems, that "platform" group also can be deleted (at present
> time
> > we
> > > > > have
> > > > > > <10 platform-dependent tests and this amount should not increase
> > > > > > dramatically so the platform-detection can be included to the
> each
> > > > such
> > > > > > test).
> > > > > > Also "cpu" groups can be deleted (while we have not
> cpu-dependent
> > > > test).
> > > > > > At the end we need only "state" groups to support test exclusion
> > on
> > > > the
> > > > > > 'one-element' level (while we have unresolved entries in the
> > current
> > > > > > exclude
> > > > > > list).
> > > > > >
> > > > > > So, after small update of unit (aka integration, aka regression
> > etc)
> > > > > tests
> > > > > > and resolution of all entries in the exclude list we don't need
> > any
> > > > > groups
> > > > > > and pure JUnit covers all our needs :)
> > > > > >
> > > > > > On the other side, if we define some groups it will nice to
> define
> > > > *all*
> > > > > > reasonable groups at the begin of the process.
> > > > >
> > > > >
> > > > > Yes. We should figure out all possible groups. And it can be
> > > > consolidated
> > > > > during applying TestNG.
> > > >
> > > >
> > > > Agree.
> > > > thanks, Vladimir
> > > >
> > > > thanks, Vladimir
> > > > > >
> > > > > > By the way, our regression tests are 'classic' regression tests
> > that
> > > > > > demonstrate some issues which were not resolved by initial code.
> > But
> > > > it
> > > > > > provides less coverage than 'regression tests' + unit tests, of
> > > cause.
> > > > > >
> > > > > > On 9/5/06, Richard Liang <ri...@gmail.com> wrote:
> > > > > > >
> > > > > > > On 9/5/06, Alex Blewitt <al...@gmail.com> wrote:
> > > > > > > > On 04/09/06, Richard Liang <ri...@gmail.com>
> wrote:
> > > > > > > > > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com > wrote:
> > > > > > > > > >
> > > > > > > > > > If you've got fast and slow tests, then have a group for
> > > fast
> > > > > and
> > > > > > > slow
> > > > > > > > > > tests. Then you can choose to just run the fast tests,
> and
> > > any
> > > > > > > > > > automated build system can handle running the slow
> tests.
> > > > > > > > >
> > > > > > > > > IMHO, "fast or slow" may not be the key point. The
> question
> > is
> > > > > > whether
> > > > > > > we
> > > > > > > > > have any requirements to run only the regression tests.
> > > > > > > >
> > > > > > > > No, probably not the key point, but (a) groups don't have to
> > be
> > > > > > > > mutually exclusive (so you can decorate it with whatever
> > groups
> > > > you
> > > > > > > > want)
> > > > > > >
> > > > > > > I agree. For example, os.win and os.linux are not mutually
> > > > exclusive.
> > > > > > >
> > > > > > > Thanks a lot.
> > > > > > >
> > > > > > > and (b) it might be useful for an automated build system to
> run
> > > > > > > > fast tests first, followed by slow (or non-fast) tests.
> > > > > > >
> > > > > > > That makes sense through we have not clear requirement
> > currently.
> > > > > > >
> > > > > > > > Mind you, I don't know what's going to happen with an
> > automated
> > > > > > > test'n'build
> > > > > > > > system; so it might not make sense to do it at this point.
> > > > > > >
> > > > > > > Really? ;-) We could also discuss whether it's feasible to
> move
> > to
> > > > > > > TestNG. As you may know, there are already several threads
> about
> > > > > > > TestNG & JUnit. Here I just review the open questions one by
> one
> > > so
> > > > > > > that we have sufficient preparation.
> > > > > > >
> > > > > > >
> > > > > > > [1]http://mail-
> > > > > >
> > > > >
> > > >
> > >
> >
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
> > > > > > >
> > > > > > > [2]http://mail-
> > > > > >
> > > > >
> > > >
> > >
> >
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
> > > > > > >
> > > > > > > [3]http://mail-
> > > > > >
> > > > >
> > > >
> > >
> >
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
> > > > > > >
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Richard
> > > > > > >
> > > > > > > >
> > > > > > > > Alex.
> > > > > > > >
> > > > > > > >
> > > > >
> > ---------------------------------------------------------------------
> > > > > > > > Terms of use :
> > http://incubator.apache.org/harmony/mailing.html
> > > > > > > > To unsubscribe, e-mail:
> > > > harmony-dev-unsubscribe@incubator.apache.org
> > > > > > > > For additional commands, e-mail:
> > > > > harmony-dev-help@incubator.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Richard Liang
> > > > > > > China Software Development Lab, IBM
> > > > > > >
> > > > > > >
> > > >
> ---------------------------------------------------------------------
> > > > > > > Terms of use :
> http://incubator.apache.org/harmony/mailing.html
> > > > > > > To unsubscribe, e-mail:
> > > harmony-dev-unsubscribe@incubator.apache.org
> > > > > > > For additional commands, e-mail:
> > > > harmony-dev-help@incubator.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Andrew Zhang
> > > > > China Software Development Lab, IBM
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Andrew Zhang
> > > China Software Development Lab, IBM
> > >
> > >
> >
> >
>
>
> --
> Andrew Zhang
> China Software Development Lab, IBM
>
>
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Andrew Zhang <zh...@gmail.com>.
On 9/5/06, Vladimir Ivanov <iv...@gmail.com> wrote:
>
> On 9/5/06, Andrew Zhang <zh...@gmail.com> wrote:
> >
> > > If test fails on linux than (for example):
> > > - the test/ implementation should be fixed or
> >
> >
> > What if the behaviour is different on different platforms?
>
>
> Seems, that it may be just 2 different tests.
>
> - the test should define platform and report 'passed' if it does not
> > > support current platform
> >
> >
> > How to define platform? Get platform information in test code?
>
>
> Yes. It is just one line of code. For example, from Support_Exec.java
> class:
> boolean onUnix = File.separatorChar == '/';
Yes, it does work sometimes, but ...
How to differentiate AIX, solaris, linux, unix, windows and etc...
If there's a public API or can be retrieved from system property, it may
work.
thanks, Vladimir
>
> In this context seems we don't need in any 'level' group
> > > > > (while 'stress' tests require reasonable time to pass).
> > > > > Seems, that "platform" group also can be deleted (at present time
> we
> > > > have
> > > > > <10 platform-dependent tests and this amount should not increase
> > > > > dramatically so the platform-detection can be included to the each
> > > such
> > > > > test).
> > > > > Also "cpu" groups can be deleted (while we have not cpu-dependent
> > > test).
> > > > > At the end we need only "state" groups to support test exclusion
> on
> > > the
> > > > > 'one-element' level (while we have unresolved entries in the
> current
> > > > > exclude
> > > > > list).
> > > > >
> > > > > So, after small update of unit (aka integration, aka regression
> etc)
> > > > tests
> > > > > and resolution of all entries in the exclude list we don't need
> any
> > > > groups
> > > > > and pure JUnit covers all our needs :)
> > > > >
> > > > > On the other side, if we define some groups it will nice to define
> > > *all*
> > > > > reasonable groups at the begin of the process.
> > > >
> > > >
> > > > Yes. We should figure out all possible groups. And it can be
> > > consolidated
> > > > during applying TestNG.
> > >
> > >
> > > Agree.
> > > thanks, Vladimir
> > >
> > > thanks, Vladimir
> > > > >
> > > > > By the way, our regression tests are 'classic' regression tests
> that
> > > > > demonstrate some issues which were not resolved by initial code.
> But
> > > it
> > > > > provides less coverage than 'regression tests' + unit tests, of
> > cause.
> > > > >
> > > > > On 9/5/06, Richard Liang <ri...@gmail.com> wrote:
> > > > > >
> > > > > > On 9/5/06, Alex Blewitt <al...@gmail.com> wrote:
> > > > > > > On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> > > > > > > > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com > wrote:
> > > > > > > > >
> > > > > > > > > If you've got fast and slow tests, then have a group for
> > fast
> > > > and
> > > > > > slow
> > > > > > > > > tests. Then you can choose to just run the fast tests, and
> > any
> > > > > > > > > automated build system can handle running the slow tests.
> > > > > > > >
> > > > > > > > IMHO, "fast or slow" may not be the key point. The question
> is
> > > > > whether
> > > > > > we
> > > > > > > > have any requirements to run only the regression tests.
> > > > > > >
> > > > > > > No, probably not the key point, but (a) groups don't have to
> be
> > > > > > > mutually exclusive (so you can decorate it with whatever
> groups
> > > you
> > > > > > > want)
> > > > > >
> > > > > > I agree. For example, os.win and os.linux are not mutually
> > > exclusive.
> > > > > >
> > > > > > Thanks a lot.
> > > > > >
> > > > > > and (b) it might be useful for an automated build system to run
> > > > > > > fast tests first, followed by slow (or non-fast) tests.
> > > > > >
> > > > > > That makes sense through we have not clear requirement
> currently.
> > > > > >
> > > > > > > Mind you, I don't know what's going to happen with an
> automated
> > > > > > test'n'build
> > > > > > > system; so it might not make sense to do it at this point.
> > > > > >
> > > > > > Really? ;-) We could also discuss whether it's feasible to move
> to
> > > > > > TestNG. As you may know, there are already several threads about
> > > > > > TestNG & JUnit. Here I just review the open questions one by one
> > so
> > > > > > that we have sufficient preparation.
> > > > > >
> > > > > >
> > > > > > [1]http://mail-
> > > > >
> > > >
> > >
> >
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
> > > > > >
> > > > > > [2]http://mail-
> > > > >
> > > >
> > >
> >
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
> > > > > >
> > > > > > [3]http://mail-
> > > > >
> > > >
> > >
> >
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
> > > > > >
> > > > > >
> > > > > > Best regards,
> > > > > > Richard
> > > > > >
> > > > > > >
> > > > > > > Alex.
> > > > > > >
> > > > > > >
> > > >
> ---------------------------------------------------------------------
> > > > > > > Terms of use :
> http://incubator.apache.org/harmony/mailing.html
> > > > > > > To unsubscribe, e-mail:
> > > harmony-dev-unsubscribe@incubator.apache.org
> > > > > > > For additional commands, e-mail:
> > > > harmony-dev-help@incubator.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Richard Liang
> > > > > > China Software Development Lab, IBM
> > > > > >
> > > > > >
> > > ---------------------------------------------------------------------
> > > > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > > > To unsubscribe, e-mail:
> > harmony-dev-unsubscribe@incubator.apache.org
> > > > > > For additional commands, e-mail:
> > > harmony-dev-help@incubator.apache.org
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Andrew Zhang
> > > > China Software Development Lab, IBM
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Andrew Zhang
> > China Software Development Lab, IBM
> >
> >
>
>
--
Andrew Zhang
China Software Development Lab, IBM
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Vladimir Ivanov <iv...@gmail.com>.
On 9/5/06, Andrew Zhang <zh...@gmail.com> wrote:
>
> > If test fails on linux than (for example):
> > - the test/ implementation should be fixed or
>
>
> What if the behaviour is different on different platforms?
Seems, that it may be just 2 different tests.
- the test should define platform and report 'passed' if it does not
> > support current platform
>
>
> How to define platform? Get platform information in test code?
Yes. It is just one line of code. For example, from Support_Exec.java
class:
boolean onUnix = File.separatorChar == '/';
thanks, Vladimir
In this context seems we don't need in any 'level' group
> > > > (while 'stress' tests require reasonable time to pass).
> > > > Seems, that "platform" group also can be deleted (at present time we
> > > have
> > > > <10 platform-dependent tests and this amount should not increase
> > > > dramatically so the platform-detection can be included to the each
> > such
> > > > test).
> > > > Also "cpu" groups can be deleted (while we have not cpu-dependent
> > test).
> > > > At the end we need only "state" groups to support test exclusion on
> > the
> > > > 'one-element' level (while we have unresolved entries in the current
> > > > exclude
> > > > list).
> > > >
> > > > So, after small update of unit (aka integration, aka regression etc)
> > > tests
> > > > and resolution of all entries in the exclude list we don't need any
> > > groups
> > > > and pure JUnit covers all our needs :)
> > > >
> > > > On the other side, if we define some groups it will nice to define
> > *all*
> > > > reasonable groups at the begin of the process.
> > >
> > >
> > > Yes. We should figure out all possible groups. And it can be
> > consolidated
> > > during applying TestNG.
> >
> >
> > Agree.
> > thanks, Vladimir
> >
> > thanks, Vladimir
> > > >
> > > > By the way, our regression tests are 'classic' regression tests that
> > > > demonstrate some issues which were not resolved by initial code. But
> > it
> > > > provides less coverage than 'regression tests' + unit tests, of
> cause.
> > > >
> > > > On 9/5/06, Richard Liang <ri...@gmail.com> wrote:
> > > > >
> > > > > On 9/5/06, Alex Blewitt <al...@gmail.com> wrote:
> > > > > > On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> > > > > > > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com > wrote:
> > > > > > > >
> > > > > > > > If you've got fast and slow tests, then have a group for
> fast
> > > and
> > > > > slow
> > > > > > > > tests. Then you can choose to just run the fast tests, and
> any
> > > > > > > > automated build system can handle running the slow tests.
> > > > > > >
> > > > > > > IMHO, "fast or slow" may not be the key point. The question is
> > > > whether
> > > > > we
> > > > > > > have any requirements to run only the regression tests.
> > > > > >
> > > > > > No, probably not the key point, but (a) groups don't have to be
> > > > > > mutually exclusive (so you can decorate it with whatever groups
> > you
> > > > > > want)
> > > > >
> > > > > I agree. For example, os.win and os.linux are not mutually
> > exclusive.
> > > > >
> > > > > Thanks a lot.
> > > > >
> > > > > and (b) it might be useful for an automated build system to run
> > > > > > fast tests first, followed by slow (or non-fast) tests.
> > > > >
> > > > > That makes sense through we have not clear requirement currently.
> > > > >
> > > > > > Mind you, I don't know what's going to happen with an automated
> > > > > test'n'build
> > > > > > system; so it might not make sense to do it at this point.
> > > > >
> > > > > Really? ;-) We could also discuss whether it's feasible to move to
> > > > > TestNG. As you may know, there are already several threads about
> > > > > TestNG & JUnit. Here I just review the open questions one by one
> so
> > > > > that we have sufficient preparation.
> > > > >
> > > > >
> > > > > [1]http://mail-
> > > >
> > >
> >
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
> > > > >
> > > > > [2]http://mail-
> > > >
> > >
> >
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
> > > > >
> > > > > [3]http://mail-
> > > >
> > >
> >
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
> > > > >
> > > > >
> > > > > Best regards,
> > > > > Richard
> > > > >
> > > > > >
> > > > > > Alex.
> > > > > >
> > > > > >
> > > ---------------------------------------------------------------------
> > > > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > > > To unsubscribe, e-mail:
> > harmony-dev-unsubscribe@incubator.apache.org
> > > > > > For additional commands, e-mail:
> > > harmony-dev-help@incubator.apache.org
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Richard Liang
> > > > > China Software Development Lab, IBM
> > > > >
> > > > >
> > ---------------------------------------------------------------------
> > > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > > To unsubscribe, e-mail:
> harmony-dev-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail:
> > harmony-dev-help@incubator.apache.org
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Andrew Zhang
> > > China Software Development Lab, IBM
> > >
> > >
> >
> >
>
>
> --
> Andrew Zhang
> China Software Development Lab, IBM
>
>
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Andrew Zhang <zh...@gmail.com>.
On 9/5/06, Vladimir Ivanov <iv...@gmail.com> wrote:
>
> On 9/5/06, Andrew Zhang <zh...@gmail.com> wrote:
> >
> > On 9/5/06, Vladimir Ivanov <iv...@gmail.com> wrote:
> > >
> > > OK, let's return back to the usage model.
> > > If I understood it correctly, before the commit of any changes each
> > > developer run *all* tests (at least all which we have now) on all
> > > available
> > > to him platforms.
> >
> >
> > Yes. But as you mentioned, what's "all available"?
> >
> > If a test passes on windows while fails on linux, is it available to
> > windows?
> >
> > If it is, how will we control it? TestNG groups.
>
>
> If developer has only windows box than he should run tests on windows
> only.
> If test fails on linux than (for example):
> - the test/ implementation should be fixed or
What if the behaviour is different on different platforms?
- the test should define platform and report 'passed' if it does not
> support current platform
How to define platform? Get platform information in test code?
In this context seems we don't need in any 'level' group
> > > (while 'stress' tests require reasonable time to pass).
> > > Seems, that "platform" group also can be deleted (at present time we
> > have
> > > <10 platform-dependent tests and this amount should not increase
> > > dramatically so the platform-detection can be included to the each
> such
> > > test).
> > > Also "cpu" groups can be deleted (while we have not cpu-dependent
> test).
> > > At the end we need only "state" groups to support test exclusion on
> the
> > > 'one-element' level (while we have unresolved entries in the current
> > > exclude
> > > list).
> > >
> > > So, after small update of unit (aka integration, aka regression etc)
> > tests
> > > and resolution of all entries in the exclude list we don't need any
> > groups
> > > and pure JUnit covers all our needs :)
> > >
> > > On the other side, if we define some groups it will nice to define
> *all*
> > > reasonable groups at the begin of the process.
> >
> >
> > Yes. We should figure out all possible groups. And it can be
> consolidated
> > during applying TestNG.
>
>
> Agree.
> thanks, Vladimir
>
> thanks, Vladimir
> > >
> > > By the way, our regression tests are 'classic' regression tests that
> > > demonstrate some issues which were not resolved by initial code. But
> it
> > > provides less coverage than 'regression tests' + unit tests, of cause.
> > >
> > > On 9/5/06, Richard Liang <ri...@gmail.com> wrote:
> > > >
> > > > On 9/5/06, Alex Blewitt <al...@gmail.com> wrote:
> > > > > On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> > > > > > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com > wrote:
> > > > > > >
> > > > > > > If you've got fast and slow tests, then have a group for fast
> > and
> > > > slow
> > > > > > > tests. Then you can choose to just run the fast tests, and any
> > > > > > > automated build system can handle running the slow tests.
> > > > > >
> > > > > > IMHO, "fast or slow" may not be the key point. The question is
> > > whether
> > > > we
> > > > > > have any requirements to run only the regression tests.
> > > > >
> > > > > No, probably not the key point, but (a) groups don't have to be
> > > > > mutually exclusive (so you can decorate it with whatever groups
> you
> > > > > want)
> > > >
> > > > I agree. For example, os.win and os.linux are not mutually
> exclusive.
> > > >
> > > > Thanks a lot.
> > > >
> > > > and (b) it might be useful for an automated build system to run
> > > > > fast tests first, followed by slow (or non-fast) tests.
> > > >
> > > > That makes sense through we have not clear requirement currently.
> > > >
> > > > > Mind you, I don't know what's going to happen with an automated
> > > > test'n'build
> > > > > system; so it might not make sense to do it at this point.
> > > >
> > > > Really? ;-) We could also discuss whether it's feasible to move to
> > > > TestNG. As you may know, there are already several threads about
> > > > TestNG & JUnit. Here I just review the open questions one by one so
> > > > that we have sufficient preparation.
> > > >
> > > >
> > > > [1]http://mail-
> > >
> >
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
> > > >
> > > > [2]http://mail-
> > >
> >
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
> > > >
> > > > [3]http://mail-
> > >
> >
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
> > > >
> > > >
> > > > Best regards,
> > > > Richard
> > > >
> > > > >
> > > > > Alex.
> > > > >
> > > > >
> > ---------------------------------------------------------------------
> > > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > > To unsubscribe, e-mail:
> harmony-dev-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail:
> > harmony-dev-help@incubator.apache.org
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Richard Liang
> > > > China Software Development Lab, IBM
> > > >
> > > >
> ---------------------------------------------------------------------
> > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail:
> harmony-dev-help@incubator.apache.org
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Andrew Zhang
> > China Software Development Lab, IBM
> >
> >
>
>
--
Andrew Zhang
China Software Development Lab, IBM
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Vladimir Ivanov <iv...@gmail.com>.
On 9/5/06, Andrew Zhang <zh...@gmail.com> wrote:
>
> On 9/5/06, Vladimir Ivanov <iv...@gmail.com> wrote:
> >
> > OK, let's return back to the usage model.
> > If I understood it correctly, before the commit of any changes each
> > developer run *all* tests (at least all which we have now) on all
> > available
> > to him platforms.
>
>
> Yes. But as you mentioned, what's "all available"?
>
> If a test passes on windows while fails on linux, is it available to
> windows?
>
> If it is, how will we control it? TestNG groups.
If developer has only windows box than he should run tests on windows only.
If test fails on linux than (for example):
- the test/ implementation should be fixed or
- the test should define platform and report 'passed' if it does not
support current platform
In this context seems we don't need in any 'level' group
> > (while 'stress' tests require reasonable time to pass).
> > Seems, that "platform" group also can be deleted (at present time we
> have
> > <10 platform-dependent tests and this amount should not increase
> > dramatically so the platform-detection can be included to the each such
> > test).
> > Also "cpu" groups can be deleted (while we have not cpu-dependent test).
> > At the end we need only "state" groups to support test exclusion on the
> > 'one-element' level (while we have unresolved entries in the current
> > exclude
> > list).
> >
> > So, after small update of unit (aka integration, aka regression etc)
> tests
> > and resolution of all entries in the exclude list we don't need any
> groups
> > and pure JUnit covers all our needs :)
> >
> > On the other side, if we define some groups it will nice to define *all*
> > reasonable groups at the begin of the process.
>
>
> Yes. We should figure out all possible groups. And it can be consolidated
> during applying TestNG.
Agree.
thanks, Vladimir
thanks, Vladimir
> >
> > By the way, our regression tests are 'classic' regression tests that
> > demonstrate some issues which were not resolved by initial code. But it
> > provides less coverage than 'regression tests' + unit tests, of cause.
> >
> > On 9/5/06, Richard Liang <ri...@gmail.com> wrote:
> > >
> > > On 9/5/06, Alex Blewitt <al...@gmail.com> wrote:
> > > > On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> > > > > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com > wrote:
> > > > > >
> > > > > > If you've got fast and slow tests, then have a group for fast
> and
> > > slow
> > > > > > tests. Then you can choose to just run the fast tests, and any
> > > > > > automated build system can handle running the slow tests.
> > > > >
> > > > > IMHO, "fast or slow" may not be the key point. The question is
> > whether
> > > we
> > > > > have any requirements to run only the regression tests.
> > > >
> > > > No, probably not the key point, but (a) groups don't have to be
> > > > mutually exclusive (so you can decorate it with whatever groups you
> > > > want)
> > >
> > > I agree. For example, os.win and os.linux are not mutually exclusive.
> > >
> > > Thanks a lot.
> > >
> > > and (b) it might be useful for an automated build system to run
> > > > fast tests first, followed by slow (or non-fast) tests.
> > >
> > > That makes sense through we have not clear requirement currently.
> > >
> > > > Mind you, I don't know what's going to happen with an automated
> > > test'n'build
> > > > system; so it might not make sense to do it at this point.
> > >
> > > Really? ;-) We could also discuss whether it's feasible to move to
> > > TestNG. As you may know, there are already several threads about
> > > TestNG & JUnit. Here I just review the open questions one by one so
> > > that we have sufficient preparation.
> > >
> > >
> > > [1]http://mail-
> >
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
> > >
> > > [2]http://mail-
> >
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
> > >
> > > [3]http://mail-
> >
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
> > >
> > >
> > > Best regards,
> > > Richard
> > >
> > > >
> > > > Alex.
> > > >
> > > >
> ---------------------------------------------------------------------
> > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail:
> harmony-dev-help@incubator.apache.org
> > > >
> > > >
> > >
> > >
> > > --
> > > Richard Liang
> > > China Software Development Lab, IBM
> > >
> > > ---------------------------------------------------------------------
> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >
> > >
> >
> >
>
>
> --
> Andrew Zhang
> China Software Development Lab, IBM
>
>
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Andrew Zhang <zh...@gmail.com>.
On 9/5/06, Vladimir Ivanov <iv...@gmail.com> wrote:
>
> OK, let's return back to the usage model.
> If I understood it correctly, before the commit of any changes each
> developer run *all* tests (at least all which we have now) on all
> available
> to him platforms.
Yes. But as you mentioned, what's "all available"?
If a test passes on windows while fails on linux, is it available to
windows?
If it is, how will we control it? TestNG groups.
In this context seems we don't need in any 'level' group
> (while 'stress' tests require reasonable time to pass).
> Seems, that "platform" group also can be deleted (at present time we have
> <10 platform-dependent tests and this amount should not increase
> dramatically so the platform-detection can be included to the each such
> test).
> Also "cpu" groups can be deleted (while we have not cpu-dependent test).
> At the end we need only "state" groups to support test exclusion on the
> 'one-element' level (while we have unresolved entries in the current
> exclude
> list).
>
> So, after small update of unit (aka integration, aka regression etc) tests
> and resolution of all entries in the exclude list we don't need any groups
> and pure JUnit covers all our needs :)
>
> On the other side, if we define some groups it will nice to define *all*
> reasonable groups at the begin of the process.
Yes. We should figure out all possible groups. And it can be consolidated
during applying TestNG.
thanks, Vladimir
>
> By the way, our regression tests are 'classic' regression tests that
> demonstrate some issues which were not resolved by initial code. But it
> provides less coverage than 'regression tests' + unit tests, of cause.
>
> On 9/5/06, Richard Liang <ri...@gmail.com> wrote:
> >
> > On 9/5/06, Alex Blewitt <al...@gmail.com> wrote:
> > > On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> > > > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com > wrote:
> > > > >
> > > > > If you've got fast and slow tests, then have a group for fast and
> > slow
> > > > > tests. Then you can choose to just run the fast tests, and any
> > > > > automated build system can handle running the slow tests.
> > > >
> > > > IMHO, "fast or slow" may not be the key point. The question is
> whether
> > we
> > > > have any requirements to run only the regression tests.
> > >
> > > No, probably not the key point, but (a) groups don't have to be
> > > mutually exclusive (so you can decorate it with whatever groups you
> > > want)
> >
> > I agree. For example, os.win and os.linux are not mutually exclusive.
> >
> > Thanks a lot.
> >
> > and (b) it might be useful for an automated build system to run
> > > fast tests first, followed by slow (or non-fast) tests.
> >
> > That makes sense through we have not clear requirement currently.
> >
> > > Mind you, I don't know what's going to happen with an automated
> > test'n'build
> > > system; so it might not make sense to do it at this point.
> >
> > Really? ;-) We could also discuss whether it's feasible to move to
> > TestNG. As you may know, there are already several threads about
> > TestNG & JUnit. Here I just review the open questions one by one so
> > that we have sufficient preparation.
> >
> >
> > [1]http://mail-
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
> >
> > [2]http://mail-
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
> >
> > [3]http://mail-
> archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
> >
> >
> > Best regards,
> > Richard
> >
> > >
> > > Alex.
> > >
> > > ---------------------------------------------------------------------
> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >
> > >
> >
> >
> > --
> > Richard Liang
> > China Software Development Lab, IBM
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>
--
Andrew Zhang
China Software Development Lab, IBM
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Vladimir Ivanov <iv...@gmail.com>.
OK, let's return back to the usage model.
If I understood it correctly, before the commit of any changes each
developer run *all* tests (at least all which we have now) on all available
to him platforms. In this context seems we don't need in any 'level' group
(while 'stress' tests require reasonable time to pass).
Seems, that "platform" group also can be deleted (at present time we have
<10 platform-dependent tests and this amount should not increase
dramatically so the platform-detection can be included to the each such
test).
Also "cpu" groups can be deleted (while we have not cpu-dependent test).
At the end we need only "state" groups to support test exclusion on the
'one-element' level (while we have unresolved entries in the current exclude
list).
So, after small update of unit (aka integration, aka regression etc) tests
and resolution of all entries in the exclude list we don't need any groups
and pure JUnit covers all our needs :)
On the other side, if we define some groups it will nice to define *all*
reasonable groups at the begin of the process.
thanks, Vladimir
By the way, our regression tests are 'classic' regression tests that
demonstrate some issues which were not resolved by initial code. But it
provides less coverage than 'regression tests' + unit tests, of cause.
On 9/5/06, Richard Liang <ri...@gmail.com> wrote:
>
> On 9/5/06, Alex Blewitt <al...@gmail.com> wrote:
> > On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> > > On 9/4/06, Alex Blewitt <alex.blewitt@gmail.com > wrote:
> > > >
> > > > If you've got fast and slow tests, then have a group for fast and
> slow
> > > > tests. Then you can choose to just run the fast tests, and any
> > > > automated build system can handle running the slow tests.
> > >
> > > IMHO, "fast or slow" may not be the key point. The question is whether
> we
> > > have any requirements to run only the regression tests.
> >
> > No, probably not the key point, but (a) groups don't have to be
> > mutually exclusive (so you can decorate it with whatever groups you
> > want)
>
> I agree. For example, os.win and os.linux are not mutually exclusive.
>
> Thanks a lot.
>
> and (b) it might be useful for an automated build system to run
> > fast tests first, followed by slow (or non-fast) tests.
>
> That makes sense through we have not clear requirement currently.
>
> > Mind you, I don't know what's going to happen with an automated
> test'n'build
> > system; so it might not make sense to do it at this point.
>
> Really? ;-) We could also discuss whether it's feasible to move to
> TestNG. As you may know, there are already several threads about
> TestNG & JUnit. Here I just review the open questions one by one so
> that we have sufficient preparation.
>
>
> [1]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
>
> [2]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
>
> [3]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
>
>
> Best regards,
> Richard
>
> >
> > Alex.
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>
> --
> Richard Liang
> China Software Development Lab, IBM
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
On 9/5/06, Richard Liang <ri...@gmail.com> wrote:
> On 9/5/06, Alex Blewitt <al...@gmail.com> wrote:
> > On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> > > On 9/4/06, Alex Blewitt <al...@gmail.com> wrote:
> > > >
> > > > If you've got fast and slow tests, then have a group for fast and slow
> > > > tests. Then you can choose to just run the fast tests, and any
> > > > automated build system can handle running the slow tests.
> > >
> > > IMHO, "fast or slow" may not be the key point. The question is whether we
> > > have any requirements to run only the regression tests.
> >
> > No, probably not the key point, but (a) groups don't have to be
> > mutually exclusive (so you can decorate it with whatever groups you
> > want)
>
> I agree. For example, os.win and os.linux are not mutually exclusive.
>
> Thanks a lot.
>
> and (b) it might be useful for an automated build system to run
> > fast tests first, followed by slow (or non-fast) tests.
>
> That makes sense through we have not clear requirement currently.
>
> > Mind you, I don't know what's going to happen with an automated test'n'build
> > system; so it might not make sense to do it at this point.
>
> Really? ;-) We could also discuss whether it's feasible to move to
> TestNG. As you may know, there are already several threads about
> TestNG & JUnit. Here I just review the open questions one by one so
> that we have sufficient preparation.
>
Alex, I must say sorry if I misunderstood you. I mean here we could
discuss anything related to Harmony testing. ;-)
Best regards,
Richard
>
> [1]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
> [2]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
> [3]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
>
> Best regards,
> Richard
>
> >
> > Alex.
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>
> --
> Richard Liang
> China Software Development Lab, IBM
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
On 9/5/06, Alex Blewitt <al...@gmail.com> wrote:
> On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> > On 9/4/06, Alex Blewitt <al...@gmail.com> wrote:
> > >
> > > If you've got fast and slow tests, then have a group for fast and slow
> > > tests. Then you can choose to just run the fast tests, and any
> > > automated build system can handle running the slow tests.
> >
> > IMHO, "fast or slow" may not be the key point. The question is whether we
> > have any requirements to run only the regression tests.
>
> No, probably not the key point, but (a) groups don't have to be
> mutually exclusive (so you can decorate it with whatever groups you
> want)
I agree. For example, os.win and os.linux are not mutually exclusive.
Thanks a lot.
and (b) it might be useful for an automated build system to run
> fast tests first, followed by slow (or non-fast) tests.
That makes sense through we have not clear requirement currently.
> Mind you, I don't know what's going to happen with an automated test'n'build
> system; so it might not make sense to do it at this point.
Really? ;-) We could also discuss whether it's feasible to move to
TestNG. As you may know, there are already several threads about
TestNG & JUnit. Here I just review the open questions one by one so
that we have sufficient preparation.
[1]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c44ABB451.30806@googlemail.com%3e
[2]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c7273946b0607240654i7e951260x1e803ce476821982@mail.gmail.com%3e
[3]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/%3c2c9597b90607280631p2b4f6fefldaf4ff1c5cd00406@mail.gmail.com%3e
Best regards,
Richard
>
> Alex.
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Alex Blewitt <al...@gmail.com>.
On 04/09/06, Richard Liang <ri...@gmail.com> wrote:
> On 9/4/06, Alex Blewitt <al...@gmail.com> wrote:
> >
> > If you've got fast and slow tests, then have a group for fast and slow
> > tests. Then you can choose to just run the fast tests, and any
> > automated build system can handle running the slow tests.
>
> IMHO, "fast or slow" may not be the key point. The question is whether we
> have any requirements to run only the regression tests.
No, probably not the key point, but (a) groups don't have to be
mutually exclusive (so you can decorate it with whatever groups you
want) and (b) it might be useful for an automated build system to run
fast tests first, followed by slow (or non-fast) tests. Mind you, I
don't know what's going to happen with an automated test'n'build
system; so it might not make sense to do it at this point.
Alex.
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
On 9/4/06, Andrew Zhang <zh...@gmail.com> wrote:
> On 9/4/06, Richard Liang <ri...@gmail.com> wrote:
> >
> > I sent my reply this afternoon, but I have not received it. (it seems
> > there
> > is something wrong with my smtp server). So I send it again. :-)
> >
> > On 9/4/06, Alex Blewitt <al...@gmail.com> wrote:
> > >
> > > IMNSO it doesn't make sense to arbitrarily partition the tests based
> > > on a moniker, such as 'integration test', 'unit test', 'regression
> > > test' etc. For one thing, developers are generally not good at
> > > agreeing on the difference between them :-)
> >
> >
> >
> > This is really a problem, however it might be simpler than we imagine. We
> > are open to any discussion. ;-) Anyway, developers are required to write
> > unit tests.
> >
> >
> > If you've got fast and slow tests, then have a group for fast and slow
> > > tests. Then you can choose to just run the fast tests, and any
> > > automated build system can handle running the slow tests.
> >
> >
> > IMHO, "fast or slow" may not be the key point. The question is whether we
> > have any requirements to run only the regression tests.
>
>
> I think your first proposal looks fine. Platform + os + state + api/impl
> information are enough for all tests. We have to assure all tests pass on
> every platform.
>
Thanks a lot. Well, I agree :-). TestNG annotations are most helpful
to us in the management of unit tests that are prone to change. For
example, some unit tests suddenly break on a particular platform. or
we may want to exclude some tests temporarily.
Here I just try to list "all" the possible facets, so that we could
have more thorough discussion.
Best regards,
Richard
> For quick run, developers can use their own short approach, like run single
> test in IDE... It's not business of TestNG. :-)
>
>
> Best regards,
> > Richard
> >
> > Alex.
> > >
> > > On 04/09/06, Vladimir Ivanov <iv...@gmail.com> wrote:
> > > > On 9/4/06, Richard Liang <ri...@gmail.com> wrote:
> > > > >
> > > > >
> > > > >
> > > > > Mikhail Loenko wrote:
> > > > > > Hi Vladimir
> > > > > >
> > > > > > Could you please decribe for what purpose it will be used?
> > > > > >
> > > > > > I mean why one might have to either exclude or run only regression
> > > > > tests?
> > > > >
> > > > > If running all tests takes up much time, running all regression test
> > > > > (for one particular version) may be a convenient way to verify the
> > > > > correctness of bug-fixing.
> > > >
> > > >
> > > > Thanks Richard. It is exactly what I want to say :)
> > > > On the other hand may be all proposed groups need similar explanation
> > > (smt.
> > > > like use-case for group)?
> > > >
> > > > thanks, Vladimir
> > > >
> > > > Best regards,
> > > > > Richard.
> > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Mikhail
> > > > > >
> > > > > > 2006/8/30, Vladimir Ivanov <iv...@gmail.com>:
> > > > > >> On 8/30/06, Richard Liang <ri...@gmail.com> wrote:
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > Vladimir Ivanov wrote:
> > > > > >> > > Also some tag for regression tests should be added.
> > > > > >> > Yes. Do you think we could annotate regression test as
> > > > > >> > *level.regression*? Thanks a lot.
> > > > > >>
> > > > > >>
> > > > > >> Yes, I do. While tests can have more than one group it will
> > enough.
> > > > > >> thanks, Vladimir
> > > > > >>
> > > > > >>
> > > > > >> Richard
> > > > > >> > > thanks, Vladimir
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> Richard Liang wrote:
> > > > > >> > >> > Hello All,
> > > > > >> > >> >
> > > > > >> > >> > Now let's talk about the TestNG groups. I have read the
> > > related
> > > > > >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
> > > > > >> Zakharov.
> > > > > >> > >> > All of them are good discussion about TestNG groups.
> > > > > >> > >> >
> > > > > >> > >> > IMHO, we may define Harmony test groups according the
> > > following
> > > > > 4
> > > > > >> > >> > dimensions:
> > > > > >> > >> >
> > > > > >> > >> > 1) [Platform] os.any, os.<platform id>
> > > > > >> > >> > *os.any* - group of tests which pass on any platform.
> > IMHO,
> > > > > >> most of
> > > > > >> > >> > our tests should be in this group.
> > > > > >> > >> > *os.<platform id>* - group of tests which are designed for
> > > one
> > > > > >> > >> > specific platform. A test may be in more than one of the
> > > > > >> groups. e.g
> > > > > >> > .,
> > > > > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> > > > > >> > >> >
> > > > > >> > >> > ** os.any and os.<platform id> are mutually exclusive,
> > > that
> > > > > >> is,
> > > > > >> > >> > tests in os.any group should not be in os.win.IA32.
> > > > > >> > >> >
> > > > > >> > >> > 2) [Test state] state.broken, state.broken.<platform id>
> > > > > >> > >> > *state.broken* - group of tests which fail on every
> > > platform,
> > > > > >> because
> > > > > >> > >> > of bugs of tests or implementation. We need to fix the
> > bugs
> > > of
> > > > > >> tests
> > > > > >> > >> > or implementation to make them pass.
> > > > > >> > >> > *state.broken.<platform id>* - groups of test which only
> > > fail
> > > > > >> on one
> > > > > >> > >> > specific platform. A test may be in more than one of the
> > > > > >> groups. e.g
> > > > > >> > .,
> > > > > >> > >> > @Test(groups={"state.broken.linux.IA32", "
> > > os.broken.linux.IA64
> > > > > "})
> > > > > >> > >> >
> > > > > >> > >> > **state.broken.<platform id> group may be used as a
> > > > > >> convenient
> > > > > >> > way
> > > > > >> > >> > to indicate that a test is platform-specific. e.g., If we
> > > > > >> support 10
> > > > > >> > >> > platforms, and one test are designed for 9 platforms
> > except
> > > for
> > > > > >> > MacOS,
> > > > > >> > >> > instead of list 9 os.<platform id>, we can just use
> > > > > >> > state.broken.MacOS
> > > > > >> > >> >
> > > > > >> > >> > 3) [Test type] type.api , type.impl
> > > > > >> > >> > *type.api* - group of tests which are tests for APIs in
> > the
> > > Java
> > > > > >> > >> > Specification
> > > > > >> > >> > *type.impl* - groups of tests which are tests for
> > > > > >> Harmony-specific
> > > > > >> > >> > implementation
> > > > > >> > >> >
> > > > > >> > >> > ** type.api and type.impl are also mutually exclusive.
> > > > > >> > >> >
> > > > > >> > >> > 4) [Test Level] level.unit, level.integration,
> > level.system,
> > > > > >> > >> > level.stress, etc. (Levels of Test refer to the increase
> > in
> > > > > >> > complexity
> > > > > >> > >> > as moving through test cycle. )
> > > > > >> > >> > ** A test may be in more than one of the groups.
> > > > > >> > >> > ** In fact, some tests such as System tests are the
> > > > > >> verification
> > > > > >> > of
> > > > > >> > >> > the entire system. Maybe we'll put them into a separate
> > > > > project.
> > > > > >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
> > > > > >> > >> >
> > > > > >> > >> > If we want to run all the unit test for APIs on windows,
> > we
> > > > > >> may use
> > > > > >> > >> > TestNG groups to select the tests:
> > > > > >> > >> > <groups>
> > > > > >> > >> > <run>
> > > > > >> > >> > <include name="os.any" />
> > > > > >> > >> > <include name="type.api" />
> > > > > >> > >> > <include name="os.win.IA32" />
> > > > > >> > >> > <exclude name=" state.broken" />
> > > > > >> > >> > <exclude name="state.broken.win.IA32" />
> > > > > >> > >> > </run>
> > > > > >> > >> > </groups>
> > > > > >> > >> >
> > > > > >> > >> Hello All,
> > > > > >> > >>
> > > > > >> > >> I'm sorry. It seems that the example does not work. I will
> > try
> > > to
> > > > > >> > figure
> > > > > >> > >> another example soon. ;-)
> > > > > >> > >>
> > > > > >> > >> Best regards,
> > > > > >> > >> Richard
> > > > > >> > >> >
> > > > > >> > >> > Well, I think our most of existing tests are in the groups
> > > of
> > > > > >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked
> > > TestNG
> > > > > >> to add
> > > > > >> > a
> > > > > >> > >> > new option "-groups" for its JUnitConverter which allow us
> > > to
> > > > > >> specify
> > > > > >> > >> > the test groups when migrate from JUnit test to TestNG
> > test.
> > > > > >> > >> >
> > > > > >> > >> > Thanks for reading so far, and I will highly appreciate
> > your
> > > > > >> comments
> > > > > >> > >> > or suggestion. ;-)
> > > > > >> > >> >
> > > > > >> > >>
> > > > > >> > >> --
> > > > > >> > >> Richard Liang
> > > > > >> > >> China Software Development Lab, IBM
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >>
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > >> Terms of use :
> > > http://incubator.apache.org/harmony/mailing.html
> > > > > >> > >> To unsubscribe, e-mail:
> > > > > >> harmony-dev-unsubscribe@incubator.apache.org
> > > > > >> > >> For additional commands, e-mail:
> > > > > >> harmony-dev-help@incubator.apache.org
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >
> > > > > >> >
> > > > > >> > --
> > > > > >> > Richard Liang
> > > > > >> > China Software Development Lab, IBM
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > ---------------------------------------------------------------------
> > > > > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > > >> > To unsubscribe, e-mail:
> > > harmony-dev-unsubscribe@incubator.apache.org
> > > > > >> > For additional commands, e-mail:
> > > > > harmony-dev-help@incubator.apache.org
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >
> > > ---------------------------------------------------------------------
> > > > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > > > To unsubscribe, e-mail:
> > harmony-dev-unsubscribe@incubator.apache.org
> > > > > > For additional commands, e-mail:
> > > harmony-dev-help@incubator.apache.org
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Richard Liang
> > > > > China Software Development Lab, IBM
> > > > >
> > > > >
> > > > >
> > > > >
> > ---------------------------------------------------------------------
> > > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail:
> > harmony-dev-help@incubator.apache.org
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >
> > >
> >
> >
> > --
> > Richard Liang
> > China Software Development Lab, IBM
> >
> >
>
>
> --
> Andrew Zhang
> China Software Development Lab, IBM
>
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Andrew Zhang <zh...@gmail.com>.
On 9/4/06, Richard Liang <ri...@gmail.com> wrote:
>
> I sent my reply this afternoon, but I have not received it. (it seems
> there
> is something wrong with my smtp server). So I send it again. :-)
>
> On 9/4/06, Alex Blewitt <al...@gmail.com> wrote:
> >
> > IMNSO it doesn't make sense to arbitrarily partition the tests based
> > on a moniker, such as 'integration test', 'unit test', 'regression
> > test' etc. For one thing, developers are generally not good at
> > agreeing on the difference between them :-)
>
>
>
> This is really a problem, however it might be simpler than we imagine. We
> are open to any discussion. ;-) Anyway, developers are required to write
> unit tests.
>
>
> If you've got fast and slow tests, then have a group for fast and slow
> > tests. Then you can choose to just run the fast tests, and any
> > automated build system can handle running the slow tests.
>
>
> IMHO, "fast or slow" may not be the key point. The question is whether we
> have any requirements to run only the regression tests.
I think your first proposal looks fine. Platform + os + state + api/impl
information are enough for all tests. We have to assure all tests pass on
every platform.
For quick run, developers can use their own short approach, like run single
test in IDE... It's not business of TestNG. :-)
Best regards,
> Richard
>
> Alex.
> >
> > On 04/09/06, Vladimir Ivanov <iv...@gmail.com> wrote:
> > > On 9/4/06, Richard Liang <ri...@gmail.com> wrote:
> > > >
> > > >
> > > >
> > > > Mikhail Loenko wrote:
> > > > > Hi Vladimir
> > > > >
> > > > > Could you please decribe for what purpose it will be used?
> > > > >
> > > > > I mean why one might have to either exclude or run only regression
> > > > tests?
> > > >
> > > > If running all tests takes up much time, running all regression test
> > > > (for one particular version) may be a convenient way to verify the
> > > > correctness of bug-fixing.
> > >
> > >
> > > Thanks Richard. It is exactly what I want to say :)
> > > On the other hand may be all proposed groups need similar explanation
> > (smt.
> > > like use-case for group)?
> > >
> > > thanks, Vladimir
> > >
> > > Best regards,
> > > > Richard.
> > > >
> > > > >
> > > > > Thanks,
> > > > > Mikhail
> > > > >
> > > > > 2006/8/30, Vladimir Ivanov <iv...@gmail.com>:
> > > > >> On 8/30/06, Richard Liang <ri...@gmail.com> wrote:
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > Vladimir Ivanov wrote:
> > > > >> > > Also some tag for regression tests should be added.
> > > > >> > Yes. Do you think we could annotate regression test as
> > > > >> > *level.regression*? Thanks a lot.
> > > > >>
> > > > >>
> > > > >> Yes, I do. While tests can have more than one group it will
> enough.
> > > > >> thanks, Vladimir
> > > > >>
> > > > >>
> > > > >> Richard
> > > > >> > > thanks, Vladimir
> > > > >> > >
> > > > >> > >
> > > > >> > > On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> Richard Liang wrote:
> > > > >> > >> > Hello All,
> > > > >> > >> >
> > > > >> > >> > Now let's talk about the TestNG groups. I have read the
> > related
> > > > >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
> > > > >> Zakharov.
> > > > >> > >> > All of them are good discussion about TestNG groups.
> > > > >> > >> >
> > > > >> > >> > IMHO, we may define Harmony test groups according the
> > following
> > > > 4
> > > > >> > >> > dimensions:
> > > > >> > >> >
> > > > >> > >> > 1) [Platform] os.any, os.<platform id>
> > > > >> > >> > *os.any* - group of tests which pass on any platform.
> IMHO,
> > > > >> most of
> > > > >> > >> > our tests should be in this group.
> > > > >> > >> > *os.<platform id>* - group of tests which are designed for
> > one
> > > > >> > >> > specific platform. A test may be in more than one of the
> > > > >> groups. e.g
> > > > >> > .,
> > > > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> > > > >> > >> >
> > > > >> > >> > ** os.any and os.<platform id> are mutually exclusive,
> > that
> > > > >> is,
> > > > >> > >> > tests in os.any group should not be in os.win.IA32.
> > > > >> > >> >
> > > > >> > >> > 2) [Test state] state.broken, state.broken.<platform id>
> > > > >> > >> > *state.broken* - group of tests which fail on every
> > platform,
> > > > >> because
> > > > >> > >> > of bugs of tests or implementation. We need to fix the
> bugs
> > of
> > > > >> tests
> > > > >> > >> > or implementation to make them pass.
> > > > >> > >> > *state.broken.<platform id>* - groups of test which only
> > fail
> > > > >> on one
> > > > >> > >> > specific platform. A test may be in more than one of the
> > > > >> groups. e.g
> > > > >> > .,
> > > > >> > >> > @Test(groups={"state.broken.linux.IA32", "
> > os.broken.linux.IA64
> > > > "})
> > > > >> > >> >
> > > > >> > >> > **state.broken.<platform id> group may be used as a
> > > > >> convenient
> > > > >> > way
> > > > >> > >> > to indicate that a test is platform-specific. e.g., If we
> > > > >> support 10
> > > > >> > >> > platforms, and one test are designed for 9 platforms
> except
> > for
> > > > >> > MacOS,
> > > > >> > >> > instead of list 9 os.<platform id>, we can just use
> > > > >> > state.broken.MacOS
> > > > >> > >> >
> > > > >> > >> > 3) [Test type] type.api , type.impl
> > > > >> > >> > *type.api* - group of tests which are tests for APIs in
> the
> > Java
> > > > >> > >> > Specification
> > > > >> > >> > *type.impl* - groups of tests which are tests for
> > > > >> Harmony-specific
> > > > >> > >> > implementation
> > > > >> > >> >
> > > > >> > >> > ** type.api and type.impl are also mutually exclusive.
> > > > >> > >> >
> > > > >> > >> > 4) [Test Level] level.unit, level.integration,
> level.system,
> > > > >> > >> > level.stress, etc. (Levels of Test refer to the increase
> in
> > > > >> > complexity
> > > > >> > >> > as moving through test cycle. )
> > > > >> > >> > ** A test may be in more than one of the groups.
> > > > >> > >> > ** In fact, some tests such as System tests are the
> > > > >> verification
> > > > >> > of
> > > > >> > >> > the entire system. Maybe we'll put them into a separate
> > > > project.
> > > > >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
> > > > >> > >> >
> > > > >> > >> > If we want to run all the unit test for APIs on windows,
> we
> > > > >> may use
> > > > >> > >> > TestNG groups to select the tests:
> > > > >> > >> > <groups>
> > > > >> > >> > <run>
> > > > >> > >> > <include name="os.any" />
> > > > >> > >> > <include name="type.api" />
> > > > >> > >> > <include name="os.win.IA32" />
> > > > >> > >> > <exclude name=" state.broken" />
> > > > >> > >> > <exclude name="state.broken.win.IA32" />
> > > > >> > >> > </run>
> > > > >> > >> > </groups>
> > > > >> > >> >
> > > > >> > >> Hello All,
> > > > >> > >>
> > > > >> > >> I'm sorry. It seems that the example does not work. I will
> try
> > to
> > > > >> > figure
> > > > >> > >> another example soon. ;-)
> > > > >> > >>
> > > > >> > >> Best regards,
> > > > >> > >> Richard
> > > > >> > >> >
> > > > >> > >> > Well, I think our most of existing tests are in the groups
> > of
> > > > >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked
> > TestNG
> > > > >> to add
> > > > >> > a
> > > > >> > >> > new option "-groups" for its JUnitConverter which allow us
> > to
> > > > >> specify
> > > > >> > >> > the test groups when migrate from JUnit test to TestNG
> test.
> > > > >> > >> >
> > > > >> > >> > Thanks for reading so far, and I will highly appreciate
> your
> > > > >> comments
> > > > >> > >> > or suggestion. ;-)
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >> --
> > > > >> > >> Richard Liang
> > > > >> > >> China Software Development Lab, IBM
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> > >> Terms of use :
> > http://incubator.apache.org/harmony/mailing.html
> > > > >> > >> To unsubscribe, e-mail:
> > > > >> harmony-dev-unsubscribe@incubator.apache.org
> > > > >> > >> For additional commands, e-mail:
> > > > >> harmony-dev-help@incubator.apache.org
> > > > >> > >>
> > > > >> > >>
> > > > >> > >
> > > > >> >
> > > > >> > --
> > > > >> > Richard Liang
> > > > >> > China Software Development Lab, IBM
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > ---------------------------------------------------------------------
> > > > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > >> > To unsubscribe, e-mail:
> > harmony-dev-unsubscribe@incubator.apache.org
> > > > >> > For additional commands, e-mail:
> > > > harmony-dev-help@incubator.apache.org
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >
> > > > >
> > ---------------------------------------------------------------------
> > > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > > To unsubscribe, e-mail:
> harmony-dev-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail:
> > harmony-dev-help@incubator.apache.org
> > > > >
> > > > >
> > > >
> > > > --
> > > > Richard Liang
> > > > China Software Development Lab, IBM
> > > >
> > > >
> > > >
> > > >
> ---------------------------------------------------------------------
> > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail:
> harmony-dev-help@incubator.apache.org
> > > >
> > > >
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>
> --
> Richard Liang
> China Software Development Lab, IBM
>
>
--
Andrew Zhang
China Software Development Lab, IBM
Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
I sent my reply this afternoon, but I have not received it. (it seems there
is something wrong with my smtp server). So I send it again. :-)
On 9/4/06, Alex Blewitt <al...@gmail.com> wrote:
>
> IMNSO it doesn't make sense to arbitrarily partition the tests based
> on a moniker, such as 'integration test', 'unit test', 'regression
> test' etc. For one thing, developers are generally not good at
> agreeing on the difference between them :-)
This is really a problem, however it might be simpler than we imagine. We
are open to any discussion. ;-) Anyway, developers are required to write
unit tests.
If you've got fast and slow tests, then have a group for fast and slow
> tests. Then you can choose to just run the fast tests, and any
> automated build system can handle running the slow tests.
IMHO, "fast or slow" may not be the key point. The question is whether we
have any requirements to run only the regression tests.
Best regards,
Richard
Alex.
>
> On 04/09/06, Vladimir Ivanov <iv...@gmail.com> wrote:
> > On 9/4/06, Richard Liang <ri...@gmail.com> wrote:
> > >
> > >
> > >
> > > Mikhail Loenko wrote:
> > > > Hi Vladimir
> > > >
> > > > Could you please decribe for what purpose it will be used?
> > > >
> > > > I mean why one might have to either exclude or run only regression
> > > tests?
> > >
> > > If running all tests takes up much time, running all regression test
> > > (for one particular version) may be a convenient way to verify the
> > > correctness of bug-fixing.
> >
> >
> > Thanks Richard. It is exactly what I want to say :)
> > On the other hand may be all proposed groups need similar explanation
> (smt.
> > like use-case for group)?
> >
> > thanks, Vladimir
> >
> > Best regards,
> > > Richard.
> > >
> > > >
> > > > Thanks,
> > > > Mikhail
> > > >
> > > > 2006/8/30, Vladimir Ivanov <iv...@gmail.com>:
> > > >> On 8/30/06, Richard Liang <ri...@gmail.com> wrote:
> > > >> >
> > > >> >
> > > >> >
> > > >> > Vladimir Ivanov wrote:
> > > >> > > Also some tag for regression tests should be added.
> > > >> > Yes. Do you think we could annotate regression test as
> > > >> > *level.regression*? Thanks a lot.
> > > >>
> > > >>
> > > >> Yes, I do. While tests can have more than one group it will enough.
> > > >> thanks, Vladimir
> > > >>
> > > >>
> > > >> Richard
> > > >> > > thanks, Vladimir
> > > >> > >
> > > >> > >
> > > >> > > On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >> Richard Liang wrote:
> > > >> > >> > Hello All,
> > > >> > >> >
> > > >> > >> > Now let's talk about the TestNG groups. I have read the
> related
> > > >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
> > > >> Zakharov.
> > > >> > >> > All of them are good discussion about TestNG groups.
> > > >> > >> >
> > > >> > >> > IMHO, we may define Harmony test groups according the
> following
> > > 4
> > > >> > >> > dimensions:
> > > >> > >> >
> > > >> > >> > 1) [Platform] os.any, os.<platform id>
> > > >> > >> > *os.any* - group of tests which pass on any platform. IMHO,
> > > >> most of
> > > >> > >> > our tests should be in this group.
> > > >> > >> > *os.<platform id>* - group of tests which are designed for
> one
> > > >> > >> > specific platform. A test may be in more than one of the
> > > >> groups. e.g
> > > >> > .,
> > > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> > > >> > >> >
> > > >> > >> > ** os.any and os.<platform id> are mutually exclusive,
> that
> > > >> is,
> > > >> > >> > tests in os.any group should not be in os.win.IA32.
> > > >> > >> >
> > > >> > >> > 2) [Test state] state.broken, state.broken.<platform id>
> > > >> > >> > *state.broken* - group of tests which fail on every
> platform,
> > > >> because
> > > >> > >> > of bugs of tests or implementation. We need to fix the bugs
> of
> > > >> tests
> > > >> > >> > or implementation to make them pass.
> > > >> > >> > *state.broken.<platform id>* - groups of test which only
> fail
> > > >> on one
> > > >> > >> > specific platform. A test may be in more than one of the
> > > >> groups. e.g
> > > >> > .,
> > > >> > >> > @Test(groups={"state.broken.linux.IA32", "
> os.broken.linux.IA64
> > > "})
> > > >> > >> >
> > > >> > >> > **state.broken.<platform id> group may be used as a
> > > >> convenient
> > > >> > way
> > > >> > >> > to indicate that a test is platform-specific. e.g., If we
> > > >> support 10
> > > >> > >> > platforms, and one test are designed for 9 platforms except
> for
> > > >> > MacOS,
> > > >> > >> > instead of list 9 os.<platform id>, we can just use
> > > >> > state.broken.MacOS
> > > >> > >> >
> > > >> > >> > 3) [Test type] type.api , type.impl
> > > >> > >> > *type.api* - group of tests which are tests for APIs in the
> Java
> > > >> > >> > Specification
> > > >> > >> > *type.impl* - groups of tests which are tests for
> > > >> Harmony-specific
> > > >> > >> > implementation
> > > >> > >> >
> > > >> > >> > ** type.api and type.impl are also mutually exclusive.
> > > >> > >> >
> > > >> > >> > 4) [Test Level] level.unit, level.integration, level.system,
> > > >> > >> > level.stress, etc. (Levels of Test refer to the increase in
> > > >> > complexity
> > > >> > >> > as moving through test cycle. )
> > > >> > >> > ** A test may be in more than one of the groups.
> > > >> > >> > ** In fact, some tests such as System tests are the
> > > >> verification
> > > >> > of
> > > >> > >> > the entire system. Maybe we'll put them into a separate
> > > project.
> > > >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
> > > >> > >> >
> > > >> > >> > If we want to run all the unit test for APIs on windows, we
> > > >> may use
> > > >> > >> > TestNG groups to select the tests:
> > > >> > >> > <groups>
> > > >> > >> > <run>
> > > >> > >> > <include name="os.any" />
> > > >> > >> > <include name="type.api" />
> > > >> > >> > <include name="os.win.IA32" />
> > > >> > >> > <exclude name=" state.broken" />
> > > >> > >> > <exclude name="state.broken.win.IA32" />
> > > >> > >> > </run>
> > > >> > >> > </groups>
> > > >> > >> >
> > > >> > >> Hello All,
> > > >> > >>
> > > >> > >> I'm sorry. It seems that the example does not work. I will try
> to
> > > >> > figure
> > > >> > >> another example soon. ;-)
> > > >> > >>
> > > >> > >> Best regards,
> > > >> > >> Richard
> > > >> > >> >
> > > >> > >> > Well, I think our most of existing tests are in the groups
> of
> > > >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked
> TestNG
> > > >> to add
> > > >> > a
> > > >> > >> > new option "-groups" for its JUnitConverter which allow us
> to
> > > >> specify
> > > >> > >> > the test groups when migrate from JUnit test to TestNG test.
> > > >> > >> >
> > > >> > >> > Thanks for reading so far, and I will highly appreciate your
> > > >> comments
> > > >> > >> > or suggestion. ;-)
> > > >> > >> >
> > > >> > >>
> > > >> > >> --
> > > >> > >> Richard Liang
> > > >> > >> China Software Development Lab, IBM
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> > >> Terms of use :
> http://incubator.apache.org/harmony/mailing.html
> > > >> > >> To unsubscribe, e-mail:
> > > >> harmony-dev-unsubscribe@incubator.apache.org
> > > >> > >> For additional commands, e-mail:
> > > >> harmony-dev-help@incubator.apache.org
> > > >> > >>
> > > >> > >>
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > Richard Liang
> > > >> > China Software Development Lab, IBM
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> ---------------------------------------------------------------------
> > > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > >> > To unsubscribe, e-mail:
> harmony-dev-unsubscribe@incubator.apache.org
> > > >> > For additional commands, e-mail:
> > > harmony-dev-help@incubator.apache.org
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >
> > > >
> ---------------------------------------------------------------------
> > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail:
> harmony-dev-help@incubator.apache.org
> > > >
> > > >
> > >
> > > --
> > > Richard Liang
> > > China Software Development Lab, IBM
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >
> > >
> >
> >
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Richard Liang
China Software Development Lab, IBM
Re: [classlib][TestNG] groups of Harmony test
Posted by Alexey Varlamov <al...@gmail.com>.
2006/9/4, Richard Liang <ri...@gmail.com>:
>
>
> Alex Blewitt wrote:
> > IMNSO it doesn't make sense to arbitrarily partition the tests based
> > on a moniker, such as 'integration test', 'unit test', 'regression
> > test' etc. For one thing, developers are generally not good at
> > agreeing on the difference between them :-)
>
> This is really a problem, however it might be simpler than we imagine.
> We are open to any discussion. ;-) Anyway, developers are required to
> write unit tests.
>
> >
> > If you've got fast and slow tests, then have a group for fast and slow
> > tests. Then you can choose to just run the fast tests, and any
> > automated build system can handle running the slow tests.
> IMHO, "fast or slow" may not be the key point. The question is whether
> we have any requirements to run only the regression tests.
I believe we have not. If a testcase was added to prevent regression,
it basically means that there was a hole in test coverage for some
reason. Provided that such "holes" are scattered randomly through the
given module, what for we may want to run such a sieve test suite?
I can think of the sole reason, that regression tests may
*potentially* highlight weak spots in the code (or design) which are
more prone to fail during code evolution. But still I see no reason to
run only regression tests ignoring others. I'd rather second Alex in
fast/slow grouping orthogonal to regressions.
As for information purposes, Mikhail (?) suggested good idea -
explicitly specify no. of the issue in descripton comment (or
annotation).
[snip]
--
Alexey
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
Alex Blewitt wrote:
> IMNSO it doesn't make sense to arbitrarily partition the tests based
> on a moniker, such as 'integration test', 'unit test', 'regression
> test' etc. For one thing, developers are generally not good at
> agreeing on the difference between them :-)
This is really a problem, however it might be simpler than we imagine.
We are open to any discussion. ;-) Anyway, developers are required to
write unit tests.
>
> If you've got fast and slow tests, then have a group for fast and slow
> tests. Then you can choose to just run the fast tests, and any
> automated build system can handle running the slow tests.
IMHO, "fast or slow" may not be the key point. The question is whether
we have any requirements to run only the regression tests.
>
> Alex.
>
> On 04/09/06, Vladimir Ivanov <iv...@gmail.com> wrote:
>> On 9/4/06, Richard Liang <ri...@gmail.com> wrote:
>> >
>> >
>> >
>> > Mikhail Loenko wrote:
>> > > Hi Vladimir
>> > >
>> > > Could you please decribe for what purpose it will be used?
>> > >
>> > > I mean why one might have to either exclude or run only regression
>> > tests?
>> >
>> > If running all tests takes up much time, running all regression test
>> > (for one particular version) may be a convenient way to verify the
>> > correctness of bug-fixing.
>>
>>
>> Thanks Richard. It is exactly what I want to say :)
>> On the other hand may be all proposed groups need similar explanation
>> (smt.
>> like use-case for group)?
>>
>> thanks, Vladimir
>>
>> Best regards,
>> > Richard.
>> >
>> > >
>> > > Thanks,
>> > > Mikhail
>> > >
>> > > 2006/8/30, Vladimir Ivanov <iv...@gmail.com>:
>> > >> On 8/30/06, Richard Liang <ri...@gmail.com> wrote:
>> > >> >
>> > >> >
>> > >> >
>> > >> > Vladimir Ivanov wrote:
>> > >> > > Also some tag for regression tests should be added.
>> > >> > Yes. Do you think we could annotate regression test as
>> > >> > *level.regression*? Thanks a lot.
>> > >>
>> > >>
>> > >> Yes, I do. While tests can have more than one group it will enough.
>> > >> thanks, Vladimir
>> > >>
>> > >>
>> > >> Richard
>> > >> > > thanks, Vladimir
>> > >> > >
>> > >> > >
>> > >> > > On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
>> > >> > >>
>> > >> > >>
>> > >> > >>
>> > >> > >> Richard Liang wrote:
>> > >> > >> > Hello All,
>> > >> > >> >
>> > >> > >> > Now let's talk about the TestNG groups. I have read the
>> related
>> > >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
>> > >> Zakharov.
>> > >> > >> > All of them are good discussion about TestNG groups.
>> > >> > >> >
>> > >> > >> > IMHO, we may define Harmony test groups according the
>> following
>> > 4
>> > >> > >> > dimensions:
>> > >> > >> >
>> > >> > >> > 1) [Platform] os.any, os.<platform id>
>> > >> > >> > *os.any* - group of tests which pass on any platform. IMHO,
>> > >> most of
>> > >> > >> > our tests should be in this group.
>> > >> > >> > *os.<platform id>* - group of tests which are designed
>> for one
>> > >> > >> > specific platform. A test may be in more than one of the
>> > >> groups. e.g
>> > >> > .,
>> > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
>> > >> > >> >
>> > >> > >> > ** os.any and os.<platform id> are mutually exclusive,
>> that
>> > >> is,
>> > >> > >> > tests in os.any group should not be in os.win.IA32.
>> > >> > >> >
>> > >> > >> > 2) [Test state] state.broken, state.broken.<platform id>
>> > >> > >> > *state.broken* - group of tests which fail on every
>> platform,
>> > >> because
>> > >> > >> > of bugs of tests or implementation. We need to fix the
>> bugs of
>> > >> tests
>> > >> > >> > or implementation to make them pass.
>> > >> > >> > *state.broken.<platform id>* - groups of test which only
>> fail
>> > >> on one
>> > >> > >> > specific platform. A test may be in more than one of the
>> > >> groups. e.g
>> > >> > .,
>> > >> > >> > @Test(groups={"state.broken.linux.IA32",
>> "os.broken.linux.IA64
>> > "})
>> > >> > >> >
>> > >> > >> > **state.broken.<platform id> group may be used as a
>> > >> convenient
>> > >> > way
>> > >> > >> > to indicate that a test is platform-specific. e.g., If we
>> > >> support 10
>> > >> > >> > platforms, and one test are designed for 9 platforms
>> except for
>> > >> > MacOS,
>> > >> > >> > instead of list 9 os.<platform id>, we can just use
>> > >> > state.broken.MacOS
>> > >> > >> >
>> > >> > >> > 3) [Test type] type.api , type.impl
>> > >> > >> > *type.api* - group of tests which are tests for APIs in
>> the Java
>> > >> > >> > Specification
>> > >> > >> > *type.impl* - groups of tests which are tests for
>> > >> Harmony-specific
>> > >> > >> > implementation
>> > >> > >> >
>> > >> > >> > ** type.api and type.impl are also mutually exclusive.
>> > >> > >> >
>> > >> > >> > 4) [Test Level] level.unit, level.integration, level.system,
>> > >> > >> > level.stress, etc. (Levels of Test refer to the increase in
>> > >> > complexity
>> > >> > >> > as moving through test cycle. )
>> > >> > >> > ** A test may be in more than one of the groups.
>> > >> > >> > ** In fact, some tests such as System tests are the
>> > >> verification
>> > >> > of
>> > >> > >> > the entire system. Maybe we'll put them into a separate
>> > project.
>> > >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
>> > >> > >> >
>> > >> > >> > If we want to run all the unit test for APIs on windows, we
>> > >> may use
>> > >> > >> > TestNG groups to select the tests:
>> > >> > >> > <groups>
>> > >> > >> > <run>
>> > >> > >> > <include name="os.any" />
>> > >> > >> > <include name="type.api" />
>> > >> > >> > <include name="os.win.IA32" />
>> > >> > >> > <exclude name=" state.broken" />
>> > >> > >> > <exclude name="state.broken.win.IA32" />
>> > >> > >> > </run>
>> > >> > >> > </groups>
>> > >> > >> >
>> > >> > >> Hello All,
>> > >> > >>
>> > >> > >> I'm sorry. It seems that the example does not work. I will
>> try to
>> > >> > figure
>> > >> > >> another example soon. ;-)
>> > >> > >>
>> > >> > >> Best regards,
>> > >> > >> Richard
>> > >> > >> >
>> > >> > >> > Well, I think our most of existing tests are in the
>> groups of
>> > >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked
>> TestNG
>> > >> to add
>> > >> > a
>> > >> > >> > new option "-groups" for its JUnitConverter which allow
>> us to
>> > >> specify
>> > >> > >> > the test groups when migrate from JUnit test to TestNG test.
>> > >> > >> >
>> > >> > >> > Thanks for reading so far, and I will highly appreciate your
>> > >> comments
>> > >> > >> > or suggestion. ;-)
>> > >> > >> >
>> > >> > >>
>> > >> > >> --
>> > >> > >> Richard Liang
>> > >> > >> China Software Development Lab, IBM
>> > >> > >>
>> > >> > >>
>> > >> > >>
>> > >> > >>
>> > >>
>> ---------------------------------------------------------------------
>> > >> > >> Terms of use :
>> http://incubator.apache.org/harmony/mailing.html
>> > >> > >> To unsubscribe, e-mail:
>> > >> harmony-dev-unsubscribe@incubator.apache.org
>> > >> > >> For additional commands, e-mail:
>> > >> harmony-dev-help@incubator.apache.org
>> > >> > >>
>> > >> > >>
>> > >> > >
>> > >> >
>> > >> > --
>> > >> > Richard Liang
>> > >> > China Software Development Lab, IBM
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> ---------------------------------------------------------------------
>> > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>> > >> > To unsubscribe, e-mail:
>> harmony-dev-unsubscribe@incubator.apache.org
>> > >> > For additional commands, e-mail:
>> > harmony-dev-help@incubator.apache.org
>> > >> >
>> > >> >
>> > >>
>> > >>
>> > >
>> > >
>> ---------------------------------------------------------------------
>> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
>> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> > > For additional commands, e-mail:
>> harmony-dev-help@incubator.apache.org
>> > >
>> > >
>> >
>> > --
>> > Richard Liang
>> > China Software Development Lab, IBM
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>> >
>> >
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: Re: [classlib][TestNG] groups of Harmony test
Posted by Alex Blewitt <al...@gmail.com>.
IMNSO it doesn't make sense to arbitrarily partition the tests based
on a moniker, such as 'integration test', 'unit test', 'regression
test' etc. For one thing, developers are generally not good at
agreeing on the difference between them :-)
If you've got fast and slow tests, then have a group for fast and slow
tests. Then you can choose to just run the fast tests, and any
automated build system can handle running the slow tests.
Alex.
On 04/09/06, Vladimir Ivanov <iv...@gmail.com> wrote:
> On 9/4/06, Richard Liang <ri...@gmail.com> wrote:
> >
> >
> >
> > Mikhail Loenko wrote:
> > > Hi Vladimir
> > >
> > > Could you please decribe for what purpose it will be used?
> > >
> > > I mean why one might have to either exclude or run only regression
> > tests?
> >
> > If running all tests takes up much time, running all regression test
> > (for one particular version) may be a convenient way to verify the
> > correctness of bug-fixing.
>
>
> Thanks Richard. It is exactly what I want to say :)
> On the other hand may be all proposed groups need similar explanation (smt.
> like use-case for group)?
>
> thanks, Vladimir
>
> Best regards,
> > Richard.
> >
> > >
> > > Thanks,
> > > Mikhail
> > >
> > > 2006/8/30, Vladimir Ivanov <iv...@gmail.com>:
> > >> On 8/30/06, Richard Liang <ri...@gmail.com> wrote:
> > >> >
> > >> >
> > >> >
> > >> > Vladimir Ivanov wrote:
> > >> > > Also some tag for regression tests should be added.
> > >> > Yes. Do you think we could annotate regression test as
> > >> > *level.regression*? Thanks a lot.
> > >>
> > >>
> > >> Yes, I do. While tests can have more than one group it will enough.
> > >> thanks, Vladimir
> > >>
> > >>
> > >> Richard
> > >> > > thanks, Vladimir
> > >> > >
> > >> > >
> > >> > > On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >> Richard Liang wrote:
> > >> > >> > Hello All,
> > >> > >> >
> > >> > >> > Now let's talk about the TestNG groups. I have read the related
> > >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
> > >> Zakharov.
> > >> > >> > All of them are good discussion about TestNG groups.
> > >> > >> >
> > >> > >> > IMHO, we may define Harmony test groups according the following
> > 4
> > >> > >> > dimensions:
> > >> > >> >
> > >> > >> > 1) [Platform] os.any, os.<platform id>
> > >> > >> > *os.any* - group of tests which pass on any platform. IMHO,
> > >> most of
> > >> > >> > our tests should be in this group.
> > >> > >> > *os.<platform id>* - group of tests which are designed for one
> > >> > >> > specific platform. A test may be in more than one of the
> > >> groups. e.g
> > >> > .,
> > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> > >> > >> >
> > >> > >> > ** os.any and os.<platform id> are mutually exclusive, that
> > >> is,
> > >> > >> > tests in os.any group should not be in os.win.IA32.
> > >> > >> >
> > >> > >> > 2) [Test state] state.broken, state.broken.<platform id>
> > >> > >> > *state.broken* - group of tests which fail on every platform,
> > >> because
> > >> > >> > of bugs of tests or implementation. We need to fix the bugs of
> > >> tests
> > >> > >> > or implementation to make them pass.
> > >> > >> > *state.broken.<platform id>* - groups of test which only fail
> > >> on one
> > >> > >> > specific platform. A test may be in more than one of the
> > >> groups. e.g
> > >> > .,
> > >> > >> > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64
> > "})
> > >> > >> >
> > >> > >> > **state.broken.<platform id> group may be used as a
> > >> convenient
> > >> > way
> > >> > >> > to indicate that a test is platform-specific. e.g., If we
> > >> support 10
> > >> > >> > platforms, and one test are designed for 9 platforms except for
> > >> > MacOS,
> > >> > >> > instead of list 9 os.<platform id>, we can just use
> > >> > state.broken.MacOS
> > >> > >> >
> > >> > >> > 3) [Test type] type.api , type.impl
> > >> > >> > *type.api* - group of tests which are tests for APIs in the Java
> > >> > >> > Specification
> > >> > >> > *type.impl* - groups of tests which are tests for
> > >> Harmony-specific
> > >> > >> > implementation
> > >> > >> >
> > >> > >> > ** type.api and type.impl are also mutually exclusive.
> > >> > >> >
> > >> > >> > 4) [Test Level] level.unit, level.integration, level.system,
> > >> > >> > level.stress, etc. (Levels of Test refer to the increase in
> > >> > complexity
> > >> > >> > as moving through test cycle. )
> > >> > >> > ** A test may be in more than one of the groups.
> > >> > >> > ** In fact, some tests such as System tests are the
> > >> verification
> > >> > of
> > >> > >> > the entire system. Maybe we'll put them into a separate
> > project.
> > >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
> > >> > >> >
> > >> > >> > If we want to run all the unit test for APIs on windows, we
> > >> may use
> > >> > >> > TestNG groups to select the tests:
> > >> > >> > <groups>
> > >> > >> > <run>
> > >> > >> > <include name="os.any" />
> > >> > >> > <include name="type.api" />
> > >> > >> > <include name="os.win.IA32" />
> > >> > >> > <exclude name=" state.broken" />
> > >> > >> > <exclude name="state.broken.win.IA32" />
> > >> > >> > </run>
> > >> > >> > </groups>
> > >> > >> >
> > >> > >> Hello All,
> > >> > >>
> > >> > >> I'm sorry. It seems that the example does not work. I will try to
> > >> > figure
> > >> > >> another example soon. ;-)
> > >> > >>
> > >> > >> Best regards,
> > >> > >> Richard
> > >> > >> >
> > >> > >> > Well, I think our most of existing tests are in the groups of
> > >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked TestNG
> > >> to add
> > >> > a
> > >> > >> > new option "-groups" for its JUnitConverter which allow us to
> > >> specify
> > >> > >> > the test groups when migrate from JUnit test to TestNG test.
> > >> > >> >
> > >> > >> > Thanks for reading so far, and I will highly appreciate your
> > >> comments
> > >> > >> > or suggestion. ;-)
> > >> > >> >
> > >> > >>
> > >> > >> --
> > >> > >> Richard Liang
> > >> > >> China Software Development Lab, IBM
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >>
> > >> ---------------------------------------------------------------------
> > >> > >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> > >> > >> To unsubscribe, e-mail:
> > >> harmony-dev-unsubscribe@incubator.apache.org
> > >> > >> For additional commands, e-mail:
> > >> harmony-dev-help@incubator.apache.org
> > >> > >>
> > >> > >>
> > >> > >
> > >> >
> > >> > --
> > >> > Richard Liang
> > >> > China Software Development Lab, IBM
> > >> >
> > >> >
> > >> >
> > >> > ---------------------------------------------------------------------
> > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > >> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > >> > For additional commands, e-mail:
> > harmony-dev-help@incubator.apache.org
> > >> >
> > >> >
> > >>
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >
> > >
> >
> > --
> > Richard Liang
> > China Software Development Lab, IBM
> >
> >
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
Vladimir Ivanov wrote:
> On 9/4/06, Richard Liang <ri...@gmail.com> wrote:
>>
>>
>>
>> Mikhail Loenko wrote:
>> > Hi Vladimir
>> >
>> > Could you please decribe for what purpose it will be used?
>> >
>> > I mean why one might have to either exclude or run only regression
>> tests?
>>
>> If running all tests takes up much time, running all regression test
>> (for one particular version) may be a convenient way to verify the
>> correctness of bug-fixing.
>
>
> Thanks Richard. It is exactly what I want to say :)
> On the other hand may be all proposed groups need similar explanation
> (smt.
> like use-case for group)?
>
Good idea ;-)
I will add some use-cases for the groups. And *everyone* is welcome to
edit the wiki page [1]
[1]http://wiki.apache.org/harmony/Testing_Convention
Best regards,
Richard
> thanks, Vladimir
>
> Best regards,
>> Richard.
>>
>> >
>> > Thanks,
>> > Mikhail
>> >
>> > 2006/8/30, Vladimir Ivanov <iv...@gmail.com>:
>> >> On 8/30/06, Richard Liang <ri...@gmail.com> wrote:
>> >> >
>> >> >
>> >> >
>> >> > Vladimir Ivanov wrote:
>> >> > > Also some tag for regression tests should be added.
>> >> > Yes. Do you think we could annotate regression test as
>> >> > *level.regression*? Thanks a lot.
>> >>
>> >>
>> >> Yes, I do. While tests can have more than one group it will enough.
>> >> thanks, Vladimir
>> >>
>> >>
>> >> Richard
>> >> > > thanks, Vladimir
>> >> > >
>> >> > >
>> >> > > On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >> Richard Liang wrote:
>> >> > >> > Hello All,
>> >> > >> >
>> >> > >> > Now let's talk about the TestNG groups. I have read the
>> related
>> >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
>> >> Zakharov.
>> >> > >> > All of them are good discussion about TestNG groups.
>> >> > >> >
>> >> > >> > IMHO, we may define Harmony test groups according the
>> following
>> 4
>> >> > >> > dimensions:
>> >> > >> >
>> >> > >> > 1) [Platform] os.any, os.<platform id>
>> >> > >> > *os.any* - group of tests which pass on any platform. IMHO,
>> >> most of
>> >> > >> > our tests should be in this group.
>> >> > >> > *os.<platform id>* - group of tests which are designed for one
>> >> > >> > specific platform. A test may be in more than one of the
>> >> groups. e.g
>> >> > .,
>> >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
>> >> > >> >
>> >> > >> > ** os.any and os.<platform id> are mutually exclusive, that
>> >> is,
>> >> > >> > tests in os.any group should not be in os.win.IA32.
>> >> > >> >
>> >> > >> > 2) [Test state] state.broken, state.broken.<platform id>
>> >> > >> > *state.broken* - group of tests which fail on every platform,
>> >> because
>> >> > >> > of bugs of tests or implementation. We need to fix the bugs of
>> >> tests
>> >> > >> > or implementation to make them pass.
>> >> > >> > *state.broken.<platform id>* - groups of test which only fail
>> >> on one
>> >> > >> > specific platform. A test may be in more than one of the
>> >> groups. e.g
>> >> > .,
>> >> > >> > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64
>> "})
>> >> > >> >
>> >> > >> > **state.broken.<platform id> group may be used as a
>> >> convenient
>> >> > way
>> >> > >> > to indicate that a test is platform-specific. e.g., If we
>> >> support 10
>> >> > >> > platforms, and one test are designed for 9 platforms except
>> for
>> >> > MacOS,
>> >> > >> > instead of list 9 os.<platform id>, we can just use
>> >> > state.broken.MacOS
>> >> > >> >
>> >> > >> > 3) [Test type] type.api , type.impl
>> >> > >> > *type.api* - group of tests which are tests for APIs in the
>> Java
>> >> > >> > Specification
>> >> > >> > *type.impl* - groups of tests which are tests for
>> >> Harmony-specific
>> >> > >> > implementation
>> >> > >> >
>> >> > >> > ** type.api and type.impl are also mutually exclusive.
>> >> > >> >
>> >> > >> > 4) [Test Level] level.unit, level.integration, level.system,
>> >> > >> > level.stress, etc. (Levels of Test refer to the increase in
>> >> > complexity
>> >> > >> > as moving through test cycle. )
>> >> > >> > ** A test may be in more than one of the groups.
>> >> > >> > ** In fact, some tests such as System tests are the
>> >> verification
>> >> > of
>> >> > >> > the entire system. Maybe we'll put them into a separate
>> project.
>> >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
>> >> > >> >
>> >> > >> > If we want to run all the unit test for APIs on windows, we
>> >> may use
>> >> > >> > TestNG groups to select the tests:
>> >> > >> > <groups>
>> >> > >> > <run>
>> >> > >> > <include name="os.any" />
>> >> > >> > <include name="type.api" />
>> >> > >> > <include name="os.win.IA32" />
>> >> > >> > <exclude name=" state.broken" />
>> >> > >> > <exclude name="state.broken.win.IA32" />
>> >> > >> > </run>
>> >> > >> > </groups>
>> >> > >> >
>> >> > >> Hello All,
>> >> > >>
>> >> > >> I'm sorry. It seems that the example does not work. I will
>> try to
>> >> > figure
>> >> > >> another example soon. ;-)
>> >> > >>
>> >> > >> Best regards,
>> >> > >> Richard
>> >> > >> >
>> >> > >> > Well, I think our most of existing tests are in the groups of
>> >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked TestNG
>> >> to add
>> >> > a
>> >> > >> > new option "-groups" for its JUnitConverter which allow us to
>> >> specify
>> >> > >> > the test groups when migrate from JUnit test to TestNG test.
>> >> > >> >
>> >> > >> > Thanks for reading so far, and I will highly appreciate your
>> >> comments
>> >> > >> > or suggestion. ;-)
>> >> > >> >
>> >> > >>
>> >> > >> --
>> >> > >> Richard Liang
>> >> > >> China Software Development Lab, IBM
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >>
>> >> ---------------------------------------------------------------------
>> >> > >> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> >> > >> To unsubscribe, e-mail:
>> >> harmony-dev-unsubscribe@incubator.apache.org
>> >> > >> For additional commands, e-mail:
>> >> harmony-dev-help@incubator.apache.org
>> >> > >>
>> >> > >>
>> >> > >
>> >> >
>> >> > --
>> >> > Richard Liang
>> >> > China Software Development Lab, IBM
>> >> >
>> >> >
>> >> >
>> >> >
>> ---------------------------------------------------------------------
>> >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>> >> > To unsubscribe, e-mail:
>> harmony-dev-unsubscribe@incubator.apache.org
>> >> > For additional commands, e-mail:
>> harmony-dev-help@incubator.apache.org
>> >> >
>> >> >
>> >>
>> >>
>> >
>> > ---------------------------------------------------------------------
>> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>> >
>> >
>>
>> --
>> Richard Liang
>> China Software Development Lab, IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Vladimir Ivanov <iv...@gmail.com>.
On 9/4/06, Richard Liang <ri...@gmail.com> wrote:
>
>
>
> Mikhail Loenko wrote:
> > Hi Vladimir
> >
> > Could you please decribe for what purpose it will be used?
> >
> > I mean why one might have to either exclude or run only regression
> tests?
>
> If running all tests takes up much time, running all regression test
> (for one particular version) may be a convenient way to verify the
> correctness of bug-fixing.
Thanks Richard. It is exactly what I want to say :)
On the other hand may be all proposed groups need similar explanation (smt.
like use-case for group)?
thanks, Vladimir
Best regards,
> Richard.
>
> >
> > Thanks,
> > Mikhail
> >
> > 2006/8/30, Vladimir Ivanov <iv...@gmail.com>:
> >> On 8/30/06, Richard Liang <ri...@gmail.com> wrote:
> >> >
> >> >
> >> >
> >> > Vladimir Ivanov wrote:
> >> > > Also some tag for regression tests should be added.
> >> > Yes. Do you think we could annotate regression test as
> >> > *level.regression*? Thanks a lot.
> >>
> >>
> >> Yes, I do. While tests can have more than one group it will enough.
> >> thanks, Vladimir
> >>
> >>
> >> Richard
> >> > > thanks, Vladimir
> >> > >
> >> > >
> >> > > On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
> >> > >>
> >> > >>
> >> > >>
> >> > >> Richard Liang wrote:
> >> > >> > Hello All,
> >> > >> >
> >> > >> > Now let's talk about the TestNG groups. I have read the related
> >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
> >> Zakharov.
> >> > >> > All of them are good discussion about TestNG groups.
> >> > >> >
> >> > >> > IMHO, we may define Harmony test groups according the following
> 4
> >> > >> > dimensions:
> >> > >> >
> >> > >> > 1) [Platform] os.any, os.<platform id>
> >> > >> > *os.any* - group of tests which pass on any platform. IMHO,
> >> most of
> >> > >> > our tests should be in this group.
> >> > >> > *os.<platform id>* - group of tests which are designed for one
> >> > >> > specific platform. A test may be in more than one of the
> >> groups. e.g
> >> > .,
> >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> >> > >> >
> >> > >> > ** os.any and os.<platform id> are mutually exclusive, that
> >> is,
> >> > >> > tests in os.any group should not be in os.win.IA32.
> >> > >> >
> >> > >> > 2) [Test state] state.broken, state.broken.<platform id>
> >> > >> > *state.broken* - group of tests which fail on every platform,
> >> because
> >> > >> > of bugs of tests or implementation. We need to fix the bugs of
> >> tests
> >> > >> > or implementation to make them pass.
> >> > >> > *state.broken.<platform id>* - groups of test which only fail
> >> on one
> >> > >> > specific platform. A test may be in more than one of the
> >> groups. e.g
> >> > .,
> >> > >> > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64
> "})
> >> > >> >
> >> > >> > **state.broken.<platform id> group may be used as a
> >> convenient
> >> > way
> >> > >> > to indicate that a test is platform-specific. e.g., If we
> >> support 10
> >> > >> > platforms, and one test are designed for 9 platforms except for
> >> > MacOS,
> >> > >> > instead of list 9 os.<platform id>, we can just use
> >> > state.broken.MacOS
> >> > >> >
> >> > >> > 3) [Test type] type.api , type.impl
> >> > >> > *type.api* - group of tests which are tests for APIs in the Java
> >> > >> > Specification
> >> > >> > *type.impl* - groups of tests which are tests for
> >> Harmony-specific
> >> > >> > implementation
> >> > >> >
> >> > >> > ** type.api and type.impl are also mutually exclusive.
> >> > >> >
> >> > >> > 4) [Test Level] level.unit, level.integration, level.system,
> >> > >> > level.stress, etc. (Levels of Test refer to the increase in
> >> > complexity
> >> > >> > as moving through test cycle. )
> >> > >> > ** A test may be in more than one of the groups.
> >> > >> > ** In fact, some tests such as System tests are the
> >> verification
> >> > of
> >> > >> > the entire system. Maybe we'll put them into a separate
> project.
> >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
> >> > >> >
> >> > >> > If we want to run all the unit test for APIs on windows, we
> >> may use
> >> > >> > TestNG groups to select the tests:
> >> > >> > <groups>
> >> > >> > <run>
> >> > >> > <include name="os.any" />
> >> > >> > <include name="type.api" />
> >> > >> > <include name="os.win.IA32" />
> >> > >> > <exclude name=" state.broken" />
> >> > >> > <exclude name="state.broken.win.IA32" />
> >> > >> > </run>
> >> > >> > </groups>
> >> > >> >
> >> > >> Hello All,
> >> > >>
> >> > >> I'm sorry. It seems that the example does not work. I will try to
> >> > figure
> >> > >> another example soon. ;-)
> >> > >>
> >> > >> Best regards,
> >> > >> Richard
> >> > >> >
> >> > >> > Well, I think our most of existing tests are in the groups of
> >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked TestNG
> >> to add
> >> > a
> >> > >> > new option "-groups" for its JUnitConverter which allow us to
> >> specify
> >> > >> > the test groups when migrate from JUnit test to TestNG test.
> >> > >> >
> >> > >> > Thanks for reading so far, and I will highly appreciate your
> >> comments
> >> > >> > or suggestion. ;-)
> >> > >> >
> >> > >>
> >> > >> --
> >> > >> Richard Liang
> >> > >> China Software Development Lab, IBM
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >> ---------------------------------------------------------------------
> >> > >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> > >> To unsubscribe, e-mail:
> >> harmony-dev-unsubscribe@incubator.apache.org
> >> > >> For additional commands, e-mail:
> >> harmony-dev-help@incubator.apache.org
> >> > >>
> >> > >>
> >> > >
> >> >
> >> > --
> >> > Richard Liang
> >> > China Software Development Lab, IBM
> >> >
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> > For additional commands, e-mail:
> harmony-dev-help@incubator.apache.org
> >> >
> >> >
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
> --
> Richard Liang
> China Software Development Lab, IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
On 9/4/06, Andrew Zhang <zh...@gmail.com> wrote:
> On 9/4/06, Richard Liang <ri...@gmail.com> wrote:
> >
> > On 9/4/06, Mikhail Loenko <ml...@gmail.com> wrote:
> > > Well, my question was for what particular reason? for example?
> > >
> > > Tio verify correctness of bug-fixing IMHO all the unit, intergration,
> > api, and
> > > regression tests should be run
> >
> > Running all tests are always good to verify our code quality. And I
> > think this is what we have been doing. But what will we do if it takes
> > us 24 hours to run all Harmony tests? Anyway, running regression tests
> > could provide some confidence to our bug-fixing. We may consider it as
> > another option. :-)
>
>
> I prefer to run all tests in one module instead. :) Although it can not
> guarentee all tests would pass, it's less likey to break build system
> frequently. If the fix causes other module fails, it shows the lack of
> tests in its module. And we should add the corresponding test in the module.
>
Running all tests for one module is practical for most cases (except luni).
> Currently, Harmony regression test is a test for certain Harmony jira issue.
> So IMHO, running all regression tests for certain issue doesn't make sense.
Yes, Harmony regression test is not the so-called "regression test"
which aim to verify that no unwanted changes were introduced to one
part of the system as a
result of making changes to another part of the system.
I agree that we may not want to run all the "regression test". If
there is no objection I will remove "level.regression"
>
> But whether using annotation to mark regression test is another story. At
> least, it doesn't have any disadvantages compared with comment way, does it?
>
We may use @Test(description="Regression: Harmony-XXXX").
>
> Best regards,
> > Richard
> >
> > >
> > > Thanks,
> > > Mikhail
> > >
> > > 2006/9/4, Richard Liang <ri...@gmail.com>:
> > > >
> > > >
> > > > Mikhail Loenko wrote:
> > > > > Hi Vladimir
> > > > >
> > > > > Could you please decribe for what purpose it will be used?
> > > > >
> > > > > I mean why one might have to either exclude or run only regression
> > tests?
> > > >
> > > > If running all tests takes up much time, running all regression test
> > > > (for one particular version) may be a convenient way to verify the
> > > > correctness of bug-fixing.
> > > >
> > > > Best regards,
> > > > Richard.
> > > >
> > > > >
> > > > > Thanks,
> > > > > Mikhail
> > > > >
> > > > > 2006/8/30, Vladimir Ivanov <iv...@gmail.com>:
> > > > >> On 8/30/06, Richard Liang <ri...@gmail.com> wrote:
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > Vladimir Ivanov wrote:
> > > > >> > > Also some tag for regression tests should be added.
> > > > >> > Yes. Do you think we could annotate regression test as
> > > > >> > *level.regression*? Thanks a lot.
> > > > >>
> > > > >>
> > > > >> Yes, I do. While tests can have more than one group it will enough.
> > > > >> thanks, Vladimir
> > > > >>
> > > > >>
> > > > >> Richard
> > > > >> > > thanks, Vladimir
> > > > >> > >
> > > > >> > >
> > > > >> > > On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> Richard Liang wrote:
> > > > >> > >> > Hello All,
> > > > >> > >> >
> > > > >> > >> > Now let's talk about the TestNG groups. I have read the
> > related
> > > > >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
> > > > >> Zakharov.
> > > > >> > >> > All of them are good discussion about TestNG groups.
> > > > >> > >> >
> > > > >> > >> > IMHO, we may define Harmony test groups according the
> > following 4
> > > > >> > >> > dimensions:
> > > > >> > >> >
> > > > >> > >> > 1) [Platform] os.any, os.<platform id>
> > > > >> > >> > *os.any* - group of tests which pass on any platform. IMHO,
> > > > >> most of
> > > > >> > >> > our tests should be in this group.
> > > > >> > >> > *os.<platform id>* - group of tests which are designed for
> > one
> > > > >> > >> > specific platform. A test may be in more than one of the
> > > > >> groups. e.g
> > > > >> > .,
> > > > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> > > > >> > >> >
> > > > >> > >> > ** os.any and os.<platform id> are mutually exclusive,
> > that
> > > > >> is,
> > > > >> > >> > tests in os.any group should not be in os.win.IA32.
> > > > >> > >> >
> > > > >> > >> > 2) [Test state] state.broken, state.broken.<platform id>
> > > > >> > >> > *state.broken* - group of tests which fail on every
> > platform,
> > > > >> because
> > > > >> > >> > of bugs of tests or implementation. We need to fix the bugs
> > of
> > > > >> tests
> > > > >> > >> > or implementation to make them pass.
> > > > >> > >> > *state.broken.<platform id>* - groups of test which only
> > fail
> > > > >> on one
> > > > >> > >> > specific platform. A test may be in more than one of the
> > > > >> groups. e.g
> > > > >> > .,
> > > > >> > >> > @Test(groups={"state.broken.linux.IA32", "
> > os.broken.linux.IA64"})
> > > > >> > >> >
> > > > >> > >> > **state.broken.<platform id> group may be used as a
> > > > >> convenient
> > > > >> > way
> > > > >> > >> > to indicate that a test is platform-specific. e.g., If we
> > > > >> support 10
> > > > >> > >> > platforms, and one test are designed for 9 platforms except
> > for
> > > > >> > MacOS,
> > > > >> > >> > instead of list 9 os.<platform id>, we can just use
> > > > >> > state.broken.MacOS
> > > > >> > >> >
> > > > >> > >> > 3) [Test type] type.api , type.impl
> > > > >> > >> > *type.api* - group of tests which are tests for APIs in the
> > Java
> > > > >> > >> > Specification
> > > > >> > >> > *type.impl* - groups of tests which are tests for
> > > > >> Harmony-specific
> > > > >> > >> > implementation
> > > > >> > >> >
> > > > >> > >> > ** type.api and type.impl are also mutually exclusive.
> > > > >> > >> >
> > > > >> > >> > 4) [Test Level] level.unit, level.integration, level.system,
> > > > >> > >> > level.stress, etc. (Levels of Test refer to the increase in
> > > > >> > complexity
> > > > >> > >> > as moving through test cycle. )
> > > > >> > >> > ** A test may be in more than one of the groups.
> > > > >> > >> > ** In fact, some tests such as System tests are the
> > > > >> verification
> > > > >> > of
> > > > >> > >> > the entire system. Maybe we'll put them into a separate
> > project.
> > > > >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
> > > > >> > >> >
> > > > >> > >> > If we want to run all the unit test for APIs on windows, we
> > > > >> may use
> > > > >> > >> > TestNG groups to select the tests:
> > > > >> > >> > <groups>
> > > > >> > >> > <run>
> > > > >> > >> > <include name="os.any" />
> > > > >> > >> > <include name="type.api" />
> > > > >> > >> > <include name="os.win.IA32" />
> > > > >> > >> > <exclude name=" state.broken" />
> > > > >> > >> > <exclude name="state.broken.win.IA32" />
> > > > >> > >> > </run>
> > > > >> > >> > </groups>
> > > > >> > >> >
> > > > >> > >> Hello All,
> > > > >> > >>
> > > > >> > >> I'm sorry. It seems that the example does not work. I will try
> > to
> > > > >> > figure
> > > > >> > >> another example soon. ;-)
> > > > >> > >>
> > > > >> > >> Best regards,
> > > > >> > >> Richard
> > > > >> > >> >
> > > > >> > >> > Well, I think our most of existing tests are in the groups
> > of
> > > > >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked
> > TestNG
> > > > >> to add
> > > > >> > a
> > > > >> > >> > new option "-groups" for its JUnitConverter which allow us
> > to
> > > > >> specify
> > > > >> > >> > the test groups when migrate from JUnit test to TestNG test.
> > > > >> > >> >
> > > > >> > >> > Thanks for reading so far, and I will highly appreciate your
> > > > >> comments
> > > > >> > >> > or suggestion. ;-)
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >> --
> > > > >> > >> Richard Liang
> > > > >> > >> China Software Development Lab, IBM
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> > >> Terms of use :
> > http://incubator.apache.org/harmony/mailing.html
> > > > >> > >> To unsubscribe, e-mail:
> > > > >> harmony-dev-unsubscribe@incubator.apache.org
> > > > >> > >> For additional commands, e-mail:
> > > > >> harmony-dev-help@incubator.apache.org
> > > > >> > >>
> > > > >> > >>
> > > > >> > >
> > > > >> >
> > > > >> > --
> > > > >> > Richard Liang
> > > > >> > China Software Development Lab, IBM
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > ---------------------------------------------------------------------
> > > > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > >> > To unsubscribe, e-mail:
> > harmony-dev-unsubscribe@incubator.apache.org
> > > > >> > For additional commands, e-mail:
> > harmony-dev-help@incubator.apache.org
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >
> > > > >
> > ---------------------------------------------------------------------
> > > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > > > For additional commands, e-mail:
> > harmony-dev-help@incubator.apache.org
> > > > >
> > > > >
> > > >
> > > > --
> > > > Richard Liang
> > > > China Software Development Lab, IBM
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >
> > >
> >
> >
> > --
> > Richard Liang
> > China Software Development Lab, IBM
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>
> --
> Andrew Zhang
> China Software Development Lab, IBM
>
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Andrew Zhang <zh...@gmail.com>.
On 9/4/06, Richard Liang <ri...@gmail.com> wrote:
>
> On 9/4/06, Mikhail Loenko <ml...@gmail.com> wrote:
> > Well, my question was for what particular reason? for example?
> >
> > Tio verify correctness of bug-fixing IMHO all the unit, intergration,
> api, and
> > regression tests should be run
>
> Running all tests are always good to verify our code quality. And I
> think this is what we have been doing. But what will we do if it takes
> us 24 hours to run all Harmony tests? Anyway, running regression tests
> could provide some confidence to our bug-fixing. We may consider it as
> another option. :-)
I prefer to run all tests in one module instead. :) Although it can not
guarentee all tests would pass, it's less likey to break build system
frequently. If the fix causes other module fails, it shows the lack of
tests in its module. And we should add the corresponding test in the module.
Currently, Harmony regression test is a test for certain Harmony jira issue.
So IMHO, running all regression tests for certain issue doesn't make sense.
But whether using annotation to mark regression test is another story. At
least, it doesn't have any disadvantages compared with comment way, does it?
Best regards,
> Richard
>
> >
> > Thanks,
> > Mikhail
> >
> > 2006/9/4, Richard Liang <ri...@gmail.com>:
> > >
> > >
> > > Mikhail Loenko wrote:
> > > > Hi Vladimir
> > > >
> > > > Could you please decribe for what purpose it will be used?
> > > >
> > > > I mean why one might have to either exclude or run only regression
> tests?
> > >
> > > If running all tests takes up much time, running all regression test
> > > (for one particular version) may be a convenient way to verify the
> > > correctness of bug-fixing.
> > >
> > > Best regards,
> > > Richard.
> > >
> > > >
> > > > Thanks,
> > > > Mikhail
> > > >
> > > > 2006/8/30, Vladimir Ivanov <iv...@gmail.com>:
> > > >> On 8/30/06, Richard Liang <ri...@gmail.com> wrote:
> > > >> >
> > > >> >
> > > >> >
> > > >> > Vladimir Ivanov wrote:
> > > >> > > Also some tag for regression tests should be added.
> > > >> > Yes. Do you think we could annotate regression test as
> > > >> > *level.regression*? Thanks a lot.
> > > >>
> > > >>
> > > >> Yes, I do. While tests can have more than one group it will enough.
> > > >> thanks, Vladimir
> > > >>
> > > >>
> > > >> Richard
> > > >> > > thanks, Vladimir
> > > >> > >
> > > >> > >
> > > >> > > On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >> Richard Liang wrote:
> > > >> > >> > Hello All,
> > > >> > >> >
> > > >> > >> > Now let's talk about the TestNG groups. I have read the
> related
> > > >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
> > > >> Zakharov.
> > > >> > >> > All of them are good discussion about TestNG groups.
> > > >> > >> >
> > > >> > >> > IMHO, we may define Harmony test groups according the
> following 4
> > > >> > >> > dimensions:
> > > >> > >> >
> > > >> > >> > 1) [Platform] os.any, os.<platform id>
> > > >> > >> > *os.any* - group of tests which pass on any platform. IMHO,
> > > >> most of
> > > >> > >> > our tests should be in this group.
> > > >> > >> > *os.<platform id>* - group of tests which are designed for
> one
> > > >> > >> > specific platform. A test may be in more than one of the
> > > >> groups. e.g
> > > >> > .,
> > > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> > > >> > >> >
> > > >> > >> > ** os.any and os.<platform id> are mutually exclusive,
> that
> > > >> is,
> > > >> > >> > tests in os.any group should not be in os.win.IA32.
> > > >> > >> >
> > > >> > >> > 2) [Test state] state.broken, state.broken.<platform id>
> > > >> > >> > *state.broken* - group of tests which fail on every
> platform,
> > > >> because
> > > >> > >> > of bugs of tests or implementation. We need to fix the bugs
> of
> > > >> tests
> > > >> > >> > or implementation to make them pass.
> > > >> > >> > *state.broken.<platform id>* - groups of test which only
> fail
> > > >> on one
> > > >> > >> > specific platform. A test may be in more than one of the
> > > >> groups. e.g
> > > >> > .,
> > > >> > >> > @Test(groups={"state.broken.linux.IA32", "
> os.broken.linux.IA64"})
> > > >> > >> >
> > > >> > >> > **state.broken.<platform id> group may be used as a
> > > >> convenient
> > > >> > way
> > > >> > >> > to indicate that a test is platform-specific. e.g., If we
> > > >> support 10
> > > >> > >> > platforms, and one test are designed for 9 platforms except
> for
> > > >> > MacOS,
> > > >> > >> > instead of list 9 os.<platform id>, we can just use
> > > >> > state.broken.MacOS
> > > >> > >> >
> > > >> > >> > 3) [Test type] type.api , type.impl
> > > >> > >> > *type.api* - group of tests which are tests for APIs in the
> Java
> > > >> > >> > Specification
> > > >> > >> > *type.impl* - groups of tests which are tests for
> > > >> Harmony-specific
> > > >> > >> > implementation
> > > >> > >> >
> > > >> > >> > ** type.api and type.impl are also mutually exclusive.
> > > >> > >> >
> > > >> > >> > 4) [Test Level] level.unit, level.integration, level.system,
> > > >> > >> > level.stress, etc. (Levels of Test refer to the increase in
> > > >> > complexity
> > > >> > >> > as moving through test cycle. )
> > > >> > >> > ** A test may be in more than one of the groups.
> > > >> > >> > ** In fact, some tests such as System tests are the
> > > >> verification
> > > >> > of
> > > >> > >> > the entire system. Maybe we'll put them into a separate
> project.
> > > >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
> > > >> > >> >
> > > >> > >> > If we want to run all the unit test for APIs on windows, we
> > > >> may use
> > > >> > >> > TestNG groups to select the tests:
> > > >> > >> > <groups>
> > > >> > >> > <run>
> > > >> > >> > <include name="os.any" />
> > > >> > >> > <include name="type.api" />
> > > >> > >> > <include name="os.win.IA32" />
> > > >> > >> > <exclude name=" state.broken" />
> > > >> > >> > <exclude name="state.broken.win.IA32" />
> > > >> > >> > </run>
> > > >> > >> > </groups>
> > > >> > >> >
> > > >> > >> Hello All,
> > > >> > >>
> > > >> > >> I'm sorry. It seems that the example does not work. I will try
> to
> > > >> > figure
> > > >> > >> another example soon. ;-)
> > > >> > >>
> > > >> > >> Best regards,
> > > >> > >> Richard
> > > >> > >> >
> > > >> > >> > Well, I think our most of existing tests are in the groups
> of
> > > >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked
> TestNG
> > > >> to add
> > > >> > a
> > > >> > >> > new option "-groups" for its JUnitConverter which allow us
> to
> > > >> specify
> > > >> > >> > the test groups when migrate from JUnit test to TestNG test.
> > > >> > >> >
> > > >> > >> > Thanks for reading so far, and I will highly appreciate your
> > > >> comments
> > > >> > >> > or suggestion. ;-)
> > > >> > >> >
> > > >> > >>
> > > >> > >> --
> > > >> > >> Richard Liang
> > > >> > >> China Software Development Lab, IBM
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> > >> Terms of use :
> http://incubator.apache.org/harmony/mailing.html
> > > >> > >> To unsubscribe, e-mail:
> > > >> harmony-dev-unsubscribe@incubator.apache.org
> > > >> > >> For additional commands, e-mail:
> > > >> harmony-dev-help@incubator.apache.org
> > > >> > >>
> > > >> > >>
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > Richard Liang
> > > >> > China Software Development Lab, IBM
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> ---------------------------------------------------------------------
> > > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > >> > To unsubscribe, e-mail:
> harmony-dev-unsubscribe@incubator.apache.org
> > > >> > For additional commands, e-mail:
> harmony-dev-help@incubator.apache.org
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >
> > > >
> ---------------------------------------------------------------------
> > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail:
> harmony-dev-help@incubator.apache.org
> > > >
> > > >
> > >
> > > --
> > > Richard Liang
> > > China Software Development Lab, IBM
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>
> --
> Richard Liang
> China Software Development Lab, IBM
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Andrew Zhang
China Software Development Lab, IBM
Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
On 9/4/06, Mikhail Loenko <ml...@gmail.com> wrote:
> Well, my question was for what particular reason? for example?
>
> Tio verify correctness of bug-fixing IMHO all the unit, intergration, api, and
> regression tests should be run
Running all tests are always good to verify our code quality. And I
think this is what we have been doing. But what will we do if it takes
us 24 hours to run all Harmony tests? Anyway, running regression tests
could provide some confidence to our bug-fixing. We may consider it as
another option. :-)
Best regards,
Richard
>
> Thanks,
> Mikhail
>
> 2006/9/4, Richard Liang <ri...@gmail.com>:
> >
> >
> > Mikhail Loenko wrote:
> > > Hi Vladimir
> > >
> > > Could you please decribe for what purpose it will be used?
> > >
> > > I mean why one might have to either exclude or run only regression tests?
> >
> > If running all tests takes up much time, running all regression test
> > (for one particular version) may be a convenient way to verify the
> > correctness of bug-fixing.
> >
> > Best regards,
> > Richard.
> >
> > >
> > > Thanks,
> > > Mikhail
> > >
> > > 2006/8/30, Vladimir Ivanov <iv...@gmail.com>:
> > >> On 8/30/06, Richard Liang <ri...@gmail.com> wrote:
> > >> >
> > >> >
> > >> >
> > >> > Vladimir Ivanov wrote:
> > >> > > Also some tag for regression tests should be added.
> > >> > Yes. Do you think we could annotate regression test as
> > >> > *level.regression*? Thanks a lot.
> > >>
> > >>
> > >> Yes, I do. While tests can have more than one group it will enough.
> > >> thanks, Vladimir
> > >>
> > >>
> > >> Richard
> > >> > > thanks, Vladimir
> > >> > >
> > >> > >
> > >> > > On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >> Richard Liang wrote:
> > >> > >> > Hello All,
> > >> > >> >
> > >> > >> > Now let's talk about the TestNG groups. I have read the related
> > >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
> > >> Zakharov.
> > >> > >> > All of them are good discussion about TestNG groups.
> > >> > >> >
> > >> > >> > IMHO, we may define Harmony test groups according the following 4
> > >> > >> > dimensions:
> > >> > >> >
> > >> > >> > 1) [Platform] os.any, os.<platform id>
> > >> > >> > *os.any* - group of tests which pass on any platform. IMHO,
> > >> most of
> > >> > >> > our tests should be in this group.
> > >> > >> > *os.<platform id>* - group of tests which are designed for one
> > >> > >> > specific platform. A test may be in more than one of the
> > >> groups. e.g
> > >> > .,
> > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> > >> > >> >
> > >> > >> > ** os.any and os.<platform id> are mutually exclusive, that
> > >> is,
> > >> > >> > tests in os.any group should not be in os.win.IA32.
> > >> > >> >
> > >> > >> > 2) [Test state] state.broken, state.broken.<platform id>
> > >> > >> > *state.broken* - group of tests which fail on every platform,
> > >> because
> > >> > >> > of bugs of tests or implementation. We need to fix the bugs of
> > >> tests
> > >> > >> > or implementation to make them pass.
> > >> > >> > *state.broken.<platform id>* - groups of test which only fail
> > >> on one
> > >> > >> > specific platform. A test may be in more than one of the
> > >> groups. e.g
> > >> > .,
> > >> > >> > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
> > >> > >> >
> > >> > >> > **state.broken.<platform id> group may be used as a
> > >> convenient
> > >> > way
> > >> > >> > to indicate that a test is platform-specific. e.g., If we
> > >> support 10
> > >> > >> > platforms, and one test are designed for 9 platforms except for
> > >> > MacOS,
> > >> > >> > instead of list 9 os.<platform id>, we can just use
> > >> > state.broken.MacOS
> > >> > >> >
> > >> > >> > 3) [Test type] type.api , type.impl
> > >> > >> > *type.api* - group of tests which are tests for APIs in the Java
> > >> > >> > Specification
> > >> > >> > *type.impl* - groups of tests which are tests for
> > >> Harmony-specific
> > >> > >> > implementation
> > >> > >> >
> > >> > >> > ** type.api and type.impl are also mutually exclusive.
> > >> > >> >
> > >> > >> > 4) [Test Level] level.unit, level.integration, level.system,
> > >> > >> > level.stress, etc. (Levels of Test refer to the increase in
> > >> > complexity
> > >> > >> > as moving through test cycle. )
> > >> > >> > ** A test may be in more than one of the groups.
> > >> > >> > ** In fact, some tests such as System tests are the
> > >> verification
> > >> > of
> > >> > >> > the entire system. Maybe we'll put them into a separate project.
> > >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
> > >> > >> >
> > >> > >> > If we want to run all the unit test for APIs on windows, we
> > >> may use
> > >> > >> > TestNG groups to select the tests:
> > >> > >> > <groups>
> > >> > >> > <run>
> > >> > >> > <include name="os.any" />
> > >> > >> > <include name="type.api" />
> > >> > >> > <include name="os.win.IA32" />
> > >> > >> > <exclude name=" state.broken" />
> > >> > >> > <exclude name="state.broken.win.IA32" />
> > >> > >> > </run>
> > >> > >> > </groups>
> > >> > >> >
> > >> > >> Hello All,
> > >> > >>
> > >> > >> I'm sorry. It seems that the example does not work. I will try to
> > >> > figure
> > >> > >> another example soon. ;-)
> > >> > >>
> > >> > >> Best regards,
> > >> > >> Richard
> > >> > >> >
> > >> > >> > Well, I think our most of existing tests are in the groups of
> > >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked TestNG
> > >> to add
> > >> > a
> > >> > >> > new option "-groups" for its JUnitConverter which allow us to
> > >> specify
> > >> > >> > the test groups when migrate from JUnit test to TestNG test.
> > >> > >> >
> > >> > >> > Thanks for reading so far, and I will highly appreciate your
> > >> comments
> > >> > >> > or suggestion. ;-)
> > >> > >> >
> > >> > >>
> > >> > >> --
> > >> > >> Richard Liang
> > >> > >> China Software Development Lab, IBM
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >>
> > >> ---------------------------------------------------------------------
> > >> > >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> > >> > >> To unsubscribe, e-mail:
> > >> harmony-dev-unsubscribe@incubator.apache.org
> > >> > >> For additional commands, e-mail:
> > >> harmony-dev-help@incubator.apache.org
> > >> > >>
> > >> > >>
> > >> > >
> > >> >
> > >> > --
> > >> > Richard Liang
> > >> > China Software Development Lab, IBM
> > >> >
> > >> >
> > >> >
> > >> > ---------------------------------------------------------------------
> > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > >> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > >> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >> >
> > >> >
> > >>
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >
> > >
> >
> > --
> > Richard Liang
> > China Software Development Lab, IBM
> >
> >
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Mikhail Loenko <ml...@gmail.com>.
Well, my question was for what particular reason? for example?
Tio verify correctness of bug-fixing IMHO all the unit, intergration, api, and
regression tests should be run
Thanks,
Mikhail
2006/9/4, Richard Liang <ri...@gmail.com>:
>
>
> Mikhail Loenko wrote:
> > Hi Vladimir
> >
> > Could you please decribe for what purpose it will be used?
> >
> > I mean why one might have to either exclude or run only regression tests?
>
> If running all tests takes up much time, running all regression test
> (for one particular version) may be a convenient way to verify the
> correctness of bug-fixing.
>
> Best regards,
> Richard.
>
> >
> > Thanks,
> > Mikhail
> >
> > 2006/8/30, Vladimir Ivanov <iv...@gmail.com>:
> >> On 8/30/06, Richard Liang <ri...@gmail.com> wrote:
> >> >
> >> >
> >> >
> >> > Vladimir Ivanov wrote:
> >> > > Also some tag for regression tests should be added.
> >> > Yes. Do you think we could annotate regression test as
> >> > *level.regression*? Thanks a lot.
> >>
> >>
> >> Yes, I do. While tests can have more than one group it will enough.
> >> thanks, Vladimir
> >>
> >>
> >> Richard
> >> > > thanks, Vladimir
> >> > >
> >> > >
> >> > > On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
> >> > >>
> >> > >>
> >> > >>
> >> > >> Richard Liang wrote:
> >> > >> > Hello All,
> >> > >> >
> >> > >> > Now let's talk about the TestNG groups. I have read the related
> >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
> >> Zakharov.
> >> > >> > All of them are good discussion about TestNG groups.
> >> > >> >
> >> > >> > IMHO, we may define Harmony test groups according the following 4
> >> > >> > dimensions:
> >> > >> >
> >> > >> > 1) [Platform] os.any, os.<platform id>
> >> > >> > *os.any* - group of tests which pass on any platform. IMHO,
> >> most of
> >> > >> > our tests should be in this group.
> >> > >> > *os.<platform id>* - group of tests which are designed for one
> >> > >> > specific platform. A test may be in more than one of the
> >> groups. e.g
> >> > .,
> >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> >> > >> >
> >> > >> > ** os.any and os.<platform id> are mutually exclusive, that
> >> is,
> >> > >> > tests in os.any group should not be in os.win.IA32.
> >> > >> >
> >> > >> > 2) [Test state] state.broken, state.broken.<platform id>
> >> > >> > *state.broken* - group of tests which fail on every platform,
> >> because
> >> > >> > of bugs of tests or implementation. We need to fix the bugs of
> >> tests
> >> > >> > or implementation to make them pass.
> >> > >> > *state.broken.<platform id>* - groups of test which only fail
> >> on one
> >> > >> > specific platform. A test may be in more than one of the
> >> groups. e.g
> >> > .,
> >> > >> > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
> >> > >> >
> >> > >> > **state.broken.<platform id> group may be used as a
> >> convenient
> >> > way
> >> > >> > to indicate that a test is platform-specific. e.g., If we
> >> support 10
> >> > >> > platforms, and one test are designed for 9 platforms except for
> >> > MacOS,
> >> > >> > instead of list 9 os.<platform id>, we can just use
> >> > state.broken.MacOS
> >> > >> >
> >> > >> > 3) [Test type] type.api , type.impl
> >> > >> > *type.api* - group of tests which are tests for APIs in the Java
> >> > >> > Specification
> >> > >> > *type.impl* - groups of tests which are tests for
> >> Harmony-specific
> >> > >> > implementation
> >> > >> >
> >> > >> > ** type.api and type.impl are also mutually exclusive.
> >> > >> >
> >> > >> > 4) [Test Level] level.unit, level.integration, level.system,
> >> > >> > level.stress, etc. (Levels of Test refer to the increase in
> >> > complexity
> >> > >> > as moving through test cycle. )
> >> > >> > ** A test may be in more than one of the groups.
> >> > >> > ** In fact, some tests such as System tests are the
> >> verification
> >> > of
> >> > >> > the entire system. Maybe we'll put them into a separate project.
> >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
> >> > >> >
> >> > >> > If we want to run all the unit test for APIs on windows, we
> >> may use
> >> > >> > TestNG groups to select the tests:
> >> > >> > <groups>
> >> > >> > <run>
> >> > >> > <include name="os.any" />
> >> > >> > <include name="type.api" />
> >> > >> > <include name="os.win.IA32" />
> >> > >> > <exclude name=" state.broken" />
> >> > >> > <exclude name="state.broken.win.IA32" />
> >> > >> > </run>
> >> > >> > </groups>
> >> > >> >
> >> > >> Hello All,
> >> > >>
> >> > >> I'm sorry. It seems that the example does not work. I will try to
> >> > figure
> >> > >> another example soon. ;-)
> >> > >>
> >> > >> Best regards,
> >> > >> Richard
> >> > >> >
> >> > >> > Well, I think our most of existing tests are in the groups of
> >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked TestNG
> >> to add
> >> > a
> >> > >> > new option "-groups" for its JUnitConverter which allow us to
> >> specify
> >> > >> > the test groups when migrate from JUnit test to TestNG test.
> >> > >> >
> >> > >> > Thanks for reading so far, and I will highly appreciate your
> >> comments
> >> > >> > or suggestion. ;-)
> >> > >> >
> >> > >>
> >> > >> --
> >> > >> Richard Liang
> >> > >> China Software Development Lab, IBM
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >> ---------------------------------------------------------------------
> >> > >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> > >> To unsubscribe, e-mail:
> >> harmony-dev-unsubscribe@incubator.apache.org
> >> > >> For additional commands, e-mail:
> >> harmony-dev-help@incubator.apache.org
> >> > >>
> >> > >>
> >> > >
> >> >
> >> > --
> >> > Richard Liang
> >> > China Software Development Lab, IBM
> >> >
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >> >
> >> >
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
> --
> Richard Liang
> China Software Development Lab, IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
Mikhail Loenko wrote:
> Hi Vladimir
>
> Could you please decribe for what purpose it will be used?
>
> I mean why one might have to either exclude or run only regression tests?
If running all tests takes up much time, running all regression test
(for one particular version) may be a convenient way to verify the
correctness of bug-fixing.
Best regards,
Richard.
>
> Thanks,
> Mikhail
>
> 2006/8/30, Vladimir Ivanov <iv...@gmail.com>:
>> On 8/30/06, Richard Liang <ri...@gmail.com> wrote:
>> >
>> >
>> >
>> > Vladimir Ivanov wrote:
>> > > Also some tag for regression tests should be added.
>> > Yes. Do you think we could annotate regression test as
>> > *level.regression*? Thanks a lot.
>>
>>
>> Yes, I do. While tests can have more than one group it will enough.
>> thanks, Vladimir
>>
>>
>> Richard
>> > > thanks, Vladimir
>> > >
>> > >
>> > > On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
>> > >>
>> > >>
>> > >>
>> > >> Richard Liang wrote:
>> > >> > Hello All,
>> > >> >
>> > >> > Now let's talk about the TestNG groups. I have read the related
>> > >> > threads which posted by George, Vladimir Ivanov and Alexei
>> Zakharov.
>> > >> > All of them are good discussion about TestNG groups.
>> > >> >
>> > >> > IMHO, we may define Harmony test groups according the following 4
>> > >> > dimensions:
>> > >> >
>> > >> > 1) [Platform] os.any, os.<platform id>
>> > >> > *os.any* - group of tests which pass on any platform. IMHO,
>> most of
>> > >> > our tests should be in this group.
>> > >> > *os.<platform id>* - group of tests which are designed for one
>> > >> > specific platform. A test may be in more than one of the
>> groups. e.g
>> > .,
>> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
>> > >> >
>> > >> > ** os.any and os.<platform id> are mutually exclusive, that
>> is,
>> > >> > tests in os.any group should not be in os.win.IA32.
>> > >> >
>> > >> > 2) [Test state] state.broken, state.broken.<platform id>
>> > >> > *state.broken* - group of tests which fail on every platform,
>> because
>> > >> > of bugs of tests or implementation. We need to fix the bugs of
>> tests
>> > >> > or implementation to make them pass.
>> > >> > *state.broken.<platform id>* - groups of test which only fail
>> on one
>> > >> > specific platform. A test may be in more than one of the
>> groups. e.g
>> > .,
>> > >> > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>> > >> >
>> > >> > **state.broken.<platform id> group may be used as a
>> convenient
>> > way
>> > >> > to indicate that a test is platform-specific. e.g., If we
>> support 10
>> > >> > platforms, and one test are designed for 9 platforms except for
>> > MacOS,
>> > >> > instead of list 9 os.<platform id>, we can just use
>> > state.broken.MacOS
>> > >> >
>> > >> > 3) [Test type] type.api , type.impl
>> > >> > *type.api* - group of tests which are tests for APIs in the Java
>> > >> > Specification
>> > >> > *type.impl* - groups of tests which are tests for
>> Harmony-specific
>> > >> > implementation
>> > >> >
>> > >> > ** type.api and type.impl are also mutually exclusive.
>> > >> >
>> > >> > 4) [Test Level] level.unit, level.integration, level.system,
>> > >> > level.stress, etc. (Levels of Test refer to the increase in
>> > complexity
>> > >> > as moving through test cycle. )
>> > >> > ** A test may be in more than one of the groups.
>> > >> > ** In fact, some tests such as System tests are the
>> verification
>> > of
>> > >> > the entire system. Maybe we'll put them into a separate project.
>> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
>> > >> >
>> > >> > If we want to run all the unit test for APIs on windows, we
>> may use
>> > >> > TestNG groups to select the tests:
>> > >> > <groups>
>> > >> > <run>
>> > >> > <include name="os.any" />
>> > >> > <include name="type.api" />
>> > >> > <include name="os.win.IA32" />
>> > >> > <exclude name=" state.broken" />
>> > >> > <exclude name="state.broken.win.IA32" />
>> > >> > </run>
>> > >> > </groups>
>> > >> >
>> > >> Hello All,
>> > >>
>> > >> I'm sorry. It seems that the example does not work. I will try to
>> > figure
>> > >> another example soon. ;-)
>> > >>
>> > >> Best regards,
>> > >> Richard
>> > >> >
>> > >> > Well, I think our most of existing tests are in the groups of
>> > >> > {"os.any", "type.api", "level.unit"}, and I have asked TestNG
>> to add
>> > a
>> > >> > new option "-groups" for its JUnitConverter which allow us to
>> specify
>> > >> > the test groups when migrate from JUnit test to TestNG test.
>> > >> >
>> > >> > Thanks for reading so far, and I will highly appreciate your
>> comments
>> > >> > or suggestion. ;-)
>> > >> >
>> > >>
>> > >> --
>> > >> Richard Liang
>> > >> China Software Development Lab, IBM
>> > >>
>> > >>
>> > >>
>> > >>
>> ---------------------------------------------------------------------
>> > >> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> > >> To unsubscribe, e-mail:
>> harmony-dev-unsubscribe@incubator.apache.org
>> > >> For additional commands, e-mail:
>> harmony-dev-help@incubator.apache.org
>> > >>
>> > >>
>> > >
>> >
>> > --
>> > Richard Liang
>> > China Software Development Lab, IBM
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>> >
>> >
>>
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Mikhail Loenko <ml...@gmail.com>.
Hi Vladimir
Could you please decribe for what purpose it will be used?
I mean why one might have to either exclude or run only regression tests?
Thanks,
Mikhail
2006/8/30, Vladimir Ivanov <iv...@gmail.com>:
> On 8/30/06, Richard Liang <ri...@gmail.com> wrote:
> >
> >
> >
> > Vladimir Ivanov wrote:
> > > Also some tag for regression tests should be added.
> > Yes. Do you think we could annotate regression test as
> > *level.regression*? Thanks a lot.
>
>
> Yes, I do. While tests can have more than one group it will enough.
> thanks, Vladimir
>
>
> Richard
> > > thanks, Vladimir
> > >
> > >
> > > On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
> > >>
> > >>
> > >>
> > >> Richard Liang wrote:
> > >> > Hello All,
> > >> >
> > >> > Now let's talk about the TestNG groups. I have read the related
> > >> > threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
> > >> > All of them are good discussion about TestNG groups.
> > >> >
> > >> > IMHO, we may define Harmony test groups according the following 4
> > >> > dimensions:
> > >> >
> > >> > 1) [Platform] os.any, os.<platform id>
> > >> > *os.any* - group of tests which pass on any platform. IMHO, most of
> > >> > our tests should be in this group.
> > >> > *os.<platform id>* - group of tests which are designed for one
> > >> > specific platform. A test may be in more than one of the groups. e.g
> > .,
> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> > >> >
> > >> > ** os.any and os.<platform id> are mutually exclusive, that is,
> > >> > tests in os.any group should not be in os.win.IA32.
> > >> >
> > >> > 2) [Test state] state.broken, state.broken.<platform id>
> > >> > *state.broken* - group of tests which fail on every platform, because
> > >> > of bugs of tests or implementation. We need to fix the bugs of tests
> > >> > or implementation to make them pass.
> > >> > *state.broken.<platform id>* - groups of test which only fail on one
> > >> > specific platform. A test may be in more than one of the groups. e.g
> > .,
> > >> > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
> > >> >
> > >> > **state.broken.<platform id> group may be used as a convenient
> > way
> > >> > to indicate that a test is platform-specific. e.g., If we support 10
> > >> > platforms, and one test are designed for 9 platforms except for
> > MacOS,
> > >> > instead of list 9 os.<platform id>, we can just use
> > state.broken.MacOS
> > >> >
> > >> > 3) [Test type] type.api , type.impl
> > >> > *type.api* - group of tests which are tests for APIs in the Java
> > >> > Specification
> > >> > *type.impl* - groups of tests which are tests for Harmony-specific
> > >> > implementation
> > >> >
> > >> > ** type.api and type.impl are also mutually exclusive.
> > >> >
> > >> > 4) [Test Level] level.unit, level.integration, level.system,
> > >> > level.stress, etc. (Levels of Test refer to the increase in
> > complexity
> > >> > as moving through test cycle. )
> > >> > ** A test may be in more than one of the groups.
> > >> > ** In fact, some tests such as System tests are the verification
> > of
> > >> > the entire system. Maybe we'll put them into a separate project.
> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
> > >> >
> > >> > If we want to run all the unit test for APIs on windows, we may use
> > >> > TestNG groups to select the tests:
> > >> > <groups>
> > >> > <run>
> > >> > <include name="os.any" />
> > >> > <include name="type.api" />
> > >> > <include name="os.win.IA32" />
> > >> > <exclude name=" state.broken" />
> > >> > <exclude name="state.broken.win.IA32" />
> > >> > </run>
> > >> > </groups>
> > >> >
> > >> Hello All,
> > >>
> > >> I'm sorry. It seems that the example does not work. I will try to
> > figure
> > >> another example soon. ;-)
> > >>
> > >> Best regards,
> > >> Richard
> > >> >
> > >> > Well, I think our most of existing tests are in the groups of
> > >> > {"os.any", "type.api", "level.unit"}, and I have asked TestNG to add
> > a
> > >> > new option "-groups" for its JUnitConverter which allow us to specify
> > >> > the test groups when migrate from JUnit test to TestNG test.
> > >> >
> > >> > Thanks for reading so far, and I will highly appreciate your comments
> > >> > or suggestion. ;-)
> > >> >
> > >>
> > >> --
> > >> Richard Liang
> > >> China Software Development Lab, IBM
> > >>
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> > >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> > >>
> > >>
> > >
> >
> > --
> > Richard Liang
> > China Software Development Lab, IBM
> >
> >
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Vladimir Ivanov <iv...@gmail.com>.
On 8/30/06, Richard Liang <ri...@gmail.com> wrote:
>
>
>
> Vladimir Ivanov wrote:
> > Also some tag for regression tests should be added.
> Yes. Do you think we could annotate regression test as
> *level.regression*? Thanks a lot.
Yes, I do. While tests can have more than one group it will enough.
thanks, Vladimir
Richard
> > thanks, Vladimir
> >
> >
> > On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
> >>
> >>
> >>
> >> Richard Liang wrote:
> >> > Hello All,
> >> >
> >> > Now let's talk about the TestNG groups. I have read the related
> >> > threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
> >> > All of them are good discussion about TestNG groups.
> >> >
> >> > IMHO, we may define Harmony test groups according the following 4
> >> > dimensions:
> >> >
> >> > 1) [Platform] os.any, os.<platform id>
> >> > *os.any* - group of tests which pass on any platform. IMHO, most of
> >> > our tests should be in this group.
> >> > *os.<platform id>* - group of tests which are designed for one
> >> > specific platform. A test may be in more than one of the groups. e.g
> .,
> >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> >> >
> >> > ** os.any and os.<platform id> are mutually exclusive, that is,
> >> > tests in os.any group should not be in os.win.IA32.
> >> >
> >> > 2) [Test state] state.broken, state.broken.<platform id>
> >> > *state.broken* - group of tests which fail on every platform, because
> >> > of bugs of tests or implementation. We need to fix the bugs of tests
> >> > or implementation to make them pass.
> >> > *state.broken.<platform id>* - groups of test which only fail on one
> >> > specific platform. A test may be in more than one of the groups. e.g
> .,
> >> > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
> >> >
> >> > **state.broken.<platform id> group may be used as a convenient
> way
> >> > to indicate that a test is platform-specific. e.g., If we support 10
> >> > platforms, and one test are designed for 9 platforms except for
> MacOS,
> >> > instead of list 9 os.<platform id>, we can just use
> state.broken.MacOS
> >> >
> >> > 3) [Test type] type.api , type.impl
> >> > *type.api* - group of tests which are tests for APIs in the Java
> >> > Specification
> >> > *type.impl* - groups of tests which are tests for Harmony-specific
> >> > implementation
> >> >
> >> > ** type.api and type.impl are also mutually exclusive.
> >> >
> >> > 4) [Test Level] level.unit, level.integration, level.system,
> >> > level.stress, etc. (Levels of Test refer to the increase in
> complexity
> >> > as moving through test cycle. )
> >> > ** A test may be in more than one of the groups.
> >> > ** In fact, some tests such as System tests are the verification
> of
> >> > the entire system. Maybe we'll put them into a separate project.
> >> > e.g., harmony/enhanced/SVT (System Verification Test).
> >> >
> >> > If we want to run all the unit test for APIs on windows, we may use
> >> > TestNG groups to select the tests:
> >> > <groups>
> >> > <run>
> >> > <include name="os.any" />
> >> > <include name="type.api" />
> >> > <include name="os.win.IA32" />
> >> > <exclude name=" state.broken" />
> >> > <exclude name="state.broken.win.IA32" />
> >> > </run>
> >> > </groups>
> >> >
> >> Hello All,
> >>
> >> I'm sorry. It seems that the example does not work. I will try to
> figure
> >> another example soon. ;-)
> >>
> >> Best regards,
> >> Richard
> >> >
> >> > Well, I think our most of existing tests are in the groups of
> >> > {"os.any", "type.api", "level.unit"}, and I have asked TestNG to add
> a
> >> > new option "-groups" for its JUnitConverter which allow us to specify
> >> > the test groups when migrate from JUnit test to TestNG test.
> >> >
> >> > Thanks for reading so far, and I will highly appreciate your comments
> >> > or suggestion. ;-)
> >> >
> >>
> >> --
> >> Richard Liang
> >> China Software Development Lab, IBM
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >>
> >>
> >
>
> --
> Richard Liang
> China Software Development Lab, IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
Vladimir Ivanov wrote:
> Also some tag for regression tests should be added.
Yes. Do you think we could annotate regression test as
*level.regression*? Thanks a lot.
Richard
> thanks, Vladimir
>
>
> On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
>>
>>
>>
>> Richard Liang wrote:
>> > Hello All,
>> >
>> > Now let's talk about the TestNG groups. I have read the related
>> > threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
>> > All of them are good discussion about TestNG groups.
>> >
>> > IMHO, we may define Harmony test groups according the following 4
>> > dimensions:
>> >
>> > 1) [Platform] os.any, os.<platform id>
>> > *os.any* - group of tests which pass on any platform. IMHO, most of
>> > our tests should be in this group.
>> > *os.<platform id>* - group of tests which are designed for one
>> > specific platform. A test may be in more than one of the groups. e.g.,
>> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
>> >
>> > ** os.any and os.<platform id> are mutually exclusive, that is,
>> > tests in os.any group should not be in os.win.IA32.
>> >
>> > 2) [Test state] state.broken, state.broken.<platform id>
>> > *state.broken* - group of tests which fail on every platform, because
>> > of bugs of tests or implementation. We need to fix the bugs of tests
>> > or implementation to make them pass.
>> > *state.broken.<platform id>* - groups of test which only fail on one
>> > specific platform. A test may be in more than one of the groups. e.g.,
>> > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>> >
>> > **state.broken.<platform id> group may be used as a convenient way
>> > to indicate that a test is platform-specific. e.g., If we support 10
>> > platforms, and one test are designed for 9 platforms except for MacOS,
>> > instead of list 9 os.<platform id>, we can just use state.broken.MacOS
>> >
>> > 3) [Test type] type.api , type.impl
>> > *type.api* - group of tests which are tests for APIs in the Java
>> > Specification
>> > *type.impl* - groups of tests which are tests for Harmony-specific
>> > implementation
>> >
>> > ** type.api and type.impl are also mutually exclusive.
>> >
>> > 4) [Test Level] level.unit, level.integration, level.system,
>> > level.stress, etc. (Levels of Test refer to the increase in complexity
>> > as moving through test cycle. )
>> > ** A test may be in more than one of the groups.
>> > ** In fact, some tests such as System tests are the verification of
>> > the entire system. Maybe we'll put them into a separate project.
>> > e.g., harmony/enhanced/SVT (System Verification Test).
>> >
>> > If we want to run all the unit test for APIs on windows, we may use
>> > TestNG groups to select the tests:
>> > <groups>
>> > <run>
>> > <include name="os.any" />
>> > <include name="type.api" />
>> > <include name="os.win.IA32" />
>> > <exclude name=" state.broken" />
>> > <exclude name="state.broken.win.IA32" />
>> > </run>
>> > </groups>
>> >
>> Hello All,
>>
>> I'm sorry. It seems that the example does not work. I will try to figure
>> another example soon. ;-)
>>
>> Best regards,
>> Richard
>> >
>> > Well, I think our most of existing tests are in the groups of
>> > {"os.any", "type.api", "level.unit"}, and I have asked TestNG to add a
>> > new option "-groups" for its JUnitConverter which allow us to specify
>> > the test groups when migrate from JUnit test to TestNG test.
>> >
>> > Thanks for reading so far, and I will highly appreciate your comments
>> > or suggestion. ;-)
>> >
>>
>> --
>> Richard Liang
>> China Software Development Lab, IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Can you explain how you expect to see these used? I'm not arguing, but
people think of regression tests different ways...
geir
Vladimir Ivanov wrote:
> Also some tag for regression tests should be added.
> thanks, Vladimir
>
>
> On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
>>
>>
>>
>> Richard Liang wrote:
>> > Hello All,
>> >
>> > Now let's talk about the TestNG groups. I have read the related
>> > threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
>> > All of them are good discussion about TestNG groups.
>> >
>> > IMHO, we may define Harmony test groups according the following 4
>> > dimensions:
>> >
>> > 1) [Platform] os.any, os.<platform id>
>> > *os.any* - group of tests which pass on any platform. IMHO, most of
>> > our tests should be in this group.
>> > *os.<platform id>* - group of tests which are designed for one
>> > specific platform. A test may be in more than one of the groups. e.g.,
>> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
>> >
>> > ** os.any and os.<platform id> are mutually exclusive, that is,
>> > tests in os.any group should not be in os.win.IA32.
>> >
>> > 2) [Test state] state.broken, state.broken.<platform id>
>> > *state.broken* - group of tests which fail on every platform, because
>> > of bugs of tests or implementation. We need to fix the bugs of tests
>> > or implementation to make them pass.
>> > *state.broken.<platform id>* - groups of test which only fail on one
>> > specific platform. A test may be in more than one of the groups. e.g.,
>> > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>> >
>> > **state.broken.<platform id> group may be used as a convenient way
>> > to indicate that a test is platform-specific. e.g., If we support 10
>> > platforms, and one test are designed for 9 platforms except for MacOS,
>> > instead of list 9 os.<platform id>, we can just use state.broken.MacOS
>> >
>> > 3) [Test type] type.api , type.impl
>> > *type.api* - group of tests which are tests for APIs in the Java
>> > Specification
>> > *type.impl* - groups of tests which are tests for Harmony-specific
>> > implementation
>> >
>> > ** type.api and type.impl are also mutually exclusive.
>> >
>> > 4) [Test Level] level.unit, level.integration, level.system,
>> > level.stress, etc. (Levels of Test refer to the increase in complexity
>> > as moving through test cycle. )
>> > ** A test may be in more than one of the groups.
>> > ** In fact, some tests such as System tests are the verification of
>> > the entire system. Maybe we'll put them into a separate project.
>> > e.g., harmony/enhanced/SVT (System Verification Test).
>> >
>> > If we want to run all the unit test for APIs on windows, we may use
>> > TestNG groups to select the tests:
>> > <groups>
>> > <run>
>> > <include name="os.any" />
>> > <include name="type.api" />
>> > <include name="os.win.IA32" />
>> > <exclude name=" state.broken" />
>> > <exclude name="state.broken.win.IA32" />
>> > </run>
>> > </groups>
>> >
>> Hello All,
>>
>> I'm sorry. It seems that the example does not work. I will try to figure
>> another example soon. ;-)
>>
>> Best regards,
>> Richard
>> >
>> > Well, I think our most of existing tests are in the groups of
>> > {"os.any", "type.api", "level.unit"}, and I have asked TestNG to add a
>> > new option "-groups" for its JUnitConverter which allow us to specify
>> > the test groups when migrate from JUnit test to TestNG test.
>> >
>> > Thanks for reading so far, and I will highly appreciate your comments
>> > or suggestion. ;-)
>> >
>>
>> --
>> Richard Liang
>> China Software Development Lab, IBM
>>
>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>>
>>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Vladimir Ivanov <iv...@gmail.com>.
Also some tag for regression tests should be added.
thanks, Vladimir
On 8/28/06, Richard Liang <ri...@gmail.com> wrote:
>
>
>
> Richard Liang wrote:
> > Hello All,
> >
> > Now let's talk about the TestNG groups. I have read the related
> > threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
> > All of them are good discussion about TestNG groups.
> >
> > IMHO, we may define Harmony test groups according the following 4
> > dimensions:
> >
> > 1) [Platform] os.any, os.<platform id>
> > *os.any* - group of tests which pass on any platform. IMHO, most of
> > our tests should be in this group.
> > *os.<platform id>* - group of tests which are designed for one
> > specific platform. A test may be in more than one of the groups. e.g.,
> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> >
> > ** os.any and os.<platform id> are mutually exclusive, that is,
> > tests in os.any group should not be in os.win.IA32.
> >
> > 2) [Test state] state.broken, state.broken.<platform id>
> > *state.broken* - group of tests which fail on every platform, because
> > of bugs of tests or implementation. We need to fix the bugs of tests
> > or implementation to make them pass.
> > *state.broken.<platform id>* - groups of test which only fail on one
> > specific platform. A test may be in more than one of the groups. e.g.,
> > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
> >
> > **state.broken.<platform id> group may be used as a convenient way
> > to indicate that a test is platform-specific. e.g., If we support 10
> > platforms, and one test are designed for 9 platforms except for MacOS,
> > instead of list 9 os.<platform id>, we can just use state.broken.MacOS
> >
> > 3) [Test type] type.api , type.impl
> > *type.api* - group of tests which are tests for APIs in the Java
> > Specification
> > *type.impl* - groups of tests which are tests for Harmony-specific
> > implementation
> >
> > ** type.api and type.impl are also mutually exclusive.
> >
> > 4) [Test Level] level.unit, level.integration, level.system,
> > level.stress, etc. (Levels of Test refer to the increase in complexity
> > as moving through test cycle. )
> > ** A test may be in more than one of the groups.
> > ** In fact, some tests such as System tests are the verification of
> > the entire system. Maybe we'll put them into a separate project.
> > e.g., harmony/enhanced/SVT (System Verification Test).
> >
> > If we want to run all the unit test for APIs on windows, we may use
> > TestNG groups to select the tests:
> > <groups>
> > <run>
> > <include name="os.any" />
> > <include name="type.api" />
> > <include name="os.win.IA32" />
> > <exclude name=" state.broken" />
> > <exclude name="state.broken.win.IA32" />
> > </run>
> > </groups>
> >
> Hello All,
>
> I'm sorry. It seems that the example does not work. I will try to figure
> another example soon. ;-)
>
> Best regards,
> Richard
> >
> > Well, I think our most of existing tests are in the groups of
> > {"os.any", "type.api", "level.unit"}, and I have asked TestNG to add a
> > new option "-groups" for its JUnitConverter which allow us to specify
> > the test groups when migrate from JUnit test to TestNG test.
> >
> > Thanks for reading so far, and I will highly appreciate your comments
> > or suggestion. ;-)
> >
>
> --
> Richard Liang
> China Software Development Lab, IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
Richard Liang wrote:
> Hello All,
>
> Now let's talk about the TestNG groups. I have read the related
> threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
> All of them are good discussion about TestNG groups.
>
> IMHO, we may define Harmony test groups according the following 4
> dimensions:
>
> 1) [Platform] os.any, os.<platform id>
> *os.any* - group of tests which pass on any platform. IMHO, most of
> our tests should be in this group.
> *os.<platform id>* - group of tests which are designed for one
> specific platform. A test may be in more than one of the groups. e.g.,
> @Test(groups={"os.win.IA32", "os.linux.IA32"})
>
> ** os.any and os.<platform id> are mutually exclusive, that is,
> tests in os.any group should not be in os.win.IA32.
>
> 2) [Test state] state.broken, state.broken.<platform id>
> *state.broken* - group of tests which fail on every platform, because
> of bugs of tests or implementation. We need to fix the bugs of tests
> or implementation to make them pass.
> *state.broken.<platform id>* - groups of test which only fail on one
> specific platform. A test may be in more than one of the groups. e.g.,
> @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>
> **state.broken.<platform id> group may be used as a convenient way
> to indicate that a test is platform-specific. e.g., If we support 10
> platforms, and one test are designed for 9 platforms except for MacOS,
> instead of list 9 os.<platform id>, we can just use state.broken.MacOS
>
> 3) [Test type] type.api, type.impl
> *type.api* - group of tests which are tests for APIs in the Java
> Specification
> *type.impl* - groups of tests which are tests for Harmony-specific
> implementation
>
> ** type.api and type.impl are also mutually exclusive.
>
> 4) [Test Level] level.unit, level.integration, level.system,
> level.stress, etc. (Levels of Test refer to the increase in complexity
> as moving through test cycle. )
> ** A test may be in more than one of the groups.
> ** In fact, some tests such as System tests are the verification of
> the entire system. Maybe we'll put them into a separate project.
> e.g., harmony/enhanced/SVT (System Verification Test).
>
> If we want to run all the unit test for APIs on windows, we may use
> TestNG groups to select the tests:
> <groups>
> <run>
> <include name="os.any" />
> <include name="type.api" />
> <include name="os.win.IA32" />
> <exclude name="state.broken" />
> <exclude name="state.broken.win.IA32" />
> </run>
> </groups>
>
Hello All,
I'm sorry. It seems that the example does not work. I will try to figure
another example soon. ;-)
Best regards,
Richard
>
> Well, I think our most of existing tests are in the groups of
> {"os.any", "type.api", "level.unit"}, and I have asked TestNG to add a
> new option "-groups" for its JUnitConverter which allow us to specify
> the test groups when migrate from JUnit test to TestNG test.
>
> Thanks for reading so far, and I will highly appreciate your comments
> or suggestion. ;-)
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by Richard Liang <ri...@gmail.com>.
Geir Magnusson Jr. wrote:
> I'm just catching up after a few days of travel. Have you begun
> sketching this out on the wiki?
Sorry for my late reply. Just spent a few days looking after my
three-month-old daughter ;-) I have just recorded what we have
discussed on our wiki[1].
[1]. http://wiki.apache.org/harmony/Testing_Convention
Best regards,
Richard
>
> geir
>
>
> Richard Liang wrote:
>> Hello All,
>>
>> Now let's talk about the TestNG groups. I have read the related
>> threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
>> All of them are good discussion about TestNG groups.
>>
>> IMHO, we may define Harmony test groups according the following 4
>> dimensions:
>>
>> 1) [Platform] os.any, os.<platform id>
>> *os.any* - group of tests which pass on any platform. IMHO, most of
>> our tests should be in this group.
>> *os.<platform id>* - group of tests which are designed for one
>> specific platform. A test may be in more than one of the groups.
>> e.g., @Test(groups={"os.win.IA32", "os.linux.IA32"})
>>
>> ** os.any and os.<platform id> are mutually exclusive, that is,
>> tests in os.any group should not be in os.win.IA32.
>>
>> 2) [Test state] state.broken, state.broken.<platform id>
>> *state.broken* - group of tests which fail on every platform, because
>> of bugs of tests or implementation. We need to fix the bugs of tests
>> or implementation to make them pass.
>> *state.broken.<platform id>* - groups of test which only fail on one
>> specific platform. A test may be in more than one of the groups.
>> e.g., @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>>
>> **state.broken.<platform id> group may be used as a convenient
>> way to indicate that a test is platform-specific. e.g., If we support
>> 10 platforms, and one test are designed for 9 platforms except for
>> MacOS, instead of list 9 os.<platform id>, we can just use
>> state.broken.MacOS
>>
>> 3) [Test type] type.api, type.impl
>> *type.api* - group of tests which are tests for APIs in the Java
>> Specification
>> *type.impl* - groups of tests which are tests for Harmony-specific
>> implementation
>>
>> ** type.api and type.impl are also mutually exclusive.
>>
>> 4) [Test Level] level.unit, level.integration, level.system,
>> level.stress, etc. (Levels of Test refer to the increase in
>> complexity as moving through test cycle. )
>> ** A test may be in more than one of the groups.
>> ** In fact, some tests such as System tests are the verification
>> of the entire system. Maybe we'll put them into a separate project.
>> e.g., harmony/enhanced/SVT (System Verification Test).
>>
>> If we want to run all the unit test for APIs on windows, we may use
>> TestNG groups to select the tests:
>> <groups>
>> <run>
>> <include name="os.any" />
>> <include name="type.api" />
>> <include name="os.win.IA32" />
>> <exclude name="state.broken" />
>> <exclude name="state.broken.win.IA32" />
>> </run>
>> </groups>
>>
>>
>> Well, I think our most of existing tests are in the groups of
>> {"os.any", "type.api", "level.unit"}, and I have asked TestNG to add
>> a new option "-groups" for its JUnitConverter which allow us to
>> specify the test groups when migrate from JUnit test to TestNG test.
>>
>> Thanks for reading so far, and I will highly appreciate your comments
>> or suggestion. ;-)
>>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org
Re: [classlib][TestNG] groups of Harmony test
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
I'm just catching up after a few days of travel. Have you begun
sketching this out on the wiki?
geir
Richard Liang wrote:
> Hello All,
>
> Now let's talk about the TestNG groups. I have read the related threads
> which posted by George, Vladimir Ivanov and Alexei Zakharov. All of them
> are good discussion about TestNG groups.
>
> IMHO, we may define Harmony test groups according the following 4
> dimensions:
>
> 1) [Platform] os.any, os.<platform id>
> *os.any* - group of tests which pass on any platform. IMHO, most of our
> tests should be in this group.
> *os.<platform id>* - group of tests which are designed for one specific
> platform. A test may be in more than one of the groups. e.g.,
> @Test(groups={"os.win.IA32", "os.linux.IA32"})
>
> ** os.any and os.<platform id> are mutually exclusive, that is, tests
> in os.any group should not be in os.win.IA32.
>
> 2) [Test state] state.broken, state.broken.<platform id>
> *state.broken* - group of tests which fail on every platform, because of
> bugs of tests or implementation. We need to fix the bugs of tests or
> implementation to make them pass.
> *state.broken.<platform id>* - groups of test which only fail on one
> specific platform. A test may be in more than one of the groups. e.g.,
> @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
>
> **state.broken.<platform id> group may be used as a convenient way
> to indicate that a test is platform-specific. e.g., If we support 10
> platforms, and one test are designed for 9 platforms except for MacOS,
> instead of list 9 os.<platform id>, we can just use state.broken.MacOS
>
> 3) [Test type] type.api, type.impl
> *type.api* - group of tests which are tests for APIs in the Java
> Specification
> *type.impl* - groups of tests which are tests for Harmony-specific
> implementation
>
> ** type.api and type.impl are also mutually exclusive.
>
> 4) [Test Level] level.unit, level.integration, level.system,
> level.stress, etc. (Levels of Test refer to the increase in complexity
> as moving through test cycle. )
> ** A test may be in more than one of the groups.
> ** In fact, some tests such as System tests are the verification of
> the entire system. Maybe we'll put them into a separate project. e.g.,
> harmony/enhanced/SVT (System Verification Test).
>
> If we want to run all the unit test for APIs on windows, we may use
> TestNG groups to select the tests:
> <groups>
> <run>
> <include name="os.any" />
> <include name="type.api" />
> <include name="os.win.IA32" />
> <exclude name="state.broken" />
> <exclude name="state.broken.win.IA32" />
> </run>
> </groups>
>
>
> Well, I think our most of existing tests are in the groups of {"os.any",
> "type.api", "level.unit"}, and I have asked TestNG to add a new option
> "-groups" for its JUnitConverter which allow us to specify the test
> groups when migrate from JUnit test to TestNG test.
>
> Thanks for reading so far, and I will highly appreciate your comments or
> suggestion. ;-)
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org