You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Stepan Mishura <st...@gmail.com> on 2006/02/16 06:53:42 UTC
[classlib] do we need a global exclude list?
Did anybody think of creating a 'global' (i.e. shared by all modules)
exclude list or every module will have its own exclude list? Or Harmony
tests will always pass and we don't need it at all :-)
I see at least the following benefits of creating 'global' exclude list: all
know issues are kept in one well known place (they don't spread between
several private lists), also it is easier to create an exclude list for a
target platform, for example, linux.exclude or win.exclude.
Thoughts?
Thanks,
Stepan Mishura
Intel Middleware Products Division
Re: [classlib] do we need a global exclude list?
Posted by Geir Magnusson Jr <ge...@pobox.com>.
karan malhi wrote:
> Could there be a switch to over-ride the global exclude list
> entry/entries? This way a module by default would use a global exclude
> list and also have the freedom to ignore it and use the module exclude
> list.
I don't know what having two exclude lists would mean - a test should be
excluded for a specific reason, and therefore I can't see why one would
be in the global list, but not in the module list (and vice versa...)
geir
>
> Geir Magnusson Jr wrote:
> However, I did imagine that we'd give the modules a bit of freedom and
> independence for testing - a global exclude list might impact that?
>
>>
>>
>> Stepan Mishura wrote:
>>
>>> Did anybody think of creating a 'global' (i.e. shared by all modules)
>>> exclude list or every module will have its own exclude list? Or Harmony
>>> tests will always pass and we don't need it at all :-)
>>
>>
>> That would be the goal :)
>>
>>> I see at least the following benefits of creating 'global' exclude
>>> list: all
>>> know issues are kept in one well known place (they don't spread between
>>> several private lists)
>>
>>
>> That's true, but.... I always imagined that people would be working
>> in the modules anyway, so there isn't much gain.
>>
>> also it is easier to create an exclude list for a
>>
>>> target platform, for example, linux.exclude or win.exclude.
>>
>>
>> Yes, that could be. Interesting idea.
>>
>> However, I did imagine that we'd give the modules a bit of freedom and
>> independence for testing - a global exclude list might impact that?
>>
>> geir
>>
>>>
>>> Thoughts?
>>>
>>> Thanks,
>>> Stepan Mishura
>>> Intel Middleware Products Division
>>>
>>
>
Re: [classlib] do we need a global exclude list?
Posted by Tim Ellison <t....@gmail.com>.
karan malhi wrote:
> Could there be a switch to over-ride the global exclude list
> entry/entries? This way a module by default would use a global exclude
> list and also have the freedom to ignore it and use the module exclude
> list.
In the incoming HARMONY-57 contribution the list is 'global', but you
can override the exclusion list by specifying a replacement's location
in a java property.
In future, I see the global list as simply a dynamic amalgam of the
module lists in the target. If we put the exclude lists in a standard
location, (in the repo and in the tests jar file(s)) then they can be
found by the test framework.
Regards,
Tim
> Geir Magnusson Jr wrote:
> However, I did imagine that we'd give the modules a bit of freedom and
> independence for testing - a global exclude list might impact that?
>
>>
>>
>> Stepan Mishura wrote:
>>
>>> Did anybody think of creating a 'global' (i.e. shared by all modules)
>>> exclude list or every module will have its own exclude list? Or Harmony
>>> tests will always pass and we don't need it at all :-)
>>
>>
>> That would be the goal :)
>>
>>> I see at least the following benefits of creating 'global' exclude
>>> list: all
>>> know issues are kept in one well known place (they don't spread between
>>> several private lists)
>>
>>
>> That's true, but.... I always imagined that people would be working
>> in the modules anyway, so there isn't much gain.
>>
>> also it is easier to create an exclude list for a
>>
>>> target platform, for example, linux.exclude or win.exclude.
>>
>>
>> Yes, that could be. Interesting idea.
>>
>> However, I did imagine that we'd give the modules a bit of freedom and
>> independence for testing - a global exclude list might impact that?
>>
>> geir
>>
>>>
>>> Thoughts?
>>>
>>> Thanks,
>>> Stepan Mishura
>>> Intel Middleware Products Division
>>>
>>
>
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
Re: [classlib] do we need a global exclude list?
Posted by karan malhi <ka...@gmail.com>.
Could there be a switch to over-ride the global exclude list
entry/entries? This way a module by default would use a global exclude
list and also have the freedom to ignore it and use the module exclude list.
Geir Magnusson Jr wrote:
However, I did imagine that we'd give the modules a bit of freedom and
independence for testing - a global exclude list might impact that?
>
>
> Stepan Mishura wrote:
>
>> Did anybody think of creating a 'global' (i.e. shared by all modules)
>> exclude list or every module will have its own exclude list? Or Harmony
>> tests will always pass and we don't need it at all :-)
>
>
> That would be the goal :)
>
>> I see at least the following benefits of creating 'global' exclude
>> list: all
>> know issues are kept in one well known place (they don't spread between
>> several private lists)
>
>
> That's true, but.... I always imagined that people would be working
> in the modules anyway, so there isn't much gain.
>
> also it is easier to create an exclude list for a
>
>> target platform, for example, linux.exclude or win.exclude.
>
>
> Yes, that could be. Interesting idea.
>
> However, I did imagine that we'd give the modules a bit of freedom and
> independence for testing - a global exclude list might impact that?
>
> geir
>
>>
>> Thoughts?
>>
>> Thanks,
>> Stepan Mishura
>> Intel Middleware Products Division
>>
>
--
Karan Singh
Re: [classlib] do we need a global exclude list?
Posted by Stepan Mishura <st...@gmail.com>.
On 2/16/06, Tim Ellison <t.p.ellison <at> gmail.com> wrote:
> Applying a style sheet makes it easy to read the exclusions list in
> glorious technicolor, either in an individual module or as a combined
> global view.
I see. And global view will be very helpful
Thanks,
Stepan
On 2/16/06, Tim Ellison <t.p.ellison <at> gmail.com> wrote:
>
> Geir Magnusson Jr wrote:
> > Stepan Mishura wrote:
> >> Did anybody think of creating a 'global' (i.e. shared by all modules)
> >> exclude list or every module will have its own exclude list? Or Harmony
> >> tests will always pass and we don't need it at all :-)
> >
> > That would be the goal :)
> >
> >> I see at least the following benefits of creating 'global' exclude
> >> list: all
> >> know issues are kept in one well known place (they don't spread between
> >> several private lists)
> >
> > That's true, but.... I always imagined that people would be working in
> > the modules anyway, so there isn't much gain.
>
> Modules should be as self-sufficient as we can make them, so devolving
> tests and their exclude lists into a module makes sense to me too. They
> can be linked into a global view quite easily (read on...)
>
> > also it is easier to create an exclude list for a
> >> target platform, for example, linux.exclude or win.exclude.
> >
> > Yes, that could be. Interesting idea.
>
> The mechanism that George contributed in HARMONY-57 uses a decorator to
> implement an exclusion list on regular JUnit tests. (I happen to know
> that George is off enjoying himself for the next few days, so I hope he
> doesn't mind me describing it here.)
>
> The exclusion list is implemented as a (declarative) XML file read when
> the tests run -- in HARMONY-57 you can see one in
> Harmony_Tests/src/test/resources/config/jcltest-excludes.xml
>
> Picking an entry at random from there:
> ...
> <hy:type id="tests.api.java.io.FileTest">
> <hy:exclude id="test_Constructor_String_String_112270"
> shouldfix="true">
> <hy:reason>Undiagnosed failure</hy:reason>
> </hy:exclude>
> <hy:exclude id="test_Constructor_File_String_112270"
> shouldfix="true">
> <hy:reason>Undiagnosed failure</hy:reason>
> </hy:exclude>
> <hy:exclude id="test_deleteOnExit" shouldfix="true">
> <hy:reason>Needs investigation...</hy:reason>
> </hy:exclude>
> </hy:type>
> ...
>
> You can see how it works -- some tests are excluded because they fail
> and should be fixed, others (not shown here) can be excluded because
> they don't make sense on a particular platform, VM, etc.
>
> Applying a style sheet makes it easy to read the exclusions list in
> glorious technicolor, either in an individual module or as a combined
> global view.
>
> Take a look at the incoming contribution and see if it fits your needs.
>
> Regards,
> Tim
>
> > However, I did imagine that we'd give the modules a bit of freedom and
> > independence for testing - a global exclude list might impact that?
> >
> > geir
> >
> >>
> >> Thoughts?
> >>
> >> Thanks,
> >> Stepan Mishura
> >> Intel Middleware Products Division
> >>
> >
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>
--
Thanks,
Stepan Mishura
Intel Middleware Products Division
Re: [classlib] do we need a global exclude list?
Posted by Tim Ellison <t....@gmail.com>.
Geir Magnusson Jr wrote:
>
>
> Tim Ellison wrote:
>>
>> The exclusion list is implemented as a (declarative) XML file read when
>> the tests run -- in HARMONY-57 you can see one in
>> Harmony_Tests/src/test/resources/config/jcltest-excludes.xml
>>
>> Picking an entry at random from there:
>> ...
>> <hy:type id="tests.api.java.io.FileTest">
>> <hy:exclude id="test_Constructor_String_String_112270"
>> shouldfix="true">
>> <hy:reason>Undiagnosed failure</hy:reason>
>> </hy:exclude>
>> <hy:exclude id="test_Constructor_File_String_112270"
>> shouldfix="true">
>> <hy:reason>Undiagnosed failure</hy:reason>
>> </hy:exclude>
>> <hy:exclude id="test_deleteOnExit" shouldfix="true">
>> <hy:reason>Needs investigation...</hy:reason>
>> </hy:exclude>
>> </hy:type>
>> ...
>>
>> You can see how it works -- some tests are excluded because they fail
>> and should be fixed, others (not shown here) can be excluded because
>> they don't make sense on a particular platform, VM, etc.
>>
>> Applying a style sheet makes it easy to read the exclusions list in
>> glorious technicolor, either in an individual module or as a combined
>> global view.
>>
>
> That's cute. And if these were local to each module, in a "well-known"
> place, we still could present a global picture with a top level thingy
> that generates a list...
Yep, probably putting the exclusion list into src/test/resources in each
module, and like you say, easy to combine.
> We could also add a platform element to make exclusions platform
> specific...
Already there ;-)
<xsd:element name="exclude">
<xsd:complexType>
<xsd:sequence>
<xsd:element ref="reason" minOccurs="0" maxOccurs="1"/>
</xsd:sequence>
<xsd:attribute name="id"
type="idtype"
default="all" />
<xsd:attribute name="platform"
type="platformlisttype"
default="all" />
<xsd:attribute name="shouldfix"
type="booleanstringtype"
default="true" />
</xsd:complexType>
</xsd:element>
where
<xsd:simpleType name="platformstringtype" >
<xsd:restriction base="xsd:string">
<xsd:enumeration value="win.IA32"/>
<xsd:enumeration value="linux.IA32"/>
<xsd:enumeration value="all"/>
</xsd:restriction>
</xsd:simpleType>
but of course we can do whatever we want.
Regards,
Tim
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
Re: [classlib] do we need a global exclude list?
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Tim Ellison wrote:
>
> The exclusion list is implemented as a (declarative) XML file read when
> the tests run -- in HARMONY-57 you can see one in
> Harmony_Tests/src/test/resources/config/jcltest-excludes.xml
>
> Picking an entry at random from there:
> ...
> <hy:type id="tests.api.java.io.FileTest">
> <hy:exclude id="test_Constructor_String_String_112270"
> shouldfix="true">
> <hy:reason>Undiagnosed failure</hy:reason>
> </hy:exclude>
> <hy:exclude id="test_Constructor_File_String_112270"
> shouldfix="true">
> <hy:reason>Undiagnosed failure</hy:reason>
> </hy:exclude>
> <hy:exclude id="test_deleteOnExit" shouldfix="true">
> <hy:reason>Needs investigation...</hy:reason>
> </hy:exclude>
> </hy:type>
> ...
>
> You can see how it works -- some tests are excluded because they fail
> and should be fixed, others (not shown here) can be excluded because
> they don't make sense on a particular platform, VM, etc.
>
> Applying a style sheet makes it easy to read the exclusions list in
> glorious technicolor, either in an individual module or as a combined
> global view.
>
That's cute. And if these were local to each module, in a "well-known"
place, we still could present a global picture with a top level thingy
that generates a list...
We could also add a platform element to make exclusions platform specific...
geir
Re: [classlib] do we need a global exclude list?
Posted by Mikhail Loenko <ml...@gmail.com>.
On 2/16/06, Tim Ellison <t....@gmail.com> wrote:
> > That's true, but.... I always imagined that people would be working in
> > the modules anyway, so there isn't much gain.
>
> Modules should be as self-sufficient as we can make them, so devolving
> tests and their exclude lists into a module makes sense to me too.
As far as tests in one module can fail because of bugs in other modules,
I think the modules are not self-sufficient.
Thanks,
Mikhail
Re: [classlib] do we need a global exclude list?
Posted by Tim Ellison <t....@gmail.com>.
Geir Magnusson Jr wrote:
> Stepan Mishura wrote:
>> Did anybody think of creating a 'global' (i.e. shared by all modules)
>> exclude list or every module will have its own exclude list? Or Harmony
>> tests will always pass and we don't need it at all :-)
>
> That would be the goal :)
>
>> I see at least the following benefits of creating 'global' exclude
>> list: all
>> know issues are kept in one well known place (they don't spread between
>> several private lists)
>
> That's true, but.... I always imagined that people would be working in
> the modules anyway, so there isn't much gain.
Modules should be as self-sufficient as we can make them, so devolving
tests and their exclude lists into a module makes sense to me too. They
can be linked into a global view quite easily (read on...)
> also it is easier to create an exclude list for a
>> target platform, for example, linux.exclude or win.exclude.
>
> Yes, that could be. Interesting idea.
The mechanism that George contributed in HARMONY-57 uses a decorator to
implement an exclusion list on regular JUnit tests. (I happen to know
that George is off enjoying himself for the next few days, so I hope he
doesn't mind me describing it here.)
The exclusion list is implemented as a (declarative) XML file read when
the tests run -- in HARMONY-57 you can see one in
Harmony_Tests/src/test/resources/config/jcltest-excludes.xml
Picking an entry at random from there:
...
<hy:type id="tests.api.java.io.FileTest">
<hy:exclude id="test_Constructor_String_String_112270"
shouldfix="true">
<hy:reason>Undiagnosed failure</hy:reason>
</hy:exclude>
<hy:exclude id="test_Constructor_File_String_112270"
shouldfix="true">
<hy:reason>Undiagnosed failure</hy:reason>
</hy:exclude>
<hy:exclude id="test_deleteOnExit" shouldfix="true">
<hy:reason>Needs investigation...</hy:reason>
</hy:exclude>
</hy:type>
...
You can see how it works -- some tests are excluded because they fail
and should be fixed, others (not shown here) can be excluded because
they don't make sense on a particular platform, VM, etc.
Applying a style sheet makes it easy to read the exclusions list in
glorious technicolor, either in an individual module or as a combined
global view.
Take a look at the incoming contribution and see if it fits your needs.
Regards,
Tim
> However, I did imagine that we'd give the modules a bit of freedom and
> independence for testing - a global exclude list might impact that?
>
> geir
>
>>
>> Thoughts?
>>
>> Thanks,
>> Stepan Mishura
>> Intel Middleware Products Division
>>
>
--
Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.
Re: [classlib] do we need a global exclude list?
Posted by Stepan Mishura <st...@gmail.com>.
On 2/16/06, Geir Magnusson Jr <geir <at> pobox.com> wrote:
>
>
>
> Stepan Mishura wrote:
> > Did anybody think of creating a 'global' (i.e. shared by all modules)
> > exclude list or every module will have its own exclude list? Or Harmony
> > tests will always pass and we don't need it at all :-)
>
> That would be the goal :)
>
> > I see at least the following benefits of creating 'global' exclude list:
> all
> > know issues are kept in one well known place (they don't spread between
> > several private lists)
>
> That's true, but.... I always imagined that people would be working in
> the modules anyway, so there isn't much gain.
>
> also it is easier to create an exclude list for a
> > target platform, for example, linux.exclude or win.exclude.
>
> Yes, that could be. Interesting idea.
>
> However, I did imagine that we'd give the modules a bit of freedom and
> independence for testing - a global exclude list might impact that?
I think we won't restrict freedom too much if we ask update only one list.
And I believe it won't be updated too often that may cause frequent conflict
resolution.
Also the global exclude list may be useful to understand impact of a bug,
for example, we may organize its entries in the following way:
<!-- HARMONY-NNN, summary : a bug in luni module -->
<exclude name="java/lang/SomeTest.java"/>
<exclude name="java/security/SomeTest.java"/>
<exclude name="java/nio/SomeTest.java"/>
...
<exclude name="java/text/SomeTest.java"/>
<!-- HARMONY-ZZZ, summary : a bug in auth module -->
<exclude name="javax/security/auth/SomeTest.java"/>
Thanks,
Stepan
geir
>
> >
> > Thoughts?
> >
> > Thanks,
> > Stepan Mishura
> > Intel Middleware Products Division
> >
>
--
Thanks,
Stepan Mishura
Intel Middleware Products Division
Re: [classlib] do we need a global exclude list?
Posted by Geir Magnusson Jr <ge...@pobox.com>.
Stepan Mishura wrote:
> Did anybody think of creating a 'global' (i.e. shared by all modules)
> exclude list or every module will have its own exclude list? Or Harmony
> tests will always pass and we don't need it at all :-)
That would be the goal :)
> I see at least the following benefits of creating 'global' exclude list: all
> know issues are kept in one well known place (they don't spread between
> several private lists)
That's true, but.... I always imagined that people would be working in
the modules anyway, so there isn't much gain.
also it is easier to create an exclude list for a
> target platform, for example, linux.exclude or win.exclude.
Yes, that could be. Interesting idea.
However, I did imagine that we'd give the modules a bit of freedom and
independence for testing - a global exclude list might impact that?
geir
>
> Thoughts?
>
> Thanks,
> Stepan Mishura
> Intel Middleware Products Division
>
Re: [classlib] do we need a global exclude list?
Posted by Richard Liang <ri...@gmail.com>.
+1
Richard Liang
China Software Development Lab, IBM
Stepan Mishura wrote:
> Did anybody think of creating a 'global' (i.e. shared by all modules)
> exclude list or every module will have its own exclude list? Or Harmony
> tests will always pass and we don't need it at all :-)
> I see at least the following benefits of creating 'global' exclude list: all
> know issues are kept in one well known place (they don't spread between
> several private lists), also it is easier to create an exclude list for a
> target platform, for example, linux.exclude or win.exclude.
>
> Thoughts?
>
> Thanks,
> Stepan Mishura
> Intel Middleware Products Division
>
>