You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Stepan Mishura <st...@gmail.com> on 2008/03/04 10:39:45 UTC
[build-and-test] Thoughts about testing infra
Hi folks,
I've moved 2.0 build-and-test's branch to the trunk - the branch was
used during M2-M5 testing and IMO it proved to be very useful (also I
tagged the former trunk as 1.0.)
I think that this is a good time to collect ideas and hear opinions
what should be improved and possible directions for test infra
development.
I have to admit that the current testing infra is far from being
perfect – there a lot for work to make is more usable, cleaning up
current scripts, sort out all know issues with tests/scenarios and so
on. But I'd like to address problems I believe of different level. I
have an impression that Harmony tests suites are still in shadow (IOW,
IMHO they are undervalued) of testing automation (i.e. CruiseControl
and support scripts) – and most of folks prefer to contribute to the
code development. And I believe that it is possible to make tests
development also interesting too. IMO the missed point that most of
Harmony test suites can be used as stand alone suites to verify/test
different implementations – in other words, the suites are not tightly
coupled with DRL VM, for example, it is possible to run them against
IBM VME (BTW, may it be makes sense to run full set of suites against
IBM VME for each milestone candidate as for DRL VM. Any volunteers?)
I'm going to update the test infra home page [1] with info related to
the renewed trunk. Also I think the page should contain some words
about project's testing strategy: i.e. how Harmony QA and release
process are organized - integrity/snapshot testing, milestones, code
freeze and etc., what suites are there (their description and
purpose). I think this may encourage people to invest efforts into
developing tests.
Other ideas and thoughts?
[1] http://harmony.apache.org/subcomponents/buildtest/index.html
Thanks,
Stepan.
Re: [build-and-test] Thoughts about testing infra
Posted by Stepan Mishura <st...@gmail.com>.
On 3/5/08, Tim Ellison <t....@gmail.com> wrote:
> Stepan Mishura wrote:
> > Hi folks,
> >
> > I've moved 2.0 build-and-test's branch to the trunk - the branch was
> > used during M2-M5 testing and IMO it proved to be very useful (also I
> > tagged the former trunk as 1.0.)
>
> Hurray!
>
> > I think that this is a good time to collect ideas and hear opinions
> > what should be improved and possible directions for test infra
> > development.
> >
> > I have to admit that the current testing infra is far from being
> > perfect – there a lot for work to make is more usable, cleaning up
> > current scripts, sort out all know issues with tests/scenarios and so
> > on. But I'd like to address problems I believe of different level. I
> > have an impression that Harmony tests suites are still in shadow (IOW,
> > IMHO they are undervalued) of testing automation (i.e. CruiseControl
> > and support scripts) – and most of folks prefer to contribute to the
> > code development. And I believe that it is possible to make tests
> > development also interesting too. IMO the missed point that most of
> > Harmony test suites can be used as stand alone suites to verify/test
> > different implementations – in other words, the suites are not tightly
> > coupled with DRL VM, for example, it is possible to run them against
> > IBM VME (BTW, may it be makes sense to run full set of suites against
> > IBM VME for each milestone candidate as for DRL VM. Any volunteers?)
> >
> > I'm going to update the test infra home page [1] with info related to
> > the renewed trunk. Also I think the page should contain some words
> > about project's testing strategy: i.e. how Harmony QA and release
> > process are organized - integrity/snapshot testing, milestones, code
> > freeze and etc., what suites are there (their description and
> > purpose). I think this may encourage people to invest efforts into
> > developing tests.
> >
> > Other ideas and thoughts?
>
> Thanks for your investment in the test suites, and taking time to think
> about the improvements.
>
> The thing I struggle with is understanding the current system well
> enough to make a valuable contribution.
>
Yes, to automate the suite runs you have to study how a whole system
(i.e. BTI) works.
And my point was that you don't need to understand BTI to contribute
to a particular suite. You should only grab the suite and start
running it. This is true for classlib unit tests, drlvm suites ... ,
isn't it? But the same applies for the rest of them: Geronimo unit
tests, Eclipse unit tests, functional suite, VTS VM suite and so on.
> It would be really good to have a newbies' guide to the test suites
> document on the website with step-by-step instructions (like the quick
> help for contributors document).
>
Sure, I'm going to improve the docs.
Thanks,
Stepan.
> While the architecture documentation would be interesting, I think many
> people work by running the simple case, then get dragged into
> investigating the complex cases when their needs require it.
>
> Regards,
> Tim
>
> > [1] http://harmony.apache.org/subcomponents/buildtest/index.html
> >
> > Thanks,
> > Stepan.
> >
>
Re: [build-and-test] Thoughts about testing infra
Posted by Stepan Mishura <st...@gmail.com>.
On 3/6/08, Nathan Beyer <nb...@gmail.com> wrote:
<SNIP>> >
> > Thanks for your investment in the test suites, and taking time to think
> > about the improvements.
> >
> > The thing I struggle with is understanding the current system well
> > enough to make a valuable contribution.
> >
> > It would be really good to have a newbies' guide to the test suites
> > document on the website with step-by-step instructions (like the quick
> > help for contributors document).
>
> I concur with sentiment. I don't get a chance to continually work with
> the tool, so I start fresh every time I come back to and it seems
> every time I have to stumble over a bunch of little problems for about
> a day before getting it working again. I don't know if it's been
> resolved, but the issues with running on Ant 1.7 instead of Ant 1.6.5
> we always painful.
Yep, the system wasn't tested with 1.7. Could you file a JIRA is case
of troubles?
Thanks,
Stepan.
>
> -Nathan
>
> >
> > While the architecture documentation would be interesting, I think many
> > people work by running the simple case, then get dragged into
> > investigating the complex cases when their needs require it.
> >
> > Regards,
> > Tim
> >
> >
> >
> > > [1] http://harmony.apache.org/subcomponents/buildtest/index.html
> > >
> > > Thanks,
> > > Stepan.
> > >
> >
>
Re: [build-and-test] Thoughts about testing infra
Posted by Nathan Beyer <nb...@gmail.com>.
On Wed, Mar 5, 2008 at 11:13 AM, Tim Ellison <t....@gmail.com> wrote:
> Stepan Mishura wrote:
> > Hi folks,
> >
> > I've moved 2.0 build-and-test's branch to the trunk - the branch was
> > used during M2-M5 testing and IMO it proved to be very useful (also I
> > tagged the former trunk as 1.0.)
>
> Hurray!
>
>
>
> > I think that this is a good time to collect ideas and hear opinions
> > what should be improved and possible directions for test infra
> > development.
> >
> > I have to admit that the current testing infra is far from being
> > perfect – there a lot for work to make is more usable, cleaning up
> > current scripts, sort out all know issues with tests/scenarios and so
> > on. But I'd like to address problems I believe of different level. I
> > have an impression that Harmony tests suites are still in shadow (IOW,
> > IMHO they are undervalued) of testing automation (i.e. CruiseControl
> > and support scripts) – and most of folks prefer to contribute to the
> > code development. And I believe that it is possible to make tests
> > development also interesting too. IMO the missed point that most of
> > Harmony test suites can be used as stand alone suites to verify/test
> > different implementations – in other words, the suites are not tightly
> > coupled with DRL VM, for example, it is possible to run them against
> > IBM VME (BTW, may it be makes sense to run full set of suites against
> > IBM VME for each milestone candidate as for DRL VM. Any volunteers?)
> >
> > I'm going to update the test infra home page [1] with info related to
> > the renewed trunk. Also I think the page should contain some words
> > about project's testing strategy: i.e. how Harmony QA and release
> > process are organized - integrity/snapshot testing, milestones, code
> > freeze and etc., what suites are there (their description and
> > purpose). I think this may encourage people to invest efforts into
> > developing tests.
> >
> > Other ideas and thoughts?
>
> Thanks for your investment in the test suites, and taking time to think
> about the improvements.
>
> The thing I struggle with is understanding the current system well
> enough to make a valuable contribution.
>
> It would be really good to have a newbies' guide to the test suites
> document on the website with step-by-step instructions (like the quick
> help for contributors document).
I concur with sentiment. I don't get a chance to continually work with
the tool, so I start fresh every time I come back to and it seems
every time I have to stumble over a bunch of little problems for about
a day before getting it working again. I don't know if it's been
resolved, but the issues with running on Ant 1.7 instead of Ant 1.6.5
we always painful.
-Nathan
>
> While the architecture documentation would be interesting, I think many
> people work by running the simple case, then get dragged into
> investigating the complex cases when their needs require it.
>
> Regards,
> Tim
>
>
>
> > [1] http://harmony.apache.org/subcomponents/buildtest/index.html
> >
> > Thanks,
> > Stepan.
> >
>
Re: [build-and-test] Thoughts about testing infra
Posted by Tim Ellison <t....@gmail.com>.
Stepan Mishura wrote:
> Hi folks,
>
> I've moved 2.0 build-and-test's branch to the trunk - the branch was
> used during M2-M5 testing and IMO it proved to be very useful (also I
> tagged the former trunk as 1.0.)
Hurray!
> I think that this is a good time to collect ideas and hear opinions
> what should be improved and possible directions for test infra
> development.
>
> I have to admit that the current testing infra is far from being
> perfect – there a lot for work to make is more usable, cleaning up
> current scripts, sort out all know issues with tests/scenarios and so
> on. But I'd like to address problems I believe of different level. I
> have an impression that Harmony tests suites are still in shadow (IOW,
> IMHO they are undervalued) of testing automation (i.e. CruiseControl
> and support scripts) – and most of folks prefer to contribute to the
> code development. And I believe that it is possible to make tests
> development also interesting too. IMO the missed point that most of
> Harmony test suites can be used as stand alone suites to verify/test
> different implementations – in other words, the suites are not tightly
> coupled with DRL VM, for example, it is possible to run them against
> IBM VME (BTW, may it be makes sense to run full set of suites against
> IBM VME for each milestone candidate as for DRL VM. Any volunteers?)
>
> I'm going to update the test infra home page [1] with info related to
> the renewed trunk. Also I think the page should contain some words
> about project's testing strategy: i.e. how Harmony QA and release
> process are organized - integrity/snapshot testing, milestones, code
> freeze and etc., what suites are there (their description and
> purpose). I think this may encourage people to invest efforts into
> developing tests.
>
> Other ideas and thoughts?
Thanks for your investment in the test suites, and taking time to think
about the improvements.
The thing I struggle with is understanding the current system well
enough to make a valuable contribution.
It would be really good to have a newbies' guide to the test suites
document on the website with step-by-step instructions (like the quick
help for contributors document).
While the architecture documentation would be interesting, I think many
people work by running the simple case, then get dragged into
investigating the complex cases when their needs require it.
Regards,
Tim
> [1] http://harmony.apache.org/subcomponents/buildtest/index.html
>
> Thanks,
> Stepan.
>
Re: [build-and-test] Thoughts about testing infra
Posted by Stepan Mishura <st...@gmail.com>.
On 3/7/08, Sean Qiu <se...@gmail.com> wrote:
> I know we can all suites have a property to specify runtime to be testes. :)
> But if we want to test another runtime, we need to specify them one by one.
>
> I mean is there a way to specify it globally?
>
Currently all suites depends on
DRLVM - ${drlvm.parameters.shared.jre.dir} property
or
HDK - ${hdk.parameters.shared.binaries.jre.dir} property
If you redefine both properties to point out to your JRE dir then all
suite are run against your JRE.
But I have to admit that this is not obvious and should be fixed by
defining/documenting property common for all suites.
Thanks,
Stepan.
> �� 08-3-6��Stepan Mishura<st...@gmail.com> ���
> > On 3/5/08, Sean Qiu <se...@gmail.com> wrote:
> > <SNIP>>
> >
> > > > Harmony test suites can be used as stand alone suites to verify/test
> > > > different implementations �C in other words, the suites are not tightly
> > > > coupled with DRL VM, for example, it is possible to run them against
> > > > IBM VME (BTW, may it be makes sense to run full set of suites against
> > > > IBM VME for each milestone candidate as for DRL VM. Any volunteers?)
> > > >
> > >
> > > It sounds reasonable, in fact, i am going to plan the same thing.
> > > Maybe we can make our BTI a more flexible infrastrcture for more runtime.
> > > We can pick up a runtime before starting the BTI as well as the test
> > > suite, such as:
> > >
> > > drlvm+harmony -| |- classlib unit test
> > > j9vm+harmony -| -- BTI ----|- vm unit test
> > > ... -| |- ...
> > >
> >
> >
> > Sean,
> >
> > With the current BTI it is possible to select runtime for running test
> > suite - all suites have a property to specify runtime to be testes.
> > IMO the only thing to be improved here is to unify a property name for
> > all suites.
> >
> > Or do you mean that it should be possible to run a test suite by turns
> > on different VMs?
> >
> > Thanks,
> >
> > Stepan.
> >
> >
> > >
> > > Volunteer :)
> > >
> > > > I'm going to update the test infra home page [1] with info related to
> > > > the renewed trunk. Also I think the page should contain some words
> > > > about project's testing strategy: i.e. how Harmony QA and release
> > > > process are organized - integrity/snapshot testing, milestones, code
> > > > freeze and etc., what suites are there (their description and
> > > > purpose). I think this may encourage people to invest efforts into
> > > > developing tests.
> > > >
> > > > Other ideas and thoughts?
> > > >
> > > > [1] http://harmony.apache.org/subcomponents/buildtest/index.html
> > > >
> > > > Thanks,
> > > > Stepan.
> > > >
> > >
> > >
> > > --
> > > Sean, Xiao Xia Qiu
> > > China Software Development Lab, IBM
> > >
> >
>
>
> --
> Sean, Xiao Xia Qiu
> China Software Development Lab, IBM
>
Re: [build-and-test] Thoughts about testing infra
Posted by Sean Qiu <se...@gmail.com>.
I know we can all suites have a property to specify runtime to be testes. :)
But if we want to test another runtime, we need to specify them one by one.
I mean is there a way to specify it globally?
�� 08-3-6��Stepan Mishura<st...@gmail.com> ���
> On 3/5/08, Sean Qiu <se...@gmail.com> wrote:
> <SNIP>>
>
> > > Harmony test suites can be used as stand alone suites to verify/test
> > > different implementations �C in other words, the suites are not tightly
> > > coupled with DRL VM, for example, it is possible to run them against
> > > IBM VME (BTW, may it be makes sense to run full set of suites against
> > > IBM VME for each milestone candidate as for DRL VM. Any volunteers?)
> > >
> >
> > It sounds reasonable, in fact, i am going to plan the same thing.
> > Maybe we can make our BTI a more flexible infrastrcture for more runtime.
> > We can pick up a runtime before starting the BTI as well as the test
> > suite, such as:
> >
> > drlvm+harmony -| |- classlib unit test
> > j9vm+harmony -| -- BTI ----|- vm unit test
> > ... -| |- ...
> >
>
>
> Sean,
>
> With the current BTI it is possible to select runtime for running test
> suite - all suites have a property to specify runtime to be testes.
> IMO the only thing to be improved here is to unify a property name for
> all suites.
>
> Or do you mean that it should be possible to run a test suite by turns
> on different VMs?
>
> Thanks,
>
> Stepan.
>
>
> >
> > Volunteer :)
> >
> > > I'm going to update the test infra home page [1] with info related to
> > > the renewed trunk. Also I think the page should contain some words
> > > about project's testing strategy: i.e. how Harmony QA and release
> > > process are organized - integrity/snapshot testing, milestones, code
> > > freeze and etc., what suites are there (their description and
> > > purpose). I think this may encourage people to invest efforts into
> > > developing tests.
> > >
> > > Other ideas and thoughts?
> > >
> > > [1] http://harmony.apache.org/subcomponents/buildtest/index.html
> > >
> > > Thanks,
> > > Stepan.
> > >
> >
> >
> > --
> > Sean, Xiao Xia Qiu
> > China Software Development Lab, IBM
> >
>
--
Sean, Xiao Xia Qiu
China Software Development Lab, IBM
Re: [build-and-test] Thoughts about testing infra
Posted by Stepan Mishura <st...@gmail.com>.
On 3/5/08, Sean Qiu <se...@gmail.com> wrote:
<SNIP>>
> > Harmony test suites can be used as stand alone suites to verify/test
> > different implementations �C in other words, the suites are not tightly
> > coupled with DRL VM, for example, it is possible to run them against
> > IBM VME (BTW, may it be makes sense to run full set of suites against
> > IBM VME for each milestone candidate as for DRL VM. Any volunteers?)
> >
>
> It sounds reasonable, in fact, i am going to plan the same thing.
> Maybe we can make our BTI a more flexible infrastrcture for more runtime.
> We can pick up a runtime before starting the BTI as well as the test
> suite, such as:
>
> drlvm+harmony -| |- classlib unit test
> j9vm+harmony -| -- BTI ----|- vm unit test
> ... -| |- ...
>
Sean,
With the current BTI it is possible to select runtime for running test
suite - all suites have a property to specify runtime to be testes.
IMO the only thing to be improved here is to unify a property name for
all suites.
Or do you mean that it should be possible to run a test suite by turns
on different VMs?
Thanks,
Stepan.
>
> Volunteer :)
>
> > I'm going to update the test infra home page [1] with info related to
> > the renewed trunk. Also I think the page should contain some words
> > about project's testing strategy: i.e. how Harmony QA and release
> > process are organized - integrity/snapshot testing, milestones, code
> > freeze and etc., what suites are there (their description and
> > purpose). I think this may encourage people to invest efforts into
> > developing tests.
> >
> > Other ideas and thoughts?
> >
> > [1] http://harmony.apache.org/subcomponents/buildtest/index.html
> >
> > Thanks,
> > Stepan.
> >
>
>
> --
> Sean, Xiao Xia Qiu
> China Software Development Lab, IBM
>
Re: [build-and-test] Thoughts about testing infra
Posted by Sean Qiu <se...@gmail.com>.
2008/3/4, Stepan Mishura <st...@gmail.com>:
> Hi folks,
>
> I've moved 2.0 build-and-test's branch to the trunk - the branch was
> used during M2-M5 testing and IMO it proved to be very useful (also I
> tagged the former trunk as 1.0.)
>
Great news. Thanks.
> I think that this is a good time to collect ideas and hear opinions
> what should be improved and possible directions for test infra
> development.
>
Shall we supply a friendly front-end for user?
List all the suites available and make the configuration more easy.
> I have to admit that the current testing infra is far from being
> perfect �C there a lot for work to make is more usable, cleaning up
> current scripts, sort out all know issues with tests/scenarios and so
> on. But I'd like to address problems I believe of different level. I
> have an impression that Harmony tests suites are still in shadow (IOW,
> IMHO they are undervalued) of testing automation (i.e. CruiseControl
> and support scripts) �C and most of folks prefer to contribute to the
> code development. And I believe that it is possible to make tests
> development also interesting too. IMO the missed point that most of
Agree :)
> Harmony test suites can be used as stand alone suites to verify/test
> different implementations �C in other words, the suites are not tightly
> coupled with DRL VM, for example, it is possible to run them against
> IBM VME (BTW, may it be makes sense to run full set of suites against
> IBM VME for each milestone candidate as for DRL VM. Any volunteers?)
>
It sounds reasonable, in fact, i am going to plan the same thing.
Maybe we can make our BTI a more flexible infrastrcture for more runtime.
We can pick up a runtime before starting the BTI as well as the test
suite, such as:
drlvm+harmony -| |- classlib unit test
j9vm+harmony -| -- BTI ----|- vm unit test
... -| |- ...
Volunteer :)
> I'm going to update the test infra home page [1] with info related to
> the renewed trunk. Also I think the page should contain some words
> about project's testing strategy: i.e. how Harmony QA and release
> process are organized - integrity/snapshot testing, milestones, code
> freeze and etc., what suites are there (their description and
> purpose). I think this may encourage people to invest efforts into
> developing tests.
>
> Other ideas and thoughts?
>
> [1] http://harmony.apache.org/subcomponents/buildtest/index.html
>
> Thanks,
> Stepan.
>
--
Sean, Xiao Xia Qiu
China Software Development Lab, IBM