You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Alexander Kleymenov <kl...@gmail.com> on 2007/03/27 08:43:57 UTC
[buildtest] Proposal for Build Test Infrastructure Improvement
Hello,
With this letter I'd like to announce the new approach for BT Framework
Implementation. It could be found as an attachment to JIRA #3501 [1].
Proposed BT Infra Implementation improves and structures current one and brings
the following features allowing using the framework as a standard means for
Harmony Testing works:
- Simplicity and convenience of use allows to perform all of the testing
works (by testers and developers) by unified single way and with using
of the same means.
- Clear and Documented Architecture, Workspace Structure, Clear separation
of Functional Components of the Framework, Standard Implementation Means
allows to easy start, use, improve, and maintain the framework.
- Highly configurable Test Run:
Simple means for Test Suite selections - use property value to select
required Test Suites. Only selected suites will be checked out (or
downloaded) and executed.
Simple and easy manageable means for settings dependencies between Test
Suites. It excludes the necessity of combo-test-suites creation (such as
classlib-drlvm now)
Simple and natural means for describing of required parameters for selected
Test Suites (as path to tested JVM, Classlib, compiler checks, and so on).
It is done in one specified place and there is no need to drill into Test
Suites for setting up such a parameters. Moreover, new BT Framework performs
the correctness analysis of the provided values for parameters and reports
an error just before Test Suite execution, not after an hour of the work.
Additional important feature is an ability to provide default values for
required parameters on the base of execution results of another Test Suite.
So, for example, drlvm Test Suite (which checks that DRLVM can be built)
uses the default value for 'classlib.trunk' parameter (specifying
the path to
classlib workspace) the value provided by classlib Test Suite. If classlib
Test Suite was chosen for execution, 'classlib.trunk' parameter of drlvm
test suite will be resolved. If classlib was not chosen, user will be
asked to provide the value for this parameter. Such a feature allows to
reduce the user's job on parameters setting.
Such a simplicity of Test Run configuration (selection of the suites and
parameters tuning) allows to easy set up the BT Framework for specific
needs. So for Harmony developers it is possible to chose their own sets of
checks of the workspace under development and to run this checks on demand
in single execution mode. For Harmony testers it is possible to use
sophisticated CruiseControl configurations for continuous test runs with
notification publishing.
- Easy and Documented way to integrate new Test Suites:
There is no longer need to implement CruiseControl configurations for
each of new Test Suites being integrated into framework. Cruise Control
system is used as a framework extension allowing continuous test runs, not
as a development means. It simplifies the process and reduces the time
of new Test Suite integration.
- Implemented Approach to share External Library Dependencies
between different
Test Suites allows to perform external dependencies fetching in one place by
one means and to save time and internet traffic.
- To run custom configuration there is no need to download
all of the Test Suites integrated into BT.
- The Architecture of the Framework is open for extending by other functional
parts and 3rd party tools (such as CruiseControl etc)
The main idea of the new Framework is to use Ant's scripts called 'adaptors' as
a glue layer between Test Suites and the BT Framework executing them. All of the
Parameters of the suites (such as required values, external libraries, shared
values to be used by other suites, optional parameters for low-level tuning)
are described in 'parameters.xml' file placed near corresponding 'adaptor.xml'.
That's all.
Please, look at the archive. There is a README.txt and SPEC.txt files describing
and documenting the new Infrastructure. Also there are two integrated Test
Suites - classlib and drlvm which check that classlib and drlvm can be built.
Please look at them under adapters directory to catch the main idea of the
Framework.
Of course there are many points for improvements, but the main idea and
functionality has already been implemented. Sorry for such a bulk, but I could
not provide the framework by 'logical parts' because they together forms a
whole. Now, when all of them in one place it is possible to improve or
reimplement them (I heard about the existing approach for external libraries
sharing, but did not look at it. Probably it is better then proposed in my BT
Infra implementation. In this case it is possible to replace the implementation
of this functional part of the Framework).
If you have any questions regarding the implementation please,
provide them into this mail-thread.
Thank you,
Alexander
[1] http://issues.apache.org/jira/browse/HARMONY-3501
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Alexander Kleymenov <kl...@gmail.com>.
Hello Stepan,
> Do I understand correctly that the framework works without seting up
> classpath (by adding ant-contrib.jar) or requiring to place the jar to
> $ANT_HOME/lib?
Yes, it works in such a way and I did't meet any problem with this.
> I'm asking this because one of reasons to keep build.bat/build.sh
> files in DRL VM build is that they add ant-contrib.jar to classpath
> otherwise the build doesn't work.[1]
>
> [1]http://mail-archives.apache.org/mod_mbox/harmony-dev/200607.mbox/%3c6928c5160607140431h78e67820x868a87126e6cdcf8@mail.gmail.com%3e
Stepan, is it possible to reproduce this problem in proposed BT framework?
I think such a problem in DRLVM is somehow connected with bugs in ol
ant-contrib. I use 1.0b3 version.
What's regarding to DRLVM scripts. They don't fix the problem. Try to run:
> build.bat update build
on just checked out repository and you will reproduce it. The problem
is with cpptasks' xml declaration file. It does not allow to ant to
use it at runtime. There are some general fixes proposed over the net.
Thank you,
Alexander
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Stepan Mishura <st...@gmail.com>.
Hi Alexander,
I started reviewing your patch from HARMONY-3501and see that proposed
framework depends on ant-contrib. The framework downloads the archive
and dynamically adds and uses ant-contrib's tasks in (as far as I
understood) a standart way:
<taskdef resource="net/sf/antcontrib/antlib.xml">
<classpath>
<pathelement location="path_to_ant-contrib.jar" />
</classpath>
</taskdef>
Do I understand correctly that the framework works without seting up
classpath (by adding ant-contrib.jar) or requiring to place the jar to
$ANT_HOME/lib?
I'm asking this because one of reasons to keep build.bat/build.sh
files in DRL VM build is that they add ant-contrib.jar to classpath
otherwise the build doesn't work.[1]
[1]http://mail-archives.apache.org/mod_mbox/harmony-dev/200607.mbox/%3c6928c5160607140431h78e67820x868a87126e6cdcf8@mail.gmail.com%3e
Thanks,
Stepan.
On 3/27/07, Alexander Kleymenov <kl...@gmail.com> wrote:
> Hello,
>
> With this letter I'd like to announce the new approach for BT Framework
> Implementation. It could be found as an attachment to JIRA #3501 [1].
>
> Proposed BT Infra Implementation improves and structures current one and brings
> the following features allowing using the framework as a standard means for
> Harmony Testing works:
>
> - Simplicity and convenience of use allows to perform all of the testing
> works (by testers and developers) by unified single way and with using
> of the same means.
>
> - Clear and Documented Architecture, Workspace Structure, Clear separation
> of Functional Components of the Framework, Standard Implementation Means
> allows to easy start, use, improve, and maintain the framework.
>
> - Highly configurable Test Run:
>
> Simple means for Test Suite selections - use property value to select
> required Test Suites. Only selected suites will be checked out (or
> downloaded) and executed.
>
> Simple and easy manageable means for settings dependencies between Test
> Suites. It excludes the necessity of combo-test-suites creation (such as
> classlib-drlvm now)
>
> Simple and natural means for describing of required parameters for selected
> Test Suites (as path to tested JVM, Classlib, compiler checks, and so on).
> It is done in one specified place and there is no need to drill into Test
> Suites for setting up such a parameters. Moreover, new BT Framework performs
> the correctness analysis of the provided values for parameters and reports
> an error just before Test Suite execution, not after an hour of the work.
> Additional important feature is an ability to provide default values for
> required parameters on the base of execution results of another Test Suite.
> So, for example, drlvm Test Suite (which checks that DRLVM can be built)
> uses the default value for 'classlib.trunk' parameter (specifying
> the path to
> classlib workspace) the value provided by classlib Test Suite. If classlib
> Test Suite was chosen for execution, 'classlib.trunk' parameter of drlvm
> test suite will be resolved. If classlib was not chosen, user will be
> asked to provide the value for this parameter. Such a feature allows to
> reduce the user's job on parameters setting.
>
> Such a simplicity of Test Run configuration (selection of the suites and
> parameters tuning) allows to easy set up the BT Framework for specific
> needs. So for Harmony developers it is possible to chose their own sets of
> checks of the workspace under development and to run this checks on demand
> in single execution mode. For Harmony testers it is possible to use
> sophisticated CruiseControl configurations for continuous test runs with
> notification publishing.
>
> - Easy and Documented way to integrate new Test Suites:
>
> There is no longer need to implement CruiseControl configurations for
> each of new Test Suites being integrated into framework. Cruise Control
> system is used as a framework extension allowing continuous test runs, not
> as a development means. It simplifies the process and reduces the time
> of new Test Suite integration.
>
> - Implemented Approach to share External Library Dependencies
> between different
> Test Suites allows to perform external dependencies fetching in one place by
> one means and to save time and internet traffic.
>
> - To run custom configuration there is no need to download
> all of the Test Suites integrated into BT.
>
> - The Architecture of the Framework is open for extending by other functional
> parts and 3rd party tools (such as CruiseControl etc)
>
> The main idea of the new Framework is to use Ant's scripts called 'adaptors' as
> a glue layer between Test Suites and the BT Framework executing them. All of the
> Parameters of the suites (such as required values, external libraries, shared
> values to be used by other suites, optional parameters for low-level tuning)
> are described in 'parameters.xml' file placed near corresponding 'adaptor.xml'.
> That's all.
>
> Please, look at the archive. There is a README.txt and SPEC.txt files describing
> and documenting the new Infrastructure. Also there are two integrated Test
> Suites - classlib and drlvm which check that classlib and drlvm can be built.
> Please look at them under adapters directory to catch the main idea of the
> Framework.
>
> Of course there are many points for improvements, but the main idea and
> functionality has already been implemented. Sorry for such a bulk, but I could
> not provide the framework by 'logical parts' because they together forms a
> whole. Now, when all of them in one place it is possible to improve or
> reimplement them (I heard about the existing approach for external libraries
> sharing, but did not look at it. Probably it is better then proposed in my BT
> Infra implementation. In this case it is possible to replace the implementation
> of this functional part of the Framework).
>
> If you have any questions regarding the implementation please,
> provide them into this mail-thread.
>
> Thank you,
> Alexander
>
> [1] http://issues.apache.org/jira/browse/HARMONY-3501
>
--
Stepan Mishura
Intel Enterprise Solutions Software Division
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Stepan Mishura <st...@gmail.com>.
On 4/17/07, Stepan Mishura wrote:
> Hi Alexander,
>
> I've reviewed almost all the code. And I have some comments and questions:
>
> 1) Is it possible to add modules 'drlvm-test' and 'classlib-drlvm'
> that we have in the current infra? I'll help the difference between
> approaches.
>
> For example, the following difference for classlib confuses me a
> little bit: the current infra launches classlib by simply calling
> classlib's build with corresponding target:
> <ant target="fetch-depends rebuild" ....
> </ant>
>
> But the proposed infra does the next: calls adaptor.xml with 'run'
> target (I'm OK with it). But the classlib's adaptor not just simply
> calls classlib's build. It detects OS, set corresponding properties,
> calls classlib's target 'fetch-depends' with <ant> and then calls
> classlib's target 'rebuild' via <exec> task. So we have, for example
> for Windows:
> <ant target="fetch-depends" dir="${trunk.dir}"
> inheritAll="false">
> </ant>
> <exec executable="cmd" dir="${trunk.dir}" failonerror="true">
> <arg line="/c ${ant.home}/bin/ant.bat"/>
> <arg line="rebuild"/>
> </exec>
>
> What the reason for calling classlib's build in this way? Why we have
> to run 'ant.bat' (or 'ant.sh' for Linux) via <exec>?
>
> 2) The build files have cyclic import dependencies:
> framework.xml => cc-project.xml => framework.xml
>
> It complicates the build logic. Is it possible to avoid this?
>
> 3) In exec-adaptor.xml antconrib's wrapper 'trycatch' and 'runtarget'
> task are used for calling adaptor's(i.e. suite's) target.
>
> <trycatch property="execution.exception">
> <try>
> <runtarget target="@{target}"/>
> </try>
> <catch>
> Echo error message
> </catch>
> </trycatch>
>
> Again, why it is not possible use <ant> task? And this leads to the
> next question.
>
> 4) Using antcontrib. The proposed implementation uses it in many
> places. But from my POV it is possible to use ANT tasks/targets/macros
> instead. For example, I've picked up the first case from
> 'framework.xml' that patches ANT:
> <target name="patch-ant" depends="make-dirs, define-tasks">
> <if>
> <not><available file="${temp.files.dir}/SubAnt.java"/></not>
> <then>
> <get src="http://issues.apache.org/bugzilla/attachment.cgi?id=15681"
> dest="${temp.files.dir}/SubAnt.java"
> usetimestamp="true"
> verbose="true"
> />
> </then>
> </if>
> ...
> </ target >
>
> That IMHO can be easy converted to 'pure' ANT by creating target that
> only runs if there is no SubAnt.java file and adding the target to
> "patch-ant" decencies.
>
s/decencies/dependencies/ :-)
-Stepan.
> Actually, I don't want to start flame war - which approach is better.
> I'm just trying to understand is it possible to avoid using antcontibs
> task? IOW, is there any issue that can not be solved by using ANT
> only?
>
> 5) Patching ANT - is there any workaround?
>
> 6) And minor question: is there any strong reason for naming build
> scripts 'adaptor.xml' not 'build.xml'?
>
> Thanks,
> Stepan.
>
> On 3/27/07, Alexander Kleymenov <kl...@gmail.com> wrote:
> > Hello,
> >
> > With this letter I'd like to announce the new approach for BT Framework
> > Implementation. It could be found as an attachment to JIRA #3501 [1].
> >
> > Proposed BT Infra Implementation improves and structures current one and brings
> > the following features allowing using the framework as a standard means for
> > Harmony Testing works:
> >
> > - Simplicity and convenience of use allows to perform all of the testing
> > works (by testers and developers) by unified single way and with using
> > of the same means.
> >
> > - Clear and Documented Architecture, Workspace Structure, Clear separation
> > of Functional Components of the Framework, Standard Implementation Means
> > allows to easy start, use, improve, and maintain the framework.
> >
> > - Highly configurable Test Run:
> >
> > Simple means for Test Suite selections - use property value to select
> > required Test Suites. Only selected suites will be checked out (or
> > downloaded) and executed.
> >
> > Simple and easy manageable means for settings dependencies between Test
> > Suites. It excludes the necessity of combo-test-suites creation (such as
> > classlib-drlvm now)
> >
> > Simple and natural means for describing of required parameters for selected
> > Test Suites (as path to tested JVM, Classlib, compiler checks, and so on).
> > It is done in one specified place and there is no need to drill into Test
> > Suites for setting up such a parameters. Moreover, new BT Framework performs
> > the correctness analysis of the provided values for parameters and reports
> > an error just before Test Suite execution, not after an hour of the work.
> > Additional important feature is an ability to provide default values for
> > required parameters on the base of execution results of another Test Suite.
> > So, for example, drlvm Test Suite (which checks that DRLVM can be built)
> > uses the default value for 'classlib.trunk' parameter (specifying
> > the path to
> > classlib workspace) the value provided by classlib Test Suite. If classlib
> > Test Suite was chosen for execution, 'classlib.trunk' parameter of drlvm
> > test suite will be resolved. If classlib was not chosen, user will be
> > asked to provide the value for this parameter. Such a feature allows to
> > reduce the user's job on parameters setting.
> >
> > Such a simplicity of Test Run configuration (selection of the suites and
> > parameters tuning) allows to easy set up the BT Framework for specific
> > needs. So for Harmony developers it is possible to chose their own sets of
> > checks of the workspace under development and to run this checks on demand
> > in single execution mode. For Harmony testers it is possible to use
> > sophisticated CruiseControl configurations for continuous test runs with
> > notification publishing.
> >
> > - Easy and Documented way to integrate new Test Suites:
> >
> > There is no longer need to implement CruiseControl configurations for
> > each of new Test Suites being integrated into framework. Cruise Control
> > system is used as a framework extension allowing continuous test runs, not
> > as a development means. It simplifies the process and reduces the time
> > of new Test Suite integration.
> >
> > - Implemented Approach to share External Library Dependencies
> > between different
> > Test Suites allows to perform external dependencies fetching in one place by
> > one means and to save time and internet traffic.
> >
> > - To run custom configuration there is no need to download
> > all of the Test Suites integrated into BT.
> >
> > - The Architecture of the Framework is open for extending by other functional
> > parts and 3rd party tools (such as CruiseControl etc)
> >
> > The main idea of the new Framework is to use Ant's scripts called 'adaptors' as
> > a glue layer between Test Suites and the BT Framework executing them. All of the
> > Parameters of the suites (such as required values, external libraries, shared
> > values to be used by other suites, optional parameters for low-level tuning)
> > are described in 'parameters.xml' file placed near corresponding 'adaptor.xml'.
> > That's all.
> >
> > Please, look at the archive. There is a README.txt and SPEC.txt files describing
> > and documenting the new Infrastructure. Also there are two integrated Test
> > Suites - classlib and drlvm which check that classlib and drlvm can be built.
> > Please look at them under adapters directory to catch the main idea of the
> > Framework.
> >
> > Of course there are many points for improvements, but the main idea and
> > functionality has already been implemented. Sorry for such a bulk, but I could
> > not provide the framework by 'logical parts' because they together forms a
> > whole. Now, when all of them in one place it is possible to improve or
> > reimplement them (I heard about the existing approach for external libraries
> > sharing, but did not look at it. Probably it is better then proposed in my BT
> > Infra implementation. In this case it is possible to replace the implementation
> > of this functional part of the Framework).
> >
> > If you have any questions regarding the implementation please,
> > provide them into this mail-thread.
> >
> > Thank you,
> > Alexander
> >
> > [1] http://issues.apache.org/jira/browse/HARMONY-3501
> >
>
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Stepan Mishura <st...@gmail.com>.
On 4/23/07, Alexander Kleymenov <kl...@gmail.com> wrote:
> Stepan,
>
> Reported issues are mostly targeted to integrated Test Suites, not to BTI.
> To notify the developpers and to get these issues fixed, please report
> them to the corresponding JIRA issues containing the Test Suites.
>
Alexander,
I'm a bit confused. I believe that at least last issue I reported
relates to the infra implementation. It fails on
'make-parameter-files' infra target complaining that it can not create
'${classlib.parameters.shared.trunk.dir}/HUT_iterative.trigger' file.
My understanding that all such properties resolve by the infra. Am I
correct?
Also Dacapo scenario uses property defined in the next way:
jdk.to.test.home=${drlvm.parameters.shared.jre.dir}. But that is
defined in drlvm suite. And Dacapo scenario doesn't declare that it
depends on drlvm. I expected to see error message here. But the infra
successfully resolved it. I thought that if a suite uses properties
defined in another suite then the suite must have corresponding entry
in its dependencies. Am I correct?
Stepan Mishura
Intel Enterprise Solutions Software Division
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Alexander Kleymenov <kl...@gmail.com>.
Stepan,
Reported issues are mostly targeted to integrated Test Suites, not to BTI.
To notify the developpers and to get these issues fixed, please report
them to the corresponding JIRA issues containing the Test Suites.
Alexander
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Stepan Mishura <st...@gmail.com>.
Hi Alexander,
FYI, I succesfully set up suite from [1] and run it in 'single run
mode'. But it failed for me under CC with:
create-cc-configuration:
[delete] Deleting: /export/users/smishura/newinfra/build/cc/cc-config.xml
[copy] Copying 1 file to /export/users/smishura/newinfra/build/cc
[mkdir] Created dir:
/export/users/smishura/newinfra/build/cc/logs/HUT_iterative
[copy] Copying 1 file to /export/users/smishura/newinfra/build/cc
make-parameter-files:
[touch] Creating
/export/users/smishura/newinfra/${classlib.parameters.shared.trunk.dir}/HUT_iterative.trigger
BUILD FAILED
/export/users/smishura/newinfra/scripts/cc-project.xml:83: The
following error occurred while executing this line:
/export/users/smishura/newinfra/scripts/cc-project.xml:136: The
following error occurred while executing this line:
/export/users/smishura/newinfra/scripts/cc-project.xml:307: Could not
create /export/users/smishura/newinfra/${classlib.parameters.shared.trunk.dir}/HUT_iterative.trigger
Total time: 2 seconds
[1] http://issues.apache.org/jira/browse/HARMONY-3629
Thanks,
Stepan.
On 4/20/07, Stepan Mishura wrote:
> On 4/20/07, Stepan Mishura wrote:
> > On 4/18/07, Alexander Kleymenov wrote:
> > > I did not look at the 'Jetty Scenario' integration scripts yet.
> > > So I see they have an issue.
> > > Adaptor's run target should <fail if="test suite failed">.
> > > Alexey, please, fix your adaptor.
> > > Stepan, please, use another Test Suite.
> > > I can suggest to use 'Struts Scenario' published at HARMONY-3597 JIRA issue.
> >
> > I've tried to setup this suite and the result is the same - the
> > suite's setup failed but overall setup passed. Here is my output:
> > ...
> > -generate-test-data:
> > [junit] Running org.apache.harmony.struts.test.WebAppTest
> > [junit] ----URL 10000.URL./struts2-showcase-2.0.6/showcase.jsp:
> > testing with file520468537.html
> > [junit] +---Page Apache Tomcat/5.0.30 - Error report: testing
> > 10010.ANCHOR.3 with file1722219095.html
> > [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.524 sec
> > [junit] Test org.apache.harmony.struts.test.WebAppTest FAILED
> >
> > -stop-server-for-setup:
> >
> > setup:
> > [delete] Deleting: /export/users/smishura/newinfra/build/temp/null1434408292
> >
> > check-status:
> >
> > setup:
> >
> > BUILD SUCCESSFUL
> > Total time: 18 seconds
> >
>
> Alexander,
>
> I've tried another suite - Dacapo benchmark from [1] to test the new
> infra. According to its parameters.xml it doesn't depends on drlvm but
> requires property "${drlvm.parameters.shared.jre.dir}" to be set. But
> I don't see any errors messages about it during running setup. Is this
> a mistake?
>
> Also the framework failed to set up it with the following error:
> setup:
> [mkdir] Created dir: /export/users/smishura/newinfra/build/results/dacapo
> [mkdir] Created dir: /export/users/smishura/newinfra/build/checkouts/dacapo
>
> -dacapo.download:
> [echo]
> [echo] --------
> [echo] WARNING: Target setup of adaptor for Test Suite 'dacapo' FAILED
> [echo] -------- with the following exception:
> [echo] The following error occurred while executing this line:
> [echo] /export/users/smishura/newinfra/adaptors/dacapo/adaptor.xml:35:
> The specified destination is a directory
> [delete] Deleting: /export/users/smishura/newinfra/build/temp/null842216362
>
> check-status:
> [echo]
> [echo] ------
> [echo] ERROR: Setup of Test Suite 'dacapo' FAILED.
> [echo] ------
>
> BUILD FAILED
> /export/users/smishura/newinfra/scripts/framework.xml:443: The
> following error occurred while executing this line:
> /export/users/smishura/newinfra/scripts/framework.xml:761: There are
> Test Suites failures
>
> Total time: 1 second
>
> From the first glance it looks like there is a mistake in Dacapo
> adaptor - it failed to download some resource file. So the question is
> why the suite doesn't use the framework functionality to download the
> required file?
>
> [1] https://issues.apache.org/jira/browse/HARMONY-3681
>
> Thanks,
> Stepan Mishura
> Intel Enterprise Solutions Software Division
>
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Stepan Mishura <st...@gmail.com>.
On 4/20/07, Stepan Mishura wrote:
> On 4/18/07, Alexander Kleymenov wrote:
> > I did not look at the 'Jetty Scenario' integration scripts yet.
> > So I see they have an issue.
> > Adaptor's run target should <fail if="test suite failed">.
> > Alexey, please, fix your adaptor.
> > Stepan, please, use another Test Suite.
> > I can suggest to use 'Struts Scenario' published at HARMONY-3597 JIRA issue.
>
> I've tried to setup this suite and the result is the same - the
> suite's setup failed but overall setup passed. Here is my output:
> ...
> -generate-test-data:
> [junit] Running org.apache.harmony.struts.test.WebAppTest
> [junit] ----URL 10000.URL./struts2-showcase-2.0.6/showcase.jsp:
> testing with file520468537.html
> [junit] +---Page Apache Tomcat/5.0.30 - Error report: testing
> 10010.ANCHOR.3 with file1722219095.html
> [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.524 sec
> [junit] Test org.apache.harmony.struts.test.WebAppTest FAILED
>
> -stop-server-for-setup:
>
> setup:
> [delete] Deleting: /export/users/smishura/newinfra/build/temp/null1434408292
>
> check-status:
>
> setup:
>
> BUILD SUCCESSFUL
> Total time: 18 seconds
>
Alexander,
I've tried another suite - Dacapo benchmark from [1] to test the new
infra. According to its parameters.xml it doesn't depends on drlvm but
requires property "${drlvm.parameters.shared.jre.dir}" to be set. But
I don't see any errors messages about it during running setup. Is this
a mistake?
Also the framework failed to set up it with the following error:
setup:
[mkdir] Created dir: /export/users/smishura/newinfra/build/results/dacapo
[mkdir] Created dir: /export/users/smishura/newinfra/build/checkouts/dacapo
-dacapo.download:
[echo]
[echo] --------
[echo] WARNING: Target setup of adaptor for Test Suite 'dacapo' FAILED
[echo] -------- with the following exception:
[echo] The following error occurred while executing this line:
[echo] /export/users/smishura/newinfra/adaptors/dacapo/adaptor.xml:35:
The specified destination is a directory
[delete] Deleting: /export/users/smishura/newinfra/build/temp/null842216362
check-status:
[echo]
[echo] ------
[echo] ERROR: Setup of Test Suite 'dacapo' FAILED.
[echo] ------
BUILD FAILED
/export/users/smishura/newinfra/scripts/framework.xml:443: The
following error occurred while executing this line:
/export/users/smishura/newinfra/scripts/framework.xml:761: There are
Test Suites failures
Total time: 1 second
>From the first glance it looks like there is a mistake in Dacapo
adaptor - it failed to download some resource file. So the question is
why the suite doesn't use the framework functionality to download the
required file?
[1] https://issues.apache.org/jira/browse/HARMONY-3681
Thanks,
Stepan Mishura
Intel Enterprise Solutions Software Division
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Stepan Mishura <st...@gmail.com>.
On 4/18/07, Alexander Kleymenov <kl...@gmail.com> wrote:
> I did not look at the 'Jetty Scenario' integration scripts yet.
> So I see they have an issue.
> Adaptor's run target should <fail if="test suite failed">.
> Alexey, please, fix your adaptor.
> Stepan, please, use another Test Suite.
> I can suggest to use 'Struts Scenario' published at HARMONY-3597 JIRA issue.
I've tried to setup this suite and the result is the same - the
suite's setup failed but overall setup passed. Here is my output:
...
-generate-test-data:
[junit] Running org.apache.harmony.struts.test.WebAppTest
[junit] ----URL 10000.URL./struts2-showcase-2.0.6/showcase.jsp:
testing with file520468537.html
[junit] +---Page Apache Tomcat/5.0.30 - Error report: testing
10010.ANCHOR.3 with file1722219095.html
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.524 sec
[junit] Test org.apache.harmony.struts.test.WebAppTest FAILED
-stop-server-for-setup:
setup:
[delete] Deleting: /export/users/smishura/newinfra/build/temp/null1434408292
check-status:
setup:
BUILD SUCCESSFUL
Total time: 18 seconds
Thanks,
Stepan Mishura
Intel Enterprise Solutions Software Division
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Alexander Kleymenov <kl...@gmail.com>.
I did not look at the 'Jetty Scenario' integration scripts yet.
So I see they have an issue.
Adaptor's run target should <fail if="test suite failed">.
Alexey, please, fix your adaptor.
Stepan, please, use another Test Suite.
I can suggest to use 'Struts Scenario' published at HARMONY-3597 JIRA issue.
Thanks,
Alexander
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Stepan Mishura <st...@gmail.com>.
On 4/18/07, Ivaschenko, Aleksei V wrote:
> >-----Original Message-----
> >From: Stepan Mishura [mailto:stepan.mishura@gmail.com]
> >Sent: Wednesday, April 18, 2007 4:11 PM
> >
> >Hi Alexander,
> >
> >Next question:
> >7) Suite's dependencies.
> >
> >I've tried the framework with JettyScenario. So I've unpacked a bundle
> >with Jetty scenario into the framework dir, copied dir with tests,
> >commented out some lines in dataptor (so everything according to the
> >README.txt file) but I accidentally typed
> >$ ant -Dtest.suites="JettyScenario" setup
> >instead of
> >$ ant -Dtest.suites="classlib,drlvm,JettyScenario" setup
> >
> >I expected that the framework would set up
> >classlib,drlvm,JettyScenario because JettyScenario defines in
> >parameters.xml dependencies on classlib,drlvm. But it set up only
> >JettyScenario. I've looked into the code and my impression that
> >dependencies defined in parameters.xml affect only in which order the
> >framework runs suites. Am I correct?
> >
> >Also my understanding that we have the following dependencies between
> >suites:
> >1) classlib - no dependencies
> >2) drlvm - depends on classlib only
> >3) all other suites - depends on classlib and drlvm
> >
> >Do we have suites with other set of dependencies?
> >
> >Please see other comments below.
> >
> >BWT, FYI I run JettyScenario and found that it failed but overall run
> >passed. I see the following output:
> >run:
> > [mkdir] Created dir:
> >/export/users/smishura/newinfra/build/checkouts/JettyScenario/results
> > [junit] Running JettyScenario
> > [java] Java Result: 139
> > [junit] Tests run: 5, Failures: 1, Errors: 3, Time elapsed: 94.581
> sec
> > [junit] Test JettyScenario FAILED
> > [java] java.net.ConnectException: localhost/127.0.0.1:4444 -
> >Connection refused
> > [java] at
> >org.apache.harmony.luni.net.PlainSocketImpl.connect(PlainSocketImpl.jav
> a:23
> >0)
> > [java] at
> >org.apache.harmony.luni.net.PlainSocketImpl.connect(PlainSocketImpl.jav
> a:19
> >3)
> > [java] at java.net.Socket.startupSocket(Socket.java:662)
> > [java] at java.net.Socket.<init>(Socket.java:214)
> > [java] at org.mortbay.start.Main.stop(Main.java:526)
> > [java] at org.mortbay.start.Main.main(Main.java:104)
> > [java] at
> >java.lang.reflect.VMReflection.invokeMethod(VMReflection.java)
> > [java] at java.lang.reflect.Method.invoke(Method.java:381)
> > [java] at org.apache.harmony.vm.JarRunner.main(JarRunner.java:80)
> > [delete] Deleting:
> >/export/users/smishura/newinfra/build/temp/null1281662371
> >
> >check-status:
> >
> >run:
> >
> >BUILD SUCCESSFUL
> >Total time: 2 minutes 10 seconds
>
> This is the way Jetty scenario implemented - 'run' target of adaptor.xml
> does not ever fail. It reports scenario 'failed' or 'passed' in its log
> (and in mail notification if in CC mode), and does not lead framework to
> 'BUILD FAILED'. This is because Jetty server should be stopped after
> scenario is completed whether it failed or not.
>
But this means that the output should be browsed each time to figure
out whether the testing scenario failed or not. This is not
convenient. My understanding from discussion with Alexander was that
'run' target of adaptor.xml may fail and this is the framework task to
collect all suites status and fail if one of suites failed.
Alexander, could you clarify this?
Thanks,
Stepan.
> >> > > > > > What the reason for calling classlib's build in this way?
> Why
> >we have
> >> > > > > > to run 'ant.bat' (or 'ant.sh' for Linux) via <exec>?
> >> > > > >
> >> > > > > It's workaround for Ant's OutOfMemory problem arising after
> doing
> >the same
> >> > > > > by <ant> target. The <ant>-rebuild of classlib project causes
> big
> >memory
> >> > > > > leaks somewhere in the Ant and the further BTI execution is
> >impossible
> >> > > > > becouse of arising OOM.
> >> > > > >
> >> > > >
> >> > > > As far as I remember the solution was to increase the max
> memory
> >used
> >> > > > by Ant with setting ANT_OPTS to "-Xms256M -Xmx512M". Have you
> tried
> >> > > > this?
> >> > >
> >> > > 2Gb was not enough.
> >> > >
> >> >
> >> > Wow ... and do you have any idea why 512M is enough for the current
> >> > infra and 2Gb is not enough for the proposed one?
> >>
> >> OOE happens under CC execution on the half way of drlvm building.
> >> We didn't it before in one Ant project.
> >>
> >
> >Sorry, may be my question was unclear. I asked the following: have you
> >investigated why the current infra is able to run the same testing
> >scenario (classlib+drlvm) with 512M and new one can not do it with
> >2Gb?
> >
> >> > As I understand
> >> > certain ANT tasks can use a lot of memory - classlibrary complies
> over
> >> > 3500 with javac task, so we may assume that this requires a lot of
> >> > memory. But root cause of such essential increase (more then x4
> times)
> >> > is somewhere in the proposed infa from my POV. Can this impose some
> >> > limits, for example, to a number of testing suite that the infra
> can
> >> > setup and run?
> >>
> >> Load testing of BTI could help to answer your question.
> >> And I didn't tried Ant 1.7. Probably it fixes some of the leaks.
> >>
> >
> >So you think that this is ANT issue only. Right?
> >
> ><SNIP>
> >
> >> Yes, it is Ant-based Framework. adaptor.xml takes the
> >> responsibility to 'adapt' existing Test Suite for execution under
> this
> >> framework. So it's called adaptor.xml
> >>
> >> > > > BTW, if we rename build.xml=>buildtest.xml we have to type each
> >time:
> >> > > > ant -f buildtest.xml.
> >> > >
> >> > > The standard usage of BTI supposes launching by means of shell
> >scripts
> >> > > (buildtest.bat/.sh), not by ant launcher.
> >> > >
> >> >
> >> > Just to setup classlib and lanch ant with -f buildtest.xml (like
> DRL VM
> >build)?
> >>
> >> No, just to say
> >> > buildtest -Dtest.suites=some,selected,suites run
> >> and get the things working.
> >
> >I meant that shell script is required to set up classpath and lanch
> >ant with -f buildtest.xml
> >
> >Thanks,
> >Stepan
>
> Thanks,
> Aleksei.
>
--
Stepan Mishura
Intel Enterprise Solutions Software Division
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by "Ivaschenko, Aleksei V" <al...@intel.com>.
>-----Original Message-----
>From: Stepan Mishura [mailto:stepan.mishura@gmail.com]
>Sent: Wednesday, April 18, 2007 4:11 PM
>
>Hi Alexander,
>
>Next question:
>7) Suite's dependencies.
>
>I've tried the framework with JettyScenario. So I've unpacked a bundle
>with Jetty scenario into the framework dir, copied dir with tests,
>commented out some lines in dataptor (so everything according to the
>README.txt file) but I accidentally typed
>$ ant -Dtest.suites="JettyScenario" setup
>instead of
>$ ant -Dtest.suites="classlib,drlvm,JettyScenario" setup
>
>I expected that the framework would set up
>classlib,drlvm,JettyScenario because JettyScenario defines in
>parameters.xml dependencies on classlib,drlvm. But it set up only
>JettyScenario. I've looked into the code and my impression that
>dependencies defined in parameters.xml affect only in which order the
>framework runs suites. Am I correct?
>
>Also my understanding that we have the following dependencies between
>suites:
>1) classlib - no dependencies
>2) drlvm - depends on classlib only
>3) all other suites - depends on classlib and drlvm
>
>Do we have suites with other set of dependencies?
>
>Please see other comments below.
>
>BWT, FYI I run JettyScenario and found that it failed but overall run
>passed. I see the following output:
>run:
> [mkdir] Created dir:
>/export/users/smishura/newinfra/build/checkouts/JettyScenario/results
> [junit] Running JettyScenario
> [java] Java Result: 139
> [junit] Tests run: 5, Failures: 1, Errors: 3, Time elapsed: 94.581
sec
> [junit] Test JettyScenario FAILED
> [java] java.net.ConnectException: localhost/127.0.0.1:4444 -
>Connection refused
> [java] at
>org.apache.harmony.luni.net.PlainSocketImpl.connect(PlainSocketImpl.jav
a:23
>0)
> [java] at
>org.apache.harmony.luni.net.PlainSocketImpl.connect(PlainSocketImpl.jav
a:19
>3)
> [java] at java.net.Socket.startupSocket(Socket.java:662)
> [java] at java.net.Socket.<init>(Socket.java:214)
> [java] at org.mortbay.start.Main.stop(Main.java:526)
> [java] at org.mortbay.start.Main.main(Main.java:104)
> [java] at
>java.lang.reflect.VMReflection.invokeMethod(VMReflection.java)
> [java] at java.lang.reflect.Method.invoke(Method.java:381)
> [java] at org.apache.harmony.vm.JarRunner.main(JarRunner.java:80)
> [delete] Deleting:
>/export/users/smishura/newinfra/build/temp/null1281662371
>
>check-status:
>
>run:
>
>BUILD SUCCESSFUL
>Total time: 2 minutes 10 seconds
This is the way Jetty scenario implemented - 'run' target of adaptor.xml
does not ever fail. It reports scenario 'failed' or 'passed' in its log
(and in mail notification if in CC mode), and does not lead framework to
'BUILD FAILED'. This is because Jetty server should be stopped after
scenario is completed whether it failed or not.
>> > > > > > What the reason for calling classlib's build in this way?
Why
>we have
>> > > > > > to run 'ant.bat' (or 'ant.sh' for Linux) via <exec>?
>> > > > >
>> > > > > It's workaround for Ant's OutOfMemory problem arising after
doing
>the same
>> > > > > by <ant> target. The <ant>-rebuild of classlib project causes
big
>memory
>> > > > > leaks somewhere in the Ant and the further BTI execution is
>impossible
>> > > > > becouse of arising OOM.
>> > > > >
>> > > >
>> > > > As far as I remember the solution was to increase the max
memory
>used
>> > > > by Ant with setting ANT_OPTS to "-Xms256M -Xmx512M". Have you
tried
>> > > > this?
>> > >
>> > > 2Gb was not enough.
>> > >
>> >
>> > Wow ... and do you have any idea why 512M is enough for the current
>> > infra and 2Gb is not enough for the proposed one?
>>
>> OOE happens under CC execution on the half way of drlvm building.
>> We didn't it before in one Ant project.
>>
>
>Sorry, may be my question was unclear. I asked the following: have you
>investigated why the current infra is able to run the same testing
>scenario (classlib+drlvm) with 512M and new one can not do it with
>2Gb?
>
>> > As I understand
>> > certain ANT tasks can use a lot of memory - classlibrary complies
over
>> > 3500 with javac task, so we may assume that this requires a lot of
>> > memory. But root cause of such essential increase (more then x4
times)
>> > is somewhere in the proposed infa from my POV. Can this impose some
>> > limits, for example, to a number of testing suite that the infra
can
>> > setup and run?
>>
>> Load testing of BTI could help to answer your question.
>> And I didn't tried Ant 1.7. Probably it fixes some of the leaks.
>>
>
>So you think that this is ANT issue only. Right?
>
><SNIP>
>
>> Yes, it is Ant-based Framework. adaptor.xml takes the
>> responsibility to 'adapt' existing Test Suite for execution under
this
>> framework. So it's called adaptor.xml
>>
>> > > > BTW, if we rename build.xml=>buildtest.xml we have to type each
>time:
>> > > > ant -f buildtest.xml.
>> > >
>> > > The standard usage of BTI supposes launching by means of shell
>scripts
>> > > (buildtest.bat/.sh), not by ant launcher.
>> > >
>> >
>> > Just to setup classlib and lanch ant with -f buildtest.xml (like
DRL VM
>build)?
>>
>> No, just to say
>> > buildtest -Dtest.suites=some,selected,suites run
>> and get the things working.
>
>I meant that shell script is required to set up classpath and lanch
>ant with -f buildtest.xml
>
>Thanks,
>Stepan
Thanks,
Aleksei.
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Stepan Mishura <st...@gmail.com>.
On 4/20/07, Alexander Kleymenov wrote:
> Hi,
>
> > > There can be a HDK building test suite, so all of the suites depending
> > > on drlvm will depend and on this test suite. And there can appear
> > > another dependencies between the suites.
> >
> > This is not clear for me: currently there is no hdk suite. Could you
> > give more info?
>
> I'm not telling about the facts, but about possibilities here.
> Such a suite can arise, and BTI should handle new dependencies without
> need to be reimplemented.
>
Sorry for being persistent - I don't understand about which
possibilities you are talking about. Could you describe how suites'
drlvm-dependency is resolved in case of hdk suite only ? Also how the
framework resolves the dependency if there are classlib+drlvm+hdk all
together? And what if I want to run suites on classlib+IBM VME?
> > > The test-suite dependency managing is a simple and quite natural way to
> > > define the relationships between the suites.
> >
> > It is quite big peace of code. And I'm just trying to understand
> > whether we really need this functionality. If it is used only to
> > resolve suites dependencies on classlib and drlvm then I'd substitute
> > it with something that more simple.
> >
> > BWT, how the framework resolves cycle dependencies?
>
> It should report an error. Now it just cut the circle.
>
> > > The information on such a
> > > relationships is very useful during the test run setup and execution
> > > stages. The resolution of default values for required parameters is
> > > dependent on this information, the order in which the selected test
> > > suites will be run (for single and CC run) is determined by means of
> > > this information, CC related functionality uses dependency info to
> > > retrieve SVN modifications lead to test-suite execution.
> > >
> > > So test-suite dependency management is an important architectural part
> > > of the proposed BTI. It can't be simply removed from the system. Can you
> > > suggest substitution for it?
> >
> > My current view that the most of suites just need to know where is JRE
> > for testing.
>
> No, there are other things. For example - to run classlib unit tests we
> define test suite 'classlib-test' which depends on 'classlib' suite.
But currently there is no 'classlib-test' suite for the proposed infra ...
> All we need to do to implement 'classlib-test' is to define the dependency on
> 'classlib', use its shared 'trunk.dir' parameter as a default for
> 'built.classlib.workspace' required parameter, and write single line in
> adaptor's run target:
> <ant dir="${built.classlib.workspace}" target="test"/>
Right. And the current infra does this without a need to build a tree
of suites dependencies and traversing the tree to optimize it by
looking for cycles and unselected nodes. Again could give an example?
We mentioned hdk suite above. It might be a perfect real case to
demonstrate new functionality.
> It's quite simple and natural solution in the current BTI approach
> with dependency management means presented.
>
> > Then the framework can do the following
> > 1) if classlib is specified then test.jre.home=${classlib}/deploy/jdk/jre
> > In this case the framework user should just place VM implementation to
> > ${classlib}/deploy/jdk/jre/bin/default. (for example, it may be IBM
> > VME). So the framework first rebuild classlib and then runs other
> > suites.
> >
> > 2) if classlib+drlvm are specified then
> > test.jre.home=${drlvm}/build/deploy/jdk/jre
> > In this case the framework first rebuild classlib, then drlvm suite
> > and after that runs other suites.
> >
> > 3) classlib and drlvm are not specified. test.jre.home is specified by
> > framework user.
> >
> > Does this work for you?
>
> I think we should keep the general approach to dependency management and
> do not tie with particular intersection points (I mean tested jre in
> your case).
> I think we should keep the information about such intersections on the
> level of test suites (as it's done now). In the current approach the
> appearance of a new test suite with the product which can be shared does
> not cause the fixes inside the framework. In your case such a new
> 'shared' product causes drilling into framework.
>
No drill into the framework is required - the suite implements
ant-targets to set-up it, run it and publish its results. And the
framework provides to the suite a location of JRE to run its tests. I
want just keep things simple.
Thanks,
Stepan
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Alexander Kleymenov <kl...@gmail.com>.
Hi,
> > There can be a HDK building test suite, so all of the suites depending
> > on drlvm will depend and on this test suite. And there can appear
> > another dependencies between the suites.
>
> This is not clear for me: currently there is no hdk suite. Could you
> give more info?
I'm not telling about the facts, but about possibilities here.
Such a suite can arise, and BTI should handle new dependencies without
need to be reimplemented.
> > The test-suite dependency managing is a simple and quite natural way to
> > define the relationships between the suites.
>
> It is quite big peace of code. And I'm just trying to understand
> whether we really need this functionality. If it is used only to
> resolve suites dependencies on classlib and drlvm then I'd substitute
> it with something that more simple.
>
> BWT, how the framework resolves cycle dependencies?
It should report an error. Now it just cut the circle.
> > The information on such a
> > relationships is very useful during the test run setup and execution
> > stages. The resolution of default values for required parameters is
> > dependent on this information, the order in which the selected test
> > suites will be run (for single and CC run) is determined by means of
> > this information, CC related functionality uses dependency info to
> > retrieve SVN modifications lead to test-suite execution.
> >
> > So test-suite dependency management is an important architectural part
> > of the proposed BTI. It can't be simply removed from the system. Can you
> > suggest substitution for it?
>
> My current view that the most of suites just need to know where is JRE
> for testing.
No, there are other things. For example - to run classlib unit tests we
define test suite 'classlib-test' which depends on 'classlib' suite.
All we need to do to implement 'classlib-test' is to define the dependency on
'classlib', use its shared 'trunk.dir' parameter as a default for
'built.classlib.workspace' required parameter, and write single line in
adaptor's run target:
<ant dir="${built.classlib.workspace}" target="test"/>
It's quite simple and natural solution in the current BTI approach
with dependency management means presented.
> Then the framework can do the following
> 1) if classlib is specified then test.jre.home=${classlib}/deploy/jdk/jre
> In this case the framework user should just place VM implementation to
> ${classlib}/deploy/jdk/jre/bin/default. (for example, it may be IBM
> VME). So the framework first rebuild classlib and then runs other
> suites.
>
> 2) if classlib+drlvm are specified then
> test.jre.home=${drlvm}/build/deploy/jdk/jre
> In this case the framework first rebuild classlib, then drlvm suite
> and after that runs other suites.
>
> 3) classlib and drlvm are not specified. test.jre.home is specified by
> framework user.
>
> Does this work for you?
I think we should keep the general approach to dependency management and
do not tie with particular intersection points (I mean tested jre in
your case).
I think we should keep the information about such intersections on the
level of test suites (as it's done now). In the current approach the
appearance of a new test suite with the product which can be shared does
not cause the fixes inside the framework. In your case such a new
'shared' product causes drilling into framework.
Alexander
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Stepan Mishura <st...@gmail.com>.
On 4/19/07, Alexander Kleymenov wrote:
> Stepan,
>
> On 4/19/07, Stepan Mishura <st...@gmail.com> wrote:
> >
> > Do we have suites with set of dependencies other then the following:
> > 1) classlib - no dependencies
> > 2) drlvm - depends on classlib only
> > 3) all other suites - depends on classlib and drlvm
> >
> > If no then ant code that manages dependencies between suites looks
> > redundant for me.
Hi Alexander,
>
> There can be a HDK building test suite, so all of the suites depending
> on drlvm will depend and on this test suite. And there can appear
> another dependencies between the suites.
>
This is not clear for me: currently there is no hdk suite. Could you
give more info?
> The test-suite dependency managing is a simple and quite natural way to
> define the relationships between the suites.
It is quite big peace of code. And I'm just trying to understand
whether we really need this functionality. If it is used only to
resolve suites dependencies on classlib and drlvm then I'd substitute
it with something that more simple.
BWT, how the framework resolves cycle dependencies?
> The information on such a
> relationships is very useful during the test run setup and execution
> stages. The resolution of default values for required parameters is
> dependent on this information, the order in which the selected test
> suites will be run (for single and CC run) is determined by means of
> this information, CC related functionality uses dependency info to
> retrieve SVN modifications lead to test-suite execution.
>
> So test-suite dependency management is an important architectural part
> of the proposed BTI. It can't be simply removed from the system. Can you
> suggest substitution for it?
My current view that the most of suites just need to know where is JRE
for testing.
Then the framework can do the following
1) if classlib is specified then test.jre.home=${classlib}/deploy/jdk/jre
In this case the framework user should just place VM implementation to
${classlib}/deploy/jdk/jre/bin/default. (for example, it may be IBM
VME). So the framework first rebuild classlib and then runs other
suites.
2) if classlib+drlvm are specified then
test.jre.home=${drlvm}/build/deploy/jdk/jre
In this case the framework first rebuild classlib, then drlvm suite
and after that runs other suites.
3) classlib and drlvm are not specified. test.jre.home is specified by
framework user.
Does this work for you?
Thanks,
Stepan
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Alexander Kleymenov <kl...@gmail.com>.
Stepan,
On 4/19/07, Stepan Mishura <st...@gmail.com> wrote:
>
> Do we have suites with set of dependencies other then the following:
> 1) classlib - no dependencies
> 2) drlvm - depends on classlib only
> 3) all other suites - depends on classlib and drlvm
>
> If no then ant code that manages dependencies between suites looks
> redundant for me.
There can be a HDK building test suite, so all of the suites depending
on drlvm will depend and on this test suite. And there can appear
another dependencies between the suites.
The test-suite dependency managing is a simple and quite natural way to
define the relationships between the suites. The information on such a
relationships is very useful during the test run setup and execution
stages. The resolution of default values for required parameters is
dependent on this information, the order in which the selected test
suites will be run (for single and CC run) is determined by means of
this information, CC related functionality uses dependency info to
retrieve SVN modifications lead to test-suite execution.
So test-suite dependency management is an important architectural part
of the proposed BTI. It can't be simply removed from the system. Can you
suggest substitution for it?
Alexander
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Stepan Mishura <st...@gmail.com>.
On 4/19/07, Alexander Kleymenov <kl...@gmail.com> wrote:
> Hi,
>
> On 4/18/07, Stepan Mishura <st...@gmail.com> wrote:
> > Hi Alexander,
> >
> > Next question:
> > 7) Suite's dependencies.
> >
> > I've tried the framework with JettyScenario. So I've unpacked a bundle
> > with Jetty scenario into the framework dir, copied dir with tests,
> > commented out some lines in dataptor (so everything according to the
> > README.txt file) but I accidentally typed
> > $ ant -Dtest.suites="JettyScenario" setup
> > instead of
> > $ ant -Dtest.suites="classlib,drlvm,JettyScenario" setup
> >
> > I expected that the framework would set up
> > classlib,drlvm,JettyScenario because JettyScenario defines in
> > parameters.xml dependencies on classlib,drlvm. But it set up only
> > JettyScenario. I've looked into the code and my impression that
> > dependencies defined in parameters.xml affect only in which order the
> > framework runs suites. Am I correct?
>
> Yes, dependency of 'test-suite' on 'superior-suite' means 'test-suite'
> uses some product of 'superior-suite' described there as a shared parameter
> and used as a default value for some required parameter of 'test-suite'.
> If we didn't select the 'superior-suite' for execution such a required
> parameter will not be resolved and will be demanded from user.
>
Hi Alexander,
You've missed one my question. I'd like to repeat it:
Do we have suites with set of dependencies other then the following:
1) classlib - no dependencies
2) drlvm - depends on classlib only
3) all other suites - depends on classlib and drlvm
If no then ant code that manages dependencies between suites looks
redundant for me.
Thanks,
> > > > > > > > What the reason for calling classlib's build in this way? Why we have
> > > > > > > > to run 'ant.bat' (or 'ant.sh' for Linux) via <exec>?
> > > > > > >
> > > > > > > It's workaround for Ant's OutOfMemory problem arising after doing the same
> > > > > > > by <ant> target. The <ant>-rebuild of classlib project causes big memory
> > > > > > > leaks somewhere in the Ant and the further BTI execution is impossible
> > > > > > > becouse of arising OOM.
> > > > > > >
> > > > > >
> > > > > > As far as I remember the solution was to increase the max memory used
> > > > > > by Ant with setting ANT_OPTS to "-Xms256M -Xmx512M". Have you tried
> > > > > > this?
> > > > >
> > > > > 2Gb was not enough.
> > > > >
> > > >
> > > > Wow ... and do you have any idea why 512M is enough for the current
> > > > infra and 2Gb is not enough for the proposed one?
> > >
> > > OOE happens under CC execution on the half way of drlvm building.
> > > We didn't it before in one Ant project.
> > >
> >
> > Sorry, may be my question was unclear. I asked the following: have you
> > investigated why the current infra is able to run the same testing
> > scenario (classlib+drlvm) with 512M and new one can not do it with
> > 2Gb?
>
> I didn't make deep investigation but I think this is because we did not try
> an Ant's project containing the launching of CC
> which subsequently calls <ant classlib build>, and then calls <ant drlvm build>
> - and all these in one project executed on single JVM. <Exec> creates
> separate ant process for it - it's almost the same what we did before.
>
> > > > As I understand
> > > > certain ANT tasks can use a lot of memory - classlibrary complies over
> > > > 3500 with javac task, so we may assume that this requires a lot of
> > > > memory. But root cause of such essential increase (more then x4 times)
> > > > is somewhere in the proposed infa from my POV. Can this impose some
> > > > limits, for example, to a number of testing suite that the infra can
> > > > setup and run?
> > >
> > > Load testing of BTI could help to answer your question.
> > > And I didn't tried Ant 1.7. Probably it fixes some of the leaks.
> > >
> >
> > So you think that this is ANT issue only. Right?
>
> Maybe Ant, maybe CC, maybe single-jvm-mix of them.
>
> > One minor question: I re-set up the scenarion as README.txt recommends
> > (i.e.by specifying -Dtest.suites="classlib,drlvm,JettyScenario"). And
> > run it under CC. But I received notifications about JettyScenario
> > status only. There were no notifications from classlib and drlvm. As
> > far as I see this is not CC configuration problem - the publishers for
> > them just were not implemented. Right? Otherwise what I should do to
> > see notifications from classlib and drlvm?
>
> There no publisher for classlib and drlvm build status notifications.
> These suites are mostly executed as an intermediate before others.
>
> But it's possible to provide custom publisher and use it. The steps to
> do it for classlib are:
>
> 1. Implement Ant script containing 'publish' target. You can use
> some of existing publisher implementations as a base. Name it as you
> want, say, publisher.xml.
>
> 2. In framework.local.properties define the following property:
> classlib.parameters.cc.publisher=/path/to/your/publisher.xml
>
> 3. If everything is OK with your implementation, it will sent
> status notifications while running under CC.
>
> Thanks,
> Alexander
>
--
Stepan Mishura
Intel Enterprise Solutions Software Division
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Alexander Kleymenov <kl...@gmail.com>.
Hi,
On 4/18/07, Stepan Mishura <st...@gmail.com> wrote:
> Hi Alexander,
>
> Next question:
> 7) Suite's dependencies.
>
> I've tried the framework with JettyScenario. So I've unpacked a bundle
> with Jetty scenario into the framework dir, copied dir with tests,
> commented out some lines in dataptor (so everything according to the
> README.txt file) but I accidentally typed
> $ ant -Dtest.suites="JettyScenario" setup
> instead of
> $ ant -Dtest.suites="classlib,drlvm,JettyScenario" setup
>
> I expected that the framework would set up
> classlib,drlvm,JettyScenario because JettyScenario defines in
> parameters.xml dependencies on classlib,drlvm. But it set up only
> JettyScenario. I've looked into the code and my impression that
> dependencies defined in parameters.xml affect only in which order the
> framework runs suites. Am I correct?
Yes, dependency of 'test-suite' on 'superior-suite' means 'test-suite'
uses some product of 'superior-suite' described there as a shared parameter
and used as a default value for some required parameter of 'test-suite'.
If we didn't select the 'superior-suite' for execution such a required
parameter will not be resolved and will be demanded from user.
> > > > > > > What the reason for calling classlib's build in this way? Why we have
> > > > > > > to run 'ant.bat' (or 'ant.sh' for Linux) via <exec>?
> > > > > >
> > > > > > It's workaround for Ant's OutOfMemory problem arising after doing the same
> > > > > > by <ant> target. The <ant>-rebuild of classlib project causes big memory
> > > > > > leaks somewhere in the Ant and the further BTI execution is impossible
> > > > > > becouse of arising OOM.
> > > > > >
> > > > >
> > > > > As far as I remember the solution was to increase the max memory used
> > > > > by Ant with setting ANT_OPTS to "-Xms256M -Xmx512M". Have you tried
> > > > > this?
> > > >
> > > > 2Gb was not enough.
> > > >
> > >
> > > Wow ... and do you have any idea why 512M is enough for the current
> > > infra and 2Gb is not enough for the proposed one?
> >
> > OOE happens under CC execution on the half way of drlvm building.
> > We didn't it before in one Ant project.
> >
>
> Sorry, may be my question was unclear. I asked the following: have you
> investigated why the current infra is able to run the same testing
> scenario (classlib+drlvm) with 512M and new one can not do it with
> 2Gb?
I didn't make deep investigation but I think this is because we did not try
an Ant's project containing the launching of CC
which subsequently calls <ant classlib build>, and then calls <ant drlvm build>
- and all these in one project executed on single JVM. <Exec> creates
separate ant process for it - it's almost the same what we did before.
> > > As I understand
> > > certain ANT tasks can use a lot of memory - classlibrary complies over
> > > 3500 with javac task, so we may assume that this requires a lot of
> > > memory. But root cause of such essential increase (more then x4 times)
> > > is somewhere in the proposed infa from my POV. Can this impose some
> > > limits, for example, to a number of testing suite that the infra can
> > > setup and run?
> >
> > Load testing of BTI could help to answer your question.
> > And I didn't tried Ant 1.7. Probably it fixes some of the leaks.
> >
>
> So you think that this is ANT issue only. Right?
Maybe Ant, maybe CC, maybe single-jvm-mix of them.
> One minor question: I re-set up the scenarion as README.txt recommends
> (i.e.by specifying -Dtest.suites="classlib,drlvm,JettyScenario"). And
> run it under CC. But I received notifications about JettyScenario
> status only. There were no notifications from classlib and drlvm. As
> far as I see this is not CC configuration problem - the publishers for
> them just were not implemented. Right? Otherwise what I should do to
> see notifications from classlib and drlvm?
There no publisher for classlib and drlvm build status notifications.
These suites are mostly executed as an intermediate before others.
But it's possible to provide custom publisher and use it. The steps to
do it for classlib are:
1. Implement Ant script containing 'publish' target. You can use
some of existing publisher implementations as a base. Name it as you
want, say, publisher.xml.
2. In framework.local.properties define the following property:
classlib.parameters.cc.publisher=/path/to/your/publisher.xml
3. If everything is OK with your implementation, it will sent
status notifications while running under CC.
Thanks,
Alexander
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Stepan Mishura <st...@gmail.com>.
On 4/18/07, Stepan Mishura wrote:
> Hi Alexander,
>
> Next question:
> 7) Suite's dependencies.
>
> I've tried the framework with JettyScenario. So I've unpacked a bundle
> with Jetty scenario into the framework dir, copied dir with tests,
> commented out some lines in dataptor (so everything according to the
> README.txt file) but I accidentally typed
> $ ant -Dtest.suites="JettyScenario" setup
> instead of
> $ ant -Dtest.suites="classlib,drlvm,JettyScenario" setup
>
One minor question: I re-set up the scenarion as README.txt recommends
(i.e.by specifying -Dtest.suites="classlib,drlvm,JettyScenario"). And
run it under CC. But I received notifications about JettyScenario
status only. There were no notifications from classlib and drlvm. As
far as I see this is not CC configuration problem - the publishers for
them just were not implemented. Right? Otherwise what I should do to
see notifications from classlib and drlvm?
Thanks,
Stepan.
> I expected that the framework would set up
> classlib,drlvm,JettyScenario because JettyScenario defines in
> parameters.xml dependencies on classlib,drlvm. But it set up only
> JettyScenario. I've looked into the code and my impression that
> dependencies defined in parameters.xml affect only in which order the
> framework runs suites. Am I correct?
>
> Also my understanding that we have the following dependencies between suites:
> 1) classlib - no dependencies
> 2) drlvm - depends on classlib only
> 3) all other suites - depends on classlib and drlvm
>
> Do we have suites with other set of dependencies?
>
> Please see other comments below.
>
> BWT, FYI I run JettyScenario and found that it failed but overall run
> passed. I see the following output:
> run:
> [mkdir] Created dir:
> /export/users/smishura/newinfra/build/checkouts/JettyScenario/results
> [junit] Running JettyScenario
> [java] Java Result: 139
> [junit] Tests run: 5, Failures: 1, Errors: 3, Time elapsed: 94.581 sec
> [junit] Test JettyScenario FAILED
> [java] java.net.ConnectException: localhost/127.0.0.1:4444 -
> Connection refused
> [java] at org.apache.harmony.luni.net.PlainSocketImpl.connect(PlainSocketImpl.java:230)
> [java] at org.apache.harmony.luni.net.PlainSocketImpl.connect(PlainSocketImpl.java:193)
> [java] at java.net.Socket.startupSocket(Socket.java:662)
> [java] at java.net.Socket.<init>(Socket.java:214)
> [java] at org.mortbay.start.Main.stop(Main.java:526)
> [java] at org.mortbay.start.Main.main(Main.java:104)
> [java] at java.lang.reflect.VMReflection.invokeMethod(VMReflection.java)
> [java] at java.lang.reflect.Method.invoke(Method.java:381)
> [java] at org.apache.harmony.vm.JarRunner.main(JarRunner.java:80)
> [delete] Deleting: /export/users/smishura/newinfra/build/temp/null1281662371
>
> check-status:
>
> run:
>
> BUILD SUCCESSFUL
> Total time: 2 minutes 10 seconds
>
> > > > > > > What the reason for calling classlib's build in this way? Why we have
> > > > > > > to run 'ant.bat' (or 'ant.sh' for Linux) via <exec>?
> > > > > >
> > > > > > It's workaround for Ant's OutOfMemory problem arising after doing the same
> > > > > > by <ant> target. The <ant>-rebuild of classlib project causes big memory
> > > > > > leaks somewhere in the Ant and the further BTI execution is impossible
> > > > > > becouse of arising OOM.
> > > > > >
> > > > >
> > > > > As far as I remember the solution was to increase the max memory used
> > > > > by Ant with setting ANT_OPTS to "-Xms256M -Xmx512M". Have you tried
> > > > > this?
> > > >
> > > > 2Gb was not enough.
> > > >
> > >
> > > Wow ... and do you have any idea why 512M is enough for the current
> > > infra and 2Gb is not enough for the proposed one?
> >
> > OOE happens under CC execution on the half way of drlvm building.
> > We didn't it before in one Ant project.
> >
>
> Sorry, may be my question was unclear. I asked the following: have you
> investigated why the current infra is able to run the same testing
> scenario (classlib+drlvm) with 512M and new one can not do it with
> 2Gb?
>
> > > As I understand
> > > certain ANT tasks can use a lot of memory - classlibrary complies over
> > > 3500 with javac task, so we may assume that this requires a lot of
> > > memory. But root cause of such essential increase (more then x4 times)
> > > is somewhere in the proposed infa from my POV. Can this impose some
> > > limits, for example, to a number of testing suite that the infra can
> > > setup and run?
> >
> > Load testing of BTI could help to answer your question.
> > And I didn't tried Ant 1.7. Probably it fixes some of the leaks.
> >
>
> So you think that this is ANT issue only. Right?
>
> <SNIP>
>
> > Yes, it is Ant-based Framework. adaptor.xml takes the
> > responsibility to 'adapt' existing Test Suite for execution under this
> > framework. So it's called adaptor.xml
> >
> > > > > BTW, if we rename build.xml=>buildtest.xml we have to type each time:
> > > > > ant -f buildtest.xml.
> > > >
> > > > The standard usage of BTI supposes launching by means of shell scripts
> > > > (buildtest.bat/.sh), not by ant launcher.
> > > >
> > >
> > > Just to setup classlib and lanch ant with -f buildtest.xml (like DRL VM build)?
> >
> > No, just to say
> > > buildtest -Dtest.suites=some,selected,suites run
> > and get the things working.
>
> I meant that shell script is required to set up classpath and lanch
> ant with -f buildtest.xml
>
> Thanks,
> Stepan
>
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Stepan Mishura <st...@gmail.com>.
Hi Alexander,
Next question:
7) Suite's dependencies.
I've tried the framework with JettyScenario. So I've unpacked a bundle
with Jetty scenario into the framework dir, copied dir with tests,
commented out some lines in dataptor (so everything according to the
README.txt file) but I accidentally typed
$ ant -Dtest.suites="JettyScenario" setup
instead of
$ ant -Dtest.suites="classlib,drlvm,JettyScenario" setup
I expected that the framework would set up
classlib,drlvm,JettyScenario because JettyScenario defines in
parameters.xml dependencies on classlib,drlvm. But it set up only
JettyScenario. I've looked into the code and my impression that
dependencies defined in parameters.xml affect only in which order the
framework runs suites. Am I correct?
Also my understanding that we have the following dependencies between suites:
1) classlib - no dependencies
2) drlvm - depends on classlib only
3) all other suites - depends on classlib and drlvm
Do we have suites with other set of dependencies?
Please see other comments below.
BWT, FYI I run JettyScenario and found that it failed but overall run
passed. I see the following output:
run:
[mkdir] Created dir:
/export/users/smishura/newinfra/build/checkouts/JettyScenario/results
[junit] Running JettyScenario
[java] Java Result: 139
[junit] Tests run: 5, Failures: 1, Errors: 3, Time elapsed: 94.581 sec
[junit] Test JettyScenario FAILED
[java] java.net.ConnectException: localhost/127.0.0.1:4444 -
Connection refused
[java] at org.apache.harmony.luni.net.PlainSocketImpl.connect(PlainSocketImpl.java:230)
[java] at org.apache.harmony.luni.net.PlainSocketImpl.connect(PlainSocketImpl.java:193)
[java] at java.net.Socket.startupSocket(Socket.java:662)
[java] at java.net.Socket.<init>(Socket.java:214)
[java] at org.mortbay.start.Main.stop(Main.java:526)
[java] at org.mortbay.start.Main.main(Main.java:104)
[java] at java.lang.reflect.VMReflection.invokeMethod(VMReflection.java)
[java] at java.lang.reflect.Method.invoke(Method.java:381)
[java] at org.apache.harmony.vm.JarRunner.main(JarRunner.java:80)
[delete] Deleting: /export/users/smishura/newinfra/build/temp/null1281662371
check-status:
run:
BUILD SUCCESSFUL
Total time: 2 minutes 10 seconds
> > > > > > What the reason for calling classlib's build in this way? Why we have
> > > > > > to run 'ant.bat' (or 'ant.sh' for Linux) via <exec>?
> > > > >
> > > > > It's workaround for Ant's OutOfMemory problem arising after doing the same
> > > > > by <ant> target. The <ant>-rebuild of classlib project causes big memory
> > > > > leaks somewhere in the Ant and the further BTI execution is impossible
> > > > > becouse of arising OOM.
> > > > >
> > > >
> > > > As far as I remember the solution was to increase the max memory used
> > > > by Ant with setting ANT_OPTS to "-Xms256M -Xmx512M". Have you tried
> > > > this?
> > >
> > > 2Gb was not enough.
> > >
> >
> > Wow ... and do you have any idea why 512M is enough for the current
> > infra and 2Gb is not enough for the proposed one?
>
> OOE happens under CC execution on the half way of drlvm building.
> We didn't it before in one Ant project.
>
Sorry, may be my question was unclear. I asked the following: have you
investigated why the current infra is able to run the same testing
scenario (classlib+drlvm) with 512M and new one can not do it with
2Gb?
> > As I understand
> > certain ANT tasks can use a lot of memory - classlibrary complies over
> > 3500 with javac task, so we may assume that this requires a lot of
> > memory. But root cause of such essential increase (more then x4 times)
> > is somewhere in the proposed infa from my POV. Can this impose some
> > limits, for example, to a number of testing suite that the infra can
> > setup and run?
>
> Load testing of BTI could help to answer your question.
> And I didn't tried Ant 1.7. Probably it fixes some of the leaks.
>
So you think that this is ANT issue only. Right?
<SNIP>
> Yes, it is Ant-based Framework. adaptor.xml takes the
> responsibility to 'adapt' existing Test Suite for execution under this
> framework. So it's called adaptor.xml
>
> > > > BTW, if we rename build.xml=>buildtest.xml we have to type each time:
> > > > ant -f buildtest.xml.
> > >
> > > The standard usage of BTI supposes launching by means of shell scripts
> > > (buildtest.bat/.sh), not by ant launcher.
> > >
> >
> > Just to setup classlib and lanch ant with -f buildtest.xml (like DRL VM build)?
>
> No, just to say
> > buildtest -Dtest.suites=some,selected,suites run
> and get the things working.
I meant that shell script is required to set up classpath and lanch
ant with -f buildtest.xml
Thanks,
Stepan
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Alexander Kleymenov <kl...@gmail.com>.
Hi,
> > > > > What the reason for calling classlib's build in this way? Why we have
> > > > > to run 'ant.bat' (or 'ant.sh' for Linux) via <exec>?
> > > >
> > > > It's workaround for Ant's OutOfMemory problem arising after doing the same
> > > > by <ant> target. The <ant>-rebuild of classlib project causes big memory
> > > > leaks somewhere in the Ant and the further BTI execution is impossible
> > > > becouse of arising OOM.
> > > >
> > >
> > > As far as I remember the solution was to increase the max memory used
> > > by Ant with setting ANT_OPTS to "-Xms256M -Xmx512M". Have you tried
> > > this?
> >
> > 2Gb was not enough.
> >
>
> Wow ... and do you have any idea why 512M is enough for the current
> infra and 2Gb is not enough for the proposed one?
OOE happens under CC execution on the half way of drlvm building.
We didn't it before in one Ant project.
> As I understand
> certain ANT tasks can use a lot of memory - classlibrary complies over
> 3500 with javac task, so we may assume that this requires a lot of
> memory. But root cause of such essential increase (more then x4 times)
> is somewhere in the proposed infa from my POV. Can this impose some
> limits, for example, to a number of testing suite that the infra can
> setup and run?
Load testing of BTI could help to answer your question.
And I didn't tried Ant 1.7. Probably it fixes some of the leaks.
> > > > >
> > > > > 3) In exec-adaptor.xml antconrib's wrapper 'trycatch' and 'runtarget'
> > > > > task are used for calling adaptor's(i.e. suite's) target.
> > > > >
> > > > > <trycatch property="execution.exception">
> > > > > <try>
> > > > > <runtarget target="@{target}"/>
> > > > > </try>
> > > > > <catch>
> > > > > Echo error message
> > > > > </catch>
> > > > > </trycatch>
> > > > >
> > > > > Again, why it is not possible use <ant> task? And this leads to the
> > > > > next question.
> > > >
> > > > Infra needs some failure processing means here. Default <ant> task does
> > > > not allow to do it. Using of <runtarget> instead of <ant> saves memory
> > > > as it does not fork the ant's project.
> > > >
> > >
> > > What kind of failure processing it does?
> >
> > Sets up the property, prints trace, and keeps execution going on.
> >
>
> Did I understand it correctly that if suites are run not under
> CruiseControl it lets to collect exit status for all suites without
> stopping the build (even there are failed suites)?
Yes, You did. BTI doesn't stop test suites executing after failure of one of
them.
> > > > > 5) Patching ANT - is there any workaround?
> > > >
> > > > It's a workaround itself. For Ant's basedir problem. Ant does not set
> > > > basedir properly if <ant> task executes the code containing <subant>
> > > > task. This path is taken from the bug report filed on both versions of
> > > > ant. Here it's supposed that 1.6.5 version is used. To be more correct
> > > > here we should check current ant's version and take suitable patch.
> > > >
> > >
> > > Oops ... does it mean that this workaround binds the infra to selected
> > > ANT version (in our case 1.6.5)? For example, if I have Ant 1.7
> > > installed the infra won't work?
> >
> > Yes, I'm composing TODO list for BTI and it already has this item.
> >
>
> Frankly, I'm not quite comfort with this. So you are going to detect
> ANT version and apply corresponding patch. Correct? Let me think a bit
> if there are any potential issues with this approach. For example, if
> I setup the infra with ANT 1.6.5 and after that switch to ANT 1.7 then
> I need to re-set up the infra. Right? Other issues?
>
No, it will be self-patched automatically, without re-setup.
> > > > > 6) And minor question: is there any strong reason for naming build
> > > > > scripts 'adaptor.xml' not 'build.xml'?
> > > >
> > > > I'm positioning the BTI framework not as an Ant-based Build System,
> > > > but as an Application implemented in Ant Scripting Language.
> > > > In build systems build.xml is a self-documenting file name.
> > > > In BTI the name adaptor.xml does the same - it documents itself.
> > > > I think it would be better to rename root build.xml to buildtest.xml.
> > >
> > > Sorry, I didn't catch about proposed infra positioning. Could you clarify?
> >
> > What exactly needs clarification?
> >
>
> Why this is not Ant-based framework? What is Ant Scripting Language
> (because it uses antcontrib)?
Yes, it is Ant-based Framework. adaptor.xml takes the
responsibility to 'adapt' existing Test Suite for execution under this
framework. So it's called adaptor.xml
> > > BTW, if we rename build.xml=>buildtest.xml we have to type each time:
> > > ant -f buildtest.xml.
> >
> > The standard usage of BTI supposes launching by means of shell scripts
> > (buildtest.bat/.sh), not by ant launcher.
> >
>
> Just to setup classlib and lanch ant with -f buildtest.xml (like DRL VM build)?
No, just to say
> buildtest -Dtest.suites=some,selected,suites run
and get the things working.
Thanks,
Alexander
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Stepan Mishura <st...@gmail.com>.
On 4/17/07, Alexander Kleymenov wrote:
> Stepan,
>
> > > > What the reason for calling classlib's build in this way? Why we have
> > > > to run 'ant.bat' (or 'ant.sh' for Linux) via <exec>?
> > >
> > > It's workaround for Ant's OutOfMemory problem arising after doing the same
> > > by <ant> target. The <ant>-rebuild of classlib project causes big memory
> > > leaks somewhere in the Ant and the further BTI execution is impossible
> > > becouse of arising OOM.
> > >
> >
> > As far as I remember the solution was to increase the max memory used
> > by Ant with setting ANT_OPTS to "-Xms256M -Xmx512M". Have you tried
> > this?
>
> 2Gb was not enough.
>
Wow ... and do you have any idea why 512M is enough for the current
infra and 2Gb is not enough for the proposed one? As I understand
certain ANT tasks can use a lot of memory - classlibrary complies over
3500 with javac task, so we may assume that this requires a lot of
memory. But root cause of such essential increase (more then x4 times)
is somewhere in the proposed infa from my POV. Can this impose some
limits, for example, to a number of testing suite that the infra can
setup and run?
> > > >
> > > > 3) In exec-adaptor.xml antconrib's wrapper 'trycatch' and 'runtarget'
> > > > task are used for calling adaptor's(i.e. suite's) target.
> > > >
> > > > <trycatch property="execution.exception">
> > > > <try>
> > > > <runtarget target="@{target}"/>
> > > > </try>
> > > > <catch>
> > > > Echo error message
> > > > </catch>
> > > > </trycatch>
> > > >
> > > > Again, why it is not possible use <ant> task? And this leads to the
> > > > next question.
> > >
> > > Infra needs some failure processing means here. Default <ant> task does
> > > not allow to do it. Using of <runtarget> instead of <ant> saves memory
> > > as it does not fork the ant's project.
> > >
> >
> > What kind of failure processing it does?
>
> Sets up the property, prints trace, and keeps execution going on.
>
Did I understand it correctly that if suites are run not under
CruiseControl it lets to collect exit status for all suites without
stopping the build (even there are failed suites)?
> > > > 5) Patching ANT - is there any workaround?
> > >
> > > It's a workaround itself. For Ant's basedir problem. Ant does not set
> > > basedir properly if <ant> task executes the code containing <subant>
> > > task. This path is taken from the bug report filed on both versions of
> > > ant. Here it's supposed that 1.6.5 version is used. To be more correct
> > > here we should check current ant's version and take suitable patch.
> > >
> >
> > Oops ... does it mean that this workaround binds the infra to selected
> > ANT version (in our case 1.6.5)? For example, if I have Ant 1.7
> > installed the infra won't work?
>
> Yes, I'm composing TODO list for BTI and it already has this item.
>
Frankly, I'm not quite comfort with this. So you are going to detect
ANT version and apply corresponding patch. Correct? Let me think a bit
if there are any potential issues with this approach. For example, if
I setup the infra with ANT 1.6.5 and after that switch to ANT 1.7 then
I need to re-set up the infra. Right? Other issues?
> > Also ... I wonder why we haven't meet this issue in classlib or drlvm
> > builds yet? It might mean that we use this Ant bug as a feature and
> > once Ant get fixed ... need to check classlib and drlvm.
>
> I've executed classlib's and drlvm's build system under patched ant -
> everything worked fine. So there are no 'bug as feature' substitution.
>
Thanks for your investigation!
> > > > 6) And minor question: is there any strong reason for naming build
> > > > scripts 'adaptor.xml' not 'build.xml'?
> > >
> > > I'm positioning the BTI framework not as an Ant-based Build System,
> > > but as an Application implemented in Ant Scripting Language.
> > > In build systems build.xml is a self-documenting file name.
> > > In BTI the name adaptor.xml does the same - it documents itself.
> > > I think it would be better to rename root build.xml to buildtest.xml.
> >
> > Sorry, I didn't catch about proposed infra positioning. Could you clarify?
>
> What exactly needs clarification?
>
Why this is not Ant-based framework? What is Ant Scripting Language
(because it uses antcontrib)?
> > BTW, if we rename build.xml=>buildtest.xml we have to type each time:
> > ant -f buildtest.xml.
>
> The standard usage of BTI supposes launching by means of shell scripts
> (buildtest.bat/.sh), not by ant launcher.
>
Just to setup classlib and lanch ant with -f buildtest.xml (like DRL VM build)?
Thanks,
Stepan Mishura
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Alexander Kleymenov <kl...@gmail.com>.
Stepan,
> > > What the reason for calling classlib's build in this way? Why we have
> > > to run 'ant.bat' (or 'ant.sh' for Linux) via <exec>?
> >
> > It's workaround for Ant's OutOfMemory problem arising after doing the same
> > by <ant> target. The <ant>-rebuild of classlib project causes big memory
> > leaks somewhere in the Ant and the further BTI execution is impossible
> > becouse of arising OOM.
> >
>
> As far as I remember the solution was to increase the max memory used
> by Ant with setting ANT_OPTS to "-Xms256M -Xmx512M". Have you tried
> this?
2Gb was not enough.
> > >
> > > 3) In exec-adaptor.xml antconrib's wrapper 'trycatch' and 'runtarget'
> > > task are used for calling adaptor's(i.e. suite's) target.
> > >
> > > <trycatch property="execution.exception">
> > > <try>
> > > <runtarget target="@{target}"/>
> > > </try>
> > > <catch>
> > > Echo error message
> > > </catch>
> > > </trycatch>
> > >
> > > Again, why it is not possible use <ant> task? And this leads to the
> > > next question.
> >
> > Infra needs some failure processing means here. Default <ant> task does
> > not allow to do it. Using of <runtarget> instead of <ant> saves memory
> > as it does not fork the ant's project.
> >
>
> What kind of failure processing it does?
Sets up the property, prints trace, and keeps execution going on.
> > > 5) Patching ANT - is there any workaround?
> >
> > It's a workaround itself. For Ant's basedir problem. Ant does not set
> > basedir properly if <ant> task executes the code containing <subant>
> > task. This path is taken from the bug report filed on both versions of
> > ant. Here it's supposed that 1.6.5 version is used. To be more correct
> > here we should check current ant's version and take suitable patch.
> >
>
> Oops ... does it mean that this workaround binds the infra to selected
> ANT version (in our case 1.6.5)? For example, if I have Ant 1.7
> installed the infra won't work?
Yes, I'm composing TODO list for BTI and it already has this item.
> Also ... I wonder why we haven't meet this issue in classlib or drlvm
> builds yet? It might mean that we use this Ant bug as a feature and
> once Ant get fixed ... need to check classlib and drlvm.
I've executed classlib's and drlvm's build system under patched ant -
everything worked fine. So there are no 'bug as feature' substitution.
> > > 6) And minor question: is there any strong reason for naming build
> > > scripts 'adaptor.xml' not 'build.xml'?
> >
> > I'm positioning the BTI framework not as an Ant-based Build System,
> > but as an Application implemented in Ant Scripting Language.
> > In build systems build.xml is a self-documenting file name.
> > In BTI the name adaptor.xml does the same - it documents itself.
> > I think it would be better to rename root build.xml to buildtest.xml.
>
> Sorry, I didn't catch about proposed infra positioning. Could you clarify?
What exactly needs clarification?
> BTW, if we rename build.xml=>buildtest.xml we have to type each time:
> ant -f buildtest.xml.
The standard usage of BTI supposes launching by means of shell scripts
(buildtest.bat/.sh), not by ant launcher.
Thanks,
Alexander
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Stepan Mishura <st...@gmail.com>.
On 4/17/07, Alexander Kleymenov wrote:
> Hi Stepan,
>
> Thanks for comments and your work with it!
> Please, look at my responses bellow.
>
> On 4/17/07, Stepan Mishura <st...@gmail.com> wrote:
> > Hi Alexander,
> >
> > I've reviewed almost all the code. And I have some comments and questions:
> >
> > 1) Is it possible to add modules 'drlvm-test' and 'classlib-drlvm'
> > that we have in the current infra? I'll help the difference between
> > approaches.
>
> I'll look at drlvm-test.
> I think classlib-drlvm is the same as launching of the new BTI with
> drlvm and classlib selected.
>
I'd like to have both of them :-)
> >
> > For example, the following difference for classlib confuses me a
> > little bit: the current infra launches classlib by simply calling
> > classlib's build with corresponding target:
> > <ant target="fetch-depends rebuild" ....
> > </ant>
> >
> > But the proposed infra does the next: calls adaptor.xml with 'run'
> > target (I'm OK with it). But the classlib's adaptor not just simply
> > calls classlib's build. It detects OS, set corresponding properties,
> > calls classlib's target 'fetch-depends' with <ant> and then calls
> > classlib's target 'rebuild' via <exec> task. So we have, for example
> > for Windows:
> > <ant target="fetch-depends" dir="${trunk.dir}"
> > inheritAll="false">
> > </ant>
> > <exec executable="cmd" dir="${trunk.dir}" failonerror="true">
> > <arg line="/c ${ant.home}/bin/ant.bat"/>
> > <arg line="rebuild"/>
> > </exec>
> >
> > What the reason for calling classlib's build in this way? Why we have
> > to run 'ant.bat' (or 'ant.sh' for Linux) via <exec>?
>
> It's workaround for Ant's OutOfMemory problem arising after doing the same
> by <ant> target. The <ant>-rebuild of classlib project causes big memory
> leaks somewhere in the Ant and the further BTI execution is impossible
> becouse of arising OOM.
>
As far as I remember the solution was to increase the max memory used
by Ant with setting ANT_OPTS to "-Xms256M -Xmx512M". Have you tried
this?
> >
> > 2) The build files have cyclic import dependencies:
> > framework.xml => cc-project.xml => framework.xml
> >
> > It complicates the build logic. Is it possible to avoid this?
>
> I agree, we should avoid such loops. Here it is implementation rather
> than design issue, so it should not take much efforts and time.
> Should I provide the fix for this?
>
Well, I think we can fix this latter.
> >
> > 3) In exec-adaptor.xml antconrib's wrapper 'trycatch' and 'runtarget'
> > task are used for calling adaptor's(i.e. suite's) target.
> >
> > <trycatch property="execution.exception">
> > <try>
> > <runtarget target="@{target}"/>
> > </try>
> > <catch>
> > Echo error message
> > </catch>
> > </trycatch>
> >
> > Again, why it is not possible use <ant> task? And this leads to the
> > next question.
>
> Infra needs some failure processing means here. Default <ant> task does
> not allow to do it. Using of <runtarget> instead of <ant> saves memory
> as it does not fork the ant's project.
>
What kind of failure processing it does?
> > 4) Using antcontrib. The proposed implementation uses it in many
> > places. But from my POV it is possible to use ANT tasks/targets/macros
> > instead. For example, I've picked up the first case from
> > 'framework.xml' that patches ANT:
> > <target name="patch-ant" depends="make-dirs, define-tasks">
> > <if>
> > <not><available file="${temp.files.dir}/SubAnt.java"/></not>
> > <then>
> > <get src="http://issues.apache.org/bugzilla/attachment.cgi?id=15681"
> > dest="${temp.files.dir}/SubAnt.java"
> > usetimestamp="true"
> > verbose="true"
> > />
> > </then>
> > </if>
> > ...
> > </ target >
> >
> > That IMHO can be easy converted to 'pure' ANT by creating target that
> > only runs if there is no SubAnt.java file and adding the target to
> > "patch-ant" decencies.
>
> Yes, it's possible to rewrite this simple case. But such a notation
> looks quite natural. And execution thread went to here assumes that
> ant-contrib has already been installed. So I chose simple notation
> rather than ant-contrib independent.
>
> > Actually, I don't want to start flame war - which approach is better.
> > I'm just trying to understand is it possible to avoid using antcontibs
> > task? IOW, is there any issue that can not be solved by using ANT
> > only?
>
> I tried to implement the framework without ant-contrib but such a way
> led to really complicated and unmaintainable code.
> I thought 'why to reinvent ant-contrib' and started to use it.
>
OK. I'll continue reviewing and if I find that it possible to avoid
using it I'll let you know.
> > 5) Patching ANT - is there any workaround?
>
> It's a workaround itself. For Ant's basedir problem. Ant does not set
> basedir properly if <ant> task executes the code containing <subant>
> task. This path is taken from the bug report filed on both versions of
> ant. Here it's supposed that 1.6.5 version is used. To be more correct
> here we should check current ant's version and take suitable patch.
>
Oops ... does it mean that this workaround binds the infra to selected
ANT version (in our case 1.6.5)? For example, if I have Ant 1.7
installed the infra won't work?
Also ... I wonder why we haven't meet this issue in classlib or drlvm
builds yet? It might mean that we use this Ant bug as a feature and
once Ant get fixed ... need to check classlib and drlvm.
> > 6) And minor question: is there any strong reason for naming build
> > scripts 'adaptor.xml' not 'build.xml'?
>
> I'm positioning the BTI framework not as an Ant-based Build System,
> but as an Application implemented in Ant Scripting Language.
> In build systems build.xml is a self-documenting file name.
> In BTI the name adaptor.xml does the same - it documents itself.
> I think it would be better to rename root build.xml to buildtest.xml.
>
Sorry, I didn't catch about proposed infra positioning. Could you clarify?
BTW, if we rename build.xml=>buildtest.xml we have to type each time:
ant -f buildtest.xml.
Thanks,
Stepan.
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Alexander Kleymenov <kl...@gmail.com>.
Hi Stepan,
Thanks for comments and your work with it!
Please, look at my responses bellow.
On 4/17/07, Stepan Mishura <st...@gmail.com> wrote:
> Hi Alexander,
>
> I've reviewed almost all the code. And I have some comments and questions:
>
> 1) Is it possible to add modules 'drlvm-test' and 'classlib-drlvm'
> that we have in the current infra? I'll help the difference between
> approaches.
I'll look at drlvm-test.
I think classlib-drlvm is the same as launching of the new BTI with
drlvm and classlib selected.
>
> For example, the following difference for classlib confuses me a
> little bit: the current infra launches classlib by simply calling
> classlib's build with corresponding target:
> <ant target="fetch-depends rebuild" ....
> </ant>
>
> But the proposed infra does the next: calls adaptor.xml with 'run'
> target (I'm OK with it). But the classlib's adaptor not just simply
> calls classlib's build. It detects OS, set corresponding properties,
> calls classlib's target 'fetch-depends' with <ant> and then calls
> classlib's target 'rebuild' via <exec> task. So we have, for example
> for Windows:
> <ant target="fetch-depends" dir="${trunk.dir}"
> inheritAll="false">
> </ant>
> <exec executable="cmd" dir="${trunk.dir}" failonerror="true">
> <arg line="/c ${ant.home}/bin/ant.bat"/>
> <arg line="rebuild"/>
> </exec>
>
> What the reason for calling classlib's build in this way? Why we have
> to run 'ant.bat' (or 'ant.sh' for Linux) via <exec>?
It's workaround for Ant's OutOfMemory problem arising after doing the same
by <ant> target. The <ant>-rebuild of classlib project causes big memory
leaks somewhere in the Ant and the further BTI execution is impossible
becouse of arising OOM.
>
> 2) The build files have cyclic import dependencies:
> framework.xml => cc-project.xml => framework.xml
>
> It complicates the build logic. Is it possible to avoid this?
I agree, we should avoid such loops. Here it is implementation rather
than design issue, so it should not take much efforts and time.
Should I provide the fix for this?
>
> 3) In exec-adaptor.xml antconrib's wrapper 'trycatch' and 'runtarget'
> task are used for calling adaptor's(i.e. suite's) target.
>
> <trycatch property="execution.exception">
> <try>
> <runtarget target="@{target}"/>
> </try>
> <catch>
> Echo error message
> </catch>
> </trycatch>
>
> Again, why it is not possible use <ant> task? And this leads to the
> next question.
Infra needs some failure processing means here. Default <ant> task does
not allow to do it. Using of <runtarget> instead of <ant> saves memory
as it does not fork the ant's project.
> 4) Using antcontrib. The proposed implementation uses it in many
> places. But from my POV it is possible to use ANT tasks/targets/macros
> instead. For example, I've picked up the first case from
> 'framework.xml' that patches ANT:
> <target name="patch-ant" depends="make-dirs, define-tasks">
> <if>
> <not><available file="${temp.files.dir}/SubAnt.java"/></not>
> <then>
> <get src="http://issues.apache.org/bugzilla/attachment.cgi?id=15681"
> dest="${temp.files.dir}/SubAnt.java"
> usetimestamp="true"
> verbose="true"
> />
> </then>
> </if>
> ...
> </ target >
>
> That IMHO can be easy converted to 'pure' ANT by creating target that
> only runs if there is no SubAnt.java file and adding the target to
> "patch-ant" decencies.
Yes, it's possible to rewrite this simple case. But such a notation
looks quite natural. And execution thread went to here assumes that
ant-contrib has already been installed. So I chose simple notation
rather than ant-contrib independent.
> Actually, I don't want to start flame war - which approach is better.
> I'm just trying to understand is it possible to avoid using antcontibs
> task? IOW, is there any issue that can not be solved by using ANT
> only?
I tried to implement the framework without ant-contrib but such a way
led to really complicated and unmaintainable code.
I thought 'why to reinvent ant-contrib' and started to use it.
> 5) Patching ANT - is there any workaround?
It's a workaround itself. For Ant's basedir problem. Ant does not set
basedir properly if <ant> task executes the code containing <subant>
task. This path is taken from the bug report filed on both versions of
ant. Here it's supposed that 1.6.5 version is used. To be more correct
here we should check current ant's version and take suitable patch.
> 6) And minor question: is there any strong reason for naming build
> scripts 'adaptor.xml' not 'build.xml'?
I'm positioning the BTI framework not as an Ant-based Build System,
but as an Application implemented in Ant Scripting Language.
In build systems build.xml is a self-documenting file name.
In BTI the name adaptor.xml does the same - it documents itself.
I think it would be better to rename root build.xml to buildtest.xml.
Thank you,
Alexander
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by Stepan Mishura <st...@gmail.com>.
Hi Alexander,
I've reviewed almost all the code. And I have some comments and questions:
1) Is it possible to add modules 'drlvm-test' and 'classlib-drlvm'
that we have in the current infra? I'll help the difference between
approaches.
For example, the following difference for classlib confuses me a
little bit: the current infra launches classlib by simply calling
classlib's build with corresponding target:
<ant target="fetch-depends rebuild" ....
</ant>
But the proposed infra does the next: calls adaptor.xml with 'run'
target (I'm OK with it). But the classlib's adaptor not just simply
calls classlib's build. It detects OS, set corresponding properties,
calls classlib's target 'fetch-depends' with <ant> and then calls
classlib's target 'rebuild' via <exec> task. So we have, for example
for Windows:
<ant target="fetch-depends" dir="${trunk.dir}"
inheritAll="false">
</ant>
<exec executable="cmd" dir="${trunk.dir}" failonerror="true">
<arg line="/c ${ant.home}/bin/ant.bat"/>
<arg line="rebuild"/>
</exec>
What the reason for calling classlib's build in this way? Why we have
to run 'ant.bat' (or 'ant.sh' for Linux) via <exec>?
2) The build files have cyclic import dependencies:
framework.xml => cc-project.xml => framework.xml
It complicates the build logic. Is it possible to avoid this?
3) In exec-adaptor.xml antconrib's wrapper 'trycatch' and 'runtarget'
task are used for calling adaptor's(i.e. suite's) target.
<trycatch property="execution.exception">
<try>
<runtarget target="@{target}"/>
</try>
<catch>
Echo error message
</catch>
</trycatch>
Again, why it is not possible use <ant> task? And this leads to the
next question.
4) Using antcontrib. The proposed implementation uses it in many
places. But from my POV it is possible to use ANT tasks/targets/macros
instead. For example, I've picked up the first case from
'framework.xml' that patches ANT:
<target name="patch-ant" depends="make-dirs, define-tasks">
<if>
<not><available file="${temp.files.dir}/SubAnt.java"/></not>
<then>
<get src="http://issues.apache.org/bugzilla/attachment.cgi?id=15681"
dest="${temp.files.dir}/SubAnt.java"
usetimestamp="true"
verbose="true"
/>
</then>
</if>
...
</ target >
That IMHO can be easy converted to 'pure' ANT by creating target that
only runs if there is no SubAnt.java file and adding the target to
"patch-ant" decencies.
Actually, I don't want to start flame war - which approach is better.
I'm just trying to understand is it possible to avoid using antcontibs
task? IOW, is there any issue that can not be solved by using ANT
only?
5) Patching ANT - is there any workaround?
6) And minor question: is there any strong reason for naming build
scripts 'adaptor.xml' not 'build.xml'?
Thanks,
Stepan.
On 3/27/07, Alexander Kleymenov <kl...@gmail.com> wrote:
> Hello,
>
> With this letter I'd like to announce the new approach for BT Framework
> Implementation. It could be found as an attachment to JIRA #3501 [1].
>
> Proposed BT Infra Implementation improves and structures current one and brings
> the following features allowing using the framework as a standard means for
> Harmony Testing works:
>
> - Simplicity and convenience of use allows to perform all of the testing
> works (by testers and developers) by unified single way and with using
> of the same means.
>
> - Clear and Documented Architecture, Workspace Structure, Clear separation
> of Functional Components of the Framework, Standard Implementation Means
> allows to easy start, use, improve, and maintain the framework.
>
> - Highly configurable Test Run:
>
> Simple means for Test Suite selections - use property value to select
> required Test Suites. Only selected suites will be checked out (or
> downloaded) and executed.
>
> Simple and easy manageable means for settings dependencies between Test
> Suites. It excludes the necessity of combo-test-suites creation (such as
> classlib-drlvm now)
>
> Simple and natural means for describing of required parameters for selected
> Test Suites (as path to tested JVM, Classlib, compiler checks, and so on).
> It is done in one specified place and there is no need to drill into Test
> Suites for setting up such a parameters. Moreover, new BT Framework performs
> the correctness analysis of the provided values for parameters and reports
> an error just before Test Suite execution, not after an hour of the work.
> Additional important feature is an ability to provide default values for
> required parameters on the base of execution results of another Test Suite.
> So, for example, drlvm Test Suite (which checks that DRLVM can be built)
> uses the default value for 'classlib.trunk' parameter (specifying
> the path to
> classlib workspace) the value provided by classlib Test Suite. If classlib
> Test Suite was chosen for execution, 'classlib.trunk' parameter of drlvm
> test suite will be resolved. If classlib was not chosen, user will be
> asked to provide the value for this parameter. Such a feature allows to
> reduce the user's job on parameters setting.
>
> Such a simplicity of Test Run configuration (selection of the suites and
> parameters tuning) allows to easy set up the BT Framework for specific
> needs. So for Harmony developers it is possible to chose their own sets of
> checks of the workspace under development and to run this checks on demand
> in single execution mode. For Harmony testers it is possible to use
> sophisticated CruiseControl configurations for continuous test runs with
> notification publishing.
>
> - Easy and Documented way to integrate new Test Suites:
>
> There is no longer need to implement CruiseControl configurations for
> each of new Test Suites being integrated into framework. Cruise Control
> system is used as a framework extension allowing continuous test runs, not
> as a development means. It simplifies the process and reduces the time
> of new Test Suite integration.
>
> - Implemented Approach to share External Library Dependencies
> between different
> Test Suites allows to perform external dependencies fetching in one place by
> one means and to save time and internet traffic.
>
> - To run custom configuration there is no need to download
> all of the Test Suites integrated into BT.
>
> - The Architecture of the Framework is open for extending by other functional
> parts and 3rd party tools (such as CruiseControl etc)
>
> The main idea of the new Framework is to use Ant's scripts called 'adaptors' as
> a glue layer between Test Suites and the BT Framework executing them. All of the
> Parameters of the suites (such as required values, external libraries, shared
> values to be used by other suites, optional parameters for low-level tuning)
> are described in 'parameters.xml' file placed near corresponding 'adaptor.xml'.
> That's all.
>
> Please, look at the archive. There is a README.txt and SPEC.txt files describing
> and documenting the new Infrastructure. Also there are two integrated Test
> Suites - classlib and drlvm which check that classlib and drlvm can be built.
> Please look at them under adapters directory to catch the main idea of the
> Framework.
>
> Of course there are many points for improvements, but the main idea and
> functionality has already been implemented. Sorry for such a bulk, but I could
> not provide the framework by 'logical parts' because they together forms a
> whole. Now, when all of them in one place it is possible to improve or
> reimplement them (I heard about the existing approach for external libraries
> sharing, but did not look at it. Probably it is better then proposed in my BT
> Infra implementation. In this case it is possible to replace the implementation
> of this functional part of the Framework).
>
> If you have any questions regarding the implementation please,
> provide them into this mail-thread.
>
> Thank you,
> Alexander
>
> [1] http://issues.apache.org/jira/browse/HARMONY-3501
>
--
Stepan Mishura
Intel Enterprise Solutions Software Division
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by tatyana doubtsova <ta...@gmail.com>.
Hi,
I also adjusted class library tests iterative runs to BT 2.0.
It is avalabe in https://issues.apache.org/jira/browse/HARMONY-3629
Thanks,
Tanya
On 4/1/07, tatyana doubtsova <ta...@gmail.com> wrote:
>
> Hi,
>
> I would like to inform you that I connected reliability test suite (
> https://issues.apache.org/jira/browse/HARMONY-2918) to BT 2.0.
> The archive with the adaptor and custom configuration to run underCruise
> Control is available in https://issues.apache.org/jira/browse/HARMONY-3539
> .
>
> The suggested approach seems to be user friendly and clear. Thanks to
> Alexander!
>
> Tanya.
>
>
> On 3/30/07, Ivaschenko, Aleksei V <al...@intel.com> wrote:
> >
> > I've just tried out new BT Framework with my automated scenario for
> > Jetty application server. I found new framework quite usable - I've
> > integrated my scenario without studying cruise control tool, and
> > implemented dependencies on other workloads such as classlib and drlvm
> > builds without big problems.
> >
> > The working implementation of integration scripts could be found here:
> > https://issues.apache.org/jira/browse/HARMONY-3530
> >
> > Thanks,
> > Aleksei.
> >
> > >-----Original Message-----
> > >From: Alexander Kleymenov [mailto: kleymenov@gmail.com]
> > >Sent: Tuesday, March 27, 2007 1:44 PM
> > >To: dev@harmony.apache.org
> > >Subject: [buildtest] Proposal for Build Test Infrastructure Improvement
> >
> > >
> > >Hello,
> > >
> > >With this letter I'd like to announce the new approach for BT Framework
> > >Implementation. It could be found as an attachment to JIRA #3501 [1].
> > >
> > >Proposed BT Infra Implementation improves and structures current one
> > and
> > >brings
> > >the following features allowing using the framework as a standard means
> > for
> > >Harmony Testing works:
> > >
> > > - Simplicity and convenience of use allows to perform all of the
> > testing
> > > works (by testers and developers) by unified single way and with
> > using
> > > of the same means.
> > >
> > > - Clear and Documented Architecture, Workspace Structure, Clear
> > >separation
> > > of Functional Components of the Framework, Standard Implementation
> > >Means
> > > allows to easy start, use, improve, and maintain the framework.
> > >
> > > - Highly configurable Test Run:
> > >
> > > Simple means for Test Suite selections - use property value to
> > select
> > > required Test Suites. Only selected suites will be checked out (or
> > > downloaded) and executed.
> > >
> > > Simple and easy manageable means for settings dependencies between
> > Test
> > > Suites. It excludes the necessity of combo-test-suites creation
> > (such
> > >as
> > > classlib-drlvm now)
> > >
> > > Simple and natural means for describing of required parameters for
> > >selected
> > > Test Suites (as path to tested JVM, Classlib, compiler checks, and
> > so
> > >on).
> > > It is done in one specified place and there is no need to drill
> > into
> > >Test
> > > Suites for setting up such a parameters. Moreover, new BT Framework
> >
> > >performs
> > > the correctness analysis of the provided values for parameters and
> > >reports
> > > an error just before Test Suite execution, not after an hour of the
> > >work.
> > > Additional important feature is an ability to provide default
> > values
> > >for
> > > required parameters on the base of execution results of another
> > Test
> > >Suite.
> > > So, for example, drlvm Test Suite (which checks that DRLVM can be
> > >built)
> > > uses the default value for ' classlib.trunk' parameter (specifying
> > >the path to
> > > classlib workspace) the value provided by classlib Test Suite. If
> > >classlib
> > > Test Suite was chosen for execution, 'classlib.trunk ' parameter of
> > >drlvm
> > > test suite will be resolved. If classlib was not chosen, user will
> > be
> > > asked to provide the value for this parameter. Such a feature
> > allows to
> > > reduce the user's job on parameters setting.
> > >
> > > Such a simplicity of Test Run configuration (selection of the
> > suites
> > >and
> > > parameters tuning) allows to easy set up the BT Framework for
> > specific
> > > needs. So for Harmony developers it is possible to chose their own
> > sets
> > >of
> > > checks of the workspace under development and to run this checks on
> > >demand
> > > in single execution mode. For Harmony testers it is possible to use
> > > sophisticated CruiseControl configurations for continuous test runs
> >
> > >with
> > > notification publishing.
> > >
> > > - Easy and Documented way to integrate new Test Suites:
> > >
> > > There is no longer need to implement CruiseControl configurations
> > for
> > > each of new Test Suites being integrated into framework. Cruise
> > Control
> > > system is used as a framework extension allowing continuous test
> > runs,
> > >not
> > > as a development means. It simplifies the process and reduces the
> > time
> > > of new Test Suite integration.
> > >
> > > - Implemented Approach to share External Library Dependencies
> > >between different
> > > Test Suites allows to perform external dependencies fetching in one
> > >place by
> > > one means and to save time and internet traffic.
> > >
> > > - To run custom configuration there is no need to download
> > > all of the Test Suites integrated into BT.
> > >
> > > - The Architecture of the Framework is open for extending by other
> > >functional
> > > parts and 3rd party tools (such as CruiseControl etc)
> > >
> > >The main idea of the new Framework is to use Ant's scripts called
> > >'adaptors' as
> > >a glue layer between Test Suites and the BT Framework executing them.
> > All
> > >of the
> > >Parameters of the suites (such as required values, external libraries,
> > >shared
> > >values to be used by other suites, optional parameters for low-level
> > >tuning)
> > >are described in ' parameters.xml' file placed near corresponding
> > >'adaptor.xml'.
> > >That's all.
> > >
> > >Please, look at the archive. There is a README.txt and SPEC.txt files
> > >describing
> > >and documenting the new Infrastructure. Also there are two integrated
> > Test
> > >Suites - classlib and drlvm which check that classlib and drlvm can be
> > >built.
> > >Please look at them under adapters directory to catch the main idea of
> > the
> > >Framework.
> > >
> > >Of course there are many points for improvements, but the main idea and
> >
> > >functionality has already been implemented. Sorry for such a bulk, but
> > I
> > >could
> > >not provide the framework by 'logical parts' because they together
> > forms a
> > >whole. Now, when all of them in one place it is possible to improve or
> > >reimplement them (I heard about the existing approach for external
> > >libraries
> > >sharing, but did not look at it. Probably it is better then proposed
> > in my
> > >BT
> > >Infra implementation. In this case it is possible to replace the
> > >implementation
> > >of this functional part of the Framework).
> > >
> > >If you have any questions regarding the implementation please,
> > >provide them into this mail-thread.
> > >
> > >Thank you,
> > >Alexander
> > >
> > >[1] http://issues.apache.org/jira/browse/HARMONY-3501
> >
>
>
Re: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by tatyana doubtsova <ta...@gmail.com>.
Hi,
I would like to inform you that I connected reliability test suite (
https://issues.apache.org/jira/browse/HARMONY-2918) to BT 2.0.
The archive with the adaptor and custom configuration to run underCruise
Control is available in https://issues.apache.org/jira/browse/HARMONY-3539.
The suggested approach seems to be user friendly and clear. Thanks to
Alexander!
Tanya.
On 3/30/07, Ivaschenko, Aleksei V <al...@intel.com> wrote:
>
> I've just tried out new BT Framework with my automated scenario for
> Jetty application server. I found new framework quite usable - I've
> integrated my scenario without studying cruise control tool, and
> implemented dependencies on other workloads such as classlib and drlvm
> builds without big problems.
>
> The working implementation of integration scripts could be found here:
> https://issues.apache.org/jira/browse/HARMONY-3530
>
> Thanks,
> Aleksei.
>
> >-----Original Message-----
> >From: Alexander Kleymenov [mailto:kleymenov@gmail.com]
> >Sent: Tuesday, March 27, 2007 1:44 PM
> >To: dev@harmony.apache.org
> >Subject: [buildtest] Proposal for Build Test Infrastructure Improvement
> >
> >Hello,
> >
> >With this letter I'd like to announce the new approach for BT Framework
> >Implementation. It could be found as an attachment to JIRA #3501 [1].
> >
> >Proposed BT Infra Implementation improves and structures current one
> and
> >brings
> >the following features allowing using the framework as a standard means
> for
> >Harmony Testing works:
> >
> > - Simplicity and convenience of use allows to perform all of the
> testing
> > works (by testers and developers) by unified single way and with
> using
> > of the same means.
> >
> > - Clear and Documented Architecture, Workspace Structure, Clear
> >separation
> > of Functional Components of the Framework, Standard Implementation
> >Means
> > allows to easy start, use, improve, and maintain the framework.
> >
> > - Highly configurable Test Run:
> >
> > Simple means for Test Suite selections - use property value to
> select
> > required Test Suites. Only selected suites will be checked out (or
> > downloaded) and executed.
> >
> > Simple and easy manageable means for settings dependencies between
> Test
> > Suites. It excludes the necessity of combo-test-suites creation
> (such
> >as
> > classlib-drlvm now)
> >
> > Simple and natural means for describing of required parameters for
> >selected
> > Test Suites (as path to tested JVM, Classlib, compiler checks, and
> so
> >on).
> > It is done in one specified place and there is no need to drill
> into
> >Test
> > Suites for setting up such a parameters. Moreover, new BT Framework
> >performs
> > the correctness analysis of the provided values for parameters and
> >reports
> > an error just before Test Suite execution, not after an hour of the
> >work.
> > Additional important feature is an ability to provide default
> values
> >for
> > required parameters on the base of execution results of another
> Test
> >Suite.
> > So, for example, drlvm Test Suite (which checks that DRLVM can be
> >built)
> > uses the default value for 'classlib.trunk' parameter (specifying
> >the path to
> > classlib workspace) the value provided by classlib Test Suite. If
> >classlib
> > Test Suite was chosen for execution, 'classlib.trunk' parameter of
> >drlvm
> > test suite will be resolved. If classlib was not chosen, user will
> be
> > asked to provide the value for this parameter. Such a feature
> allows to
> > reduce the user's job on parameters setting.
> >
> > Such a simplicity of Test Run configuration (selection of the
> suites
> >and
> > parameters tuning) allows to easy set up the BT Framework for
> specific
> > needs. So for Harmony developers it is possible to chose their own
> sets
> >of
> > checks of the workspace under development and to run this checks on
> >demand
> > in single execution mode. For Harmony testers it is possible to use
> > sophisticated CruiseControl configurations for continuous test runs
> >with
> > notification publishing.
> >
> > - Easy and Documented way to integrate new Test Suites:
> >
> > There is no longer need to implement CruiseControl configurations
> for
> > each of new Test Suites being integrated into framework. Cruise
> Control
> > system is used as a framework extension allowing continuous test
> runs,
> >not
> > as a development means. It simplifies the process and reduces the
> time
> > of new Test Suite integration.
> >
> > - Implemented Approach to share External Library Dependencies
> >between different
> > Test Suites allows to perform external dependencies fetching in one
> >place by
> > one means and to save time and internet traffic.
> >
> > - To run custom configuration there is no need to download
> > all of the Test Suites integrated into BT.
> >
> > - The Architecture of the Framework is open for extending by other
> >functional
> > parts and 3rd party tools (such as CruiseControl etc)
> >
> >The main idea of the new Framework is to use Ant's scripts called
> >'adaptors' as
> >a glue layer between Test Suites and the BT Framework executing them.
> All
> >of the
> >Parameters of the suites (such as required values, external libraries,
> >shared
> >values to be used by other suites, optional parameters for low-level
> >tuning)
> >are described in 'parameters.xml' file placed near corresponding
> >'adaptor.xml'.
> >That's all.
> >
> >Please, look at the archive. There is a README.txt and SPEC.txt files
> >describing
> >and documenting the new Infrastructure. Also there are two integrated
> Test
> >Suites - classlib and drlvm which check that classlib and drlvm can be
> >built.
> >Please look at them under adapters directory to catch the main idea of
> the
> >Framework.
> >
> >Of course there are many points for improvements, but the main idea and
> >functionality has already been implemented. Sorry for such a bulk, but
> I
> >could
> >not provide the framework by 'logical parts' because they together
> forms a
> >whole. Now, when all of them in one place it is possible to improve or
> >reimplement them (I heard about the existing approach for external
> >libraries
> >sharing, but did not look at it. Probably it is better then proposed
> in my
> >BT
> >Infra implementation. In this case it is possible to replace the
> >implementation
> >of this functional part of the Framework).
> >
> >If you have any questions regarding the implementation please,
> >provide them into this mail-thread.
> >
> >Thank you,
> >Alexander
> >
> >[1] http://issues.apache.org/jira/browse/HARMONY-3501
>
RE: [buildtest] Proposal for Build Test Infrastructure Improvement
Posted by "Ivaschenko, Aleksei V" <al...@intel.com>.
I've just tried out new BT Framework with my automated scenario for
Jetty application server. I found new framework quite usable - I've
integrated my scenario without studying cruise control tool, and
implemented dependencies on other workloads such as classlib and drlvm
builds without big problems.
The working implementation of integration scripts could be found here:
https://issues.apache.org/jira/browse/HARMONY-3530
Thanks,
Aleksei.
>-----Original Message-----
>From: Alexander Kleymenov [mailto:kleymenov@gmail.com]
>Sent: Tuesday, March 27, 2007 1:44 PM
>To: dev@harmony.apache.org
>Subject: [buildtest] Proposal for Build Test Infrastructure Improvement
>
>Hello,
>
>With this letter I'd like to announce the new approach for BT Framework
>Implementation. It could be found as an attachment to JIRA #3501 [1].
>
>Proposed BT Infra Implementation improves and structures current one
and
>brings
>the following features allowing using the framework as a standard means
for
>Harmony Testing works:
>
> - Simplicity and convenience of use allows to perform all of the
testing
> works (by testers and developers) by unified single way and with
using
> of the same means.
>
> - Clear and Documented Architecture, Workspace Structure, Clear
>separation
> of Functional Components of the Framework, Standard Implementation
>Means
> allows to easy start, use, improve, and maintain the framework.
>
> - Highly configurable Test Run:
>
> Simple means for Test Suite selections - use property value to
select
> required Test Suites. Only selected suites will be checked out (or
> downloaded) and executed.
>
> Simple and easy manageable means for settings dependencies between
Test
> Suites. It excludes the necessity of combo-test-suites creation
(such
>as
> classlib-drlvm now)
>
> Simple and natural means for describing of required parameters for
>selected
> Test Suites (as path to tested JVM, Classlib, compiler checks, and
so
>on).
> It is done in one specified place and there is no need to drill
into
>Test
> Suites for setting up such a parameters. Moreover, new BT Framework
>performs
> the correctness analysis of the provided values for parameters and
>reports
> an error just before Test Suite execution, not after an hour of the
>work.
> Additional important feature is an ability to provide default
values
>for
> required parameters on the base of execution results of another
Test
>Suite.
> So, for example, drlvm Test Suite (which checks that DRLVM can be
>built)
> uses the default value for 'classlib.trunk' parameter (specifying
>the path to
> classlib workspace) the value provided by classlib Test Suite. If
>classlib
> Test Suite was chosen for execution, 'classlib.trunk' parameter of
>drlvm
> test suite will be resolved. If classlib was not chosen, user will
be
> asked to provide the value for this parameter. Such a feature
allows to
> reduce the user's job on parameters setting.
>
> Such a simplicity of Test Run configuration (selection of the
suites
>and
> parameters tuning) allows to easy set up the BT Framework for
specific
> needs. So for Harmony developers it is possible to chose their own
sets
>of
> checks of the workspace under development and to run this checks on
>demand
> in single execution mode. For Harmony testers it is possible to use
> sophisticated CruiseControl configurations for continuous test runs
>with
> notification publishing.
>
> - Easy and Documented way to integrate new Test Suites:
>
> There is no longer need to implement CruiseControl configurations
for
> each of new Test Suites being integrated into framework. Cruise
Control
> system is used as a framework extension allowing continuous test
runs,
>not
> as a development means. It simplifies the process and reduces the
time
> of new Test Suite integration.
>
> - Implemented Approach to share External Library Dependencies
>between different
> Test Suites allows to perform external dependencies fetching in one
>place by
> one means and to save time and internet traffic.
>
> - To run custom configuration there is no need to download
> all of the Test Suites integrated into BT.
>
> - The Architecture of the Framework is open for extending by other
>functional
> parts and 3rd party tools (such as CruiseControl etc)
>
>The main idea of the new Framework is to use Ant's scripts called
>'adaptors' as
>a glue layer between Test Suites and the BT Framework executing them.
All
>of the
>Parameters of the suites (such as required values, external libraries,
>shared
>values to be used by other suites, optional parameters for low-level
>tuning)
>are described in 'parameters.xml' file placed near corresponding
>'adaptor.xml'.
>That's all.
>
>Please, look at the archive. There is a README.txt and SPEC.txt files
>describing
>and documenting the new Infrastructure. Also there are two integrated
Test
>Suites - classlib and drlvm which check that classlib and drlvm can be
>built.
>Please look at them under adapters directory to catch the main idea of
the
>Framework.
>
>Of course there are many points for improvements, but the main idea and
>functionality has already been implemented. Sorry for such a bulk, but
I
>could
>not provide the framework by 'logical parts' because they together
forms a
>whole. Now, when all of them in one place it is possible to improve or
>reimplement them (I heard about the existing approach for external
>libraries
>sharing, but did not look at it. Probably it is better then proposed
in my
>BT
>Infra implementation. In this case it is possible to replace the
>implementation
>of this functional part of the Framework).
>
>If you have any questions regarding the implementation please,
>provide them into this mail-thread.
>
>Thank you,
>Alexander
>
>[1] http://issues.apache.org/jira/browse/HARMONY-3501