You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Alexander Kleymenov <kl...@gmail.com> on 2007/04/27 08:09:50 UTC
[bti] Harmony Build and Testing Infrastructure - Directions for further development
Hi,
Here I'm trying to describe the main directions of further BTI development.
Along with bug fixing and improvement works I see the following high priority
directions affecting general BTI usage model and the works on new Test Suites
(TS) integrations:
1. As BTI is under SVN, its usage model should be actualized to behave
according to Proposed Use Cases.
(for more info see README.txt, section "Test Execution (post
integration vision)")
2. At present each of the TS integrations implements its own Execution Status
Notification functionality. It could be better to implement publishing
functionality
as a separate BTI component and [re]use it by TS integrations.
Among the other obvious gains it will:
- eliminate code doubling among the suites,
- speed up and simplify the integration scripts development process,
- make the integration scripts more clear and less error-prone,
- unify the notification message format,
- allow to concentrate all the publishing configuration functionality
in one place.
3. Documentation, Documentation, Documentation.
Another actual works on BTI are:
- Support for multiple default values for required parameters.
- Optional parameters management/processing.
- Support for conditional external dependencies fetching.
- Allocate the functionality 'supporting' Test Suites Integration Scripts
in the special BTI module (say support.xml). This module could contain the
functionality performing the work with SVN, Ant execution, and other
procedures reused among the many Test Suites integration scripts.
- Bug fixing
Thanks,
Alexander
Re: [bti] Harmony Build and Testing Infrastructure - Directions for further development
Posted by Alexander Kleymenov <kl...@gmail.com>.
On 5/17/07, Nathan Beyer <nd...@apache.org> wrote:
> Can we work on getting some valid instructions for what's currently
> there? I've tried doing what the README [1] suggests, but it just
> doesn't work.
> Under "Using BT Infrastructure" it has steps for setting stuff up, but
> I can't even get passed step 2. Granted, the document says something
> about "vision", but I'd like to see some instructions that actually
> work, so I can get it to work.
You are right, the documentation is obsolete. I'll try to find a time
to update it. Now, to start work with BTI the following steps should be
performed:
1. Checkout the BTI repository:
$ svn checkout -r HEAD
http://svn.apache.org/repos/asf/harmony/enhanced/buildtest/branches/2.0
2. Tune environment variables in buildtest.(bat|sh)
3. Perform BTI installation (now, it just creates
framework.local.properties file)
$ buildtest install
4. Configure framework in framework.local.properties file (specify mail
addresses for notifications)
5. Do setup step. It will generate required-parameters.properties file
and ask to provide values for required parameters (if they are needed).
$ buildtest -Dtest.suites="classlib,drlvm,scimark" setup
Note: this combination of selected test suites won't demand any required
parameters because BTI will automatically set them up.
6. Do run:
$ buildtest -Dtest.suites="classlib,drlvm,scimark" run
for single run, or
$ buildtest -Dtest.suites="classlib,drlvm,scimark" run-cc
for CC-controlled run.
7. Enjoy?
>
> Additionally, I find that having two targets "install" and "setup" in
> the main build file to be very confusing. Which should be run first?
> "install" doesn't seem to do anything. "setup" runs, but bombs out the
> first time saying that required parameters are needed, but the
> 'required-parameters.properties' file isn't generated until after
> "setup" is run. Ahh, the insanity!
The supposed (but not currently fully implemented) BTI usage is as follows:
1. User makes non-recursive checkout of BTI:
$ svn checkout -N ...
it helps us to avoid checkout of all the test sources from ${BTI_ROOT}/tests
directory (we just do not need all of them).
2. Then User tunes environment in buildtest.(bat|sh) and performs
the BTI installation:
$ buildtest install
it:
- instals the BTI on local workstation (does checkout of scripts
directory and makes necessary configuration files such as
framework.local.properties)
- checkouts test suites adaptors (${BTI_ROOT}/adaptors) which are the
glue between BTI execution framework and actual Test Suites.
3. Then User performs the setup of test run (selects the Test Suites to execute)
$ buildtest -Dtest.suites="classlib,drlvm,scimark" setup
it:
- selects test suites to launch
- downloads all required external dependencies (if local substitution
were not provided)
- process required parameters for selected suites, place them into
required-parameters.properties file and, if there are some unspecified
parameter values, demand them from User (you have been asked for
them exactly at this point).
4. If necessary, User:
- provides the values for required parameters in
required-parameters.properties file
- configures BTI framework in framework.local.properties file (such as
mail addresses for notifications, ports, log level, and so on)
5. Finally, User launches the Test Run execution:
$ buildtest run
- it will call adaptors' "setup" targets for all of the selected test
suites, and then consequently 'run' them
or
$ buildtest run-cc
- it will call adaptors "setup" targets for all of the selected test
suites, and then launch CC-controlled continuous execution
process.
6. To select and execute another suites, go to step 3.
This is an ideal view of BTI's Use-Case. I've described this task as a
high-priority direction of further BTI development.
I'm going to work on it after several bugfixings and there is a time to
discuss this Use-Case.
Thanks for feedback,
Alexander
Re: [bti] Harmony Build and Testing Infrastructure - Directions for further development
Posted by Nathan Beyer <nd...@apache.org>.
Can we work on getting some valid instructions for what's currently
there? I've tried doing what the README [1] suggests, but it just
doesn't work.
Under "Using BT Infrastructure" it has steps for setting stuff up, but
I can't even get passed step 2. Granted, the document says something
about "vision", but I'd like to see some instructions that actually
work, so I can get it to work.
Additionally, I find that having two targets "install" and "setup" in
the main build file to be very confusing. Which should be run first?
"install" doesn't seem to do anything. "setup" runs, but bombs out the
first time saying that required parameters are needed, but the
'required-parameters.properties' file isn't generated until after
"setup" is run. Ahh, the insanity!
-Nathan
[1] http://svn.apache.org/repos/asf/harmony/enhanced/buildtest/branches/2.0/README.txt
On 5/16/07, Alexander Kleymenov <kl...@gmail.com> wrote:
> Hi,
>
> As a BTI Status Update I'd like to describe the works currently done on it.
>
> First of all - notification publishing functionality has been implemented on
> the framework's side. Now, if Integration Scripts for some Test Suite
> (TS) do not implement status notification publisher, the default framework's
> means will be used. The notifications sent by default publisher include
> the following information: build status, build failure message,
> errors/warnings occurred during the execution, and files changed in SVN
> repository since last successful build.
>
> The following notification configuration parameters are provided by
> framework:
>
> - it's possible to switch between notification sending mode: to send
> notifications every time after the TS execution, or only if its
> execution status has been changed from failed to passed, or from
> passed
> to failed. This configuration is done by means of the following
> framework's parameter property:
> framework.parameters.usemail.always=true
> specified in framework.local.properties file in the root BTI
> directory.
>
> - it's possible set up mail server, 'to' and 'from' addresses,
> notification
> subject prefix by means of the following framework's parameter
> properties:
> framework.parameters.usemail=smtp.server
> framework.parameters.usemail.to=to@address.org
> framework.parameters.usemail.from=from@address.org
> framework.parameters.usemail.subject.prefix=[build-test alert]
>
> Each particular Test Suite can customize the default message by specific
> summary information, can provide attachments to the notification, and
> substitute
> default notification message with its own.
>
> All this is configured by the following TS-specific parameters:
>
> ${test.suite}.parameters.cc.summary
> - if contains the path to file, the content of this file will be
> prepended
> to the status notification message.
> ${test.suite}.parameters.cc.message
> - if contains the path to file, the content of this file will be used
> instead of the default message.
> ${test.suite}.parameters.cc.attachs
> - coma or space separated list of absolute paths to files to send with
> notification.
> ${test.suite}.parameters.cc.to
> ${test.suite}.parameters.cc.from
> - if defined and contains email addresses, will be used for mail
> notification instead of corresponding global framework's parameters
> values.
>
> The values for summary, message, and attachs parameters should be provided
> in test suite's parameters.xml files (as it is mostly Test Suite's
> implementation-specific information). The values for 'to' and 'from'
> parameters should be provided in framework.local.properties file as it is
> local BTI configuration information.
>
> There is 'test-mail' target in the root build.xml intended to test the
> correctness of mail configuration before BTI execution.
>
> Secondly, some improvements supporting Test Suites execution process have
> been
> implemented. So <exec-ant> subroutine launching the ant build process has
> been
> implemented. It wraps the logic used to call and process the execution
> status
> and output of external ant scripts in one place, and implements some
> workarounds
> on common problems with absence of junit.jar in ANT_HOME/lib directory,
> proxy
> settings, and other.
>
> Now along with further TODO's implementation I'm checking the integration
> scripts for test suites not integrated in SVN. I'm actualizing them, making
> them more BTI compliant and, if it is possible, simplifying them.
>
> If you have any comments regarding implemented improvements, TODO list,
> further development directions, and Test Suite integration process,
> please, provide them.
>
> Thanks,
> Alexander
>
Re: [bti] Harmony Build and Testing Infrastructure - Directions for further development
Posted by Alexander Kleymenov <kl...@gmail.com>.
Hi,
As a BTI Status Update I'd like to describe the works currently done on it.
First of all - notification publishing functionality has been implemented on
the framework's side. Now, if Integration Scripts for some Test Suite
(TS) do not implement status notification publisher, the default framework's
means will be used. The notifications sent by default publisher include
the following information: build status, build failure message,
errors/warnings occurred during the execution, and files changed in SVN
repository since last successful build.
The following notification configuration parameters are provided by
framework:
- it's possible to switch between notification sending mode: to send
notifications every time after the TS execution, or only if its
execution status has been changed from failed to passed, or from
passed
to failed. This configuration is done by means of the following
framework's parameter property:
framework.parameters.usemail.always=true
specified in framework.local.properties file in the root BTI
directory.
- it's possible set up mail server, 'to' and 'from' addresses,
notification
subject prefix by means of the following framework's parameter
properties:
framework.parameters.usemail=smtp.server
framework.parameters.usemail.to=to@address.org
framework.parameters.usemail.from=from@address.org
framework.parameters.usemail.subject.prefix=[build-test alert]
Each particular Test Suite can customize the default message by specific
summary information, can provide attachments to the notification, and
substitute
default notification message with its own.
All this is configured by the following TS-specific parameters:
${test.suite}.parameters.cc.summary
- if contains the path to file, the content of this file will be
prepended
to the status notification message.
${test.suite}.parameters.cc.message
- if contains the path to file, the content of this file will be used
instead of the default message.
${test.suite}.parameters.cc.attachs
- coma or space separated list of absolute paths to files to send with
notification.
${test.suite}.parameters.cc.to
${test.suite}.parameters.cc.from
- if defined and contains email addresses, will be used for mail
notification instead of corresponding global framework's parameters
values.
The values for summary, message, and attachs parameters should be provided
in test suite's parameters.xml files (as it is mostly Test Suite's
implementation-specific information). The values for 'to' and 'from'
parameters should be provided in framework.local.properties file as it is
local BTI configuration information.
There is 'test-mail' target in the root build.xml intended to test the
correctness of mail configuration before BTI execution.
Secondly, some improvements supporting Test Suites execution process have
been
implemented. So <exec-ant> subroutine launching the ant build process has
been
implemented. It wraps the logic used to call and process the execution
status
and output of external ant scripts in one place, and implements some
workarounds
on common problems with absence of junit.jar in ANT_HOME/lib directory,
proxy
settings, and other.
Now along with further TODO's implementation I'm checking the integration
scripts for test suites not integrated in SVN. I'm actualizing them, making
them more BTI compliant and, if it is possible, simplifying them.
If you have any comments regarding implemented improvements, TODO list,
further development directions, and Test Suite integration process,
please, provide them.
Thanks,
Alexander