[jira] Updated: (HARMONY-2283) [drlvm][regression] Regression Testing Infrastructure proposal

["Alexander Kleymenov (JIRA)" <ji...@apache.org>]

     [ http://issues.apache.org/jira/browse/HARMONY-2283?page=all ]

Alexander Kleymenov updated HARMONY-2283:
-----------------------------------------

    Attachment: patch-reg-tests.txt

Attached patch contains initial implementation of regression testing infrastructure. Apache Ant was chosen as a base for implementation because:
    1.  It is platform independent tool installed on every workstation used for Harmony/DRLVM development. 
    2.  It provides all the means to build/execute "heterogeneous" tests. 
    3.  It allows configuring of tests execution. 
    4.  For default test format (such as JUnit) it allows automatic test building/executing (after the test source have been added to the test tree it will automatically be built and executed by build target) 
    5.  For some sophisticated test demanding non-standard launching method it is possible to create special Ant target (possible in little ant file placed next to the test) describing the way of launching it. These targets can use common parameters and macrodefinitions defined for regression testing. 

Several regression tests are provided as a usage example. They are placed in src/test/regression directory of DRLVM workspace. Subdirectories containing the tests are named according to the JIRA report describing the problem. The test examples were chosen to demonstrate some of possible cases: they are in different formats (JUnit and pure Java), include heterogeneous sources (Java, C/C++, Java assembler), and some of them are using special launching methods for execution (i.e. custom launchers overwriting default test launching way).

The package for regression tests was chosen as org.apache.harmony.drlvm.tests.regression.h<xxxx>. Where <xxxx> is a number of JIRA report.

Currently the implementation does not support automatic test integration and all of the tests to be executed are explicitly listed in build/make/targets/reg.test.run.xml file. Automatic integration can be easily done after defining of test naming conventions and directory structure.

Please review the proposed infrastructure and provide your comments.

Thanks,
Alexander

> [drlvm][regression] Regression Testing Infrastructure proposal
> --------------------------------------------------------------
>
>                 Key: HARMONY-2283
>                 URL: http://issues.apache.org/jira/browse/HARMONY-2283
>             Project: Harmony
>          Issue Type: Improvement
>          Components: DRLVM
>            Reporter: Alexander Kleymenov
>         Attachments: patch-reg-tests.txt
>
>
> There is no infrastructure for automated regression testing in Harmony DRLVM project.
> This JIRA report is intended to initiate the discussions and works on it.
> With taking into account the specific of the product, the following requirements to infrastructure can be defined:
>     1. It should support tests in different formats (JUnit, pure Java, etc) and implemented in heterogeneous sources (Java, C/C++, Java assembler etc). 
>     2. It should provide convenient means for test run configuration (parallel/sequential execution, execution timeouts, command line parameters specifying, environment variables presetting). 
>     3. It should provide means for rapid and convenient integration of new test. 
>     4. It should be based on commonly used means. 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira