You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@river.apache.org by Peter Firmstone <ji...@zeus.net.au> on 2012/12/01 12:24:21 UTC
QA Test suite refactoring.
Presently most tests have a lot of shared mutable state.
This isn't helped by the setup(QAConfig) mutator method in the Test
interface appended.
An alternative might be:
public Test setup(QAConfig config) throws Exception;
This would allow the test to return another Test object, fully
constructed with immutable state.
It would also allow the test to create an object that can run several
other Test objects, rather than relying on inheritance.
Thoughts?
/**
* This interface must implemented by all tests supported by the test
* harness. The following sequence of events is performed for each
* test:
* <p><ul>
* <li> the test class is instantiated and it's no-arg constructor called.
* The constructor will typically perform minimal initialization,
* since the test does not have access to the test environment
* <li> the setup method is called, passing the config object.
* This provides an opportunity for performing any test setup
* that relies on accessing configuration parameters.
* <li> the run method is called to run the test
* <li> the teardown method is called to clean up state or services
* created by the test. This method is called
* even if <code>setup</code> or <code>run</code> throws an
exception.
* </ul>
*/
public interface Test {
/**
* Failure type. Indicates that a test failed because the product
failed,
* or an unrecognized environmental failure occured.
*/
public final static int TEST = 0;
/**
* Failure type. Indicates that a test failed because of a recognized
* environmental problem. These will typically be due to
* intermittent or recurring configuration dependent failures
* which have been determined to result from problems external
* to the product.
*/
public final static int ENV = 1;
/**
* Failure type. Indicates that a test failed but the reason for
* failure may be external to the product being tested.
*/
public final static int INDEF = 2;
/** Failure type indicating that the test should not be run. */
public final static int SKIP = 3;
/** Failure type indicating that the test should be rerun. */
public final static int RERUN = 4;
/** Failure type (for analyzers) indicating failure not recognized */
public final static int UNKNOWN = 5;
/**
* Failure type indicating no failure occured. This is a special
* value which should be returned by a FailureAnalyzer to indicate
* that the exception represents a passing condition for the test
*/
public final static int PASSED = 6;
/**
* Performs setup for this test. If an exception is thrown, the
* <code>tearDown</code> method will still be called.
*
* @param config the test properties
* @throws Exception if setup fails for any reason
*/
public void setup(QAConfig config) throws Exception;
/**
* Execute the body of the test.
*
* @throws Exception if the test fails for any reason
*/
public void run() throws Exception;
/**
* Tears down any setup that was needed to run the test. This method is
* called even if setup throws an exception, and so must be designed to
* be tolerant of test state that is not completely initialized.
*/
public void tearDown();
}
Re: QA Test suite refactoring.
Posted by Peter Firmstone <ji...@zeus.net.au>.
The second stage of refactoring included separating out concerns from
base test classes into new objects, and using object construction
visibility guarantees to reduce mutable state. I've also fixed a number
of occurrences of insufficient synchronization.
The tests will be complete tomorrow, after the test run is complete,
I'll upload the changes, provided everything passes, then run against
all platforms on Jenkins.
Development is easier and progresses faster when all tests remain
stable, it allows earlier and simpler determination of changes that
cause or induce an error.
The third stage of refactoring will separate common operations out from
methods in super classes into task objects, used to set up services
required for testing and to monitor events etc, during testing. This
will be done so the tests can be broken out from test construction,
minimising mutable state and making writing new tests easier.
I've also spent some time reading the test documentation, it seems there
were parts of the specification where qa tests remain outstanding, these
are documented in the qa/doc directory. This might also explain why I
've discovered dead code in some test classes.
I can test against the current trunk branch by specifying the river home
directory when I run the test suite, if trunk passes, then we can merge,
otherwise, we'll probably need to move trunk to dev and merge back
changes incrementally in an effort to determine causes of test failure.
Cheers,
Peter.
Peter Firmstone wrote:
> First stage of refactoring complete.
>
> [java] -----------------------------------------
> [java]
> [java] # of tests started = 1412
> [java] # of tests completed = 1412
> [java] # of tests skipped = 46
> [java] # of tests passed = 1412
> [java] # of tests failed = 0
> [java]
> [java] -----------------------------------------
> [java]
> [java] Date finished:
> [java] Mon Dec 24 16:54:31 EST 2012
> [java] Time elapsed:
> [java] 91393 seconds
> [java]
>
> collect-result:
>
> BUILD SUCCESSFUL
> Total time: 1,539 minutes 24 seconds
>
> Peter Firmstone wrote:
>> I propose a minor change in semantics for TestFactory.
>> TestDescription includes the following:
>>
>> /**
>> * The <code>TestDescription</code> is an object which describes a test;
>> * it is a factory for the test object and provides support facilities
>> * for setting up and executing the test.
>> * <p>
>> * The description contains of a set of properties. Some of these
>> * properties have well defined names which are used to construct
>> * the execution environment of the test. These well defined names are:
>> * <ul>
>>
>> So we could rename TestFactory and setup method to this:
>>
>> public interface TestExecutionEnvironment {
>>
>> /**
>> * Performs construction of execution environment for this test.
>> * If an exception is thrown, the <code>tearDown</code> method will
>> * still be called.
>> *
>> * @param config the test properties
>> * @throws Exception if construct fails for any reason
>> */
>> Test construct(QAConfig config) throws Exception;
>>
>> /**
>> * Tears down any construct that was needed to run the test. This
>> method is
>> * called even if construct throws an exception, and so must be
>> designed to
>> * be tolerant of test state that is not completely initialized.
>> */
>> void tearDown();
>> }
>>
>> Peter Firmstone wrote:
>>> I've completed the first phase of refactoring, separating out the
>>> environment setup and test run into separate interfaces as per
>>> appended. This should allow the setup and teardown and execution
>>> of the tests to remain separate concerns. At present all tests
>>> implement both interfaces, so nothing has really changed, yet.
>>>
>>> When the qa test run is complete and has passed all tests, I'll
>>> upload the changes for review.
>>>
>>> The next stage will be the separation of setup and test execution
>>> for the tests that have experienced failures recently in trunk.
>>> This will simplify test execution and should make it easier to
>>> manage shared state.
>>>
>>> From there, perhaps we should merge the changes made in trunk back
>>> in incrementally.
>>>
>>> Since a considerable amount of work is required to refactor all
>>> tests, I suggest we migrate the tests to the new format slowly over
>>> time on an as needed basis.
>>>
>>> The names of interfaces and methods can be changed if necessary:
>>>
>>> public interface TestFactory {
>>>
>>> /**
>>> * Performs setup for this test. If an exception is thrown, the
>>> * <code>tearDown</code> method will still be called.
>>> *
>>> * @param config the test properties
>>> * @throws Exception if setup fails for any reason
>>> */
>>> Test setup(QAConfig config) throws Exception;
>>>
>>> /**
>>> * Tears down any setup that was needed to run the test. This
>>> method is
>>> * called even if setup throws an exception, and so must be
>>> designed to
>>> * be tolerant of test state that is not completely initialized.
>>> */
>>> void tearDown();
>>> }
>>>
>>>
>>> public interface Test {
>>> /**
>>> * Failure type. Indicates that a test failed because of a
>>> recognized
>>> * environmental problem. These will typically be due to
>>> * intermittent or recurring configuration dependent failures
>>> * which have been determined to result from problems external
>>> * to the product.
>>> */
>>> int ENV = 1;
>>> /**
>>> * Failure type. Indicates that a test failed but the reason for
>>> * failure may be external to the product being tested.
>>> */
>>> int INDEF = 2;
>>> /**
>>> * Failure type indicating no failure occured. This is a special
>>> * value which should be returned by a FailureAnalyzer to indicate
>>> * that the exception represents a passing condition for the test
>>> */
>>> int PASSED = 6;
>>> /**
>>> * Failure type indicating that the test should be rerun.
>>> */
>>> int RERUN = 4;
>>> /**
>>> * Failure type indicating that the test should not be run.
>>> */
>>> int SKIP = 3;
>>> /**
>>> * Failure type. Indicates that a test failed because the product
>>> failed,
>>> * or an unrecognized environmental failure occured.
>>> */
>>> int TEST = 0;
>>> /**
>>> * Failure type (for analyzers) indicating failure not recognized
>>> */
>>> int UNKNOWN = 5;
>>>
>>> /**
>>> * Execute the body of the test.
>>> *
>>> * @throws Exception if the test fails for any reason
>>> */
>>> void run() throws Exception;
>>> }
>>>
>>> Regards,
>>>
>>> Peter.
>>>
>>> Peter Firmstone wrote:
>>>> Presently most tests have a lot of shared mutable state.
>>>>
>>>> This isn't helped by the setup(QAConfig) mutator method in the Test
>>>> interface appended.
>>>>
>>>> An alternative might be:
>>>>
>>>> public Test setup(QAConfig config) throws Exception;
>>>>
>>>> This would allow the test to return another Test object, fully
>>>> constructed with immutable state.
>>>>
>>>> It would also allow the test to create an object that can run
>>>> several other Test objects, rather than relying on inheritance.
>>>>
>>>> Thoughts?
>>>>
>>>>
>>>>
>>>> /**
>>>> * This interface must implemented by all tests supported by the test
>>>> * harness. The following sequence of events is performed for each
>>>> * test:
>>>> * <p><ul>
>>>> * <li> the test class is instantiated and it's no-arg constructor
>>>> called.
>>>> * The constructor will typically perform minimal
>>>> initialization,
>>>> * since the test does not have access to the test environment
>>>> * <li> the setup method is called, passing the config object.
>>>> * This provides an opportunity for performing any test setup
>>>> * that relies on accessing configuration parameters.
>>>> * <li> the run method is called to run the test
>>>> * <li> the teardown method is called to clean up state or services
>>>> * created by the test. This method is called
>>>> * even if <code>setup</code> or <code>run</code> throws an
>>>> exception.
>>>> * </ul>
>>>> */
>>>> public interface Test {
>>>>
>>>> /**
>>>> * Failure type. Indicates that a test failed because the
>>>> product failed,
>>>> * or an unrecognized environmental failure occured.
>>>> */
>>>> public final static int TEST = 0;
>>>> /**
>>>> * Failure type. Indicates that a test failed because of a
>>>> recognized
>>>> * environmental problem. These will typically be due to
>>>> * intermittent or recurring configuration dependent failures
>>>> * which have been determined to result from problems external
>>>> * to the product.
>>>> */
>>>> public final static int ENV = 1;
>>>> /**
>>>> * Failure type. Indicates that a test failed but the reason for
>>>> * failure may be external to the product being tested.
>>>> */
>>>> public final static int INDEF = 2;
>>>>
>>>> /** Failure type indicating that the test should not be run. */
>>>> public final static int SKIP = 3;
>>>>
>>>> /** Failure type indicating that the test should be rerun. */
>>>> public final static int RERUN = 4;
>>>>
>>>> /** Failure type (for analyzers) indicating failure not
>>>> recognized */
>>>> public final static int UNKNOWN = 5;
>>>>
>>>> /**
>>>> * Failure type indicating no failure occured. This is a special
>>>> * value which should be returned by a FailureAnalyzer to indicate
>>>> * that the exception represents a passing condition for the test
>>>> */
>>>> public final static int PASSED = 6;
>>>>
>>>> /**
>>>> * Performs setup for this test. If an exception is thrown, the
>>>> * <code>tearDown</code> method will still be called.
>>>> *
>>>> * @param config the test properties
>>>> * @throws Exception if setup fails for any reason
>>>> */
>>>> public void setup(QAConfig config) throws Exception;
>>>>
>>>> /**
>>>> * Execute the body of the test.
>>>> *
>>>> * @throws Exception if the test fails for any reason
>>>> */
>>>> public void run() throws Exception;
>>>>
>>>> /**
>>>> * Tears down any setup that was needed to run the test. This
>>>> method is
>>>> * called even if setup throws an exception, and so must be
>>>> designed to
>>>> * be tolerant of test state that is not completely initialized.
>>>> */
>>>> public void tearDown();
>>>> }
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: QA Test suite refactoring.
Posted by Peter Firmstone <ji...@zeus.net.au>.
First stage of refactoring complete.
[java] -----------------------------------------
[java]
[java] # of tests started = 1412
[java] # of tests completed = 1412
[java] # of tests skipped = 46
[java] # of tests passed = 1412
[java] # of tests failed = 0
[java]
[java] -----------------------------------------
[java]
[java] Date finished:
[java] Mon Dec 24 16:54:31 EST 2012
[java] Time elapsed:
[java] 91393 seconds
[java]
collect-result:
BUILD SUCCESSFUL
Total time: 1,539 minutes 24 seconds
Peter Firmstone wrote:
> I propose a minor change in semantics for TestFactory.
> TestDescription includes the following:
>
> /**
> * The <code>TestDescription</code> is an object which describes a test;
> * it is a factory for the test object and provides support facilities
> * for setting up and executing the test.
> * <p>
> * The description contains of a set of properties. Some of these
> * properties have well defined names which are used to construct
> * the execution environment of the test. These well defined names are:
> * <ul>
>
> So we could rename TestFactory and setup method to this:
>
> public interface TestExecutionEnvironment {
>
> /**
> * Performs construction of execution environment for this test.
> * If an exception is thrown, the <code>tearDown</code> method will
> * still be called.
> *
> * @param config the test properties
> * @throws Exception if construct fails for any reason
> */
> Test construct(QAConfig config) throws Exception;
>
> /**
> * Tears down any construct that was needed to run the test. This
> method is
> * called even if construct throws an exception, and so must be
> designed to
> * be tolerant of test state that is not completely initialized.
> */
> void tearDown();
> }
>
> Peter Firmstone wrote:
>> I've completed the first phase of refactoring, separating out the
>> environment setup and test run into separate interfaces as per
>> appended. This should allow the setup and teardown and execution of
>> the tests to remain separate concerns. At present all tests
>> implement both interfaces, so nothing has really changed, yet.
>>
>> When the qa test run is complete and has passed all tests, I'll
>> upload the changes for review.
>>
>> The next stage will be the separation of setup and test execution for
>> the tests that have experienced failures recently in trunk. This
>> will simplify test execution and should make it easier to manage
>> shared state.
>>
>> From there, perhaps we should merge the changes made in trunk back in
>> incrementally.
>>
>> Since a considerable amount of work is required to refactor all
>> tests, I suggest we migrate the tests to the new format slowly over
>> time on an as needed basis.
>>
>> The names of interfaces and methods can be changed if necessary:
>>
>> public interface TestFactory {
>>
>> /**
>> * Performs setup for this test. If an exception is thrown, the
>> * <code>tearDown</code> method will still be called.
>> *
>> * @param config the test properties
>> * @throws Exception if setup fails for any reason
>> */
>> Test setup(QAConfig config) throws Exception;
>>
>> /**
>> * Tears down any setup that was needed to run the test. This
>> method is
>> * called even if setup throws an exception, and so must be
>> designed to
>> * be tolerant of test state that is not completely initialized.
>> */
>> void tearDown();
>> }
>>
>>
>> public interface Test {
>> /**
>> * Failure type. Indicates that a test failed because of a recognized
>> * environmental problem. These will typically be due to
>> * intermittent or recurring configuration dependent failures
>> * which have been determined to result from problems external
>> * to the product.
>> */
>> int ENV = 1;
>> /**
>> * Failure type. Indicates that a test failed but the reason for
>> * failure may be external to the product being tested.
>> */
>> int INDEF = 2;
>> /**
>> * Failure type indicating no failure occured. This is a special
>> * value which should be returned by a FailureAnalyzer to indicate
>> * that the exception represents a passing condition for the test
>> */
>> int PASSED = 6;
>> /**
>> * Failure type indicating that the test should be rerun.
>> */
>> int RERUN = 4;
>> /**
>> * Failure type indicating that the test should not be run.
>> */
>> int SKIP = 3;
>> /**
>> * Failure type. Indicates that a test failed because the product
>> failed,
>> * or an unrecognized environmental failure occured.
>> */
>> int TEST = 0;
>> /**
>> * Failure type (for analyzers) indicating failure not recognized
>> */
>> int UNKNOWN = 5;
>>
>> /**
>> * Execute the body of the test.
>> *
>> * @throws Exception if the test fails for any reason
>> */
>> void run() throws Exception;
>> }
>>
>> Regards,
>>
>> Peter.
>>
>> Peter Firmstone wrote:
>>> Presently most tests have a lot of shared mutable state.
>>>
>>> This isn't helped by the setup(QAConfig) mutator method in the Test
>>> interface appended.
>>>
>>> An alternative might be:
>>>
>>> public Test setup(QAConfig config) throws Exception;
>>>
>>> This would allow the test to return another Test object, fully
>>> constructed with immutable state.
>>>
>>> It would also allow the test to create an object that can run
>>> several other Test objects, rather than relying on inheritance.
>>>
>>> Thoughts?
>>>
>>>
>>>
>>> /**
>>> * This interface must implemented by all tests supported by the test
>>> * harness. The following sequence of events is performed for each
>>> * test:
>>> * <p><ul>
>>> * <li> the test class is instantiated and it's no-arg constructor
>>> called.
>>> * The constructor will typically perform minimal initialization,
>>> * since the test does not have access to the test environment
>>> * <li> the setup method is called, passing the config object.
>>> * This provides an opportunity for performing any test setup
>>> * that relies on accessing configuration parameters.
>>> * <li> the run method is called to run the test
>>> * <li> the teardown method is called to clean up state or services
>>> * created by the test. This method is called
>>> * even if <code>setup</code> or <code>run</code> throws an
>>> exception.
>>> * </ul>
>>> */
>>> public interface Test {
>>>
>>> /**
>>> * Failure type. Indicates that a test failed because the product
>>> failed,
>>> * or an unrecognized environmental failure occured.
>>> */
>>> public final static int TEST = 0;
>>> /**
>>> * Failure type. Indicates that a test failed because of a
>>> recognized
>>> * environmental problem. These will typically be due to
>>> * intermittent or recurring configuration dependent failures
>>> * which have been determined to result from problems external
>>> * to the product.
>>> */
>>> public final static int ENV = 1;
>>> /**
>>> * Failure type. Indicates that a test failed but the reason for
>>> * failure may be external to the product being tested.
>>> */
>>> public final static int INDEF = 2;
>>>
>>> /** Failure type indicating that the test should not be run. */
>>> public final static int SKIP = 3;
>>>
>>> /** Failure type indicating that the test should be rerun. */
>>> public final static int RERUN = 4;
>>>
>>> /** Failure type (for analyzers) indicating failure not
>>> recognized */
>>> public final static int UNKNOWN = 5;
>>>
>>> /**
>>> * Failure type indicating no failure occured. This is a special
>>> * value which should be returned by a FailureAnalyzer to indicate
>>> * that the exception represents a passing condition for the test
>>> */
>>> public final static int PASSED = 6;
>>>
>>> /**
>>> * Performs setup for this test. If an exception is thrown, the
>>> * <code>tearDown</code> method will still be called.
>>> *
>>> * @param config the test properties
>>> * @throws Exception if setup fails for any reason
>>> */
>>> public void setup(QAConfig config) throws Exception;
>>>
>>> /**
>>> * Execute the body of the test.
>>> *
>>> * @throws Exception if the test fails for any reason
>>> */
>>> public void run() throws Exception;
>>>
>>> /**
>>> * Tears down any setup that was needed to run the test. This
>>> method is
>>> * called even if setup throws an exception, and so must be
>>> designed to
>>> * be tolerant of test state that is not completely initialized.
>>> */
>>> public void tearDown();
>>> }
>>>
>>>
>>
>>
>
>
Re: QA Test suite refactoring.
Posted by Peter Firmstone <ji...@zeus.net.au>.
I propose a minor change in semantics for TestFactory. TestDescription
includes the following:
/**
* The <code>TestDescription</code> is an object which describes a test;
* it is a factory for the test object and provides support facilities
* for setting up and executing the test.
* <p>
* The description contains of a set of properties. Some of these
* properties have well defined names which are used to construct
* the execution environment of the test. These well defined names are:
* <ul>
So we could rename TestFactory and setup method to this:
public interface TestExecutionEnvironment {
/**
* Performs construction of execution environment for this test.
* If an exception is thrown, the <code>tearDown</code> method will
* still be called.
*
* @param config the test properties
* @throws Exception if construct fails for any reason
*/
Test construct(QAConfig config) throws Exception;
/**
* Tears down any construct that was needed to run the test. This
method is
* called even if construct throws an exception, and so must be
designed to
* be tolerant of test state that is not completely initialized.
*/
void tearDown();
}
Peter Firmstone wrote:
> I've completed the first phase of refactoring, separating out the
> environment setup and test run into separate interfaces as per
> appended. This should allow the setup and teardown and execution of
> the tests to remain separate concerns. At present all tests implement
> both interfaces, so nothing has really changed, yet.
>
> When the qa test run is complete and has passed all tests, I'll upload
> the changes for review.
>
> The next stage will be the separation of setup and test execution for
> the tests that have experienced failures recently in trunk. This will
> simplify test execution and should make it easier to manage shared state.
>
> From there, perhaps we should merge the changes made in trunk back in
> incrementally.
>
> Since a considerable amount of work is required to refactor all tests,
> I suggest we migrate the tests to the new format slowly over time on
> an as needed basis.
>
> The names of interfaces and methods can be changed if necessary:
>
> public interface TestFactory {
>
> /**
> * Performs setup for this test. If an exception is thrown, the
> * <code>tearDown</code> method will still be called.
> *
> * @param config the test properties
> * @throws Exception if setup fails for any reason
> */
> Test setup(QAConfig config) throws Exception;
>
> /**
> * Tears down any setup that was needed to run the test. This
> method is
> * called even if setup throws an exception, and so must be
> designed to
> * be tolerant of test state that is not completely initialized.
> */
> void tearDown();
> }
>
>
> public interface Test {
> /**
> * Failure type. Indicates that a test failed because of a recognized
> * environmental problem. These will typically be due to
> * intermittent or recurring configuration dependent failures
> * which have been determined to result from problems external
> * to the product.
> */
> int ENV = 1;
> /**
> * Failure type. Indicates that a test failed but the reason for
> * failure may be external to the product being tested.
> */
> int INDEF = 2;
> /**
> * Failure type indicating no failure occured. This is a special
> * value which should be returned by a FailureAnalyzer to indicate
> * that the exception represents a passing condition for the test
> */
> int PASSED = 6;
> /**
> * Failure type indicating that the test should be rerun.
> */
> int RERUN = 4;
> /**
> * Failure type indicating that the test should not be run.
> */
> int SKIP = 3;
> /**
> * Failure type. Indicates that a test failed because the product
> failed,
> * or an unrecognized environmental failure occured.
> */
> int TEST = 0;
> /**
> * Failure type (for analyzers) indicating failure not recognized
> */
> int UNKNOWN = 5;
>
> /**
> * Execute the body of the test.
> *
> * @throws Exception if the test fails for any reason
> */
> void run() throws Exception;
> }
>
> Regards,
>
> Peter.
>
> Peter Firmstone wrote:
>> Presently most tests have a lot of shared mutable state.
>>
>> This isn't helped by the setup(QAConfig) mutator method in the Test
>> interface appended.
>>
>> An alternative might be:
>>
>> public Test setup(QAConfig config) throws Exception;
>>
>> This would allow the test to return another Test object, fully
>> constructed with immutable state.
>>
>> It would also allow the test to create an object that can run several
>> other Test objects, rather than relying on inheritance.
>>
>> Thoughts?
>>
>>
>>
>> /**
>> * This interface must implemented by all tests supported by the test
>> * harness. The following sequence of events is performed for each
>> * test:
>> * <p><ul>
>> * <li> the test class is instantiated and it's no-arg constructor
>> called.
>> * The constructor will typically perform minimal initialization,
>> * since the test does not have access to the test environment
>> * <li> the setup method is called, passing the config object.
>> * This provides an opportunity for performing any test setup
>> * that relies on accessing configuration parameters.
>> * <li> the run method is called to run the test
>> * <li> the teardown method is called to clean up state or services
>> * created by the test. This method is called
>> * even if <code>setup</code> or <code>run</code> throws an
>> exception.
>> * </ul>
>> */
>> public interface Test {
>>
>> /**
>> * Failure type. Indicates that a test failed because the product
>> failed,
>> * or an unrecognized environmental failure occured.
>> */
>> public final static int TEST = 0;
>> /**
>> * Failure type. Indicates that a test failed because of a recognized
>> * environmental problem. These will typically be due to
>> * intermittent or recurring configuration dependent failures
>> * which have been determined to result from problems external
>> * to the product.
>> */
>> public final static int ENV = 1;
>> /**
>> * Failure type. Indicates that a test failed but the reason for
>> * failure may be external to the product being tested.
>> */
>> public final static int INDEF = 2;
>>
>> /** Failure type indicating that the test should not be run. */
>> public final static int SKIP = 3;
>>
>> /** Failure type indicating that the test should be rerun. */
>> public final static int RERUN = 4;
>>
>> /** Failure type (for analyzers) indicating failure not recognized */
>> public final static int UNKNOWN = 5;
>>
>> /**
>> * Failure type indicating no failure occured. This is a special
>> * value which should be returned by a FailureAnalyzer to indicate
>> * that the exception represents a passing condition for the test
>> */
>> public final static int PASSED = 6;
>>
>> /**
>> * Performs setup for this test. If an exception is thrown, the
>> * <code>tearDown</code> method will still be called.
>> *
>> * @param config the test properties
>> * @throws Exception if setup fails for any reason
>> */
>> public void setup(QAConfig config) throws Exception;
>>
>> /**
>> * Execute the body of the test.
>> *
>> * @throws Exception if the test fails for any reason
>> */
>> public void run() throws Exception;
>>
>> /**
>> * Tears down any setup that was needed to run the test. This
>> method is
>> * called even if setup throws an exception, and so must be
>> designed to
>> * be tolerant of test state that is not completely initialized.
>> */
>> public void tearDown();
>> }
>>
>>
>
>
Re: QA Test suite refactoring.
Posted by Peter Firmstone <ji...@zeus.net.au>.
I've completed the first phase of refactoring, separating out the
environment setup and test run into separate interfaces as per
appended. This should allow the setup and teardown and execution of
the tests to remain separate concerns. At present all tests implement
both interfaces, so nothing has really changed, yet.
When the qa test run is complete and has passed all tests, I'll upload
the changes for review.
The next stage will be the separation of setup and test execution for
the tests that have experienced failures recently in trunk. This will
simplify test execution and should make it easier to manage shared state.
From there, perhaps we should merge the changes made in trunk back in
incrementally.
Since a considerable amount of work is required to refactor all tests, I
suggest we migrate the tests to the new format slowly over time on an as
needed basis.
The names of interfaces and methods can be changed if necessary:
public interface TestFactory {
/**
* Performs setup for this test. If an exception is thrown, the
* <code>tearDown</code> method will still be called.
*
* @param config the test properties
* @throws Exception if setup fails for any reason
*/
Test setup(QAConfig config) throws Exception;
/**
* Tears down any setup that was needed to run the test. This method is
* called even if setup throws an exception, and so must be designed to
* be tolerant of test state that is not completely initialized.
*/
void tearDown();
}
public interface Test {
/**
* Failure type. Indicates that a test failed because of a recognized
* environmental problem. These will typically be due to
* intermittent or recurring configuration dependent failures
* which have been determined to result from problems external
* to the product.
*/
int ENV = 1;
/**
* Failure type. Indicates that a test failed but the reason for
* failure may be external to the product being tested.
*/
int INDEF = 2;
/**
* Failure type indicating no failure occured. This is a special
* value which should be returned by a FailureAnalyzer to indicate
* that the exception represents a passing condition for the test
*/
int PASSED = 6;
/**
* Failure type indicating that the test should be rerun.
*/
int RERUN = 4;
/**
* Failure type indicating that the test should not be run.
*/
int SKIP = 3;
/**
* Failure type. Indicates that a test failed because the product
failed,
* or an unrecognized environmental failure occured.
*/
int TEST = 0;
/**
* Failure type (for analyzers) indicating failure not recognized
*/
int UNKNOWN = 5;
/**
* Execute the body of the test.
*
* @throws Exception if the test fails for any reason
*/
void run() throws Exception;
}
Regards,
Peter.
Peter Firmstone wrote:
> Presently most tests have a lot of shared mutable state.
>
> This isn't helped by the setup(QAConfig) mutator method in the Test
> interface appended.
>
> An alternative might be:
>
> public Test setup(QAConfig config) throws Exception;
>
> This would allow the test to return another Test object, fully
> constructed with immutable state.
>
> It would also allow the test to create an object that can run several
> other Test objects, rather than relying on inheritance.
>
> Thoughts?
>
>
>
> /**
> * This interface must implemented by all tests supported by the test
> * harness. The following sequence of events is performed for each
> * test:
> * <p><ul>
> * <li> the test class is instantiated and it's no-arg constructor
> called.
> * The constructor will typically perform minimal initialization,
> * since the test does not have access to the test environment
> * <li> the setup method is called, passing the config object.
> * This provides an opportunity for performing any test setup
> * that relies on accessing configuration parameters.
> * <li> the run method is called to run the test
> * <li> the teardown method is called to clean up state or services
> * created by the test. This method is called
> * even if <code>setup</code> or <code>run</code> throws an
> exception.
> * </ul>
> */
> public interface Test {
>
> /**
> * Failure type. Indicates that a test failed because the product
> failed,
> * or an unrecognized environmental failure occured.
> */
> public final static int TEST = 0;
> /**
> * Failure type. Indicates that a test failed because of a recognized
> * environmental problem. These will typically be due to
> * intermittent or recurring configuration dependent failures
> * which have been determined to result from problems external
> * to the product.
> */
> public final static int ENV = 1;
> /**
> * Failure type. Indicates that a test failed but the reason for
> * failure may be external to the product being tested.
> */
> public final static int INDEF = 2;
>
> /** Failure type indicating that the test should not be run. */
> public final static int SKIP = 3;
>
> /** Failure type indicating that the test should be rerun. */
> public final static int RERUN = 4;
>
> /** Failure type (for analyzers) indicating failure not recognized */
> public final static int UNKNOWN = 5;
>
> /**
> * Failure type indicating no failure occured. This is a special
> * value which should be returned by a FailureAnalyzer to indicate
> * that the exception represents a passing condition for the test
> */
> public final static int PASSED = 6;
>
> /**
> * Performs setup for this test. If an exception is thrown, the
> * <code>tearDown</code> method will still be called.
> *
> * @param config the test properties
> * @throws Exception if setup fails for any reason
> */
> public void setup(QAConfig config) throws Exception;
>
> /**
> * Execute the body of the test.
> *
> * @throws Exception if the test fails for any reason
> */
> public void run() throws Exception;
>
> /**
> * Tears down any setup that was needed to run the test. This
> method is
> * called even if setup throws an exception, and so must be
> designed to
> * be tolerant of test state that is not completely initialized.
> */
> public void tearDown();
> }
>
>
Re: QA Test suite refactoring.
Posted by Dan Creswell <da...@gmail.com>.
Well that's less crap than my suggestions! :)
On 2 December 2012 10:33, Gregg Wonderly <gr...@gmail.com> wrote:
> How about something like initTest?
>
> Gregg Wonderly
>
> On Dec 2, 2012, at 12:07 PM, Dan Creswell <da...@gmail.com> wrote:
>
>> On 1 December 2012 03:24, Peter Firmstone <ji...@zeus.net.au> wrote:
>>> Presently most tests have a lot of shared mutable state.
>>>
>>> This isn't helped by the setup(QAConfig) mutator method in the Test
>>> interface appended.
>>>
>>> An alternative might be:
>>>
>>> public Test setup(QAConfig config) throws Exception;
>>>
>>> This would allow the test to return another Test object, fully constructed
>>> with immutable state.
>>>
>>> It would also allow the test to create an object that can run several other
>>> Test objects, rather than relying on inheritance.
>>>
>>> Thoughts?
>>>
>>>
>>
>> Reasonable IMO, I'd observe what we've got now is a factory method
>> which kinda doesn't jibe with the original intentions behind the Test
>> interface. The Test object has already been constructed in order for
>> setup to be called.
>>
>> Tricky as making more than basic changes will cause much work although
>> I think that's the way this is going to go anyway. I'm somewhat
>> tempted to say something radical like "let's just move this stuf over
>> to junit" but for now how about we come up with a better name than
>> setup that fits the intention better?
>>
>> I've only got crap names off the top of my head to suggest like
>> "construct" or "build"...
>
Re: QA Test suite refactoring.
Posted by Gregg Wonderly <gr...@gmail.com>.
How about something like initTest?
Gregg Wonderly
On Dec 2, 2012, at 12:07 PM, Dan Creswell <da...@gmail.com> wrote:
> On 1 December 2012 03:24, Peter Firmstone <ji...@zeus.net.au> wrote:
>> Presently most tests have a lot of shared mutable state.
>>
>> This isn't helped by the setup(QAConfig) mutator method in the Test
>> interface appended.
>>
>> An alternative might be:
>>
>> public Test setup(QAConfig config) throws Exception;
>>
>> This would allow the test to return another Test object, fully constructed
>> with immutable state.
>>
>> It would also allow the test to create an object that can run several other
>> Test objects, rather than relying on inheritance.
>>
>> Thoughts?
>>
>>
>
> Reasonable IMO, I'd observe what we've got now is a factory method
> which kinda doesn't jibe with the original intentions behind the Test
> interface. The Test object has already been constructed in order for
> setup to be called.
>
> Tricky as making more than basic changes will cause much work although
> I think that's the way this is going to go anyway. I'm somewhat
> tempted to say something radical like "let's just move this stuf over
> to junit" but for now how about we come up with a better name than
> setup that fits the intention better?
>
> I've only got crap names off the top of my head to suggest like
> "construct" or "build"...
Re: QA Test suite refactoring.
Posted by Dan Creswell <da...@gmail.com>.
On 1 December 2012 03:24, Peter Firmstone <ji...@zeus.net.au> wrote:
> Presently most tests have a lot of shared mutable state.
>
> This isn't helped by the setup(QAConfig) mutator method in the Test
> interface appended.
>
> An alternative might be:
>
> public Test setup(QAConfig config) throws Exception;
>
> This would allow the test to return another Test object, fully constructed
> with immutable state.
>
> It would also allow the test to create an object that can run several other
> Test objects, rather than relying on inheritance.
>
> Thoughts?
>
>
Reasonable IMO, I'd observe what we've got now is a factory method
which kinda doesn't jibe with the original intentions behind the Test
interface. The Test object has already been constructed in order for
setup to be called.
Tricky as making more than basic changes will cause much work although
I think that's the way this is going to go anyway. I'm somewhat
tempted to say something radical like "let's just move this stuf over
to junit" but for now how about we come up with a better name than
setup that fits the intention better?
I've only got crap names off the top of my head to suggest like
"construct" or "build"...