[Db-derby Wiki] Update of "DerbyPerfTesting" by SunithaKambhampati

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Db-derby Wiki" for change notification.

The following page has been changed by SunithaKambhampati:
http://wiki.apache.org/db-derby/DerbyPerfTesting

New page:
This page will describe how to write new Derby performance tests within the Junit framework that is already available.  For more details of the test harness related to Junit testing, please see details on [http://wiki.apache.org/db-derby/DerbyJunitTesting DerbyJunitTesting] page.

[[TableOfContents]]

== JDBCPerfTestCase class ==
As part of DERBY-1970, an initial framework for adding performance tests for Derby has been added.
This is
 * `org.apache.derbyTesting.junit.JDBCPerfTestCase`

The `JDBCPerfTestCase` extends the functionality provided by `TestCase`. `JDBCPerfTestCase` extends `BaseJDBCTestCase`. The following functionality is added in `JDBCPerfTestCase`
   * Time the execution of the test fixture
   * Add mechanism to run a test fixture for I iterations
   * Adds mechanism to run a test including setup, test fixture, tearDown  for R times.

For a standard JUnit test, the method fixture is run once.
{{{ 
{
     setUp run once
     fixture method is run once
     tearDown run once
}
}}}

For JDBCPerfTestCase with R repeats and I iterations
 
This cycle is repeated R times
{{{ 
{
    setUp run once
    fixture method is run I times(and timed)
    tearDown is run once
}
}}}

== Location of performance tests ==
The base directory for adding performance tests is org/apache/derbyTesting/perf/
For simple individual tests,  please add them to org/apache/derbyTesting/perf/basic
Depending on what area the performance test is supposed to cover, you can also add subpackages under basic.   For e.g, if the performance test is testing some area in lobs, then create a sub directory `lobs` and add a test there.

For example, see the ValuesTest in org/apache/derbyTesting/perf/basic/jdbc

Discussion on the list about location of performance tests [http://www.nabble.com/Location-for-performance-tests.-tf2483748.html#a6927831]

== Writing a performance Junit Derby test ==
To write a performance JUnit test that uses JDBC, make your test class extend `JDBCPerfTestCase`.  Add the test fixtures that you want to measure performance for, optionally write your `setUp` and `tearDown` methods and finally the `suite` method that returns a suite of tests to run.  In the suite method, add the number of iterations and repeats that you want the to run the test.   If you do not specify the iterations and repeats for the test, then by default , the test is run for one iteration and the entire cycle is run once.

For an example, see the tests in org/apache/derbyTesting/perf/basic/jdbc

If a particular test fixture is to be run for I iterations and R repeats, some careful consideration is required to ensure that, there are no unwanted side effects. For example, if the test method tries to insert data into a table with primary key from one range, then it must be able to handle that the particular test fixture will be able to run for I iterations.

Add the suite of tests for the particular test class, also to the _Suite.suite() method  in org/apache/derbyTesting/perf/basic

Since `JDBCPerfTestCase` extends the `BaseJDBCTestCase`, please use the standard practises for setting up connections, etc as described on the DerbyJunitTesting page.  For instance the method `getConnection` to obtain a connection to the default database, so there is no need for your own test class to have a Connection field.

If you are missing something, ask on derby-dev or create a sub-task/link a Jira issue to DERBY-1970

== Running Tests ==

=== Running tests using Junit directly ===

Tests can be run using the Junit 3.8 !TestRunners. The classpath needs to include the junit.jar and the Derby classes.
Most of Derby's JUnit test cases and suites run successfully using the !TestRunners directly, as that is the eventual goal.

Also note that the top level suite `org.apache.derbyTesting.functionTests.suites.All` does not run the performance suite.

The class name of the test passed to the runners can be an individual test case or one one of the Derby suites, see examples.

<!> No system properties are required when running directly with !TestRunners

=== batch run ===
{{{
# Single test case
> java -cp '../../tools/java/junit.jar:../../classes' junit.textui.TestRunner
        org.apache.derbyTesting.perf.basic.jdbc.ValuesTest
...........................

# The basic package _Suite
java junit.textui.TestRunner org.apache.derbyTesting.perf.basic._Suite

}}}

== Adding performance tests to Derby ==

Goal is to add performance tests to Derby so that anyone in the community can run these tests easily and provide numbers, identify areas for improvement, be encouraged to work on performance issues and increase the quality of derby. 

This would gain all the benefits of JUnit, such as running tests from ant, integration with IDEs, ability to hook into other JUnit suites,easier understanding of how Derby tests are run etc.

== Features and Improvements for the performance junit framework ==

'''Please add new items to the end so that the numbering of items is kept consistent.'''

 1. Improve reporting for the JDBC!PerfTestCase..
     Extend the !TestResult, and enhance reporting detail in JDBC!PerfTestCase to gather the performance results in xml format for flexibility and ease to generate cool reports using xsl.  One idea is to use the java.beans.XMLEncoder to write out XML output to a file. 

 1. A way to have a suite run all tests in client/server mode and a suite that runs all tests in embedded mode. 
 1. Each test suite runs as part of one jvm run. Need a way to handle running the test as separate runs.
 1. Ensure that the performance tests are not broken, by adding some or parts of the tests to functional tests. Need to analyze the benefit of doing this and the time overhead of adding them to derbyall.  Since performance tests are written as junit tests, it would be easy to plug them in for functional tests.
 1. A way to add multi-user performance tests.
 1. Add tools to generate comparison numbers between different build runs.