[Db-derby Wiki] Update of "TestingFrameworks" by JohnHEmbretsen

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Db-derby Wiki" for change notification.

The following page has been changed by JohnHEmbretsen:
http://wiki.apache.org/db-derby/TestingFrameworks

------------------------------------------------------------------------------
+ = Testing frameworks and suites =
- The Derby regression test suites contain multiple tests. The
- tests are arranged into suites for ease of use in running
- multiple related tests.
  
- Some of the test suites run the tests in alternate frameworks:
+ In Derby land, the word "framework" is usually used for specifying which JDBC driver
+ to use when running a test, or a suite of multiple tests. There are currently
+ three such frameworks that are recognized by the Derby test harness and being used in
+ the developer community:
  
-  * derbynetmats: basic network server tests using the IBM Universal JDBC driver
-  * derbynetclientmats: basic network server tests using the Derby client driver
+  * `Embedded` - Using the Derby Embedded JDBC driver
+  * `DerbyNetClient` - Using the Derby Client JDBC driver
+  * `DerbyNet` - Using the IBM Universal JDBC driver (also known as JCC)
+ 
+ In the `DerbyNet` and `DerbyNetClient` frameworks the Derby Network Server is accessed
+ using one of the client drivers. In the `Embedded` framework the Network Server is
+ not used.
+ 
+ 
+ == Specifying test framework ==
+ 
+ Which framework to use can usually be specified on the command line by using the 
+ `framework` system property, e.g.:
+ 
+ {{{
+ > java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.harness.RunTest lang/wisconsin.java
+ }}}
+ 
+ The default framework is the Embedded framework. Framework names are case sensitive.
+ It is also possible to specify a specific framework when running suites, but keep in mind
+ that not all tests are suitable for all frameworks - more on that below.
+ 
+ 
+ == Test suites and frameworks ==
+ 
+ The Derby regression test suites contain multiple tests. The tests are arranged 
+ into suites for ease of use in running multiple related tests. A test suite can
+ consist of both individual tests and other test suites.
+ 
+ It can be quite confusing for a new developer to figure out in which framework a
+ specific test or test suite is being run while running "the mother of all test
+ suites", '''''derbyall''''', or some of its subsuites.
+ 
+ Some suites are set up for specific frameworks:
+ 
+  * `derbynetmats`: basic network server tests using the !DerbyNet framework. Note that this suite is also a subsuite of the derbynetclientmats suite.
+  * `derbynetclientmats`: basic network server tests using the !DerbyNetClient framework. Note that this suite is a superset of the derbynetmats suite.
+ 
+ (`mats` is a widely used testing abbreviation for "minimal acceptance test suite")
+ 
+ This means that during a `derbyall` run using default framework settings, most of the tests in 
+ the `derbynetmats` suite are run at least twice: Once in the !DerbyNet framework and once 
+ in the !DerbyNetClient framework.
+ 
+ The fact that individual tests may be excluded from being run in a certain framework 
+ by using `.exclude` files (in the [:KillDerbyTestHarness:"old" test harness] only) makes this even more 
+ complicated.
  
  The general rule for test authors is that it is best if a test
  can run in all three frameworks (embedded, and the two client configurations).
@@ -16, +62 @@

  just run in embedded framework only. Running these tests in other frameworks 
  might generate different outputs so you may have to generate different masters.
  
+ 
+ == How to find out the suites or frameworks in which a test will run? ==
+ 
- To see in which suites a particular test runs, you can examine the 'suites'
+ To see in which suite(s) a particular test runs, you can examine the 'suites'
+ directory of the code repository (e.g. [http://svn.apache.org/viewvc/db/derby/code/trunk/java/testing/org/apache/derbyTesting/functionTests/suites/ here]).
+ 
+ All non-JUnit suites are defined by a `.runall` file in the suites directory. This file lists
+ all the individual tests in the suite, for example (from derbynetclientmats.runall):
+ {{{
+ junitTests/derbyNet/CompatibilityTest.java
+ jdbcapi/xaSimplePositive.sql
+ jdbcapi/xaStateTran.sql
+ jdbcapi/lobStreams.java
+ jdbcapi/XATest.java
+ jdbcapi/checkDataSource.java
+ derbynet/ByteArrayCombinerStreamTest.junit
+ derbynet/SqlExceptionTest.junit
+ lang/grantRevoke.java
+ }}}
+ In addition, any subsuites to be included in a suite
+ are specified in a `.properties` file, also in the suites directory. For example, the
+ `suites/derbynetclientmats.properties` file includes the following line:
+ {{{
+ suites= encodingTests derbynetclientmats derbynetmats jdbc40
+ }}}
+ 
- directory. You could also invoke the tests without actually running them 
+ You can also invoke the test or a suite without actually running the test(s), 
  and then see if your test gets picked up in the framework of your interest. 
- RunTest has listOnly option.
+ The [:KillDerbyTestHarness:"old" test harness] has a listOnly option (`-DlistOnly=true`).
  
+ The JUnit tests are configured a little differently, see ["DerbyJUnitTesting"].
- There is a *lot* more information in the file: java/testing/README.htm in
- the source tree.
  
+ 
+ == More information ==
+ 
+ There is a lot more information in the file [http://svn.apache.org/viewvc/db/derby/code/trunk/java/testing/README.htm?view=co java/testing/README.htm] in the source tree.
+ 
+ Other helpful tips may be found via the DerbyTesting wiki page.
+