You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-commits@db.apache.org by dj...@apache.org on 2005/10/31 19:40:32 UTC
svn commit: r329876 [2/3] - in /db/derby/code/trunk/java/testing: README.htm
org/apache/derbyTesting/functionTests/harness/RunTest.java
org/apache/derbyTesting/functionTests/util/derby_tests.policy
Modified: db/derby/code/trunk/java/testing/README.htm
URL: http://svn.apache.org/viewcvs/db/derby/code/trunk/java/testing/README.htm?rev=329876&r1=329875&r2=329876&view=diff
==============================================================================
--- db/derby/code/trunk/java/testing/README.htm (original)
+++ db/derby/code/trunk/java/testing/README.htm Mon Oct 31 10:40:29 2005
@@ -1,1340 +1,1239 @@
-<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
-<html>
-<head>
- <meta content="text/html; charset=ISO-8859-1"
- http-equiv="content-type">
- <title>readme.htm</title>
-</head>
-<body>
-<h1><a class="mozTocH1" name="mozTocId934928"></a>Derby Functional Tests<br>
-</h1>
-<h2><a class="mozTocH2" name="mozTocId504000"></a>Package:
-org.apache.derbyTesting<!--mozToc h1 1 h2 2 h3 3 h4 4 h5 5 h6 6--><br>
-</h2>
-<p>
-<small>created by myrna@golux.com<br>
-last updated on 04/18/2005 by: m.v.lunteren@gmail.com<br>
-</small>
-</p>
-<ul>
- <li><a href="#intro">1. Introduction</a></li>
- <li><a href="#quickstart">2. Quickstart</a></li>
- <li style="margin-left: 40px;"><a
- href="#2.1_running_with_derby_classes_">2.1 running tests<br>
- </a></li>
- <li style="margin-left: 40px;"><a
- href="#building_derbyTesting__running_with">2.2 building
-derbyTesting package</a><br>
- </li>
- <li><a href="#run">3. More details on running the derby functional
-tests</a></li>
- <li style="margin-left: 40px;"><a href="#run1">3.1 Running 1 test</a></li>
- <li style="margin-left: 40px;"><a href="#run2">3.2 Running suites of
-tests</a></li>
- <li><a href="#overview">4. Harness internals for developers</a> </li>
- <li style="margin-left: 40px;"><a href="#ov1">4.1 Test types</a></li>
- <li style="margin-left: 40px;"><a href="#ov2">4.2 Supporting files
-for tests</a></li>
- <li style="margin-left: 40px;"><a href="#ov3">4.3
-<testname>_app.properties</a></li>
- <li style="margin-left: 40px;"><a href="#ov4">4.4
-<testname>_derby.properties</a></li>
- <li style="margin-left: 40px;"><a href="#ov5">4.5 tmp files, out
-files, master files, and canons</a></li>
- <li style="margin-left: 40px;"><a href="#ov6">4.6 Masking and
-comparing</a></li>
- <li style="margin-left: 40px;"><a href="#Adding_a_new_test">4.7
-Adding a new test</a></li>
- <li style="margin-left: 40px;"><a
- href="#4.8_Suites_and_Adding_a_new_suite">4.8 Suites and adding a
-new suite</a></li>
- <li style="margin-left: 40px;"><a href="#ov9">4.9 Running with a new
-jvm</a></li>
- <li style="margin-left: 40px;"><a href="#skipping">4.10 Skipping a
-test</a></li>
- <li style="margin-left: 40px;"><a href="#frameworks">4.11 Frameworks</a></li>
- <li style="margin-left: 40px;"><a href="#props">4.12 Some test
-harness properties</a> </li>
-</ul>
-<br>
-<h2>1. <a name="intro"></a>Introduction</h2>
-<p>
-This document describes functionality of the derby
-functional testing package org.apache.derbyTesting. This package is
-based on the functional tests in use at IBM for testing the Cloudscape
-product before its contribution to ASF.</p>
-<p>In the following, instructions are geared towards a unix
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
+<HTML>
+<HEAD>
+ <META HTTP-EQUIV="CONTENT-TYPE" CONTENT="text/html; charset=windows-1252">
+ <TITLE>readme.htm</TITLE>
+ <META NAME="GENERATOR" CONTENT="OpenOffice.org 1.1.4 (Win32)">
+ <META NAME="CREATED" CONTENT="20051031;9095275">
+ <META NAME="CHANGED" CONTENT="20051031;9274086">
+</HEAD>
+<BODY LANG="en-US" DIR="LTR">
+<H1><A NAME="mozTocId934928"></A>Derby Functional Tests</H1>
+<H2><A NAME="mozTocId504000"></A>Package: org.apache.derbyTesting<!--mozToc h1 1 h2 2 h3 3 h4 4 h5 5 h6 6--></H2>
+<P><FONT SIZE=2>created by myrna@golux.com<BR>last updated on
+04/18/2005 by: m.v.lunteren@gmail.com</FONT></P>
+<UL>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#intro">1. Introduction</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#quickstart">2.
+ Quickstart</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#2.1_running_with_derby_classes_">2.1
+ running tests</A></P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#building_derbyTesting__running_with">2.2
+ building derbyTesting package</A></P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#run">3. More details on
+ running the derby functional tests</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#run1">3.1 Running 1 test</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#run2">3.2 Running suites
+ of tests</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#overview">4. Harness
+ internals for developers</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#ov1">4.1 Test types</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#ov2">4.2 Supporting
+ files for tests</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#ov3">4.3
+ <testname>_app.properties</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#ov4">4.4
+ <testname>_derby.properties</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#ov5">4.5 tmp files, out
+ files, master files, and canons</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#ov6">4.6 Masking and
+ comparing</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#Adding_a_new_test">4.7
+ Adding a new test</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#4.8_Suites_and_Adding_a_new_suite">4.8
+ Suites and adding a new suite</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#ov9">4.9 Running with a
+ new jvm</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#skipping">4.10 Skipping
+ a test</A>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"><A HREF="#frameworks">4.11
+ Frameworks</A>
+ </P>
+ <LI><P><A HREF="#props">4.12 Some test harness properties</A>
+ </P>
+ <LI><P><A HREF="#security">4.13 SecuirtyManager testing by default</A></P>
+</UL>
+<P><BR><BR>
+</P>
+<H2><A NAME="intro"></A>1. Introduction</H2>
+<P>This document describes functionality of the derby functional
+testing package org.apache.derbyTesting. This package is based on the
+functional tests in use at IBM for testing the Cloudscape product
+before its contribution to ASF.</P>
+<P>In the following, instructions are geared towards a unix
environment. For other environments, some details may need to be
-adjusted. For instance, the document may
-refer to $ANT_HOME, for DOS, this would be %ANT_HOME%.<br>
-</p>
-<p>In the following the top
-directory under which the subversion tree is placed is referred to as
-${derby.source} - see also the
-derby <a href="http://svn.apache.org/repos/asf/db/derby/code/trunk/BUILDING.txt">BUILDING.txt</a>.<br>
-</p>
-<p>The version of the classes and supporting files of the derbyTesting
-package have to match the version of the classes of the derby package.
-Thus you either need to build all jars yourself, or get all jar files
-from the Derby site at the same time when available. <br>
-<br>
-</p>
-<span style="font-weight: bold;">
-</span>
-<h2><a class="mozTocH2" name="mozTocId191589"></a>2. <a
- name="quickstart"></a>QuickStart<br>
-</h2>
-<h3><a name="2.1_running_with_derby_classes_"></a>2.1 running tests</h3>
-<p>
-The derbyTesting package enables you to run 1 test or a suite of tests.
-Before you can run, you need to setup your environment:<br>
-</p>
-<ul>
- <li>Obtain a jdk or jre (based on jdk 1.3.1 specification <small><a
- href="README.htm#Note1:"><small>See Note1</small></a></small> or
-higher).
-Add the bin directory to your $PATH. Currently supported are:<br>
- </li>
-</ul>
-<table
- style="text-align: left; width: 727px; height: 40px; margin-left: 40px;"
- border="1" cellpadding="2" cellspacing="2">
- <tbody>
- <tr>
- <td style="vertical-align: top;"><small> jdk131
-- Sun
-HotSpot jdk1.3.1</small><br>
- <small> jdk141 - Sun HotSpot jdk1.4.1</small><br>
- <small> jdk142 - Sun HotSpot jdk1.4.2</small><br>
- <small> jdk15 - Sun HotSpot jdk1.5</small><br>
- <small> ibm131 - IBM Classic jdk1.3.1</small><br>
- <small> ibm141 - IBM Classic jdk1.4.1</small><br>
- <small> ibm142 - IBM Classic jdk1.4.2</small><br>
- <small> j9_13 - WCTME jvm (available with IBM
- Websphere Studio Device Developer, 5.6), version 2.1</small><br>
- <small> j9_22 - WCTME jvm (available with IBM
- Websphere Client Technology Micro Edition, 5.7), version 2.2</small><br>
- <small> j9_foundation - WCTME jvm (available with IBM
- Websphere Client Technology Micro Edition, 5.7), version 2.2, foundation library <br>
- </small></td>
- </tr>
- </tbody>
-</table>
-<ul>
- <li>cd into a directory that does not have any colons or spaces in it.<br>
- </li>
- <li>set $CLASSPATH to include the following jars:</li>
- <ul>
- <li><small>jakarta-oro-2.0.8.jar</small></li>
- <small> oromatcher, obtain from <a
- href="http://jakarta.apache.org/oro/index.html">http://jakarta.apache.org/oro/index.html</a>,
-or follow this link for <a
- href="http://apache.roweboat.net/jakarta/oro/source/jakarta-oro-2.0.8.zip">zip</a>
-file, or <a
- href="http://apache.roweboat.net/jakarta/oro/source/jakarta-oro-2.0.8.tar.gz">tar</a>.gz)</small><li><small>derbyTesting.jar<br>
- </small></li>
- </ul>
- <ul>
- <small> test files and classes</small><li><small>derby.jar<br>
- </small></li>
- <small> main derby package classes</small><li><small>derbytools.jar<br>
- </small></li>
- <small> derby tools classes for tools like ij
-and dblook</small><li><small>derbynet.jar<br>
- </small></li>
- <small> derby network server classes</small><li><small>derbyclient.jar<br>
- </small></li>
- <small> derby client classes</small><li><small>db2jcc.jar
-and db2jcc_license_c.jar<br>
- </small></li>
- <small> IBM Universal JDBC Driver classes. (See
-IBM <a href="http://www-106.ibm.com/developerworks/db2/downloads/jcc/">developerworks</a>
-for download)</small><br>
- <li><small>derbyLocale_*.jar</small></li>
- <small> locale files holding translated messages.</small><br>
- </ul>
-</ul>
-<small></small>
-<ul>
- <ul>
- </ul>
-</ul>
-<p>
-For example:<br>
-</p>
-<div style="margin-left: 40px;">
-<table style="text-align: left; width: 484px; height: 32px;" border="1"
- cellpadding="2" cellspacing="2">
- <tbody>
- <tr>
- <td style="vertical-align: top;"><small>(note that $jardir is
-only a convenience variable and that the set CLASSPATH command below
-has carriage
-returns for formatting reasons):<br>
- </small><small>set jardir=/local/derbyjar<br>
-set
-CLASSPATH="$jardir/derby.jar:$jardir/derbytools.jar:$jardir/derbynet.jar:$jardir/derbyclient.jar:<br>
-$jardir/db2jcc.jar:$jardir/db2jcc_license_c.jar:<br>
-$jardir/derbyTesting.jar:/local/derby/tools/java/jakarta-oro-2.0.8.jar:<br>
-$jardir/derbyLocale_de_DE.jar:$jardir/derbyLocale_es.jar:$jardir/derbyLocale_fr.jar:<br>
-$jardir/derbyLocale_it.jar:$jardir/derbyLocale_ja_JP.jar:$jardir/derbyLocale_ko_KR.jar:<br>
-$jardir/derbyLocale_pt_BR.jar:$jardir/derbyLocale_zh_CN.jar:$jardir/derbyLocale_zh_TW.jar:<br>
-$CLASSPATH</small><br>
- <small>set PATH=/local/jdk141/bin:$PATH</small><br>
- </td>
- </tr>
- </tbody>
-</table>
-</div>
-<p>
-To run 1 test:
-</p>
-<table
- style="text-align: left; width: 514px; height: 32px; margin-left: 40px;"
- border="1" cellpadding="2" cellspacing="2">
- <tbody>
- <tr>
- <td style="vertical-align: top;">syntax:<br>
- <small>java
--D<testproperty>
-org.apache.derbyTesting.functionTests.harness.RunTest
-<testdir>/<testname></small><br>
- <small>where <br>
- </small>
- <ul>
- <li><small> <testproperty> are test specific
-properties, such as
-'framework' for the RunTest class. </small></li>
- <li><small> <testdir> is one of the
-directories under
-functionTests/tests where the actual test is located</small></li>
- <li><small> <testname> is the actual name of
-the test</small></li>
- </ul>
- <small>examples:<br>
-to run the test supersimple against the embedded driver:<br>
- </small> <small>java
-org.apache.derbyTesting.functionTests.harness.RunTest
-lang/supersimple.sql<br>
- <br>
-To run a test with network server, using the derbyclient driver, add
--Dframework=DerbyNetClient to the run.
-The test harness will to start
-network server at port 1527 or connect to a running one, run the test,
-and stop network server thereafter.<br>
-for example:<br>
- java </small><small>-Dframework=DerbyNetClient
- </small><small>org.apache.derbyTesting.functionTests.harness.RunTest
-lang/supersimple.sql<br>
- </small><small> </small></td>
- </tr>
- </tbody>
-</table>
-<p>
-A successful run will have a .pass file, and the output to the
+adjusted. For instance, the document may refer to $ANT_HOME, for DOS,
+this would be %ANT_HOME%.</P>
+<P>In the following the top directory under which the subversion tree
+is placed is referred to as ${derby.source} - see also the derby
+<A HREF="http://svn.apache.org/repos/asf/db/derby/code/trunk/BUILDING.txt">BUILDING.txt</A>.</P>
+<P>The version of the classes and supporting files of the
+derbyTesting package have to match the version of the classes of the
+derby package. Thus you either need to build all jars yourself, or
+get all jar files from the Derby site at the same time when
+available.
+</P>
+<H2><A NAME="mozTocId191589"></A><A NAME="quickstart"></A>2.
+QuickStart</H2>
+<H3><A NAME="2.1_running_with_derby_classes_"></A>2.1 running tests</H3>
+<P>The derbyTesting package enables you to run 1 test or a suite of
+tests. Before you can run, you need to setup your environment:</P>
+<UL>
+ <LI><P>Obtain a jdk or jre (based on jdk 1.3.1 specification <A HREF="#Note1:"><FONT SIZE=1>See
+ Note1</FONT></A> or higher). Add the bin directory to your $PATH.
+ Currently supported are:</P>
+</UL>
+<TABLE BORDER=1 CELLPADDING=2 CELLSPACING=2>
+ <TR>
+ <TD>
+ <P> <FONT SIZE=2>jdk131 - Sun HotSpot
+ jdk1.3.1</FONT><BR> <FONT SIZE=2>jdk141 - Sun
+ HotSpot jdk1.4.1</FONT><BR> <FONT SIZE=2>jdk142
+ - Sun HotSpot jdk1.4.2</FONT><BR> <FONT SIZE=2>jdk15
+ - Sun HotSpot jdk1.5</FONT><BR> <FONT SIZE=2>ibm131
+ - IBM Classic jdk1.3.1</FONT><BR> <FONT SIZE=2>ibm141
+ - IBM Classic jdk1.4.1</FONT><BR> <FONT SIZE=2>ibm142
+ - IBM Classic jdk1.4.2</FONT><BR> <FONT SIZE=2>j9_13
+ - WCTME jvm (available with IBM Websphere Studio Device Developer,
+ 5.6), version 2.1</FONT><BR> <FONT SIZE=2>j9_22
+ - WCTME jvm (available with IBM Websphere Client Technology Micro
+ Edition, 5.7), version 2.2</FONT><BR>
+ <FONT SIZE=2>j9_foundation - WCTME jvm (available with IBM
+ Websphere Client Technology Micro Edition, 5.7), version 2.2,
+ foundation library </FONT>
+ </P>
+ </TD>
+ </TR>
+</TABLE>
+<UL>
+ <LI><P STYLE="margin-bottom: 0in">cd into a directory that does not
+ have any colons or spaces in it.</P>
+ <LI><P STYLE="margin-bottom: 0in">set $CLASSPATH to include the
+ following jars:
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in"><FONT SIZE=2>jakarta-oro-2.0.8.jar</FONT>
+ <FONT SIZE=2>oromatcher, obtain from
+ <A HREF="http://jakarta.apache.org/oro/index.html">http://jakarta.apache.org/oro/index.html</A>,
+ or follow this link for <A HREF="http://apache.roweboat.net/jakarta/oro/source/jakarta-oro-2.0.8.zip">zip</A>
+ file, or <A HREF="http://apache.roweboat.net/jakarta/oro/source/jakarta-oro-2.0.8.tar.gz">tar</A>.gz)</FONT></P>
+ <LI><P STYLE="margin-bottom: 0in"><FONT SIZE=2>derbyTesting.jar</FONT></P>
+ <P STYLE="margin-bottom: 0in"> <FONT SIZE=2>test
+ files and classes</FONT></P>
+ <LI><P STYLE="margin-bottom: 0in"><FONT SIZE=2>derby.jar<BR></FONT>
+ <FONT SIZE=2>main derby package classes</FONT></P>
+ <LI><P STYLE="margin-bottom: 0in"><FONT SIZE=2>derbytools.jar<BR></FONT>
+ <FONT SIZE=2>derby tools classes for tools like ij and dblook</FONT></P>
+ <LI><P STYLE="margin-bottom: 0in"><FONT SIZE=2>derbynet.jar<BR></FONT>
+ <FONT SIZE=2>derby network server classes</FONT></P>
+ <LI><P STYLE="margin-bottom: 0in"><FONT SIZE=2>derbyclient.jar<BR></FONT>
+ <FONT SIZE=2>derby client classes</FONT></P>
+ <LI><P STYLE="margin-bottom: 0in"><FONT SIZE=2>db2jcc.jar and
+ db2jcc_license_c.jar<BR></FONT> <FONT SIZE=2>IBM
+ Universal JDBC Driver classes. (See IBM <A HREF="http://www-106.ibm.com/developerworks/db2/downloads/jcc/">developerworks</A>
+ for download)</FONT></P>
+ <LI><P><FONT SIZE=2>derbyLocale_*.jar</FONT>
+ <FONT SIZE=2>locale files holding translated messages.</FONT></P>
+ </UL>
+</UL>
+<P>For example:</P>
+<DL>
+ <DD>
+ <TABLE BORDER=1 CELLPADDING=2 CELLSPACING=2>
+ <TR>
+ <TD>
+ <P><FONT SIZE=2>(note that $jardir is only a convenience variable
+ and that the set CLASSPATH command below has carriage returns for
+ formatting reasons):<BR>set jardir=/local/derbyjar<BR>set
+ CLASSPATH="$jardir/derby.jar:$jardir/derbytools.jar:$jardir/derbynet.jar:$jardir/derbyclient.jar:<BR>$jardir/db2jcc.jar:$jardir/db2jcc_license_c.jar:<BR>$jardir/derbyTesting.jar:/local/derby/tools/java/jakarta-oro-2.0.8.jar:<BR>$jardir/derbyLocale_de_DE.jar:$jardir/derbyLocale_es.jar:$jardir/derbyLocale_fr.jar:<BR>$jardir/derbyLocale_it.jar:$jardir/derbyLocale_ja_JP.jar:$jardir/derbyLocale_ko_KR.jar:<BR>$jardir/derbyLocale_pt_BR.jar:$jardir/derbyLocale_zh_CN.jar:$jardir/derbyLocale_zh_TW.jar:<BR>$CLASSPATH</FONT><BR><FONT SIZE=2>set
+ PATH=/local/jdk141/bin:$PATH</FONT></P>
+ </TD>
+ </TR>
+ </TABLE>
+</DL>
+<P>To run 1 test:
+</P>
+<TABLE BORDER=1 CELLPADDING=2 CELLSPACING=2>
+ <TR>
+ <TD>
+ <P>syntax:<BR> <FONT SIZE=2>java
+ -D<testproperty>
+ org.apache.derbyTesting.functionTests.harness.RunTest
+ <testdir>/<testname></FONT><BR><FONT SIZE=2>where </FONT>
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in"> <FONT SIZE=2><testproperty>
+ are test specific properties, such as 'framework' for the RunTest
+ class. </FONT>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"> <FONT SIZE=2><testdir>
+ is one of the directories under functionTests/tests where the
+ actual test is located</FONT>
+ </P>
+ <LI><P> <FONT SIZE=2><testname> is the actual
+ name of the test</FONT>
+ </P>
+ </UL>
+ <P><FONT SIZE=2>examples:<BR>to run the test supersimple against
+ the embedded driver:<BR></FONT> <FONT SIZE=2>java
+ org.apache.derbyTesting.functionTests.harness.RunTest
+ lang/supersimple.sql<BR><BR>To run a test with network server,
+ using the derbyclient driver, add -Dframework=DerbyNetClient to
+ the run. The test harness will to start network server at port
+ 1527 or connect to a running one, run the test, and stop network
+ server thereafter.<BR>for example:<BR>
+ java -Dframework=DerbyNetClient
+ org.apache.derbyTesting.functionTests.harness.RunTest
+ lang/supersimple.sql</FONT></P>
+ </TD>
+ </TR>
+</TABLE>
+<P>A successful run will have a .pass file, and the output to the
console will show no difference between expected and actual test
result. A failed test run will have at least a .fail file and the
output to the console will show the difference between expected and
-actual result.<br>
-</p>
-<p>
-To run a suite:
-</p>
-<table
- style="text-align: left; width: 546px; height: 32px; margin-left: 40px;"
- border="1" cellpadding="2" cellspacing="2">
- <tbody>
- <tr>
- <td style="vertical-align: top;">syntax:<br>
- <small>java
--D<testproperty>
-org.apache.derbyTesting.functionTests.harness.RunSuite
-<testsuite></small><br>
- <small>where <br>
- </small>
- <ul>
- <li><small> <testproperty> are test specific
-properties, such as
-'verbose' for the RunSuite class. </small></li>
- <li><small> <testsuite> is one of the suites
-under
-org/apache/derbyTesting/suites</small></li>
- </ul>
- <small>for example for running the suite derbylang:<br>
- </small><small> java -Dverbose=true
-org.apache.derbyTesting.functionTests.harness.RunSuite derbylang</small><br>
- </td>
- </tr>
- </tbody>
-</table>
-<p>
-Each suite run should be started in a clean directory. The test
-output directory will not be emptied out before testing is
-begun, although individual test files and result files will be cleaned
-out and overwritten.
-</p>
-<p>
-To run a suite with J2ME/CDC/Foundation:
-</p>
-<table
- style="text-align: left; width: 546px; height: 32px; margin-left: 40px;"
- border="1" cellpadding="2" cellspacing="2">
- <tbody>
- <tr>
- <td style="vertical-align: top;"><small>(note that the command below
-has carriage returns for formatting reasons):<br>
- syntax:<br></small>
- <small><java>
--Xbootclasspath/a:$jardir/<jdbc_jar> -Dbootcp=$jardir/<jdbc_jar> <br>
--Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource <br>
--Djvmflags="-Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource" <br>
-org.apache.derbyTesting.functionTests.harness.RunSuite
-<testsuite></small><br>
- <small>where <br>
- </small>
- <ul>
- <li><small> <java> is the jvm with any jvm flags.
- For testing with CDC/Foundation Profile using j9 jvm from IBM,
- this will be 'j9 -jcl:foun10'. </small></li>
- <li><small> <jdbc_jar> is the jar file which implements
- the JDBC API for CDC/Foundation Profile (JSR169). </small></li>
- <li><small> <testsuite> is one of the suites under
-org/apache/derbyTesting/suites</small></li>
- </ul>
- <small>for example for running the suite derbyall:<br>
- </small><small> j9 -jcl:foun10 -Xbootclasspath/a:$jardir/jdbc.jar
- -Dbootcp=$jardir/jdbc.jar <br>
- -Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource <br>
- -Djvmflags="-Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource" <br>
- org.apache.derbyTesting.functionTests.harness.RunSuite derbyall</small><br>
- </td>
- </tr>
- </tbody>
-</table>
-<p>
-The suites provided are:
-</p>
-<ul>
- <li>derbylang: <br>
- </li>
- <ul>
- <li>basic functionality of
-language implementation in derby. <br>
- </li>
- <li>Mostly .sql type tests. <br>
- </li>
- <li>tested on a variety of hardware takes from 1.15m to 2.00 hours<br>
- </li>
- </ul>
- <li>derbynetclientmats</li>
- <ul>
- <li>basic network server tests using the derby client<br>
- </li>
- <li>variety of tests, including some from derbylang suite</li>
- <li>tested on a variety of hardware takes from 15 to 30 minutes</li>
- </ul>
- <li>derbynetmats</li>
- <ul>
- <li>basic network server tests using the IBM Universal JDBC driver<br>
- </li>
- <li>variety of tests, including some from derbylang suite</li>
- <li>tested on a variety of hardware takes from 15 to 30 minutes</li>
- </ul>
- <li>derbynetautostart</li>
- <ul>
- <li>tests network server functionality without requiring
-network server framework<br>
- </li>
- </ul>
- <li>propertyinfo</li>
- <ul>
- <li>runs test to get property information<br>
- </li>
- </ul>
- <li>storeall</li>
- <ul>
- <li>tests for storage area</li>
- <li>includes:</li>
- <ul>
- <li>storemats: most basic quick verification tests.<br>
- </li>
- <li>storemore: more extensive storage tests</li>
- <li>storetests: set of store tests grouped together because they
-do not each need to create a new database</li>
- </ul>
- <li>tested on a variety of hardware takes from 25 to 50 minutes<br>
- </li>
- </ul>
- <li>xa</li>
- <ul>
- <li>tests the xa implementation. There is both a storage and
-language element to these tests</li>
- <li>tested on a variety of hardware takes from 2 to 4 minutes<br>
- </li>
- </ul>
- <li>storeunit</li>
- <ul>
- <li>tests store-related unit tests. Runs from 8 to 15 minutes<br>
- </li>
- </ul>
- <li>unit <br>
- </li>
- <ul>
- <li>tests 4 general functionality unit tests. runs from 5 to 10
-minutes<br>
- </li>
- </ul>
- <li>jdbcapi</li>
- <ul>
- <li>tests implementation of jdbc api such as Connection class
-implementation, Metadata etc.</li>
- <li>takes from 20 to 40 minutes<br>
- </li>
- </ul>
- <li>jdbc20<br>
- </li>
- <ul>
- <li>tests implementation of features from the jdbc 20 specification</li>
- <li>takes 2 to 5 minutes<br>
- </li>
- </ul>
- <li>jdk14</li>
- <ul>
- <li>tests implementation of features from the jdk14 specification</li>
- <li>takes 2 to 5 minutes<br>
- </li>
- </ul>
- <li>demo, simpledemo<br>
- </li>
- <ul>
- <li>tests the SimpleApp example <br>
- </li>
- <li>simpledemo runs SimpleApp itself - and thus has a different
-default resource package name (namely, no package) than all the other
-tests. Hence it needed its own suite.properties file.</li>
- <li>takes 30 to 1 minute</li>
- </ul>
- <li>nist</li>
- <ul>
- <li>test obtained from the NIST SQL suite v 6.0</li>
- <li>takes 5 to 10 minutes<br>
- </li>
- </ul>
- <li>encryptionAll <br>
- </li>
- <ul>
- <li>takes 30 to 55 minutes</li>
- <li>runs a few encryption tests plus the following encryption tests
-suites<br>
- </li>
- </ul>
- <ul>
- <ul>
- <li>encryption</li>
- </ul>
- <ul>
- <ul>
- <li>runs the storemats, sysinfo and multi suites in encryption
-scheme DESede<br>
- </li>
- </ul>
- <ul>
- <li>takes 25 to 40 minutes</li>
- </ul>
- </ul>
- <ul>
- <li>encryptionAES - tests AES encryption scheme<br>
- </li>
- </ul>
- <ul>
- <li>encryptionBlowfish - tests Blowfish encryption scheme<br>
- </li>
- </ul>
- <ul>
- <li>encryptionCFB - tests CFB encryption scheme<br>
- </li>
- </ul>
- <ul>
- <li>encryptionDES - tests DES encryption scheme<br>
- </li>
- </ul>
- <ul>
- <li>encryptionECB - tests ECB encryption scheme<br>
- </li>
- </ul>
- <ul>
- <li>encryptionOFB - tests OFB encryption scheme</li>
- </ul>
- </ul>
- <ul>
- </ul>
- <li>multi</li>
- <ul>
- <li>runs a simple test case with 10 threads</li>
- <li>runs for 10 minutes, then shuts down all threads<br>
- </li>
- </ul>
- <li>derbytools<br>
- </li>
- <ul>
- <li>tests for dblook, ij, and import/export utilities</li>
- <li>takes 5 to 10 minutes<br>
- </li>
- </ul>
- <li>i18nTest</li>
- <ul>
- <li>tests that characters outside simple ascii scope do not result
-in errors. </li>
- <li>takes 5 to 10 minutes<br>
- </li>
- </ul>
- <li>derbyall</li>
- <ul>
- <li>contains all suites typically run by all developers<br>
- </li>
- </ul>
- <ul>
- <li>tested on a variety of hardware takes from 3.00 - 6.00 hours</li>
- </ul>
- <li>largeData<br>
- </li>
- <ul>
- <li>Contains tests that deal with large amounts of data and thus
-require more machine resources. This suite is NOT run as part of
-'derbyall' because the tests it contains require either 1) more machine
-resources than what the typical Derby developer might have, and/or 2) a
-significant amount of time to run, and thus shouldn't be run every
-night.<br>
- </li>
- </ul>
- <ul>
- <li>As tests are added to this quite, it could require more and
-more
-time to run (several minutes to several hours to several days), which
-is why it is NOT included as part of the derbyall suite.<br>
- </li>
- </ul>
- <li><a href="#Note2:"><small>See Note2</small></a><br>
- </li>
-</ul>
-<p>
-A successful run with all tests passing will have no *.fail files
+actual result.</P>
+<P>To run a suite:
+</P>
+<TABLE BORDER=1 CELLPADDING=2 CELLSPACING=2>
+ <TR>
+ <TD>
+ <P>syntax:<BR> <FONT SIZE=2>java -D<testproperty>
+ org.apache.derbyTesting.functionTests.harness.RunSuite
+ <testsuite></FONT><BR><FONT SIZE=2>where </FONT>
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in"> <FONT SIZE=2><testproperty>
+ are test specific properties, such as 'verbose' for the RunSuite
+ class. </FONT>
+ </P>
+ <LI><P> <FONT SIZE=2><testsuite> is one of the
+ suites under org/apache/derbyTesting/suites</FONT>
+ </P>
+ </UL>
+ <P><FONT SIZE=2>for example for running the suite
+ derbylang:<BR></FONT> <FONT SIZE=2>java -Dverbose=true
+ org.apache.derbyTesting.functionTests.harness.RunSuite derbylang</FONT></P>
+ </TD>
+ </TR>
+</TABLE>
+<P>Each suite run should be started in a clean directory. The test
+output directory will not be emptied out before testing is begun,
+although individual test files and result files will be cleaned out
+and overwritten.
+</P>
+<P>To run a suite with J2ME/CDC/Foundation:
+</P>
+<TABLE BORDER=1 CELLPADDING=2 CELLSPACING=2>
+ <TR>
+ <TD>
+ <P><FONT SIZE=2>(note that the command below has carriage returns
+ for formatting reasons):<BR>syntax:<BR></FONT> <FONT SIZE=2><java>
+ -Xbootclasspath/a:$jardir/<jdbc_jar>
+ -Dbootcp=$jardir/<jdbc_jar>
+ <BR>-Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource
+ <BR>-Djvmflags="-Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource"
+ <BR>org.apache.derbyTesting.functionTests.harness.RunSuite
+ <testsuite></FONT><BR><FONT SIZE=2>where </FONT>
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in"> <FONT SIZE=2><java>
+ is the jvm with any jvm flags. For testing with CDC/Foundation
+ Profile using j9 jvm from IBM, this will be 'j9 -jcl:foun10'. </FONT>
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"> <FONT SIZE=2><jdbc_jar>
+ is the jar file which implements the JDBC API for CDC/Foundation
+ Profile (JSR169). </FONT>
+ </P>
+ <LI><P> <FONT SIZE=2><testsuite> is one of the
+ suites under org/apache/derbyTesting/suites</FONT>
+ </P>
+ </UL>
+ <P><FONT SIZE=2>for example for running the suite
+ derbyall:<BR></FONT> <FONT SIZE=2>j9 -jcl:foun10
+ -Xbootclasspath/a:$jardir/jdbc.jar -Dbootcp=$jardir/jdbc.jar
+ <BR>-Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource
+ <BR>-Djvmflags="-Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource"
+ <BR>org.apache.derbyTesting.functionTests.harness.RunSuite
+ derbyall</FONT></P>
+ </TD>
+ </TR>
+</TABLE>
+<P>The suites provided are:
+</P>
+<UL>
+ <LI><P STYLE="margin-bottom: 0in">derbylang:
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">basic functionality of
+ language implementation in derby.
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">Mostly .sql type tests.
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">tested on a variety of hardware
+ takes from 1.15m to 2.00 hours</P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">derbynetclientmats
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">basic network server tests using
+ the derby client</P>
+ <LI><P STYLE="margin-bottom: 0in">variety of tests, including some
+ from derbylang suite
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">tested on a variety of hardware
+ takes from 15 to 30 minutes
+ </P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">derbynetmats
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">basic network server tests using
+ the IBM Universal JDBC driver</P>
+ <LI><P STYLE="margin-bottom: 0in">variety of tests, including some
+ from derbylang suite
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">tested on a variety of hardware
+ takes from 15 to 30 minutes
+ </P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">derbynetautostart
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">tests network server
+ functionality without requiring network server framework</P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">propertyinfo
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">runs test to get property
+ information</P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">storeall
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">tests for storage area
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">includes:
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">storemats: most basic quick
+ verification tests.</P>
+ <LI><P STYLE="margin-bottom: 0in">storemore: more extensive
+ storage tests
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">storetests: set of store tests
+ grouped together because they do not each need to create a new
+ database
+ </P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">tested on a variety of hardware
+ takes from 25 to 50 minutes</P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">xa
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">tests the xa implementation.
+ There is both a storage and language element to these tests
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">tested on a variety of hardware
+ takes from 2 to 4 minutes</P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">storeunit
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">tests store-related unit tests.
+ Runs from 8 to 15 minutes</P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">unit
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">tests 4 general functionality
+ unit tests. runs from 5 to 10 minutes</P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">jdbcapi
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">tests implementation of jdbc api
+ such as Connection class implementation, Metadata etc.
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">takes from 20 to 40 minutes</P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">jdbc20</P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">tests implementation of features
+ from the jdbc 20 specification
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">takes 2 to 5 minutes</P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">jdk14
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">tests implementation of features
+ from the jdk14 specification
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">takes 2 to 5 minutes</P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">demo, simpledemo</P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">tests the SimpleApp example
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">simpledemo runs SimpleApp itself
+ - and thus has a different default resource package name (namely,
+ no package) than all the other tests. Hence it needed its own
+ suite.properties file.
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">takes 30 to 1 minute
+ </P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">nist
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">test obtained from the NIST SQL
+ suite v 6.0
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">takes 5 to 10 minutes</P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">encryptionAll
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">takes 30 to 55 minutes
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">runs a few encryption tests plus
+ the following encryption tests suites</P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">encryption
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">runs the storemats, sysinfo and
+ multi suites in encryption scheme DESede</P>
+ <LI><P STYLE="margin-bottom: 0in">takes 25 to 40 minutes
+ </P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">encryptionAES - tests AES
+ encryption scheme</P>
+ <LI><P STYLE="margin-bottom: 0in">encryptionBlowfish - tests
+ Blowfish encryption scheme</P>
+ <LI><P STYLE="margin-bottom: 0in">encryptionCFB - tests CFB
+ encryption scheme</P>
+ <LI><P STYLE="margin-bottom: 0in">encryptionDES - tests DES
+ encryption scheme</P>
+ <LI><P STYLE="margin-bottom: 0in">encryptionECB - tests ECB
+ encryption scheme</P>
+ <LI><P STYLE="margin-bottom: 0in">encryptionOFB - tests OFB
+ encryption scheme
+ </P>
+ </UL>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">multi
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">runs a simple test case with 10
+ threads
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">runs for 10 minutes, then shuts
+ down all threads</P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">derbytools</P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">tests for dblook, ij, and
+ import/export utilities
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">takes 5 to 10 minutes</P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">i18nTest
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">tests that characters outside
+ simple ascii scope do not result in errors.
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">takes 5 to 10 minutes</P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">derbyall
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">contains all suites typically run
+ by all developers</P>
+ <LI><P STYLE="margin-bottom: 0in">tested on a variety of hardware
+ takes from 3.00 - 6.00 hours
+ </P>
+ </UL>
+ <LI><P STYLE="margin-bottom: 0in">largeData</P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">Contains tests that deal with
+ large amounts of data and thus require more machine resources.
+ This suite is NOT run as part of 'derbyall' because the tests it
+ contains require either 1) more machine resources than what the
+ typical Derby developer might have, and/or 2) a significant amount
+ of time to run, and thus shouldn't be run every night.</P>
+ <LI><P STYLE="margin-bottom: 0in">As tests are added to this quite,
+ it could require more and more time to run (several minutes to
+ several hours to several days), which is why it is NOT included as
+ part of the derbyall suite.</P>
+ </UL>
+ <LI><P><A HREF="#Note2:"><FONT SIZE=2>See Note2</FONT></A></P>
+</UL>
+<P>A successful run with all tests passing will have no *.fail files
created, the <testsuite>_fail.txt file will be empty, and the
-<testsuite>_report.txt file will show no failures in the Summary
-results section.
-</p>
-<table
- style="text-align: left; width: 556px; height: 32px; margin-left: 40px;"
- border="1" cellpadding="2" cellspacing="2">
- <tbody>
- <tr>
- <td style="vertical-align: top;"><small>-------snippet from
-derbylang_report.txt -----<br>
------------------------------------------------------------<br>
-Summary results:<br>
- <br>
-Test Run Started: 2004-11-10 11:27:55.0<br>
-Test Run Duration: 00:04:09<br>
- <br>
-129 Tests Run<br>
-100% Pass (129 tests passed)<br>
- 0% Fail (0 tests failed)<br>
-0 Suites skipped</small><br>
- </td>
- </tr>
- </tbody>
-</table>
-<br>
-<h3><a name="building_derbyTesting__running_with"></a>2.2 building
-derbyTesting package<br>
-</h3>
-<p>
-To build the derbyTesting package:<br>
-</p>
-<ul>
- <li>follow all the steps in the derby <a
- href="http://svn.apache.org/repos/asf/db/derby/code/trunk/BUILDING.txt">BUILDING.txt</a>.</li>
-</ul>
-<p>This is some typical
-output for the ant build process.<br>
-</p>
-<table
- style="text-align: left; width: 516px; height: 32px; margin-left: 40px;"
- border="1" cellpadding="2" cellspacing="2">
- <tbody>
- <tr>
- <td style="vertical-align: top;"><small>> cd
-/local/derby/java/testing<br>
-> ant.ksh<br>
-Searching for build.xml ...<br>
-Buildfile: /local/derby/java/testing/build.xml<br>
- <br>
-compile:<br>
- [javac] Compiling 30 source files to
-/local/derby/classes<br>
-...<br>
- [copy] Copying 1 file to
-/local/derby/classes/org/apache/derbyTesting/funct<br>
-ionTests<br>
- <br>
-BUILD SUCCESSFUL<br>
-Total time: 10 minutes 3 seconds</small></td>
- </tr>
- </tbody>
-</table>
-<p>Building using the ant all target places all files, that is,
-classes, but also supporting files such as expected output (*.out), sql
-test files (*.sql), properties files and any data files used in
+<testsuite>_report.txt file will show no failures in the
+Summary results section.
+</P>
+<TABLE BORDER=1 CELLPADDING=2 CELLSPACING=2>
+ <TR>
+ <TD>
+ <P><FONT SIZE=2>-------snippet from derbylang_report.txt
+ -----<BR>-----------------------------------------------------------<BR>Summary
+ results:<BR><BR>Test Run Started: 2004-11-10 11:27:55.0<BR>Test
+ Run Duration: 00:04:09<BR><BR>129 Tests Run<BR>100% Pass (129
+ tests passed)<BR> 0% Fail (0 tests failed)<BR>0 Suites
+ skipped</FONT></P>
+ </TD>
+ </TR>
+</TABLE>
+<P><BR><BR>
+</P>
+<H3><A NAME="building_derbyTesting__running_with"></A>2.2 building
+derbyTesting package</H3>
+<P>To build the derbyTesting package:</P>
+<UL>
+ <LI><P>follow all the steps in the derby <A HREF="http://svn.apache.org/repos/asf/db/derby/code/trunk/BUILDING.txt">BUILDING.txt</A>.
+ </P>
+</UL>
+<P>This is some typical output for the ant build process.</P>
+<TABLE BORDER=1 CELLPADDING=2 CELLSPACING=2>
+ <TR>
+ <TD>
+ <P><FONT SIZE=2>> cd /local/derby/java/testing<BR>>
+ ant.ksh<BR>Searching for build.xml ...<BR>Buildfile:
+ /local/derby/java/testing/build.xml<BR><BR>compile:<BR>
+ [javac] Compiling 30 source files to /local/derby/classes<BR>...<BR>
+ [copy] Copying 1 file to
+ /local/derby/classes/org/apache/derbyTesting/funct<BR>ionTests<BR><BR>BUILD
+ SUCCESSFUL<BR>Total time: 10 minutes 3 seconds</FONT></P>
+ </TD>
+ </TR>
+</TABLE>
+<P>Building using the ant all target places all files, that is,
+classes, but also supporting files such as expected output (*.out),
+sql test files (*.sql), properties files and any data files used in
individual tests into the classes directory so they can all be found
-using the CLASSPATH. <br>
-</p>
-<p>Once you have built the derbyTesting package, you can make a
+using the CLASSPATH.
+</P>
+<P>Once you have built the derbyTesting package, you can make a
derbyTesting.jar using the jar build target at the
-${derby.source}level.
-</p>
-<p>
-This will look something like:
-</p>
-<table
- style="text-align: left; width: 528px; height: 32px; margin-left: 40px;"
- border="1" cellpadding="2" cellspacing="2">
- <tbody>
- <tr>
- <td style="vertical-align: top;"><small>c:> ant derbytestingjar<br>
-Searching for build.xml ...<br>
-Buildfile: C:\derby\build.xml<br>
- <br>
-initjars:<br>
- [mkdir] Created dir: C:\derby\jars\<br>
- [mkdir] Created dir: C:\derby\jars\lists<br>
- [echo] Revision number set to exported<br>
- [echo] .<br>
- <br>
-derbytestingjar:<br>
- [echo] Beginning derbytesting.jar build<br>
-.....<br>
-BUILD SUCCESSFULL<br>
- </small></td>
- </tr>
- </tbody>
-</table>
-<br>
-<br>
-<h2><a class="mozTocH2" name="mozTocId582299"></a>3. <a name="run"></a>More
-details on running the derby functional tests</h2>
-<p>
-The functional tests are run using a class called 'RunTest'. This class
-calls a number of other classes. A group of tests, called a 'suite' is
-executed using a class called 'RunSuite'.<br>
-</p>
-<h3><a class="mozTocH3" name="mozTocId595945"></a>3.1 <a name="run1"></a>Running
-1 test</h3>
-<p>See section 2.1 for the basic steps to run 1 test.
-</p>
-<p>To pass on system level properties to the test harness, use the test
-harness property -DtestSpecialProps. For example, to ensure extra
-information is appended to the log: <br>
-</p>
-<table
- style="text-align: left; width: 558px; height: 32px; margin-left: 40px;"
- border="1" cellpadding="2" cellspacing="2">
- <tbody>
- <tr>
- <td style="vertical-align: top;"> <small>java
--Dframework=DerbyNetClient
--DtestSpecialProps=derby.infolog.append=true
-org.apache.derbyTesting.functionTests.harness.RunTest
-lang/supersimple.sql</small></td>
- </tr>
- </tbody>
-</table>
-<p><br>
-Tests will be executed in the current directory. When
-running a test
-using the network server and derby client, i.e.
--Dframework=DerbyNetClient, the test will run
-in a subdirectory (automatically created) 'DerbyNetClient'. <small> <a
- href="#Note3:"></a><br>
-</small></p>
-<p>
-The test will normally create the following:<br>
-</p>
-<ul>
- <li>a database. The default name is 'wombat'. However, the name may
-be different depending on certain properties passed in to the test
-harness.</li>
- <li>a .out file: the final result file</li>
- <li>a .tmp file; the initial result file, before any modification to
-prevent irrelevant differences has been applied (before 'masking').</li>
- <li>a .diff file; the differences between the .out and the master
-file with expected output it is compared to.</li>
- <li>a .pass or .fail file. This file lists the test if it passes
-under .pass, and under .fail if the output in .out is different from
-the expected output in the master.</li>
-</ul>
-<p>
-possibly created:<br>
-</p>
-<ul>
- <li>additional files used in a specific test may get copied over to
-the test directory. These normally do not get cleaned up.</li>
- <li>.tmpstr file is created for network server tests and is a
-possibly
-massaged copy of the master file the output needs to be compared with.</li>
- <li>.err and .out files in network server database files for any
-additional error output.</li>
-</ul>
-<p>
-When the test is successful, cleanup will occur unless the test harness
-property -Dkeepfiles=true is used. Cleanup will attempt to cleanup all
-files except for .pass. <small><br>
-<a href="#Note3:">See Note3.</a></small>
-</p>
-<p>
-A successful run (this example is from a dos environment) would
-look for instance like:
-</p>
-<table
- style="text-align: left; width: 414px; height: 45px; margin-left: 80px;"
- border="1" cellpadding="2" cellspacing="2">
- <tbody>
- <tr>
- <td style="vertical-align: top;"><small>c:>
-derbyTesting.functionTests.harness.RunTest lang/supersimple.sql<br>
-C:\derby\run2<br>
-supersimple<br>
--- listing properties --<br>
-derby.locks.deadlockTimeout=3<br>
-derby.locks.waitTimeout=3<br>
-*** Start: supersimple jdk1.4.2_03 2004-11-10 16:51:02 ***<br>
-The test should be running...<br>
-MasterFileName = master/supersimple.out<br>
-*** End: supersimple jdk1.4.2_03 2004-11-10 16:51:25 ***<br>
- </small></td>
- </tr>
- </tbody>
-</table>
-<br>
-<p>A Test Failure shows the diff, creates a .fail file, does not create
-a .pass file, and does not cleanup any files upon completion. The
-output might look like this:<br>
-</p>
-<table
- style="text-align: left; width: 442px; height: 32px; margin-left: 80px;"
- border="1" cellpadding="2" cellspacing="2">
- <tbody>
- <tr>
- <td style="vertical-align: top;"><small> </small><small>c:>
-derbyTesting.functionTests.harness.RunTest lang/supersimple.sql<br>
-C:\derby\run2</small><small><br>
-supersimple<br>
--- listing properties --<br>
-derby.locks.deadlockTimeout=3<br>
-derby.locks.waitTimeout=3<br>
-*** Start: supersimple jdk1.4.2_03 2004-11-10 16:54:39 ***<br>
-The test should be running...<br>
-MasterFileName = master/supersimple.out<br>
-10 del<br>
-< 10<br>
-10a10<br>
-> 1<br>
-Test Failed.<br>
-*** End: supersimple jdk1.4.2_03 2004-11-10 16:55:02 ***</small><br>
- </td>
- </tr>
- </tbody>
-</table>
-<br>
-<h3><a class="mozTocH3" name="mozTocId368566"></a>3.2 <a name="run2"></a>Running
-a suite of tests</h3>
-<p>
-See section 2.1 for a basic explanation on how to run a suite of tests.<br>
-</p>
-<p>
-Tests will be run in a subdirectory with the name of the test
-suite under the current directory. Eg. for derbylang suite, a directory
-derbylang will be created. While the tests are run, information about
-the run is inserted into a <testsuite>.sum file. When all tests
-have completed summary files are created <testsuite>_pass.txt,
-_fail.txt, and _diff.txt files are created as well as a
-<testsuite>_report.txt
-with additional details. Some of the information is duplicate. Also, a
-.skip file will be created holding a list of the tests that were
-skipped (for more details on this, see the section on <a
- href="#skipping">skipping tests</a>).
-</p>
-<p>
-RunSuite does not empty the top level directory before running. Thus,
-if another suite was run in the same directory at an earlier time, the
-resulting summary files might contain results for more than the current
-run. Therefore it is important to run each suite in a clean directory.
-</p>
-<p>Sample output from RunSuite:<br>
-</p>
-<table
- style="text-align: left; width: 471px; height: 32px; margin-left: 80px;"
- border="1" cellpadding="2" cellspacing="2">
- <tbody>
- <tr>
- <td style="vertical-align: top;"><small><small>c:> $ java
-org.apache.derbyTesting.functionTests.harness.RunSuite derbylang<br>
-Top suite: derbylang<br>
-Suite to run: derbylang:derbylang<br>
-Now do RunList<br>
-Now run the suite's tests<br>
-Run the tests...<br>
-Execute command: java -DjavaCmd=java
--Doutputdir=C:\derbyt1\derbylang\derbylang
--Dtopsuitedir=C:\derbyt1\derbylang -Dtoprepo<br>
-rtdir=C:\derbyt1\derbylang -Drundir=C:\derbyt1
--Dsuitename=derbylang:derbylang -Dtopsuitename=derbylang
-org.apache.derbyTesting.functionTests.harness.RunTest
-lang/altertable.sql<br>
-Execute command: java -DjavaCmd=java
--Doutputdir=C:\derbyt1\derbylang\derbylang
--Dtopsuitedir=C:\derbyt1\derbylang -Dtopreportdir=C:\derbyt1\derbylang
--Drundir=C:\derbyt1 -Dsuitename=derbylang:derbylang
--Dtopsuitename=derbylang
-org.apache.derbyTesting.functionTests.harness.RunTest
-lang/arithmetic.sql<br>
-...(.more tests)....<br>
-Generated report: derbylang_report.txt</small></small><small><br>
- </small></td>
- </tr>
- </tbody>
-</table>
-<p>
-This output does not show whether the tests passed or failed. The
-Summary section in <testsuite>_report.txt shows the statistics of
-the passed vs. failed tests, the summary <testsuite>_*.txt files
-list the tests that passed and failed.
-</p>
-<br>
-<h2><a class="mozTocH2" name="mozTocId635355"></a>4. <a name="overview"></a>
-Harness internals for developers<br>
-</h2>
-<p>
-The following is intended for people who have the subversion tree
-available and want to add or modify tests.
-</p>
-<p>
-The test harness executing one test basically does the following in
-sequence:
-</p>
-<ul>
- <li>identify test to run</li>
- <li>identify properties to run with</li>
- <li>copy needed support files</li>
- <li>find the expected output</li>
- <li>if network server, start network server</li>
- <li>run the test, creating the database</li>
- <li>if network server, shutdown the server</li>
- <li>modify the output based on Sed class and _sed.properties file for
-the test</li>
- <li>compare expected output with actual output</li>
- <li>if pass, cleanup.</li>
-</ul>
-<br>
-<h3><a class="mozTocH3" name="mozTocId344499"></a>4.1 <a name="ov1"></a>Test
-types</h3>
-<p>
-The test harness recognizes, or will recognize tests with the following
-extensions:<br>
-</p>
-<ul>
- <li> .java tests that run in a separate jvm.</li>
- <li> .sql tests that run using ij</li>
- <li> .sql2 related to .sql</li>
- <li> .multi multi threaded tests. There is
-currently only 1 test being run. The multi test functions a little
-differently from .java and .sql* tests in that RunTest starts a
-separate harness class called MultiTest to control the details of the
-run. Also, the actual test files live under
-org/apache/derbyTesting/functionTests/multiTests, rather than
-org/apache/derbyTesting/functionTests/tests. <br>
- </li>
- <li> .unit unit tests. The unit tests
-actually refer to <testname>_derby.properties files under
-org/apache/derbyTesting/functionTests/tests/unit that activate the
-actual unit test harness and tests under
-org/apache/derbyTesting/unitTests. These tests test more underlying
-functionality than the (rest of the) functionTests, which are more
-geared toward how end-users might use functionality.<br>
- </li>
-</ul>
-<br>
-<h3><a class="mozTocH3" name="mozTocId809770"></a>4.2 <a name="ov2"></a>Supporting
-files for tests</h3>
-<p>
-Various additional files may be used by a test, for instance, to create
-large data values, to test using of jar files and the like. Any files
-that need to be accessed by a particular test that are not accessed
-from the classpath need to be listed under supportfiles= in the
-<testname>_app.properties file.<br>
-Tests can refer to classes without being in the classpath, and sql
-tests can use the ij command 'run resource ' to execute additional .sql
-files without changes to the _app.properties files.
-</p>
-<p>For example, in the file
-(org/apache/derbyTesting/functionTests/tests/)tools/dblook_test_app.properties:<br>
-<small>
-supportfiles=tools/dblook_makeDB.sql,tools/dblook_test.jar</small><br>
-</p>
-<h3><a class="mozTocH3" name="mozTocId427577"></a>4.3 <a name="ov3"></a><testname>_app.properties</h3>
-<p>
-Every test directory has a default_app.properties. This file is for
-system level properties generic to all the tests in that test
-directory. </p>
-<p>
-If a test requires different system level properties, a test specific
-properties file can be created to overwrite the defaults. The test
-specific properties file needs to have a name starting with the
-test file name, followed with _app.properties</p>
-<p>For example, for the test tools/dblook_test.java, there is a
-properties file called tools/dblook_test_app.properties<br>
-</p>
-<h3><a class="mozTocH3" name="mozTocId715566"></a>4.4 <a name="ov4"></a><testname>_derby.properties</h3>
-<p>
-Every test directory has a default_derby.properties. This file is for
-derby specific properties common to all the tests in that test
-directory.<br>
-If a test requires different derby properties, a test specific
-properties file can be created to overwrite the defaults. The test
-specific properties file needs to have a name starting with the
-test file name, followed with _derby.properties<br>
-<br>
-</p>
-<h3><a class="mozTocH3" name="mozTocId874096"></a>4.5 <a name="ov5"></a>tmp
-files, out files, master files, and canons</h3>
-<p>
-The test's output will be put into a file testname.tmp. Then the output
-is modified if masking is required and the result is put into a .out
-file.<br>
-The expected output is found by examining the following directories,
-based on certain input<br>
-</p>
-<ul>
- <li>functionTests/master/framework/jcc_version/jvmcode</li>
- <li>functionTests/master/framework/jcc_version/earlier_jvmcode</li>
- <li>functionTests/master/framework/jcc_version</li>
- <li>functionTests/master/framework/jvmcode</li>
- <li>functionTests/master/framework/earlier_jvmcode</li>
- <li>functionTests/master/jvmcode</li>
- <li>functionTests/master</li>
-</ul>
-<p>
-For example, if we are running a test and the flag -Dframework=DerbyNet
-is used, to use network server and the IBM Universal JDBC Driver
-('jcc'), and the jvm we are
-using is Sun's jdk 142, and the jcc version is 2.4 (not available at
-time of writing) then the search for the master to compare with
-starts in the functionTests/derbynet/jcc2.4/jdk14 directory. If a .out
-file with the same name as the test is found in that directory, that
-master is taken. If there is no such file in that directory, search
-continues in the directory functionTests/derbynet/jcc2.4/jdk13 if it
-exists.</p>
-<p>If there is no file there, nor for any other jcc directory, it will
-continue to derbynet/jdk14, and the search is continued for earlier jvm
-versions.<br>
-If we are not running network server, the DerbyNet and
-jcc_version directories are not traversed.<br>
-</p>
-<p>The version details do not go into the subversion level, i.e.
-running
-with jdk141 or jdk142 is expected to have the same behavior.
-</p>
-<p>
-This functionality supports dealing with minor differences in behavior
-caused by minor differences in behavior in the underlying jvms, jcc
-versions, differences between results returned through network server
-vs. embedded and minor differences between a debug and non debug (jar)
-build. </p>
-<p>
-However, having a large number of these files means a maintenance
+${derby.source}level.
+</P>
+<P>This will look something like:
+</P>
+<TABLE BORDER=1 CELLPADDING=2 CELLSPACING=2>
+ <TR>
+ <TD>
+ <P><FONT SIZE=2>c:> ant derbytestingjar<BR>Searching for
+ build.xml ...<BR>Buildfile: C:\derby\build.xml<BR><BR>initjars:<BR>
+ [mkdir] Created dir: C:\derby\jars\<BR> [mkdir]
+ Created dir: C:\derby\jars\lists<BR>
+ [echo] Revision number set to exported<BR>
+ [echo] .<BR><BR>derbytestingjar:<BR>
+ [echo] Beginning derbytesting.jar build<BR>.....<BR>BUILD
+ SUCCESSFULL</FONT></P>
+ </TD>
+ </TR>
+</TABLE>
+<P><BR><BR>
+</P>
+<H2><A NAME="mozTocId582299"></A><A NAME="run"></A>3. More details on
+running the derby functional tests</H2>
+<P>The functional tests are run using a class called 'RunTest'. This
+class calls a number of other classes. A group of tests, called a
+'suite' is executed using a class called 'RunSuite'.</P>
+<H3><A NAME="mozTocId595945"></A><A NAME="run1"></A>3.1 Running 1
+test</H3>
+<P>See section 2.1 for the basic steps to run 1 test.
+</P>
+<P>To pass on system level properties to the test harness, use the
+test harness property -DtestSpecialProps. For example, to ensure
+extra information is appended to the log: </P>
+<TABLE BORDER=1 CELLPADDING=2 CELLSPACING=2>
+ <TR>
+ <TD>
+ <P><FONT SIZE=2>java -Dframework=DerbyNetClient
+ -DtestSpecialProps=derby.infolog.append=true
+ org.apache.derbyTesting.functionTests.harness.RunTest
+ lang/supersimple.sql</FONT></P>
+ </TD>
+ </TR>
+</TABLE>
+<P><BR>Tests will be executed in the current directory. When running
+a test using the network server and derby client, i.e.
+-Dframework=DerbyNetClient, the test will run in a subdirectory
+(automatically created) 'DerbyNetClient'.
+</P>
+<P>The test will normally create the following:</P>
+<UL>
+ <LI><P STYLE="margin-bottom: 0in">a database. The default name is
+ 'wombat'. However, the name may be different depending on certain
+ properties passed in to the test harness.
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">a .out file: the final result file
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">a .tmp file; the initial result
+ file, before any modification to prevent irrelevant differences has
+ been applied (before 'masking').
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">a .diff file; the differences
+ between the .out and the master file with expected output it is
+ compared to.
+ </P>
+ <LI><P>a .pass or .fail file. This file lists the test if it passes
+ under .pass, and under .fail if the output in .out is different from
+ the expected output in the master.
+ </P>
+</UL>
+<P>possibly created:</P>
+<UL>
+ <LI><P STYLE="margin-bottom: 0in">additional files used in a
+ specific test may get copied over to the test directory. These
+ normally do not get cleaned up.
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">.tmpstr file is created for
+ network server tests and is a possibly massaged copy of the master
+ file the output needs to be compared with.
+ </P>
+ <LI><P>.err and .out files in network server database files for any
+ additional error output.
+ </P>
+</UL>
+<P>When the test is successful, cleanup will occur unless the test
+harness property -Dkeepfiles=true is used. Cleanup will attempt to
+cleanup all files except for .pass. <FONT SIZE=2><BR><A HREF="#Note3:">See
+Note3.</A></FONT>
+</P>
+<P>A successful run (this example is from a dos environment) would
+look for instance like:
+</P>
+<TABLE BORDER=1 CELLPADDING=2 CELLSPACING=2>
+ <TR>
+ <TD>
+ <P><FONT SIZE=2>c:> derbyTesting.functionTests.harness.RunTest
+ lang/supersimple.sql<BR>C:\derby\run2<BR>supersimple<BR>-- listing
+ properties
+ --<BR>derby.locks.deadlockTimeout=3<BR>derby.locks.waitTimeout=3<BR>***
+ Start: supersimple jdk1.4.2_03 2004-11-10 16:51:02 ***<BR>The test
+ should be running...<BR>MasterFileName =
+ master/supersimple.out<BR>*** End: supersimple
+ jdk1.4.2_03 2004-11-10 16:51:25 ***</FONT></P>
+ </TD>
+ </TR>
+</TABLE>
+<P><BR><BR>
+</P>
+<P>A Test Failure shows the diff, creates a .fail file, does not
+create a .pass file, and does not cleanup any files upon completion.
+The output might look like this:</P>
+<TABLE BORDER=1 CELLPADDING=2 CELLSPACING=2>
+ <TR>
+ <TD>
+ <P> <FONT SIZE=2>c:>
+ derbyTesting.functionTests.harness.RunTest
+ lang/supersimple.sql<BR>C:\derby\run2<BR>supersimple<BR>-- listing
+ properties
+ --<BR>derby.locks.deadlockTimeout=3<BR>derby.locks.waitTimeout=3<BR>***
+ Start: supersimple jdk1.4.2_03 2004-11-10 16:54:39 ***<BR>The test
+ should be running...<BR>MasterFileName = master/supersimple.out<BR>10
+ del<BR>< 10<BR>10a10<BR>> 1<BR>Test Failed.<BR>*** End:
+ supersimple jdk1.4.2_03 2004-11-10 16:55:02 ***</FONT></P>
+ </TD>
+ </TR>
+</TABLE>
+<P><BR><BR>
+</P>
+<H3><A NAME="mozTocId368566"></A><A NAME="run2"></A>3.2 Running a
+suite of tests</H3>
+<P>See section 2.1 for a basic explanation on how to run a suite of
+tests.</P>
+<P>Tests will be run in a subdirectory with the name of the test
+suite under the current directory. Eg. for derbylang suite, a
+directory derbylang will be created. While the tests are run,
+information about the run is inserted into a <testsuite>.sum
+file. When all tests have completed summary files are created
+<testsuite>_pass.txt, _fail.txt, and _diff.txt files are
+created as well as a <testsuite>_report.txt with additional
+details. Some of the information is duplicate. Also, a .skip file
+will be created holding a list of the tests that were skipped (for
+more details on this, see the section on <A HREF="#skipping">skipping
+tests</A>).
+</P>
+<P>RunSuite does not empty the top level directory before running.
+Thus, if another suite was run in the same directory at an earlier
+time, the resulting summary files might contain results for more than
+the current run. Therefore it is important to run each suite in a
+clean directory.
+</P>
+<P>Sample output from RunSuite:</P>
+<TABLE BORDER=1 CELLPADDING=2 CELLSPACING=2>
+ <TR>
+ <TD>
+ <P><FONT SIZE=1>c:> $ java
+ org.apache.derbyTesting.functionTests.harness.RunSuite
+ derbylang<BR>Top suite: derbylang<BR>Suite to run:
+ derbylang:derbylang<BR>Now do RunList<BR>Now run the suite's
+ tests<BR>Run the tests...<BR>Execute command: java -DjavaCmd=java
+ -Doutputdir=C:\derbyt1\derbylang\derbylang
+ -Dtopsuitedir=C:\derbyt1\derbylang
+ -Dtoprepo<BR>rtdir=C:\derbyt1\derbylang -Drundir=C:\derbyt1
+ -Dsuitename=derbylang:derbylang -Dtopsuitename=derbylang
+ org.apache.derbyTesting.functionTests.harness.RunTest
+ lang/altertable.sql<BR>Execute command: java -DjavaCmd=java
+ -Doutputdir=C:\derbyt1\derbylang\derbylang
+ -Dtopsuitedir=C:\derbyt1\derbylang
+ -Dtopreportdir=C:\derbyt1\derbylang -Drundir=C:\derbyt1
+ -Dsuitename=derbylang:derbylang -Dtopsuitename=derbylang
+ org.apache.derbyTesting.functionTests.harness.RunTest
+ lang/arithmetic.sql<BR>...(.more tests)....<BR>Generated report:
+ derbylang_report.txt</FONT></P>
+ </TD>
+ </TR>
+</TABLE>
+<P>This output does not show whether the tests passed or failed. The
+Summary section in <testsuite>_report.txt shows the statistics
+of the passed vs. failed tests, the summary <testsuite>_*.txt
+files list the tests that passed and failed.
+</P>
+<P><BR><BR>
+</P>
+<H2><A NAME="mozTocId635355"></A><A NAME="overview"></A>4. Harness
+internals for developers</H2>
+<P>The following is intended for people who have the subversion tree
+available and want to add or modify tests.
+</P>
+<P>The test harness executing one test basically does the following
+in sequence:
+</P>
+<UL>
+ <LI><P STYLE="margin-bottom: 0in">identify test to run
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">identify properties to run with
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">copy needed support files
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">find the expected output
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">if network server, start network
+ server
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">run the test, creating the
+ database
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">if network server, shutdown the
+ server
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">modify the output based on Sed
+ class and _sed.properties file for the test
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">compare expected output with
+ actual output
+ </P>
+ <LI><P>if pass, cleanup.
+ </P>
+</UL>
+<P><BR><BR>
+</P>
+<H3><A NAME="mozTocId344499"></A><A NAME="ov1"></A>4.1 Test types</H3>
+<P>The test harness recognizes, or will recognize tests with the
+following extensions:</P>
+<UL>
+ <LI><P STYLE="margin-bottom: 0in"> .java
+ tests that run in a separate jvm.
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"> .sql
+ tests that run using ij
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"> .sql2
+ related to .sql
+ </P>
+ <LI><P STYLE="margin-bottom: 0in"> .multi
+ multi threaded tests. There is currently only 1 test being run. The
+ multi test functions a little differently from .java and .sql* tests
+ in that RunTest starts a separate harness class called MultiTest to
+ control the details of the run. Also, the actual test files live
+ under org/apache/derbyTesting/functionTests/multiTests, rather than
+ org/apache/derbyTesting/functionTests/tests.
+ </P>
+ <LI><P> .unit unit tests. The unit tests
+ actually refer to <testname>_derby.properties files under
+ org/apache/derbyTesting/functionTests/tests/unit that activate the
+ actual unit test harness and tests under
+ org/apache/derbyTesting/unitTests. These tests test more underlying
+ functionality than the (rest of the) functionTests, which are more
+ geared toward how end-users might use functionality.</P>
+</UL>
+<P><BR><BR>
+</P>
+<H3><A NAME="mozTocId809770"></A><A NAME="ov2"></A>4.2 Supporting
+files for tests</H3>
+<P>Various additional files may be used by a test, for instance, to
+create large data values, to test using of jar files and the like.
+Any files that need to be accessed by a particular test that are not
+accessed from the classpath need to be listed under supportfiles= in
+the <testname>_app.properties file.<BR>Tests can refer to
+classes without being in the classpath, and sql tests can use the ij
+command 'run resource ' to execute additional .sql files without
+changes to the _app.properties files.
+</P>
+<P>For example, in the file
+(org/apache/derbyTesting/functionTests/tests/)tools/dblook_test_app.properties:<BR>
+<FONT SIZE=2>supportfiles=tools/dblook_makeDB.sql,tools/dblook_test.jar</FONT></P>
+<H3><A NAME="mozTocId427577"></A><A NAME="ov3"></A>4.3
+<testname>_app.properties</H3>
+<P>Every test directory has a default_app.properties. This file is
+for system level properties generic to all the tests in that test
+directory.
+</P>
+<P>If a test requires different system level properties, a test
+specific properties file can be created to overwrite the defaults.
+The test specific properties file needs to have a name starting with
+the test file name, followed with _app.properties</P>
+<P>For example, for the test tools/dblook_test.java, there is a
+properties file called tools/dblook_test_app.properties</P>
+<H3><A NAME="mozTocId715566"></A><A NAME="ov4"></A>4.4
+<testname>_derby.properties</H3>
+<P>Every test directory has a default_derby.properties. This file is
+for derby specific properties common to all the tests in that test
+directory.<BR>If a test requires different derby properties, a test
+specific properties file can be created to overwrite the defaults.
+The test specific properties file needs to have a name starting with
+the test file name, followed with _derby.properties</P>
+<H3><A NAME="mozTocId874096"></A><A NAME="ov5"></A>4.5 tmp files, out
+files, master files, and canons</H3>
+<P>The test's output will be put into a file testname.tmp. Then the
+output is modified if masking is required and the result is put into
+a .out file.<BR>The expected output is found by examining the
+following directories, based on certain input</P>
+<UL>
+ <LI><P STYLE="margin-bottom: 0in">functionTests/master/framework/jcc_version/jvmcode
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">functionTests/master/framework/jcc_version/earlier_jvmcode
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">functionTests/master/framework/jcc_version
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">functionTests/master/framework/jvmcode
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">functionTests/master/framework/earlier_jvmcode
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">functionTests/master/jvmcode
+ </P>
+ <LI><P>functionTests/master
+ </P>
+</UL>
+<P>For example, if we are running a test and the flag
+-Dframework=DerbyNet is used, to use network server and the IBM
+Universal JDBC Driver ('jcc'), and the jvm we are using is Sun's jdk
+142, and the jcc version is 2.4 (not available at time of writing)
+then the search for the master to compare with starts in the
+functionTests/derbynet/jcc2.4/jdk14 directory. If a .out file with
+the same name as the test is found in that directory, that master is
+taken. If there is no such file in that directory, search continues
+in the directory functionTests/derbynet/jcc2.4/jdk13 if it exists.</P>
+<P>If there is no file there, nor for any other jcc directory, it
+will continue to derbynet/jdk14, and the search is continued for
+earlier jvm versions.<BR>If we are not running network server, the
+DerbyNet and jcc_version directories are not traversed.</P>
+<P>The version details do not go into the subversion level, i.e.
+running with jdk141 or jdk142 is expected to have the same behavior.
+</P>
+<P>This functionality supports dealing with minor differences in
+behavior caused by minor differences in behavior in the underlying
+jvms, jcc versions, differences between results returned through
+network server vs. embedded and minor differences between a debug and
+non debug (jar) build.
+</P>
+<P>However, having a large number of these files means a maintenance
problem. Every time test output changes due to modifications to derby
-or to the test, all output files in all directories need to be updated
-accordingly. If at all possible, irrelevant differences should be
-masked out, or the test should be written so that the output does not
-reflect such items. </p>
-<p>
-Suggestions to minimize canons:
-</p>
-<ul>
- <li>create test specific masking</li>
- <li>ensure test data has a specific correct returned order; or an
-order by should be added to a query</li>
- <li>when writing java tests, ensure only pertinent output is
-reflected.</li>
-</ul>
-<br>
-<h3><a class="mozTocH3" name="mozTocId68107"></a>4.6 <a name="ov6"></a>Masking
-and comparing</h3>
-<p>
-Tests often fail because of unimportant differences, such as process
-ids, statement ids, timestamps. The derby functional test harness
-provides for masking of these differences at 2 levels:<br>
-</p>
-<ol>
- <li>overall level. Masking required in all, or many tests can be
-achieved using the class Sed in the test harness directory. This class
-can either delete a reference present in the .tmp file from the .out
-file, or replace it with a generic string. </li>
- <li>test specific level. To make masking for only one test, a
-(testname)_sed.properties file can be created which allows to either
-remove a string from the output or to replace it.</li>
-</ol>
-<p>
-The diff is executed between the final resulting output and the master
-file found.<br>
-<br>
-</p>
-<h3><a name="Adding_a_new_test"></a>4.7<span style="font-weight: bold;">
-</span>Adding
-a new test</h3>
-<p>
-To add a new test:
-</p>
-<ul>
- <li>create the test file (e.g. newfunctest.java or newfunctest.sql)
-in the appropriate tests subdirectory</li>
- <li>list any files needed that are not .sql or .java files in a
-supportfiles entry in a test specific _app.properties file. e.g.
-newfunctest_app.properties: supportfiles=xyz.jar<br>
- </li>
- <li>list any specific derby properties in a test specific
-_derby.properties file.</li>
- <li>add the properties files to the copyfiles.ant file in the test
-specific directory</li>
- <li>run the test. The first time around, the test will fail because
-no master file will be found. </li>
- <li>if the output is correct, copy it to the master directory. Note
-that there is no copyfiles.ant file needed for the master directory,
-all .out files are automatically copied.</li>
- <li>run the test again. Investigate if any differences need to be
-masked out using a test specific sed.properties file (e.g.
-newfunctest_sed.properties). If so, ensure this is added to
-copyfiles.ant.</li>
- <li>add the test to a specific suites/*.xml file, maintaining proper
-xml syntax. </li>
- <li>run the suite, and correct any problems found.</li>
-</ul>
-<br>
-<h3><a name="4.8_Suites_and_Adding_a_new_suite"></a>4.8 Suites and
-Adding a new suite</h3>
-<p>
-A suite constitutes of a <suitename>.properties file and/or a
-<suitename>.runall file in the
-org/apache/derbyTesting/functionTests/suites directory. The .properties
-files hold references to other .properties files, or .runall files, the
-.runall files are the actual lists of tests.
-</p>
-<p>
-The lowest level suite always needs to have a .runall file.
-</p>
-<p>
-For example, the derbyall suite is only a derbyall.properties file that
-refers to other suites in the 'suites' property:
-</p>
-<table
- style="text-align: left; width: 527px; height: 32px; margin-left: 40px;"
- border="1" cellpadding="2" cellspacing="2">
- <tbody>
- <tr>
- <td style="vertical-align: top;"><small>-----------------------derbyall.properties---------------<br>
-suites=derbylang derbynetclientmats derbynetmats storeall xa derbytools<br>
-derby.debug.true=enableBtreeConsistencyCheck<br>
-derby.stream.error.logSeverityLevel=0<br>
- </small></td>
- </tr>
- </tbody>
-</table>
-<p>
-The derbylang suite is only a derbylang.runall, which lists the tests.
-The derbynetclientmats suite has both a .runall and a .properties file,
-so
-some additional properties can be specified that are true for all tests
-in that suite. </p>
-<table
- style="text-align: left; width: 521px; height: 32px; margin-left: 40px;"
- border="1" cellpadding="2" cellspacing="2">
- <tbody>
- <tr>
- <td style="vertical-align: top;"><small>------------------derbynetclientmats.properties-----------------<br>
-framework=DerbyNetClient<br>
-suites=derbynetclientmats derbynetmats<br>
-jdk12test=true<br>
-runwithj9=false<br>
-timeout=60<br>
- </small></td>
- </tr>
- </tbody>
-</table>
-<p>
-To add a suite, you need to create at least a <suite>.runall
-file, which lists the actual tests, or a properties file that refers to
-other suites that do have a .runall file. The suite should be added
-into the directory
-${derby.source}/java/testing/org/apache/derbyTesting/functionTests/suites.<br>
-<br>
-</p>
-<h3><a name="4.9_Running_with_a_new_jvm_"></a> <a name="ov9"></a>4.9
-Running
-with a new jvm<br>
-</h3>
-<p>Currently, the supported jvms are:
-</p>
-<table style="text-align: left; width: 839px; height: 32px;" border="1"
- cellpadding="2" cellspacing="2">
- <tbody>
- <tr>
- <td style="vertical-align: top;"><small> jdk131
-- Sun
-HotSpot jdk1.3.1 - class: jdk13</small><br>
- <small> jdk141 - Sun HotSpot jdk1.4.1 - class
-jdk14</small><br>
- <small> jdk142 - Sun HotSpot jdk1.4.2 - class
-jdk14</small><br>
- <small> jdk15 - Sun HotSpot jdk1.5 - class jdk15</small><br>
- <small> ibm131 - IBM Classic jdk1.3.1 -
-class ibm13</small><br>
- <small> ibm141 - IBM Classic jdk1.4.1 - class
-ibm14</small><br>
- <small> ibm142 - IBM Classic jdk1.4.1 - class
-ibm14</small><br>
- <small> j9_13 - WCTME jvm (available with IBM
- Websphere Studio Device Developer, 5.6), version 2.1 - class j9_13</small><br>
- <small> j9_22 - WCTME jvm (available with IBM
- Websphere Client Technology Micro Edition, 5.7), version 2.2 - class j9_22</small><br>
- <small> j9_foundation - WCTME jvm (available with IBM
- Websphere Client Technology Micro Edition, 5.7), version 2.2,
- foundation library - class j9_foundation<br>
- </small></td>
- </tr>
- </tbody>
-</table>
-<p>The classes above are subclasses of
+or to the test, all output files in all directories need to be
+updated accordingly. If at all possible, irrelevant differences
+should be masked out, or the test should be written so that the
+output does not reflect such items.
+</P>
+<P>Suggestions to minimize canons:
+</P>
+<UL>
+ <LI><P STYLE="margin-bottom: 0in">create test specific masking
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">ensure test data has a specific
+ correct returned order; or an order by should be added to a query
+ </P>
+ <LI><P>when writing java tests, ensure only pertinent output is
+ reflected.
+ </P>
+</UL>
+<P><BR><BR>
+</P>
+<H3><A NAME="mozTocId68107"></A><A NAME="ov6"></A>4.6 Masking and
+comparing</H3>
+<P>Tests often fail because of unimportant differences, such as
+process ids, statement ids, timestamps. The derby functional test
+harness provides for masking of these differences at 2 levels:</P>
+<OL>
+ <LI><P STYLE="margin-bottom: 0in">overall level. Masking required in
+ all, or many tests can be achieved using the class Sed in the test
+ harness directory. This class can either delete a reference present
+ in the .tmp file from the .out file, or replace it with a generic
+ string.
+ </P>
+ <LI><P>test specific level. To make masking for only one test, a
+ (testname)_sed.properties file can be created which allows to either
+ remove a string from the output or to replace it.
+ </P>
+</OL>
+<P>The diff is executed between the final resulting output and the
+master file found.</P>
+<H3><A NAME="Adding_a_new_test"></A>4.7 Adding a new test</H3>
+<P>To add a new test:
+</P>
+<UL>
+ <LI><P STYLE="margin-bottom: 0in">create the test file (e.g.
+ newfunctest.java or newfunctest.sql) in the appropriate tests
+ subdirectory
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">list any files needed that are not
+ .sql or .java files in a supportfiles entry in a test specific
+ _app.properties file. e.g. newfunctest_app.properties:
+ supportfiles=xyz.jar</P>
+ <LI><P STYLE="margin-bottom: 0in">list any specific derby properties
+ in a test specific _derby.properties file.
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">add the properties files to the
+ copyfiles.ant file in the test specific directory
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">run the test. The first time
+ around, the test will fail because no master file will be found.
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">if the output is correct, copy it
+ to the master directory. Note that there is no copyfiles.ant file
+ needed for the master directory, all .out files are automatically
+ copied.
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">run the test again. Investigate if
+ any differences need to be masked out using a test specific
+ sed.properties file (e.g. newfunctest_sed.properties). If so, ensure
+ this is added to copyfiles.ant.
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">add the test to a specific
+ suites/*.xml file, maintaining proper xml syntax.
+ </P>
+ <LI><P>run the suite, and correct any problems found.
+ </P>
+</UL>
+<P><BR><BR>
+</P>
+<H3><A NAME="4.8_Suites_and_Adding_a_new_suite"></A>4.8 Suites and
+Adding a new suite</H3>
+<P>A suite constitutes of a <suitename>.properties file and/or
+a <suitename>.runall file in the
+org/apache/derbyTesting/functionTests/suites directory. The
+.properties files hold references to other .properties files, or
+.runall files, the .runall files are the actual lists of tests.
+</P>
+<P>The lowest level suite always needs to have a .runall file.
+</P>
+<P>For example, the derbyall suite is only a derbyall.properties file
+that refers to other suites in the 'suites' property:
+</P>
+<TABLE BORDER=1 CELLPADDING=2 CELLSPACING=2>
+ <TR>
+ <TD>
+ <P><FONT SIZE=2>-----------------------derbyall.properties---------------<BR>suites=derbylang
+ derbynetclientmats derbynetmats storeall xa
+ derbytools<BR>derby.debug.true=enableBtreeConsistencyCheck<BR>derby.stream.error.logSeverityLevel=0</FONT></P>
+ </TD>
+ </TR>
+</TABLE>
+<P>The derbylang suite is only a derbylang.runall, which lists the
+tests. The derbynetclientmats suite has both a .runall and a
+.properties file, so some additional properties can be specified that
+are true for all tests in that suite.
+</P>
+<TABLE BORDER=1 CELLPADDING=2 CELLSPACING=2>
+ <TR>
+ <TD>
+ <P><FONT SIZE=2>------------------derbynetclientmats.properties-----------------<BR>framework=DerbyNetClient<BR>suites=derbynetclientmats
+ derbynetmats<BR>jdk12test=true<BR>runwithj9=false<BR>timeout=60</FONT></P>
+ </TD>
+ </TR>
+</TABLE>
+<P>To add a suite, you need to create at least a <suite>.runall
+file, which lists the actual tests, or a properties file that refers
+to other suites that do have a .runall file. The suite should be
+added into the directory
+${derby.source}/java/testing/org/apache/derbyTesting/functionTests/suites.</P>
+<H3><A NAME="4.9_Running_with_a_new_jvm_"></A><A NAME="ov9"></A>4.9
+Running with a new jvm</H3>
+<P>Currently, the supported jvms are:
+</P>
+<TABLE BORDER=1 CELLPADDING=2 CELLSPACING=2>
+ <TR>
+ <TD>
+ <P> <FONT SIZE=2>jdk131 - Sun HotSpot jdk1.3.1 -
+ class: jdk13</FONT><BR> <FONT SIZE=2>jdk141 -
+ Sun HotSpot jdk1.4.1 - class jdk14</FONT><BR>
+ <FONT SIZE=2>jdk142 - Sun HotSpot jdk1.4.2 - class jdk14</FONT><BR>
+ <FONT SIZE=2>jdk15 - Sun HotSpot jdk1.5 - class jdk15</FONT><BR>
+ <FONT SIZE=2>ibm131 - IBM Classic jdk1.3.1 - class ibm13</FONT><BR>
+ <FONT SIZE=2>ibm141 - IBM Classic jdk1.4.1 - class ibm14</FONT><BR>
+ <FONT SIZE=2>ibm142 - IBM Classic jdk1.4.1 - class ibm14</FONT><BR>
+ <FONT SIZE=2>j9_13 - WCTME jvm (available with IBM Websphere
+ Studio Device Developer, 5.6), version 2.1 - class j9_13</FONT><BR>
+ <FONT SIZE=2>j9_22 - WCTME jvm (available with IBM Websphere
+ Client Technology Micro Edition, 5.7), version 2.2 - class
+ j9_22</FONT><BR> <FONT SIZE=2>j9_foundation -
+ WCTME jvm (available with IBM Websphere Client Technology Micro
+ Edition, 5.7), version 2.2, foundation library - class
+ j9_foundation</FONT></P>
+ </TD>
+ </TR>
+</TABLE>
+<P>The classes above are subclasses of
org.apache.derbyTesting.functionTests.harness.jvm. The name at the
-front is just a convention.<br>
-</p>
-<p>To run a test with a jvm that does not have a matching class under
-org.apache.derbyTesting.functionTests.harness, do the following:<br>
-</p>
-<ul>
- <li>just run the tests as if there is a jvm class. The harness will
-default to using
-the jdk14 class. Unlikely, but possibly there are no differences<br>
- </li>
- <li>if there are failures showing that cannot be explained any other
-way but genuine, acceptable jvm differences, do the following:</li>
- <ul>
- <li>create a subclass of
-org.apache.derbyTesting.functionTests.harness.jvm. In this class,
-specify any jvm specific property settings required </li>
- <li>compile the new jvm class and run the tests</li>
- <li>create a new canon directory for any additional canons that
-need to be created.</li>
- <li>in rare occasions, other harness changes may be required</li>
- <li>for any tests that should not run with this environment, add a
-line in the testname_app.properties file indicating this. For instance
-to add a line for a jvm called jdk29, it would be like this:
-runwithjdk29=false. Note that the versioning does not currently extend
-past 2 digits. For j9 jvms, versioning does not apply currently. For all
-j9 versions, use runwithj9=false. For j9_foundation, use runwithfoundation=false.</li>
- <li>Add code in RunTest.java to switch to the new jvm based on
-values for system and vendor properties</li>
- </ul>
-</ul>
-<br>
-<h3><a name="skipping"></a>4.10 Skipping a test</h3>
-<p>There are 2 skipping mechanisms in place for different kinds of
-skipping of a test.<br>
-</p>
-<p>Some tests are written to test specific functionality only available
-with for instance certain jvms, or, with network server, certain
-versions of the IBM Universal Driver. To control this, properties can
-be set for each test, for instance, if a test should not be run when
-using an ibm jvm, set runwithibmjvm=false. If a test should be run with
-Sun Hotspot jvm version 14, then set runwithjdk14=true.<br>
-The skip setting does not go into the subversion level, i.e. setting
-runwithjdk141=false has no effect, and setting runwithjdk14 affects
-runs with jdk141 as well as jdk142.<br>
-Other skip reasons are encryption protocols specific to a certain jvm. <br>
-</p>
-<p>The property for skipping a test based on the version of the IBM
-Universal Driver is "excludeJCC". The keywords "<span
- style="font-weight: bold;">at-or-before</span>" and "<span
- style="font-weight: bold;">at-or-after</span>" can be used to specify
-which range of JCC versions should be excluded. If neither of
-these keywords is provided, the default is "<span
- style="font-weight: bold;">at-or-before</span>". For example:<br>
-</p>
-To skip a test when running with any version of the IBM Universal
-Driver that is 2.4 or earlier:<br>
-excludeJCC=at-or-before:2.4<br>
-<br>
-To skip a test when running with any version of the IBM Universal
-Driver that is 2.0 or later:<br>
-excludeJCC=at-or-after:2.0<br>
-<p>You can also specify an (optional) jvm clause to further tune the
-exclusion criteria. This clause starts with the "<span
- style="font-weight: bold;">,when</span>" tag and is followed by a
-three-part jvm version. In this case, a test will only be skipped
-if BOTH the JCC clause AND the jvm clause are true. For example:<br>
-</p>
-<p>To skip a test when running with any version of the IBM Universal
-Driver that is 2.4 or later, but ONLY if the jvm is 1.3 or earlier:<br>
-excludeJCC=at-or-after:2.4,when-at-or-before:jdk1.3.1<br>
-</p>
-To skip a test when running with any version of the IBM Universal
-Driver that is 2.0 or earlier, but ONLY if the jvm is 1.5 or later:<br>
-excludeJCC=at-or-before:2.0,when-at-or-after:jdk1.5.1<br>
-<p>Another skipping mechanism works on entire 'frameworks'. Currently
+front is just a convention.</P>
+<P>To run a test with a jvm that does not have a matching class under
+org.apache.derbyTesting.functionTests.harness, do the following:</P>
+<UL>
+ <LI><P STYLE="margin-bottom: 0in">just run the tests as if there is
+ a jvm class. The harness will default to using the jdk14 class.
+ Unlikely, but possibly there are no differences</P>
+ <LI><P STYLE="margin-bottom: 0in">if there are failures showing that
+ cannot be explained any other way but genuine, acceptable jvm
+ differences, do the following:
+ </P>
+ <UL>
+ <LI><P STYLE="margin-bottom: 0in">create a subclass of
+ org.apache.derbyTesting.functionTests.harness.jvm. In this class,
+ specify any jvm specific property settings required
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">compile the new jvm class and run
+ the tests
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">create a new canon directory for
+ any additional canons that need to be created.
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">in rare occasions, other harness
+ changes may be required
+ </P>
+ <LI><P STYLE="margin-bottom: 0in">for any tests that should not run
+ with this environment, add a line in the testname_app.properties
+ file indicating this. For instance to add a line for a jvm called
+ jdk29, it would be like this: runwithjdk29=false. Note that the
+ versioning does not currently extend past 2 digits. For j9 jvms,
+ versioning does not apply currently. For all j9 versions, use
+ runwithj9=false. For j9_foundation, use runwithfoundation=false.
+ </P>
+ <LI><P>Add code in RunTest.java to switch to the new jvm based on
+ values for system and vendor properties
+ </P>
+ </UL>
+</UL>
+<P><BR><BR>
+</P>
+<H3><A NAME="skipping"></A>4.10 Skipping a test</H3>
+<P>There are 2 skipping mechanisms in place for different kinds of
+skipping of a test.</P>
+<P>Some tests are written to test specific functionality only
[... 333 lines stripped ...]