You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Daniel John Debrunner <dj...@apache.org> on 2006/07/20 21:22:22 UTC
JUnit testing moving forward
We seem to have made great progess on the JUnit testing front, we have
some base classes with utility methods, about 40 Junit tests and a wiki
page describing Derby's JUnit use:
http://wiki.apache.org/db-derby/DerbyJUnitTesting
I think now would be a good time to establish some of the policies and
goals around JUnit in Derby. My current fear is that we are thinking too
much about getting JUnit tests to run in the existing harness, rather
than think of JUnit as the only harness moving forward.
As an example, we have eight JUnit tests in jdbcapi package, seven of
these are in the jdbcapi suite for the existing harness as *individual*
tests. If one run these eight Junit tests as a single JUnit suite then
the time taken is ~250 seconds, running them (all eight) as a derby
harness test suite takes ~500 seconds. Imagine if that performance
improvement was across all tests, derbyall would drop to two-three hours
on my laptop and around an hour on my desktop. Great savings for
everyone. So why not run these eight tests as a single JUnit suite from
the derby test harness suite, why force them to be individual tests?
Running the single JUnit suite in the derby harness took ~310 seconds.
My goal would be every test in Derby is a Junit test and the Derby
harness is removed. I think this will take some time and will be an
incremental process, but that should be the underlying thought when
working with tests.
I've been thinking that this goal would factor down into some
requirements/expectations/policies:
- Any Derby Junit test will run sucessfully as a standalone test using
JUnit test runners (i.e. any public non-abstract class in Derby that
extends TestCase)
- Every Derby package that contains JUnit tests has a class _Suite
that runs all the Junit tests in that package (see below for an
example). This can be directly for each JUnit test or through sub-suites.
- Ideally this _Suite uses a single database for all the tests
- Higher level suites would combine all the _Suites from the packages
resulting in something like derbyall
- Some policy & utilities around cleanup and database removal.
- utility methods to automatically clean the database (remove
tables etc.) based upon database meta data
- utility method to remove the database and derby.log and anything else
- assumption tests will leave a clean database and remove
everything once complete, but optionally leave the database for the next
test (set by the calling suite?)
- some clear policy on the configuration stuff, I couldn't see from
a quick look any guidance on how configurations are meant to work, e.g.
how would I set up tests to run under embedded and client, would this be
a high level suite that runs the jdbcapi._Suite test once in embedded
and then once as client?
There's probably a list of functions the current harness performs that
needs to be implemented in JUnit (as test decorators?) but these could
be tackled incrementally. My itch would be the security manager setup.
Would be good to get a list of these.
Thoughts, comments?
Dan.
-----
package org.apache.derbyTesting.functionTests.tests.jdbcapi;
import org.apache.derbyTesting.functionTests.util.BaseTestCase;
import junit.framework.Test;
import junit.framework.TestSuite;
public class _Suite extends BaseTestCase {
public _Suite(String name)
{
super(name);
}
public static Test suite() {
TestSuite suite= new TestSuite();
suite.addTest(ProcedureTest.suite());
suite.addTestSuite(ScrollResultSetTest.class);
suite.addTest(ConcurrencyTest.suite());
suite.addTestSuite(HoldabilityTest.class);
suite.addTest(SURQueryMixTest.suite());
suite.addTest(SURTest.suite());
suite.addTestSuite(UpdateXXXTest.class);
suite.addTestSuite(URCoveringIndexTest.class);
return suite;
}
}
Re: JUnit testing moving forward
Posted by Daniel John Debrunner <dj...@apache.org>.
David Van Couvering wrote:
> I think this is a laudable goal. I agree it would need to be done
> incrementally. I think it would really help to have a Wiki page where
> work around this can be clarified and coordinated. Probably a JIRA to
> hang subtasks off of too.
>
> Here are some questions I have around this
>
> - does moving to a JUnit harness imply removing canon-based testing and
> only using assertion-based testing? Or is that a separate, orthogonal
> goal (one I which I think is important, both for master file management
> overhead as well as improving our ability to truly test backward
> compatibility without having to sift through unhelpful diffs).
Orthogonal goal (and I agree a good one). It's possible to write a Junit
test that uses sql scripts and a canon file, see my early experiments
with JUnit. May not be the best approach, but it is possible, and would
probably work for a large number of the existing tests. The downside
with moving away from canon based test is who is going to re-write all
the tests as assert based and ensure no cases were lost.
As more required functionality is moved into the Junit framework, I'm
assuming tests will move over, I see a possible staging something like:
0 (today) - all tests (incl JUnit) run as single tests in derby harness
1 - derby harness runs its tests and multiple Junit suites
2 - derby harness runs its tests and a single Junit suite
3 - some canon based tests moved to JUnit tests (using canons)
4 - majority of tests run as Junit tests in single suite from derbyall
5 - derby harness killed, all tests Junit
> - Where does the responsibility lie to manage different results in
> different environments?
Not sure what you are asking here. If a test passes in one environment
and fails in the other, I don't see how which harness is being used
affects who is responsible (ie. those with the itch to see clean runs in
that environment).
Any complete move to JUnit would have to ensure we do not regress the
community's ability to run tests where they can be run today (ie. the do
no harm mantra). J2ME/CDC/Foundation is one concern here, but I'm
assuming the current JUnit tests are running on J2ME.
Thanks,
Dan.
Re: JUnit testing moving forward
Posted by David Van Couvering <Da...@Sun.COM>.
I think this is a laudable goal. I agree it would need to be done
incrementally. I think it would really help to have a Wiki page where
work around this can be clarified and coordinated. Probably a JIRA to
hang subtasks off of too.
Here are some questions I have around this
- does moving to a JUnit harness imply removing canon-based testing and
only using assertion-based testing? Or is that a separate, orthogonal
goal (one I which I think is important, both for master file management
overhead as well as improving our ability to truly test backward
compatibility without having to sift through unhelpful diffs).
- Where does the responsibility lie to manage different results in
different environments?
I'm sure I'll have more questions, but those are the ones that come up
for me right now...
Thanks,
David
Daniel John Debrunner wrote:
> We seem to have made great progess on the JUnit testing front, we have
> some base classes with utility methods, about 40 Junit tests and a wiki
> page describing Derby's JUnit use:
>
> http://wiki.apache.org/db-derby/DerbyJUnitTesting
>
> I think now would be a good time to establish some of the policies and
> goals around JUnit in Derby. My current fear is that we are thinking too
> much about getting JUnit tests to run in the existing harness, rather
> than think of JUnit as the only harness moving forward.
>
> As an example, we have eight JUnit tests in jdbcapi package, seven of
> these are in the jdbcapi suite for the existing harness as *individual*
> tests. If one run these eight Junit tests as a single JUnit suite then
> the time taken is ~250 seconds, running them (all eight) as a derby
> harness test suite takes ~500 seconds. Imagine if that performance
> improvement was across all tests, derbyall would drop to two-three hours
> on my laptop and around an hour on my desktop. Great savings for
> everyone. So why not run these eight tests as a single JUnit suite from
> the derby test harness suite, why force them to be individual tests?
> Running the single JUnit suite in the derby harness took ~310 seconds.
>
> My goal would be every test in Derby is a Junit test and the Derby
> harness is removed. I think this will take some time and will be an
> incremental process, but that should be the underlying thought when
> working with tests.
>
> I've been thinking that this goal would factor down into some
> requirements/expectations/policies:
>
> - Any Derby Junit test will run sucessfully as a standalone test using
> JUnit test runners (i.e. any public non-abstract class in Derby that
> extends TestCase)
>
> - Every Derby package that contains JUnit tests has a class _Suite
> that runs all the Junit tests in that package (see below for an
> example). This can be directly for each JUnit test or through sub-suites.
>
> - Ideally this _Suite uses a single database for all the tests
>
> - Higher level suites would combine all the _Suites from the packages
> resulting in something like derbyall
>
> - Some policy & utilities around cleanup and database removal.
> - utility methods to automatically clean the database (remove
> tables etc.) based upon database meta data
> - utility method to remove the database and derby.log and anything else
> - assumption tests will leave a clean database and remove
> everything once complete, but optionally leave the database for the next
> test (set by the calling suite?)
>
> - some clear policy on the configuration stuff, I couldn't see from
> a quick look any guidance on how configurations are meant to work, e.g.
> how would I set up tests to run under embedded and client, would this be
> a high level suite that runs the jdbcapi._Suite test once in embedded
> and then once as client?
>
> There's probably a list of functions the current harness performs that
> needs to be implemented in JUnit (as test decorators?) but these could
> be tackled incrementally. My itch would be the security manager setup.
> Would be good to get a list of these.
>
> Thoughts, comments?
> Dan.
>
> -----
> package org.apache.derbyTesting.functionTests.tests.jdbcapi;
>
> import org.apache.derbyTesting.functionTests.util.BaseTestCase;
>
> import junit.framework.Test;
> import junit.framework.TestSuite;
>
> public class _Suite extends BaseTestCase {
>
> public _Suite(String name)
> {
> super(name);
> }
>
> public static Test suite() {
> TestSuite suite= new TestSuite();
> suite.addTest(ProcedureTest.suite());
> suite.addTestSuite(ScrollResultSetTest.class);
> suite.addTest(ConcurrencyTest.suite());
> suite.addTestSuite(HoldabilityTest.class);
> suite.addTest(SURQueryMixTest.suite());
> suite.addTest(SURTest.suite());
> suite.addTestSuite(UpdateXXXTest.class);
> suite.addTestSuite(URCoveringIndexTest.class);
>
> return suite;
> }
> }
>
Re: JUnit testing moving forward
Posted by Daniel John Debrunner <dj...@apache.org>.
Myrna van Lunteren wrote:
> And I think the name for the Wiki (killDerbyTestHarness) is just perfect.
:-) It's just my way to change the mindset of everyone.
> It may not be a surprise, but I'm very interested in this. :-)
Excellent!
> Re 3, I think we should start with converting tests where the output
> is causing problems, for instance, any of the tests that show a
> difference in output when testing with a different derby version for
> client vs. server.
Interesting, my fish to fry [1] would be to convert the tests that have
single master files and no sedding requirements because it's easier and
likely to significantely bump the percentage of tests running as JUnit
(the easy fast food)
But you are right too, fixing the problematic tests may have more of an
immediate impact on the community as folks don't need to have to worry
about multiple test runs or updating many master files (simplifying the
complex dish created by a crazy relative?).
Great thing about open source, multiple people can work on different
items for different reasons related to the same goal to make overall
end-result better.
Thanks,
Dan.
[1]
http://mail-archives.apache.org/mod_mbox/www-women/200607.mbox/%3c44B7E4DB.5090209@sbcglobal.net%3e
Re: JUnit testing moving forward
Posted by Myrna van Lunteren <m....@gmail.com>.
On 7/20/06, Daniel John Debrunner <dj...@apache.org> wrote:
> Manish Khettry wrote:
>
> > And on the issue of using ant-- why not have ant
> > targets in java/testing to run standard suites? much
> > easier to get a developer to type "ant derbyall" and
> > have it do the needful (setup classpath, invoke the
> > right class) than what we do currently. This could be
> > done right now and ease the pain for new developers
> > trying to work with Derby.
>
> Agreed, and I think that's one of the reasons to move to JUnit. I think
> this will be easier with the tests in Junit. I would prefer to spend
> time moving towards JUnit that spending time getting the existing
> harness to be runnable from ant.
>
> Dan.
>
>
Hear, hear.
And I think the name for the Wiki (killDerbyTestHarness) is just perfect.
It may not be a surprise, but I'm very interested in this. :-)
Re 3, I think we should start with converting tests where the output
is causing problems, for instance, any of the tests that show a
difference in output when testing with a different derby version for
client vs. server.
Or, like DERBY-1496 - where we end up with many canons.
Myrna
Re: JUnit testing moving forward
Posted by Daniel John Debrunner <dj...@apache.org>.
Manish Khettry wrote:
> And on the issue of using ant-- why not have ant
> targets in java/testing to run standard suites? much
> easier to get a developer to type "ant derbyall" and
> have it do the needful (setup classpath, invoke the
> right class) than what we do currently. This could be
> done right now and ease the pain for new developers
> trying to work with Derby.
Agreed, and I think that's one of the reasons to move to JUnit. I think
this will be easier with the tests in Junit. I would prefer to spend
time moving towards JUnit that spending time getting the existing
harness to be runnable from ant.
Dan.
Re: JUnit testing moving forward
Posted by Manish Khettry <ma...@yahoo.com>.
This is related to the matter of testing but somewhat
tangential to the issue being discussed.
I am really interested in understanding if Derby is
amenable to unit testing. By unit testing I mean
testing at a class or package level instead of at a
system level which most of the tests do currently,
atleast for the sql layer functionality.
There are many benefits to testing a unit as opposed
to the entire system. For one you don't have to
create a database, create and populate objects
everytime you want to test something. Thinking about
testing a class or package leads to better and more OO
design. It also means more comprehensive tests imho.
Take for example the result set implementations in
sql/execute. If we could mock out the store
functionality we could test them more comprehensively
with unit tests than what we do currently-- we could
mock large amounts of data through them; we could also
exercise code path that one-shot executions from ij
don't exercise.
Not to say that we don't need system tests or the
existing tests are not good enough. I can see that
given the complexity of the code base and how tightly
coupled many of the packages and classes are, this is
hard but is this something to keep in mind as we are
writing code and tests?
And on the issue of using ant-- why not have ant
targets in java/testing to run standard suites? much
easier to get a developer to type "ant derbyall" and
have it do the needful (setup classpath, invoke the
right class) than what we do currently. This could be
done right now and ease the pain for new developers
trying to work with Derby.
Thanks,
Manish
--- Daniel John Debrunner <dj...@apache.org> wrote:
> Rick Hillegas wrote:
>
> > There are lots of good suggestions in this
> message. I would add the
> > following: If we're going to consider a new test
> harness, then we could
> > evaluate using Ant to drive the junit tests. Ant
> already has an optional
> > <junit> task for this purpose.
>
> Maybe I wasn't clear, the new harness should be
> JUnit itself, not
> something wrapped around JUnit. Then we get all the
> benefits of Junit
> for free, integration with ant, IDEs, tests
> understood by others, etc.
>
> Dan.
>
>
>
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Re: JUnit testing moving forward
Posted by Daniel John Debrunner <dj...@apache.org>.
Rick Hillegas wrote:
> There are lots of good suggestions in this message. I would add the
> following: If we're going to consider a new test harness, then we could
> evaluate using Ant to drive the junit tests. Ant already has an optional
> <junit> task for this purpose.
Maybe I wasn't clear, the new harness should be JUnit itself, not
something wrapped around JUnit. Then we get all the benefits of Junit
for free, integration with ant, IDEs, tests understood by others, etc.
Dan.
Re: JUnit testing moving forward
Posted by Rick Hillegas <Ri...@Sun.COM>.
There are lots of good suggestions in this message. I would add the
following: If we're going to consider a new test harness, then we could
evaluate using Ant to drive the junit tests. Ant already has an optional
<junit> task for this purpose.
Regards,
-Rick
Daniel John Debrunner wrote:
>We seem to have made great progess on the JUnit testing front, we have
>some base classes with utility methods, about 40 Junit tests and a wiki
>page describing Derby's JUnit use:
>
>http://wiki.apache.org/db-derby/DerbyJUnitTesting
>
>I think now would be a good time to establish some of the policies and
>goals around JUnit in Derby. My current fear is that we are thinking too
>much about getting JUnit tests to run in the existing harness, rather
>than think of JUnit as the only harness moving forward.
>
>As an example, we have eight JUnit tests in jdbcapi package, seven of
>these are in the jdbcapi suite for the existing harness as *individual*
>tests. If one run these eight Junit tests as a single JUnit suite then
>the time taken is ~250 seconds, running them (all eight) as a derby
>harness test suite takes ~500 seconds. Imagine if that performance
>improvement was across all tests, derbyall would drop to two-three hours
>on my laptop and around an hour on my desktop. Great savings for
>everyone. So why not run these eight tests as a single JUnit suite from
>the derby test harness suite, why force them to be individual tests?
>Running the single JUnit suite in the derby harness took ~310 seconds.
>
>My goal would be every test in Derby is a Junit test and the Derby
>harness is removed. I think this will take some time and will be an
>incremental process, but that should be the underlying thought when
>working with tests.
>
>I've been thinking that this goal would factor down into some
>requirements/expectations/policies:
>
> - Any Derby Junit test will run sucessfully as a standalone test using
>JUnit test runners (i.e. any public non-abstract class in Derby that
>extends TestCase)
>
> - Every Derby package that contains JUnit tests has a class _Suite
>that runs all the Junit tests in that package (see below for an
>example). This can be directly for each JUnit test or through sub-suites.
>
> - Ideally this _Suite uses a single database for all the tests
>
> - Higher level suites would combine all the _Suites from the packages
>resulting in something like derbyall
>
> - Some policy & utilities around cleanup and database removal.
> - utility methods to automatically clean the database (remove
>tables etc.) based upon database meta data
> - utility method to remove the database and derby.log and anything else
> - assumption tests will leave a clean database and remove
>everything once complete, but optionally leave the database for the next
> test (set by the calling suite?)
>
> - some clear policy on the configuration stuff, I couldn't see from
>a quick look any guidance on how configurations are meant to work, e.g.
>how would I set up tests to run under embedded and client, would this be
>a high level suite that runs the jdbcapi._Suite test once in embedded
>and then once as client?
>
>There's probably a list of functions the current harness performs that
>needs to be implemented in JUnit (as test decorators?) but these could
>be tackled incrementally. My itch would be the security manager setup.
>Would be good to get a list of these.
>
>Thoughts, comments?
>Dan.
>
>-----
>package org.apache.derbyTesting.functionTests.tests.jdbcapi;
>
>import org.apache.derbyTesting.functionTests.util.BaseTestCase;
>
>import junit.framework.Test;
>import junit.framework.TestSuite;
>
>public class _Suite extends BaseTestCase {
>
> public _Suite(String name)
> {
> super(name);
> }
>
> public static Test suite() {
> TestSuite suite= new TestSuite();
> suite.addTest(ProcedureTest.suite());
> suite.addTestSuite(ScrollResultSetTest.class);
> suite.addTest(ConcurrencyTest.suite());
> suite.addTestSuite(HoldabilityTest.class);
> suite.addTest(SURQueryMixTest.suite());
> suite.addTest(SURTest.suite());
> suite.addTestSuite(UpdateXXXTest.class);
> suite.addTestSuite(URCoveringIndexTest.class);
>
> return suite;
> }
>}
>
>
>