You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Kristian Waagan <Kr...@Sun.COM> on 2006/01/24 15:07:52 UTC
Features of the JUnit test execution harness
Hello,
I have wanted to rewrite a old canon based test to a JUnit test for a
while, but each time I get started I always run into issues that we do
not have guidelines for yet. So, I have a few questions I would like
some feedback on.
I know there are a lot of issues to consider, far more then the one I
mention here, but I hope to get a feeling of the general direction we
want to move in.
1) Should we support specific ordering of tests (within a single class)?
This feature is wanted if a number of tests operate on shared data and
each test method can be seen as a step in the test. If the order is
mixed up, the intermediate results will not be as expected and all the
tests may fail.
In general, assuming anything about the execution order of tests is
strongly disencouraged. However, since we are testing a database system,
we might want to allow for this. This feature can influence the both the
size of the test methods and the execution time for the test/suite.
2) Should we support setting up a shared test fixture for a set of tests?
This is a common issue with JUnit, and there are mechanisms to handle
it. For instance, we could let tests that require this to wrap itself in
a TestSetup instance (as described in the JUNit FAQ). Letting
DerbyJUnitTest extend TestSetup is another solution, but it might have
unwanted side-effects.
3) Many tests just want a database to work on. Other tests want to
create their own database due to specific requirements. How do we make
this easy for the developer?
4) Should it be possible to run single JUnit test/suite from the command
line by simply invoking the main method?
Personally, I think this should be possible. This require us to remove
the current use of main for running a test in the old test harness,
which I believe is doable with a few changes (in the old harness).
5) Is the ultimate goal to end up with a pure JUnit based test harness?
If this is the goal, we should design for this and accept some extra
work for plugging JUnit tests into the old test harness.
6) Where, and how, do we want to store settings for a test?
7) How is it determined which framework(s) a JUnit test should be run
in? (i.e. embedded, client, JCC)
If you have other opinions or comments related to JUnit testing, please
put them forth :)
I hope we are able to get a discussion on all or some of these issues,
and that it results in a Wiki page describing JUnit testing for Derby. I
created a page here: http://wiki.apache.org/db-derby/DerbyJUnitTesting
--
Kristian
Re: Features of the JUnit test execution harness
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
I think there's a lot of value in having a suite-level setup mechanism,
I totally agree that doing setup for each test can be overkill and I am
often finding ways to work around it.
David
Kristian Waagan wrote:
> Rick Hillegas wrote:
>
>>>
>>> Kristian Waagan wrote:
>>>
>>>
>>>>
>>>> 2) Should we support setting up a shared test fixture for a set of
>>>> tests?
>>>>
>>>> This is a common issue with JUnit, and there are mechanisms to
>>>> handle it. For instance, we could let tests that require this to
>>>> wrap itself in a TestSetup instance (as described in the JUNit FAQ).
>>>> Letting DerbyJUnitTest extend TestSetup is another solution, but it
>>>> might have unwanted side-effects.
>>>
>>>
>>>
>> Hi Kristian,
>>
>> I'm afraid I don't understand the issues here. I suspect your JUnit
>> experience is much deeper than mine. What is the advantage to having
>> DerbyJUnitTest extend TestSetup vs TestCase? It's hard to tell from
>> the FAQ. TestCase provides setup() and teardown() brackets like
>> TestSetup. In addition, TestCase provides the assertion machinery.
>> What unwanted side-effects do you have in mind?
>>
>> Thanks,
>> -Rick
>
>
> Hello Rick,
>
> Your concern is valid. Having our Derby JUnit superclass
> (DerbyJUnitTest) extend TestSetup is not a good idea.
>
> The proper way to use it is as described in the FAQ, or to create a
> separate TestSetup subclass. Just to make sure we are on the same track,
> the reason for using TestSetup is to provide the test with a one-time
> setUp() and a one-time tearDown() method. If your JUnit class has 3 test
> methods, this sequence would be run (ordering of the test methods may
> not be fixed):
>
> TestSetup.setUp()
> TestCase.setUp
> TestCase.testMethodA()
> TestCase.tearDown()
> TestCase.setUp
> TestCase.testMethodB()
> TestCase.tearDown()
> TestCase.setUp
> TestCase.testMethodC()
> TestCase.tearDown()
> TestSetup.tearDown()
>
> My concern is that many of our tests do not want to do a complete setUp
> and tearDown for each test, as this can cause the runtime for the test
> to be too long. By using the one-time methods, we can create a common
> setup for all the tests in the suite.
>
> To illustrate the two ways to use TestSetup, consider the following
> suite() methods:
>
> From the JUnit FAQ, the following example demonstrate how to use
> TestSetup without creating a separate class.
>
> public static Test suite() {
>
> TestSuite suite = new TestSuite();
>
> suite.addTest(SomeTest.suite());
> suite.addTest(AnotherTest.suite());
>
> TestSetup wrapper = new TestSetup(suite) {
>
> protected void setUp() {
> oneTimeSetUp();
> }
>
> protected void tearDown() {
> oneTimeTearDown();
> }
> };
>
> return wrapper;
> }
>
> public static void oneTimeSetUp() {
> // one-time initialization code
> }
>
> public static void oneTimeTearDown() {
> // one-time cleanup code
> }
>
>
> The other way.
>
> public static Test suite() {
> // Add all test methods in the class.
> TestSuite suite = new TestSuite(MyJUnitTestClass.class);
> // Create a new TestSetup subclass for our test.
> // This would have a setUp() and a tearDown() method, and possibly
> // several constructors and/or methods to control the setup to be
> // created.
> return new MyJUnitTestClassSetup(suite);
> }
>
> For the example above, there are lots of variants. We can return a suite
> that runs all tests in the test with X different setups, and
> additionally run a subset of the tests with a special setup. To do this,
> we must either create several TestSetup subclasses, or parameterize the
> TestSetup subclass.
>
>
>
> Did this make any more sense?
>
>
>
> --
> Kristian
Re: Features of the JUnit test execution harness
Posted by Rick Hillegas <Ri...@Sun.COM>.
Hi Myrna,
I don't think we've discussed yet how or when we would convert existing
tests to JUnit. I agree that ij needs to continue to be tested.
Regards,
-Rick
Myrna van Lunteren wrote:
> Silly question(s):
> with the Junit tests ij is not used?
> if not, and we're going to convert all tests to junit, should there be
> a new test to test the tool ij specifically (at some point) ?
>
> Myrna
>
Re: Features of the JUnit test execution harness
Posted by Myrna van Lunteren <m....@gmail.com>.
Silly question(s):
with the Junit tests ij is not used?
if not, and we're going to convert all tests to junit, should there be a new
test to test the tool ij specifically (at some point) ?
Myrna
Re: Features of the JUnit test execution harness
Posted by Rick Hillegas <Ri...@Sun.COM>.
Thanks, Kristian. This is indeed helpful. My takeaway is that our JUnit
primer should recommend something like this as the pattern if developers
need to move the setup()/teardown() brackets outside a cluster of tests.
Regards,
-Rick
Kristian Waagan wrote:
> Rick Hillegas wrote:
>
>>>
>>> Kristian Waagan wrote:
>>>
>>>
>>>>
>>>> 2) Should we support setting up a shared test fixture for a set of
>>>> tests?
>>>>
>>>> This is a common issue with JUnit, and there are mechanisms to
>>>> handle it. For instance, we could let tests that require this to
>>>> wrap itself in a TestSetup instance (as described in the JUNit
>>>> FAQ). Letting DerbyJUnitTest extend TestSetup is another solution,
>>>> but it might have unwanted side-effects.
>>>
>>>
>>>
>> Hi Kristian,
>>
>> I'm afraid I don't understand the issues here. I suspect your JUnit
>> experience is much deeper than mine. What is the advantage to having
>> DerbyJUnitTest extend TestSetup vs TestCase? It's hard to tell from
>> the FAQ. TestCase provides setup() and teardown() brackets like
>> TestSetup. In addition, TestCase provides the assertion machinery.
>> What unwanted side-effects do you have in mind?
>>
>> Thanks,
>> -Rick
>
>
> Hello Rick,
>
> Your concern is valid. Having our Derby JUnit superclass
> (DerbyJUnitTest) extend TestSetup is not a good idea.
>
> The proper way to use it is as described in the FAQ, or to create a
> separate TestSetup subclass. Just to make sure we are on the same
> track, the reason for using TestSetup is to provide the test with a
> one-time setUp() and a one-time tearDown() method. If your JUnit class
> has 3 test methods, this sequence would be run (ordering of the test
> methods may not be fixed):
>
> TestSetup.setUp()
> TestCase.setUp
> TestCase.testMethodA()
> TestCase.tearDown()
> TestCase.setUp
> TestCase.testMethodB()
> TestCase.tearDown()
> TestCase.setUp
> TestCase.testMethodC()
> TestCase.tearDown()
> TestSetup.tearDown()
>
> My concern is that many of our tests do not want to do a complete
> setUp and tearDown for each test, as this can cause the runtime for
> the test to be too long. By using the one-time methods, we can create
> a common setup for all the tests in the suite.
>
> To illustrate the two ways to use TestSetup, consider the following
> suite() methods:
>
> From the JUnit FAQ, the following example demonstrate how to use
> TestSetup without creating a separate class.
>
> public static Test suite() {
>
> TestSuite suite = new TestSuite();
>
> suite.addTest(SomeTest.suite());
> suite.addTest(AnotherTest.suite());
>
> TestSetup wrapper = new TestSetup(suite) {
>
> protected void setUp() {
> oneTimeSetUp();
> }
>
> protected void tearDown() {
> oneTimeTearDown();
> }
> };
>
> return wrapper;
> }
>
> public static void oneTimeSetUp() {
> // one-time initialization code
> }
>
> public static void oneTimeTearDown() {
> // one-time cleanup code
> }
>
>
> The other way.
>
> public static Test suite() {
> // Add all test methods in the class.
> TestSuite suite = new TestSuite(MyJUnitTestClass.class);
> // Create a new TestSetup subclass for our test.
> // This would have a setUp() and a tearDown() method, and possibly
> // several constructors and/or methods to control the setup to be
> // created.
> return new MyJUnitTestClassSetup(suite);
> }
>
> For the example above, there are lots of variants. We can return a
> suite that runs all tests in the test with X different setups, and
> additionally run a subset of the tests with a special setup. To do
> this, we must either create several TestSetup subclasses, or
> parameterize the TestSetup subclass.
>
>
>
> Did this make any more sense?
>
>
>
> --
> Kristian
Re: Features of the JUnit test execution harness
Posted by Kristian Waagan <Kr...@Sun.COM>.
Rick Hillegas wrote:
>>
>> Kristian Waagan wrote:
>>
>>
>>>
>>> 2) Should we support setting up a shared test fixture for a set of
>>> tests?
>>>
>>> This is a common issue with JUnit, and there are mechanisms to handle
>>> it. For instance, we could let tests that require this to wrap itself
>>> in a TestSetup instance (as described in the JUNit FAQ). Letting
>>> DerbyJUnitTest extend TestSetup is another solution, but it might
>>> have unwanted side-effects.
>>
>>
> Hi Kristian,
>
> I'm afraid I don't understand the issues here. I suspect your JUnit
> experience is much deeper than mine. What is the advantage to having
> DerbyJUnitTest extend TestSetup vs TestCase? It's hard to tell from the
> FAQ. TestCase provides setup() and teardown() brackets like TestSetup.
> In addition, TestCase provides the assertion machinery. What unwanted
> side-effects do you have in mind?
>
> Thanks,
> -Rick
Hello Rick,
Your concern is valid. Having our Derby JUnit superclass
(DerbyJUnitTest) extend TestSetup is not a good idea.
The proper way to use it is as described in the FAQ, or to create a
separate TestSetup subclass. Just to make sure we are on the same track,
the reason for using TestSetup is to provide the test with a one-time
setUp() and a one-time tearDown() method. If your JUnit class has 3 test
methods, this sequence would be run (ordering of the test methods may
not be fixed):
TestSetup.setUp()
TestCase.setUp
TestCase.testMethodA()
TestCase.tearDown()
TestCase.setUp
TestCase.testMethodB()
TestCase.tearDown()
TestCase.setUp
TestCase.testMethodC()
TestCase.tearDown()
TestSetup.tearDown()
My concern is that many of our tests do not want to do a complete setUp
and tearDown for each test, as this can cause the runtime for the test
to be too long. By using the one-time methods, we can create a common
setup for all the tests in the suite.
To illustrate the two ways to use TestSetup, consider the following
suite() methods:
From the JUnit FAQ, the following example demonstrate how to use
TestSetup without creating a separate class.
public static Test suite() {
TestSuite suite = new TestSuite();
suite.addTest(SomeTest.suite());
suite.addTest(AnotherTest.suite());
TestSetup wrapper = new TestSetup(suite) {
protected void setUp() {
oneTimeSetUp();
}
protected void tearDown() {
oneTimeTearDown();
}
};
return wrapper;
}
public static void oneTimeSetUp() {
// one-time initialization code
}
public static void oneTimeTearDown() {
// one-time cleanup code
}
The other way.
public static Test suite() {
// Add all test methods in the class.
TestSuite suite = new TestSuite(MyJUnitTestClass.class);
// Create a new TestSetup subclass for our test.
// This would have a setUp() and a tearDown() method, and possibly
// several constructors and/or methods to control the setup to be
// created.
return new MyJUnitTestClassSetup(suite);
}
For the example above, there are lots of variants. We can return a suite
that runs all tests in the test with X different setups, and
additionally run a subset of the tests with a special setup. To do this,
we must either create several TestSetup subclasses, or parameterize the
TestSetup subclass.
Did this make any more sense?
--
Kristian
Re: Features of the JUnit test execution harness
Posted by Rick Hillegas <Ri...@Sun.COM>.
>
> Kristian Waagan wrote:
>
>
>>
>> 2) Should we support setting up a shared test fixture for a set of
>> tests?
>>
>> This is a common issue with JUnit, and there are mechanisms to handle
>> it. For instance, we could let tests that require this to wrap itself
>> in a TestSetup instance (as described in the JUNit FAQ). Letting
>> DerbyJUnitTest extend TestSetup is another solution, but it might
>> have unwanted side-effects.
>
Hi Kristian,
I'm afraid I don't understand the issues here. I suspect your JUnit
experience is much deeper than mine. What is the advantage to having
DerbyJUnitTest extend TestSetup vs TestCase? It's hard to tell from the
FAQ. TestCase provides setup() and teardown() brackets like TestSetup.
In addition, TestCase provides the assertion machinery. What unwanted
side-effects do you have in mind?
Thanks,
-Rick
Re: Features of the JUnit test execution harness
Posted by "Bernt M. Johnsen" <Be...@Sun.COM>.
>>>>>>>>>>>> Daniel John Debrunner wrote (2006-01-24 20:23:27):
> The other issue is that we have a great opportunity to start out with
> JUnit tests that follow a consistent pattern and provide a great example
> for others to follow.
+1
--
Bernt Marius Johnsen, Database Technology Group,
Staff Engineer, Technical Lead Derby/Java DB
Sun Microsystems, Trondheim, Norway
Re: Features of the JUnit test execution harness
Posted by Rick Hillegas <Ri...@Sun.COM>.
Hi Deepa,
Thanks for bringing this up. I agree that being able to run JUnit tests
on the small device configuration would be a good idea.
Regards,
-Rick
Deepa Remesh wrote:
>>One thing that would be great is if there was a well defined way to
>>have tests handle jvm specific, framework specific or version specific
>>logic, since there will have to be some replacement for the many
>>properties in the test harness and the individual test checks for
>>framework that we have now.
>>
>>
>>
>I would also like to know how JUnit tests will map the current test
>harness properties. My concern is specifically how to run JUnit tests
>with CDC/Foundation/jsr169. I found the following test failing in this
>environment: derbyall/derbyall.fail:junitTests/lang/LangSuite.java
>
>The reason is the test uses DerbyJUnitTest.getConnection method which
>uses Driver class which is not available in jsr169. For tests which
>use the current test harness, ij.startJBMS() method is used to get
>connections which handles getting connection on different platforms.
>Can we have a similar central method to get connections in the JUnit
>harness?
>
>I think it would be good to get JUnit tests running in this
>environment before we move more tests.
>
>Thanks,
>Deepa
>
>
Re: Features of the JUnit test execution harness
Posted by Deepa Remesh <dr...@gmail.com>.
> One thing that would be great is if there was a well defined way to
> have tests handle jvm specific, framework specific or version specific
> logic, since there will have to be some replacement for the many
> properties in the test harness and the individual test checks for
> framework that we have now.
>
I would also like to know how JUnit tests will map the current test
harness properties. My concern is specifically how to run JUnit tests
with CDC/Foundation/jsr169. I found the following test failing in this
environment: derbyall/derbyall.fail:junitTests/lang/LangSuite.java
The reason is the test uses DerbyJUnitTest.getConnection method which
uses Driver class which is not available in jsr169. For tests which
use the current test harness, ij.startJBMS() method is used to get
connections which handles getting connection on different platforms.
Can we have a similar central method to get connections in the JUnit
harness?
I think it would be good to get JUnit tests running in this
environment before we move more tests.
Thanks,
Deepa
Re: Features of the JUnit test execution harness
Posted by Kathey Marsden <km...@sbcglobal.net>.
Daniel John Debrunner wrote:
>Daniel John Debrunner wrote:
>
>
>
>>David W. Van Couvering wrote:
>>
>>
>>
>>
>>>I agree with you that is disconcerting, but can't JUnit tests be written
>>>that extend DerbyJUnitTest in parallel with getting DerbyJUnitTest
>>>cleaned up to everyone's satisfaction?
>>>
>>>
>>Depends, at some point you put the potential burden of fixing all those
>>new tests on the person doing the cleanup, that should have been part of
>>the original submission.
>>
>>
>
>The other issue is that we have a great opportunity to start out with
>JUnit tests that follow a consistent pattern and provide a great example
>for others to follow. If we add a number of tests now that haphazardly
>use the methods in DerbyJUnitTest (because they are not well commented)
>then we have a set of tests like the current ones. No pattern, no
>obvious starting point leading to people inventing their own ways to get
>connections, run ddl, handle exceptions etc.
>
>
>
One thing that would be great is if there was a well defined way to
have tests handle jvm specific, framework specific or version specific
logic, since there will have to be some replacement for the many
properties in the test harness and the individual test checks for
framework that we have now.
I suggested something like this for the client/server compatibility
testing, but did not consider JUnit. It would be good to have something
laid out for JUnit testing so that tests are checking whether
functionality is suppported and the details of whether that is because
of the framework, jvm or whatever are not spread around all the tests.
http://www.nabble.com/Server-and-client-compatibility-test-for-10.2-and-10.1-t913020.html#a2499854
Kathey
Re: Features of the JUnit test execution harness
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
I have to agree we're missing a well-defined pattern and associated
"guidebook" on how to do this right. And I agree we should get this
done before too many more JUnit tests are written.
David
Daniel John Debrunner wrote:
> Daniel John Debrunner wrote:
>
>
>>David W. Van Couvering wrote:
>>
>>
>>
>>>I agree with you that is disconcerting, but can't JUnit tests be written
>>>that extend DerbyJUnitTest in parallel with getting DerbyJUnitTest
>>>cleaned up to everyone's satisfaction?
>>
>>
>>Depends, at some point you put the potential burden of fixing all those
>>new tests on the person doing the cleanup, that should have been part of
>>the original submission.
>
>
> The other issue is that we have a great opportunity to start out with
> JUnit tests that follow a consistent pattern and provide a great example
> for others to follow. If we add a number of tests now that haphazardly
> use the methods in DerbyJUnitTest (because they are not well commented)
> then we have a set of tests like the current ones. No pattern, no
> obvious starting point leading to people inventing their own ways to get
> connections, run ddl, handle exceptions etc.
>
> Dan.
>
>
Re: Features of the JUnit test execution harness
Posted by Daniel John Debrunner <dj...@apache.org>.
Daniel John Debrunner wrote:
> David W. Van Couvering wrote:
>
>
>>I agree with you that is disconcerting, but can't JUnit tests be written
>>that extend DerbyJUnitTest in parallel with getting DerbyJUnitTest
>>cleaned up to everyone's satisfaction?
>
>
> Depends, at some point you put the potential burden of fixing all those
> new tests on the person doing the cleanup, that should have been part of
> the original submission.
The other issue is that we have a great opportunity to start out with
JUnit tests that follow a consistent pattern and provide a great example
for others to follow. If we add a number of tests now that haphazardly
use the methods in DerbyJUnitTest (because they are not well commented)
then we have a set of tests like the current ones. No pattern, no
obvious starting point leading to people inventing their own ways to get
connections, run ddl, handle exceptions etc.
Dan.
Re: Features of the JUnit test execution harness
Posted by Daniel John Debrunner <dj...@apache.org>.
David W. Van Couvering wrote:
> I agree with you that is disconcerting, but can't JUnit tests be written
> that extend DerbyJUnitTest in parallel with getting DerbyJUnitTest
> cleaned up to everyone's satisfaction?
Depends, at some point you put the potential burden of fixing all those
new tests on the person doing the cleanup, that should have been part of
the original submission.
> BTW, if you don't like the rampant catching and ignoring, is anything
> preventing you from going in and fixing it? :)
Nope, apart from time. There's also nothing stopping me from veto'ing
the original submission and reverting it until it handles exceptions in
a way that does not have the potential to hide bugs. Guess which is less
effort for me. :-)
> I suppose we need to
> hear from Rick about his feelings around this before we go and reverse
> his changes...
Actually that doesn't matter, since they are not 'his changes', it's now
code at Apache Derby.
Dan.
Re: Features of the JUnit test execution harness
Posted by Rick Hillegas <Ri...@Sun.COM>.
I apologize for the radio silence on this topic. I have been on the road
and unable to address these concerns. Today I will put some effort into
beefing up the comments in DerbyJUnitTest and handling the swallowed
exceptions.
I agree that we need a primer for writing JUnit tests. I can mock up
such a document and we can collectively polish it. I would like to start
out with something small. I believe that other people are working on a
larger JUnit framework and I would like their help in building up this
document.
Thanks for your patience,
-Rick
David W. Van Couvering wrote:
> I agree with you that is disconcerting, but can't JUnit tests be
> written that extend DerbyJUnitTest in parallel with getting
> DerbyJUnitTest cleaned up to everyone's satisfaction?
>
> BTW, if you don't like the rampant catching and ignoring, is anything
> preventing you from going in and fixing it? :) I suppose we need to
> hear from Rick about his feelings around this before we go and reverse
> his changes...
>
> David
>
> Daniel John Debrunner wrote:
>
>>
>>
>> I would like my concerns to be cleared up before too many tests use the
>> current Junit facility. The rampant catching and ignoring of
>> SQLExceptions is not good.
>>
>> Dan.
>>
>>
Re: Features of the JUnit test execution harness
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
I agree with you that is disconcerting, but can't JUnit tests be written
that extend DerbyJUnitTest in parallel with getting DerbyJUnitTest
cleaned up to everyone's satisfaction?
BTW, if you don't like the rampant catching and ignoring, is anything
preventing you from going in and fixing it? :) I suppose we need to
hear from Rick about his feelings around this before we go and reverse
his changes...
David
Daniel John Debrunner wrote:
>
>
> I would like my concerns to be cleared up before too many tests use the
> current Junit facility. The rampant catching and ignoring of
> SQLExceptions is not good.
>
> Dan.
>
>
Re: Features of the JUnit test execution harness
Posted by Daniel John Debrunner <dj...@apache.org>.
Francois Orsini wrote:
>
>
> On 1/24/06, *Kristian Waagan* <Kristian.Waagan@sun.com
> <ma...@sun.com>> wrote:
>
> David W. Van Couvering wrote:
> > Hi, Kristian, thanks for your questions. My one overriding thought is
> > we should take this in incremental steps -- do all of these questions
> > need to be answered before we can rewrite a single old canon-based
> test
> > to a JUnit test? Can some of these questions be deferred?
>
> Yes, they can be deferred. I did not intend to say that we should try to
> fix/implement all issues/features at in one step.
>
> My main point of concern, is that it is too hard to write JUnit tests
> now, because there is so little information available. Just have a look
> at the number of JUnit tests that have been added to the repository - it
> sure ain't many! The conversion process, which I understand is fully
> based on "it's my itch" initiatives, is also moving along very, very
> slowly.
>
>
> +1 - I have now written some derby jUnit tests myself and I agree that
> it may not be that obvious from the beginning (different paradigm from
> canon-based tests) - I used jUnit before so it did help. I'm currently
> scratching a few itches but I'd be glad to post some instructions unless
> some itch-scratching idle volunteer want to take into that heroic task
> ;-) - jUnit aims at making unit testing easier and more effective
> therefore we just need to make the Derby jUnit adoption easier and more
> effective ;-)
I would like my concerns to be cleared up before too many tests use the
current Junit facility. The rampant catching and ignoring of
SQLExceptions is not good.
Dan.
Re: Features of the JUnit test execution harness
Posted by Francois Orsini <fr...@gmail.com>.
On 1/24/06, Kristian Waagan <Kr...@sun.com> wrote:
>
> David W. Van Couvering wrote:
> > Hi, Kristian, thanks for your questions. My one overriding thought is
> > we should take this in incremental steps -- do all of these questions
> > need to be answered before we can rewrite a single old canon-based test
> > to a JUnit test? Can some of these questions be deferred?
>
> Yes, they can be deferred. I did not intend to say that we should try to
> fix/implement all issues/features at in one step.
>
> My main point of concern, is that it is too hard to write JUnit tests
> now, because there is so little information available. Just have a look
> at the number of JUnit tests that have been added to the repository - it
> sure ain't many! The conversion process, which I understand is fully
> based on "it's my itch" initiatives, is also moving along very, very
> slowly.
>
>
+1 - I have now written some derby jUnit tests myself and I agree that it
may not be that obvious from the beginning (different paradigm from
canon-based tests) - I used jUnit before so it did help. I'm currently
scratching a few itches but I'd be glad to post some instructions unless
some itch-scratching idle volunteer want to take into that heroic task ;-)
- jUnit aims at making unit testing easier and more effective therefore we
just need to make the Derby jUnit adoption easier and more effective ;-)
Re: Features of the JUnit test execution harness
Posted by Kristian Waagan <Kr...@Sun.COM>.
David W. Van Couvering wrote:
> Hi, Kristian, thanks for your questions. My one overriding thought is
> we should take this in incremental steps -- do all of these questions
> need to be answered before we can rewrite a single old canon-based test
> to a JUnit test? Can some of these questions be deferred?
Yes, they can be deferred. I did not intend to say that we should try to
fix/implement all issues/features at in one step.
My main point of concern, is that it is too hard to write JUnit tests
now, because there is so little information available. Just have a look
at the number of JUnit tests that have been added to the repository - it
sure ain't many! The conversion process, which I understand is fully
based on "it's my itch" initiatives, is also moving along very, very slowly.
>
> Kristian Waagan wrote:
>
>> Hello,
>>
>> I have wanted to rewrite a old canon based test to a JUnit test for a
>> while, but each time I get started I always run into issues that we do
>> not have guidelines for yet. So, I have a few questions I would like
>> some feedback on.
>> I know there are a lot of issues to consider, far more then the one I
>> mention here, but I hope to get a feeling of the general direction we
>> want to move in.
>>
>>
>> 1) Should we support specific ordering of tests (within a single class)?
>>
>> This feature is wanted if a number of tests operate on shared data and
>> each test method can be seen as a step in the test. If the order is
>> mixed up, the intermediate results will not be as expected and all the
>> tests may fail.
>> In general, assuming anything about the execution order of tests is
>> strongly disencouraged. However, since we are testing a database
>> system, we might want to allow for this. This feature can influence
>> the both the size of the test methods and the execution time for the
>> test/suite.
>
>
> I thought if you name your tests correctly (e.g. test_010, test_020,
> etc.) then they are run in that order. Or is that not the case because
> all method names on a class are hashed?
This is not the case. Tests are not ordered by method names. I do not
know the implementation details about ordering when using reflection. I
do know that when adding one test at a time in suite(), they are stored
in a Vector and thus ordered (and executed in order). But please note
that this is not a documented feature of JUnit.
>
> Assuming that's not the case, it seems to me that if you have specific
> ordering requirements, you could use multiple test classes, create a
> suite of those classes, and use the TestSetup mechanism you describe
> below to do any setup work prior to running the suite.
>
Might work. In my opinion, the best thing is to write each test to be
totally independent, but I am looking for ways to handle tests that are
dependent on each other because I suspect that need will arise.
>>
>> 2) Should we support setting up a shared test fixture for a set of tests?
>>
>> This is a common issue with JUnit, and there are mechanisms to handle
>> it. For instance, we could let tests that require this to wrap itself
>> in a TestSetup instance (as described in the JUNit FAQ). Letting
>> DerbyJUnitTest extend TestSetup is another solution, but it might have
>> unwanted side-effects.
>
>
> I think there is a lot of value to having something like this (see above
> for an example).
>
>>
>> 3) Many tests just want a database to work on. Other tests want to
>> create their own database due to specific requirements. How do we make
>> this easy for the developer?
>>
>
> I think if you have a TestSetup mechanism, then a suite of tests can
> depend on the fact that a database has been created. Creating a
> database is pretty darn easy already; are there particular things that
> are repetetive or error-prone?
No. As long as you have the driver name and the connection string
(including the correct protocol and database path/name), it is very
easy. This information must be passed on to the test setup class when
suite() is run, and with the current harness implementation (with
respect to JUnit tests) this information is not available then. There
are easy workarounds, but they are not written down anywhere...
This is actually a matter of question 6.
>
>> 4) Should it be possible to run single JUnit test/suite from the
>> command line by simply invoking the main method?
>>
>> Personally, I think this should be possible. This require us to remove
>> the current use of main for running a test in the old test harness,
>> which I believe is doable with a few changes (in the old harness).
>>
>
> I think this is essential for debugging and incremental testing. +1 on
> this idea.
>
>> 5) Is the ultimate goal to end up with a pure JUnit based test harness?
>>
>> If this is the goal, we should design for this and accept some extra
>> work for plugging JUnit tests into the old test harness.
>
>
> I can see us getting there over time, but this is a Very Big Effort. Are
> there particular things we have to decide upon now in order for this to
> be possible in the future?
I don't know of any, but someone else might.
As long as we code the tests in typical JUnit spirit, we should be good.
As soon as we start deviating from "standard" behavior, people won't be
able to run the tests in third-party tools (for instance NetBeans).
We might not be able to allow for this anyway, though.
>
>>
>> 6) Where, and how, do we want to store settings for a test?
>>
>
> In a database! :)
>
> I personally think a properties file is easier to work with than an XML
> file, *as long as* we don't load these in as System properties, gads,
> that's a nightmare.
>
>> 7) How is it determined which framework(s) a JUnit test should be run
>> in? (i.e. embedded, client, JCC)
>>
>
> I think you should have ways to run a particular suite under a
> particular framework, as you do today.
I agree. You could also have a way to run under all available
frameworks, but then a bit more is required by the harness (or manual
work - starting servers).
>
>>
>> If you have other opinions or comments related to JUnit testing,
>> please put them forth :)
>>
>
> I think ther are two separate questions:
>
> - Running tests under JUnit with a JUnit harness
> - Rewriting existing tests to no longer use canons
>
> I think we need to make sure we discuss these separately. Existing
> tests can run under JUnit with canons just fine (that's what we're
> already doing). I think that's could be big performance improvement
> right there (I remember Dan showing the difference in performance).
>
Hmm, I thought we were running JUnit tests under the old harness. That
is, the harness automatically generates an empty canon.
Anyway, I agree that if our current harness gets the job done, there is
no need to rush for a new harness capable of running only JUnit tests.
But I mean we need to produce enough information to enable people to
write JUnit tests quickly and without great suffering ;)
--
Kristian
> Rewriting tests to no longer use canons (and thus .sql files) is a Much
> Bigger Task and I suggest that be separated out, and we should discuss
> its merits/demerits separately.
>
>> I hope we are able to get a discussion on all or some of these issues,
>> and that it results in a Wiki page describing JUnit testing for Derby.
>> I created a page here: http://wiki.apache.org/db-derby/DerbyJUnitTesting
>>
>
> I am sure you will get a discussion. In terms of it "resulting" in a
> Wiki page, I think that is up to whoever has the itch to do that (it
> sounds like you already do).
>
> All the best,
>
> David
>
>>
>>
>> --
>> Kristian
Re: Features of the JUnit test execution harness
Posted by Kristian Waagan <Kr...@Sun.COM>.
Deepa Remesh wrote:
>>So I guess what you are saying is that if the test framework provides a
>>common mechanism to give a Connection to a derby database, it should go
>>through a DataSource, instead of using DriverManager ?
>>
>
>
> I think we will need both mechanisms to get connection - using
> DataSource and DriverManager.
>
> J2ME/CDC/Foundation Profile is based on JSR169 for JDBC, which does
> not have java.sql.DriverManager class. So, here we need a mechanism to
> get a connection using javax.sql.DataSource.
>
> Using DataSource to get connection will not work in case of j9 vms
> (not foundation profile). In this case, database_enabler.jar is used
> for JDBC support and this does not have javax.sql.DataSource class.
>
> I'd appreciate if someone else familiar in this area can confirm this.
>
> Thanks,
> Deepa
I'm not familiar with this area, but wouldn't it in general be best to
have a single method in the harness that returns a valid connection?
If the test is running in a J2ME/CDC/Foundation environment, it uses
DataSource to get a connection, if it is in a j9 environment then it
should use whatever is supported there. For most tests, how the
connection is obtained is irrelevant, as long as it works!
I see that a few tests, for instance a test that is testing the
different ways to get connections, would have the need to override such
a standard method, but that's something that can be added when these
tests are to be written in JUnit-style. However, depending on JVM and
(un)available jars, some of the test cases in such a suite would have to
be skipped anyway.
Instead of "learning" the harness how to get connections in X different
ways, I would consider just extracting the required connection
information (driver/framework prefix, database name, server, port etc)
from the harness and then code how I get the connection in my test *if
this is required*. The harness should only have to know about a minimal
set of ways to get connections. With minimal set, I mean it has to know
about one way to get a connection for all jvms/platforms (that we
"support" or have the itch for). It does not need to know all the ways
for all jvms/platforms.
From the previous mails, the harness would have to know how to use
DataSource and DriverManager. Are there jvms/platforms where neither of
these will be able to produce a connection?
--
Kristian
Re: Features of the JUnit test execution harness
Posted by Daniel John Debrunner <dj...@apache.org>.
Deepa Remesh wrote:
>>So I guess what you are saying is that if the test framework provides a
>>common mechanism to give a Connection to a derby database, it should go
>>through a DataSource, instead of using DriverManager ?
>>
>
>
> I think we will need both mechanisms to get connection - using
> DataSource and DriverManager.
>
> J2ME/CDC/Foundation Profile is based on JSR169 for JDBC, which does
> not have java.sql.DriverManager class. So, here we need a mechanism to
> get a connection using javax.sql.DataSource.
>
> Using DataSource to get connection will not work in case of j9 vms
> (not foundation profile). In this case, database_enabler.jar is used
> for JDBC support and this does not have javax.sql.DataSource class.
>
> I'd appreciate if someone else familiar in this area can confirm this.
Correct, the other j9 profile is implementing the standard JDBC 2.1 api
and does not supply the optional JDBC 2.0 extensions which include
DataSource etc. Derby currently does support this environment and so the
test harness needs to as well.
In addition we will need to test getting connections from DriverManager,
so it needs to be supported in some way. Ideally the majority of tests
should be written to be indepedent of how a connection is obtained,
which is how it is today. Then all of the tests could be run with a
conneciton from a DataSource, the DriverManager, a Driver, a
PooledConnection or a XAConnection.
Dan.
Re: Features of the JUnit test execution harness
Posted by Deepa Remesh <dr...@gmail.com>.
> So I guess what you are saying is that if the test framework provides a
> common mechanism to give a Connection to a derby database, it should go
> through a DataSource, instead of using DriverManager ?
>
I think we will need both mechanisms to get connection - using
DataSource and DriverManager.
J2ME/CDC/Foundation Profile is based on JSR169 for JDBC, which does
not have java.sql.DriverManager class. So, here we need a mechanism to
get a connection using javax.sql.DataSource.
Using DataSource to get connection will not work in case of j9 vms
(not foundation profile). In this case, database_enabler.jar is used
for JDBC support and this does not have javax.sql.DataSource class.
I'd appreciate if someone else familiar in this area can confirm this.
Thanks,
Deepa
Re: Features of the JUnit test execution harness
Posted by Andreas Korneliussen <An...@Sun.COM>.
Myrna van Lunteren wrote:
> On 2/2/06, *Andreas Korneliussen* <Andreas.Korneliussen@sun.com
> <ma...@sun.com>> wrote:
>
> I think the work currently done on DERBY-874 was mainly to improve the
> DerbyJUnitTest's JavaDoc, and to log exceptions. So I would not throw
> that away.
>
> However I do propose to change DerbyJUnitTest to move out everything
> about configuration into a separate class.
>
>
> cool. thx for the reply.
>
> I now noticed that the wiki says all suggestions are to be put on the
> list, so here I go rather than plopping them directly on the wiki:
>
Great feature list.
What I think is important is that we provide a common mechanism to get
access to the configuration of the test.
If someone then need to do some framework/version specific logic, they
have a mechanism to get i.e the framework enum, and can write logic
based on that.
> I think the following could qualify as 'more details' to the jvm,
> framework, version specific logic:
>
> 1. jvm-specific:
> 1.1.
> not all parameters are consistent for all jvms. Think here of jit
> settings / configurations, memory settings. For j2ME testing, that jvm
> doesn't come with a DriverManager implementation, so already from the
> start you know you have to go with DataSources.
So I guess what you are saying is that if the test framework provides a
common mechanism to give a Connection to a derby database, it should go
through a DataSource, instead of using DriverManager ?
> 1.2. Different versions of a vendor's jvm also may have slightly
> different implementations resulting in slightly different behavior -
> e.g. of the order of rows, for instance, or rounding of certain numeric
> values.
> 1.3. Some behavior will be only supported by later versions...
>
> 2. version specific.
> This really falls back to the discussion on ...(can't find right now,
> raman's working on it, I think)... re mixed version testing. I think the
> conclusion was that the harness needs a way to cope with results from
> newer servers and clients - if they differ from results with same
> versions as the harness.
>
> 3. framework specific
> The tests needs to be able to cope with the following
> 3.1. different client drivers (e.g. DerbyClient, IBM Universal JDBC Driver)
> 3.2. server may need to be started by harness, or not
> 3.3. server may be on the localhost, or be running on a remote machine.
> certain individual tests may not be able to run in with this
> mechanism...
> 3.4 should be able to have the harness start the server in a differrent jvm.
>
> 4. one thing the current harness has no way of doing is to cope with
> different OSs. For instance, sometimes there are slight differences in
> behaviour of the same jvm version on different OSs. Like slightly
> different error messages (although this may be irrelevant if we're not
> gathering & comparing output).
>
> I think the following details would be useful (in addition to the above
> and item 1 on the wiki):
> - there must be a way to skip individual tests without causing an error
> but with an informational message for certain configurations. eg.
> absence of optional jars (specifically thinking of db2jcc.jar),
> unsupported functionality with older jvms..., or when there is a problem
> that's being worked on, or that's been referred to some other
> organization ( e.g. in the case of jvm bugs, OS bugs...).
>
> - some way to compare runtimestatistics.
> Currently this is done by comparing the output, I have a hard time
> thinking of another mechanism.
>
I am not sure which runtimestatistics you think of. Which output ?
Output from a SQL statement ?
Thanks
--Andreas
Re: Features of the JUnit test execution harness
Posted by Myrna van Lunteren <m....@gmail.com>.
On 2/2/06, Andreas Korneliussen <Andreas.Korneliussen@sun.com > wrote:
>
> I think the work currently done on DERBY-874 was mainly to improve the
> DerbyJUnitTest's JavaDoc, and to log exceptions. So I would not throw
> that away.
>
> However I do propose to change DerbyJUnitTest to move out everything
> about configuration into a separate class.
cool. thx for the reply.
I now noticed that the wiki says all suggestions are to be put on the list,
so here I go rather than plopping them directly on the wiki:
I think the following could qualify as 'more details' to the jvm, framework,
version specific logic:
1. jvm-specific:
1.1.
not all parameters are consistent for all jvms. Think here of jit settings /
configurations, memory settings. For j2ME testing, that jvm doesn't come
with a DriverManager implementation, so already from the start you know you
have to go with DataSources.
1.2. Different versions of a vendor's jvm also may have slightly different
implementations resulting in slightly different behavior - e.g. of the order
of rows, for instance, or rounding of certain numeric values.
1.3. Some behavior will be only supported by later versions...
2. version specific.
This really falls back to the discussion on ...(can't find right now,
raman's working on it, I think)... re mixed version testing. I think the
conclusion was that the harness needs a way to cope with results from newer
servers and clients - if they differ from results with same versions as the
harness.
3. framework specific
The tests needs to be able to cope with the following
3.1. different client drivers (e.g. DerbyClient, IBM Universal JDBC Driver)
3.2. server may need to be started by harness, or not
3.3. server may be on the localhost, or be running on a remote machine.
certain individual tests may not be able to run in with this
mechanism...
3.4 should be able to have the harness start the server in a differrent jvm.
4. one thing the current harness has no way of doing is to cope with
different OSs. For instance, sometimes there are slight differences in
behaviour of the same jvm version on different OSs. Like slightly different
error messages (although this may be irrelevant if we're not gathering &
comparing output).
I think the following details would be useful (in addition to the above and
item 1 on the wiki):
- there must be a way to skip individual tests without causing an error but
with an informational message for certain configurations. eg. absence of
optional jars (specifically thinking of db2jcc.jar), unsupported
functionality with older jvms..., or when there is a problem that's being
worked on, or that's been referred to some other organization (e.g. in the
case of jvm bugs, OS bugs...).
- some way to compare runtimestatistics.
Currently this is done by comparing the output, I have a hard time
thinking of another mechanism.
Ok, that'll do for now...:-)
Myrna
Re: Features of the JUnit test execution harness
Posted by Andreas Korneliussen <An...@Sun.COM>.
Myrna van Lunteren wrote:
> On 1/31/06, *Kristian Waagan* <Kristian.Waagan@sun.com
> <ma...@sun.com>> wrote:
>
> Differences in output should be irrelevant. Although not what you
> mentioned above, the issue of (execution) control is very relevant. The
> logic for running the tests multiple times, each time with a different
> setup/environment must be located somewhere. I think Andreas' proposal
> of introducing a separate JUnit test type (see
> http://www.nabble.com/running-JUnit-tests-t887682.html#a2300670) makes
> sense, as it gives us more freedom w.r.t. handling of JUnit tests.
>
>
> Yes, that proposal made sense to me. I personally like the approach of
> having a class for various/different configurations. Although that could
> get out of hand.
>
> Does this 'throw away' the work that Rick is doing on DERBY-874?
>
I think the work currently done on DERBY-874 was mainly to improve the
DerbyJUnitTest's JavaDoc, and to log exceptions. So I would not throw
that away.
However I do propose to change DerbyJUnitTest to move out everything
about configuration into a separate class.
> Following Andreas' approach we'd still be able to run the individual
> tests separately, yes?
>
Yes - definetly.
Andreas
Re: Features of the JUnit test execution harness
Posted by Myrna van Lunteren <m....@gmail.com>.
On 1/31/06, Kristian Waagan <Kr...@sun.com> wrote:
>
> Differences in output should be irrelevant. Although not what you
> mentioned above, the issue of (execution) control is very relevant. The
> logic for running the tests multiple times, each time with a different
> setup/environment must be located somewhere. I think Andreas' proposal
> of introducing a separate JUnit test type (see
> http://www.nabble.com/running-JUnit-tests-t887682.html#a2300670) makes
> sense, as it gives us more freedom w.r.t. handling of JUnit tests.
Yes, that proposal made sense to me. I personally like the approach of
having a class for various/different configurations. Although that could get
out of hand.
Does this 'throw away' the work that Rick is doing on DERBY-874?
Following Andreas' approach we'd still be able to run the individual tests
separately, yes?
> Thanks for the answer Myrna. And good work on the readme!
credit due: many people have reviewed, and improved it...
> <snip>
> Right now I feel that it is very hard to start writing a new JUnit
> harness, since the old harness seem to contain a lot of functionality
> and "magic". I wonder if it would be best to just start
> converting/writing some tests and create a framework along the way.
Yes, a new harness is hard, and yes, the old harness has been made to adapt
to many different needs...and we should take those into account. (I'll add
some thoughts on that wiki soon). But the old harness has grown unwieldy and
hard to maintain and is slow. To take what we learned and craft it
onto something new will be better, though daunting.
Some ideas of Dan's are in this thread from about a year ago:
http://mail-archives.apache.org/mod_mbox/db-derby-dev/200503.mbox/%3C422605A7.6020207@debrunners.com%3E
The above approach would probably result in several major rewritings of
> "the new framework" and the existing JUnit tests, but it might be better
> than thinking out everything at once (that's hard!). Perhaps I will
> rewrite an old test I have been working a little on, and put it out for
> review and comments, unless someone else beats me to it ;)
--
> Kristian
>
Personally, I do better when I have code to prod and modify through
iterations...
I'd be interested to review.
Myrna
Re: Features of the JUnit test execution harness
Posted by Kristian Waagan <Kr...@Sun.COM>.
Myrna van Lunteren wrote:
> On 1/24/06, *Kristian Waagan* <Kristian.Waagan@sun.com
> <ma...@sun.com>> wrote:
>
> >>>Kristian Waagan wrote:
> >>>4) Should it be possible to run single JUnit test/suite from the
> >>>command line
>
> >><2 yeahs snipped, 3rd: (Dan:)>
> >> I think it's essential that a single test can be run. I don't think
> it's
> >> essential that it is from test's main method. The harness produces the
> >> useful functionality of setting up the correct environment for the
> test,
> >> not sure if that could be duplicated in evey main method for every
> test.
>
> >Good point Dan.
>
> >Citing Dan: "...harness produces the useful functionality of setting up
> >the correct environment for the test,...".
>
> >Can somebody please explain to me what is actually done by the harness?
>
> >I did have a quick look at the harness code, but I would need to study
> >it a bit more to understand what it actually does. Stating what the
> >harness does when it comes to setting up the environment for tests would
> >be good for the general understanding of the test execution process.
> ---------------
>
> I've had this messages 'starred' in gmail for a bit, sorry for
> replying only now. There's no quick way to describe what the
> 'functiontest' harness does - I've tried to describe what I understand
> in the past (see the java/testing/README.htm)...
>
> Maybe if you have specific questions I'll be able to answer better.
>
> The main thing I think that the functiontest harness does is set up
> all sorts of fun properties for the tests to use, so the same test can
> run in a number of different ways - in a suite, by itself, with
> different encryption algorithms, with encoding, locales, frameworks,
> with java.sql.DriverManager or instead with DataSource; to run in
> sequence or separately, start a new database or not...Depending on
> these properties, more or less 'magic' is done - for example, network
> server gets started or not, files needed for the test get copied
> around...
> Properties in a suite override properties set for individual tests
> lower level.
> Also, it is able to figure out what jvm is used and add specific
> (non-standard) properties as needed or applicable.
>
> A big part of it is to control, and eliminate uninteresting
> differences in the output, but that would be irrelevant with Junit, yes?
Differences in output should be irrelevant. Although not what you
mentioned above, the issue of (execution) control is very relevant. The
logic for running the tests multiple times, each time with a different
setup/environment must be located somewhere. I think Andreas' proposal
of introducing a separate JUnit test type (see
http://www.nabble.com/running-JUnit-tests-t887682.html#a2300670) makes
sense, as it gives us more freedom w.r.t. handling of JUnit tests.
>
> Does this help at all?
> But maybe that wasn't what Dan meant?
>
> Myrna
Thanks for the answer Myrna. And good work on the readme! I just skimmed
through it now, but I will take the time to read it thoroughly soon.
Right now I feel that it is very hard to start writing a new JUnit
harness, since the old harness seem to contain a lot of functionality
and "magic". I wonder if it would be best to just start
converting/writing some tests and create a framework along the way.
The above approach would probably result in several major rewritings of
"the new framework" and the existing JUnit tests, but it might be better
than thinking out everything at once (that's hard!). Perhaps I will
rewrite an old test I have been working a little on, and put it out for
review and comments, unless someone else beats me to it ;)
--
Kristian
Re: Features of the JUnit test execution harness
Posted by Myrna van Lunteren <m....@gmail.com>.
On 1/24/06, Kristian Waagan <Kr...@sun.com> wrote:
>
> >>>Kristian Waagan wrote:
> >>>4) Should it be possible to run single JUnit test/suite from the
> >>>command line
>><2 yeahs snipped, 3rd: (Dan:)>
>> I think it's essential that a single test can be run. I don't think it's
>> essential that it is from test's main method. The harness produces the
>> useful functionality of setting up the correct environment for the test,
>> not sure if that could be duplicated in evey main method for every test.
>Good point Dan.
>Citing Dan: "...harness produces the useful functionality of setting up
>the correct environment for the test,...".
>Can somebody please explain to me what is actually done by the harness?
>I did have a quick look at the harness code, but I would need to study
>it a bit more to understand what it actually does. Stating what the
>harness does when it comes to setting up the environment for tests would
>be good for the general understanding of the test execution process.
---------------
I've had this messages 'starred' in gmail for a bit, sorry for replying only
now. There's no quick way to describe what the 'functiontest' harness does -
I've tried to describe what I understand in the past (see the
java/testing/README.htm)...
Maybe if you have specific questions I'll be able to answer better.
The main thing I think that the functiontest harness does is set up all
sorts of fun properties for the tests to use, so the same test can run in a
number of different ways - in a suite, by itself, with different encryption
algorithms, with encoding, locales, frameworks, with
java.sql.DriverManageror instead with DataSource; to run in sequence
or separately, start a new
database or not...Depending on these properties, more or less 'magic' is
done - for example, network server gets started or not, files needed for the
test get copied around...
Properties in a suite override properties set for individual tests lower
level.
Also, it is able to figure out what jvm is used and add specific
(non-standard) properties as needed or applicable.
A big part of it is to control, and eliminate uninteresting differences in
the output, but that would be irrelevant with Junit, yes?
Does this help at all?
But maybe that wasn't what Dan meant?
Myrna
Re: Features of the JUnit test execution harness
Posted by Kristian Waagan <Kr...@Sun.COM>.
Daniel John Debrunner wrote:
> David W. Van Couvering wrote:
>
>
>
>>Kristian Waagan wrote:
>
>
>>>4) Should it be possible to run single JUnit test/suite from the
>>>command line by simply invoking the main method?
>>>
>>>Personally, I think this should be possible. This require us to remove
>>>the current use of main for running a test in the old test harness,
>>>which I believe is doable with a few changes (in the old harness).
>>>
>>
>>I think this is essential for debugging and incremental testing. +1 on
>>this idea.
>
>
> I think it's essential that a single test can be run. I don't think it's
> essential that it is from test's main method. The harness produces the
> useful functionality of setting up the correct environment for the test,
> not sure if that could be duplicated in evey main method for every test.
Good point Dan.
Citing Dan: "...harness produces the useful functionality of setting up
the correct environment for the test,...".
Can somebody please explain to me what is actually done by the harness?
I did have a quick look at the harness code, but I would need to study
it a bit more to understand what it actually does. Stating what the
harness does when it comes to setting up the environment for tests would
be good for the general understanding of the test execution process.
--
Kristian
>
> Dan.
>
Re: Features of the JUnit test execution harness
Posted by Daniel John Debrunner <dj...@apache.org>.
David W. Van Couvering wrote:
> Kristian Waagan wrote:
>> 4) Should it be possible to run single JUnit test/suite from the
>> command line by simply invoking the main method?
>>
>> Personally, I think this should be possible. This require us to remove
>> the current use of main for running a test in the old test harness,
>> which I believe is doable with a few changes (in the old harness).
>>
>
> I think this is essential for debugging and incremental testing. +1 on
> this idea.
I think it's essential that a single test can be run. I don't think it's
essential that it is from test's main method. The harness produces the
useful functionality of setting up the correct environment for the test,
not sure if that could be duplicated in evey main method for every test.
Dan.
Re: Features of the JUnit test execution harness
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Hi, Kristian, thanks for your questions. My one overriding thought is
we should take this in incremental steps -- do all of these questions
need to be answered before we can rewrite a single old canon-based test
to a JUnit test? Can some of these questions be deferred?
Kristian Waagan wrote:
> Hello,
>
> I have wanted to rewrite a old canon based test to a JUnit test for a
> while, but each time I get started I always run into issues that we do
> not have guidelines for yet. So, I have a few questions I would like
> some feedback on.
> I know there are a lot of issues to consider, far more then the one I
> mention here, but I hope to get a feeling of the general direction we
> want to move in.
>
>
> 1) Should we support specific ordering of tests (within a single class)?
>
> This feature is wanted if a number of tests operate on shared data and
> each test method can be seen as a step in the test. If the order is
> mixed up, the intermediate results will not be as expected and all the
> tests may fail.
> In general, assuming anything about the execution order of tests is
> strongly disencouraged. However, since we are testing a database system,
> we might want to allow for this. This feature can influence the both the
> size of the test methods and the execution time for the test/suite.
I thought if you name your tests correctly (e.g. test_010, test_020,
etc.) then they are run in that order. Or is that not the case because
all method names on a class are hashed?
Assuming that's not the case, it seems to me that if you have specific
ordering requirements, you could use multiple test classes, create a
suite of those classes, and use the TestSetup mechanism you describe
below to do any setup work prior to running the suite.
>
> 2) Should we support setting up a shared test fixture for a set of tests?
>
> This is a common issue with JUnit, and there are mechanisms to handle
> it. For instance, we could let tests that require this to wrap itself in
> a TestSetup instance (as described in the JUNit FAQ). Letting
> DerbyJUnitTest extend TestSetup is another solution, but it might have
> unwanted side-effects.
I think there is a lot of value to having something like this (see above
for an example).
>
> 3) Many tests just want a database to work on. Other tests want to
> create their own database due to specific requirements. How do we make
> this easy for the developer?
>
I think if you have a TestSetup mechanism, then a suite of tests can
depend on the fact that a database has been created. Creating a
database is pretty darn easy already; are there particular things that
are repetetive or error-prone?
> 4) Should it be possible to run single JUnit test/suite from the command
> line by simply invoking the main method?
>
> Personally, I think this should be possible. This require us to remove
> the current use of main for running a test in the old test harness,
> which I believe is doable with a few changes (in the old harness).
>
I think this is essential for debugging and incremental testing. +1 on
this idea.
> 5) Is the ultimate goal to end up with a pure JUnit based test harness?
>
> If this is the goal, we should design for this and accept some extra
> work for plugging JUnit tests into the old test harness.
I can see us getting there over time, but this is a Very Big Effort.
Are there particular things we have to decide upon now in order for this
to be possible in the future?
>
> 6) Where, and how, do we want to store settings for a test?
>
In a database! :)
I personally think a properties file is easier to work with than an XML
file, *as long as* we don't load these in as System properties, gads,
that's a nightmare.
> 7) How is it determined which framework(s) a JUnit test should be run
> in? (i.e. embedded, client, JCC)
>
I think you should have ways to run a particular suite under a
particular framework, as you do today.
>
> If you have other opinions or comments related to JUnit testing, please
> put them forth :)
>
I think ther are two separate questions:
- Running tests under JUnit with a JUnit harness
- Rewriting existing tests to no longer use canons
I think we need to make sure we discuss these separately. Existing
tests can run under JUnit with canons just fine (that's what we're
already doing). I think that's could be big performance improvement
right there (I remember Dan showing the difference in performance).
Rewriting tests to no longer use canons (and thus .sql files) is a Much
Bigger Task and I suggest that be separated out, and we should discuss
its merits/demerits separately.
> I hope we are able to get a discussion on all or some of these issues,
> and that it results in a Wiki page describing JUnit testing for Derby. I
> created a page here: http://wiki.apache.org/db-derby/DerbyJUnitTesting
>
I am sure you will get a discussion. In terms of it "resulting" in a
Wiki page, I think that is up to whoever has the itch to do that (it
sounds like you already do).
All the best,
David
>
>
> --
> Kristian