Old harness '.java' test adapter for JUnit infrastructure.

[Daniel John Debrunner <dj...@apache.org>]

I just committed HarnessJavaTest which is an experiment in running an 
old harness style '.java' test using the JUnit infrastructure. The old 
harness test (e.g. jdbcapi/connectionJdbc20.java) is unchanged and pass 
is indicated by comparing with its master file. This is very similar to 
the running of .sql scripts through JUnit using ScriptTestCase / 
LangScripts / NistScripts. As in those tests only a single master file 
is supported and no 'seding' takes place.

I also added a concrete implementation in JDBCHarnessJavaTest which runs 
'.java' tests from the jdbcapi package. It ran 17 jdbcapi '.java' 
without changes in embedded. (And a subset of those with client/server 
though that is not enabled in the committed version, I just need more 
investigation). It is not added in as part of suites.All and the tests 
listed in it continue to be run with the old harness as part of derbyall.

I'm looking at this approach as a quicker way to moving to a single 
(JUnit based) test infrastructure. While many tests have been converted, 
conversion of an individual test is slow, and thus the overall 
conversion rate is slow. The current number of tests to covert is still 
high, see:
http://wiki.apache.org/db-derby/KillDerbyTestHarness#head-a6e2c89e4bc09edf99f4a7bb6e12af333b6d01f2

While canon based tests are not ideal, I think they are acceptable as an 
interim step, especially if we manage to switch to a single 
infrastructure. A good goal would be to have the 10.3 release (mid 
year?) with a single infrastructure so that bug fixing and text cases 
are easy to merge between trunk and 10.3. Also running the tests in this 
way does not stop anyone from converting an individual test, it makes it 
no harder. In fact it *might* make it easier, as a it might be possible 
to incrementally convert tests and have them continue to run with a 
master file.

One benefit I see from this approach is that it may allow the community 
to focus on converting tests that cannot be run using this or other 
adapters, for example ones with multiple canon files, per jvm version 
for example. Those tend to be the tests that occupy the most time as the 
need to update multiple canons is costly.
It will also centralized which environments a test runs in, one of the 
issues I'm seeing with the jdbcapi tests and client server is trying to 
figure out if a test is meant to run in c/s or not. Not clear from the 
old harness setup.

I'm going to see if I can get a similar high pass rate with .java tests 
in another folder, probably store. If so I will plan on changing the 
tests that can be run with an adapter to run as part of suites.All and 
not derbyall. I would also create a wiki page that listed all the old 
tests and if they are to be run with the harness or JUnit, as it could 
get hard figuring out what state an old harness test is in.

Feedback, thoughts?
Dan.

PS. HarnessJavaTest was surprisingly easy, the only setup required by ij 
to run one of the old tests was to set the ij.database system property. 
For J2ME different properties will need to be set, but that's only a few 
lines of code.

Re: Old harness '.java' test adapter for JUnit infrastructure.

[Rajesh Kartha <ka...@gmail.com>]

Daniel John Debrunner wrote:
>
> While canon based tests are not ideal, I think they are acceptable as 
> an interim step, especially if we manage to switch to a single 
> infrastructure. A good goal would be to have the 10.3 release (mid 
> year?) with a single infrastructure so that bug fixing and text cases 
> are easy to merge between trunk and 10.3. Also running the tests in 
> this way does not stop anyone from converting an individual test, it 
> makes it no harder. In fact it *might* make it easier, as a it might 
> be possible to incrementally convert tests and have them continue to 
> run with a master file.
>
> One benefit I see from this approach is that it may allow the 
> community to focus on converting tests that cannot be run using this 
> or other adapters, for example ones with multiple canon files, per jvm 
> version for example. Those tend to be the tests that occupy the most 
> time as the need to update multiple canons is costly.
> It will also centralized which environments a test runs in, one of the 
> issues I'm seeing with the jdbcapi tests and client server is trying 
> to figure out if a test is meant to run in c/s or not. Not clear from 
> the old harness setup.
>
>
+1.

Will try it out and post my findings.

-Rajesh

Re: Old harness '.java' test adapter for JUnit infrastructure.

[Daniel John Debrunner <dj...@apache.org>]

Daniel John Debrunner wrote:
> Andrew McIntyre wrote:
>> On 2/5/07, Daniel John Debrunner <dj...@apache.org> wrote:
>>>
>>> I also added a concrete implementation in JDBCHarnessJavaTest which runs
>>> '.java' tests from the jdbcapi package. It ran 17 jdbcapi '.java'
>>> without changes in embedded. (And a subset of those with client/server
>>> though that is not enabled in the committed version, I just need more
>>> investigation).
>>
>> This is pretty nifty. I tried converting the .java tests in derbynet
>> that didn't have multiple masters over, but I've run into some
>> problems that I'm still trying to sort out - difficulty locating
>> masters, need some extra properties set on some of the tests, etc.

The individual fixtures for HarnessJavaTest (ie an old style test test) 
can be decorated with the standard decorators, to set database or system 
properties, or handle external files, etc.

>>
>> I was going to look into the lang .java tests as well, but noticed
>> your recent wiki update had some of the .java tests listed as 'running
>> under a test adapter'. If you're already working on getting the lang
>> .java tests to run with the HarnessJavaTest adapter then I don't want
>> to duplicate any effort there.
> 
> Yep, I'll commit the lang sub-class of HarnessJavaTest by tomorrow 
> morning pacific time.

Committed revision 504432.

They also depend on an earlier change to JDBCDisplayUtil. That change 
fixes SecurityExceptions reading ij.* properties you may have seen 
trying to run some tests using HarnessJavaTest.

Dan.

Re: Old harness '.java' test adapter for JUnit infrastructure.

[Daniel John Debrunner <dj...@apache.org>]

Andrew McIntyre wrote:
> On 2/5/07, Daniel John Debrunner <dj...@apache.org> wrote:
>>
>> I also added a concrete implementation in JDBCHarnessJavaTest which runs
>> '.java' tests from the jdbcapi package. It ran 17 jdbcapi '.java'
>> without changes in embedded. (And a subset of those with client/server
>> though that is not enabled in the committed version, I just need more
>> investigation).
> 
> This is pretty nifty. I tried converting the .java tests in derbynet
> that didn't have multiple masters over, but I've run into some
> problems that I'm still trying to sort out - difficulty locating
> masters, need some extra properties set on some of the tests, etc.
> 
> I was going to look into the lang .java tests as well, but noticed
> your recent wiki update had some of the .java tests listed as 'running
> under a test adapter'. If you're already working on getting the lang
> .java tests to run with the HarnessJavaTest adapter then I don't want
> to duplicate any effort there.

Yep, I'll commit the lang sub-class of HarnessJavaTest by tomorrow 
morning pacific time.

Dan.

Re: Old harness '.java' test adapter for JUnit infrastructure.

[Andrew McIntyre <mc...@gmail.com>]

On 2/6/07, Jean T. Anderson <jt...@bristowhill.com> wrote:
>
> just an fyi -- I'm working on converting derbynet/callable.java to junit.

Cool, that's one of the ones that isn't appropriate for the
HarnessJavaTest route, since it has multiple masters, one for jdk14
and another for jdk15/jdk16.

andrew

Re: Old harness '.java' test adapter for JUnit infrastructure.

["Jean T. Anderson" <jt...@bristowhill.com>]

Andrew McIntyre wrote:
> On 2/5/07, Daniel John Debrunner <dj...@apache.org> wrote:
> 
>> I also added a concrete implementation in JDBCHarnessJavaTest which runs
>> '.java' tests from the jdbcapi package. It ran 17 jdbcapi '.java'
>> without changes in embedded. (And a subset of those with client/server
>> though that is not enabled in the committed version, I just need more
>> investigation).
>  
> This is pretty nifty. I tried converting the .java tests in derbynet

just an fyi -- I'm working on converting derynet/callable.java to junit.

 -jean

> that didn't have multiple masters over, but I've run into some
> problems that I'm still trying to sort out - difficulty locating
> masters, need some extra properties set on some of the tests, etc.
> 
> I was going to look into the lang .java tests as well, but noticed
> your recent wiki update had some of the .java tests listed as 'running
> under a test adapter'. If you're already working on getting the lang
> .java tests to run with the HarnessJavaTest adapter then I don't want
> to duplicate any effort there.
> 
> andrew

Re: Old harness '.java' test adapter for JUnit infrastructure.

[Andrew McIntyre <mc...@gmail.com>]

On 2/5/07, Daniel John Debrunner <dj...@apache.org> wrote:
>
> I also added a concrete implementation in JDBCHarnessJavaTest which runs
> '.java' tests from the jdbcapi package. It ran 17 jdbcapi '.java'
> without changes in embedded. (And a subset of those with client/server
> though that is not enabled in the committed version, I just need more
> investigation).

This is pretty nifty. I tried converting the .java tests in derbynet
that didn't have multiple masters over, but I've run into some
problems that I'm still trying to sort out - difficulty locating
masters, need some extra properties set on some of the tests, etc.

I was going to look into the lang .java tests as well, but noticed
your recent wiki update had some of the .java tests listed as 'running
under a test adapter'. If you're already working on getting the lang
.java tests to run with the HarnessJavaTest adapter then I don't want
to duplicate any effort there.

andrew

Re: Old harness '.java' test adapter for JUnit infrastructure.

[Daniel John Debrunner <dj...@apache.org>]

Sunitha Kambhampati wrote:
> 
> Thanks Dan for working on this.
> Daniel John Debrunner wrote:


Thanks for the feedback.


>> It will also centralized which environments a test runs in, one of the 
>> issues I'm seeing with the jdbcapi tests and client server is trying 
>> to figure out if a test is meant to run in c/s or not. Not clear from 
>> the old harness setup.
>>
> Can you elaborate more on this.   I am thinking you mean that once we 
> move to Junit harness (using the adapters or otherwise) , it would be 
> cleaner to know which environment it is run.  When we convert the test 
> using the adapter, we do have to do some work to find out if the test 
> should be run in client /server mode or not.   Is that right ?

Yes, I was talking about once the test is running in the Junit 
environment. Currently I find when looking at a harness test for 
conversion there is a significant effort involved in figuring out if 
test runs in embedded and client/server or just one "framework". Such 
confusion will go away once the test is converted as the decision is 
made in the class that runs the test.

Dan.

Re: Old harness '.java' test adapter for JUnit infrastructure.

[Sunitha Kambhampati <ks...@gmail.com>]

Thanks Dan for working on this. 

Daniel John Debrunner wrote:

> I'm looking at this approach as a quicker way to moving to a single 
> (JUnit based) test infrastructure.

+1.

> While canon based tests are not ideal, I think they are acceptable as 
> an interim step, especially if we manage to switch to a single 
> infrastructure. A good goal would be to have the 10.3 release (mid 
> year?) with a single infrastructure so that bug fixing and text cases 
> are easy to merge between trunk and 10.3. Also running the tests in 
> this way does not stop anyone from converting an individual test, it 
> makes it no harder. In fact it *might* make it easier, as a it might 
> be possible to incrementally convert tests and have them continue to 
> run with a master file.
>
> One benefit I see from this approach is that it may allow the 
> community to focus on converting tests that cannot be run using this 
> or other adapters, for example ones with multiple canon files, per jvm 
> version for example. Those tend to be the tests that occupy the most 
> time as the need to update multiple canons is costly.

 I agree with this.  In my own experience -- I have spent quite some 
time trying to update the testSecMec test :)  for different jcc 
versions/jdk versions that I think it might be greater benefit/worth for 
the community to focus on such multiple canon files test to be converted 
first.

> It will also centralized which environments a test runs in, one of the 
> issues I'm seeing with the jdbcapi tests and client server is trying 
> to figure out if a test is meant to run in c/s or not. Not clear from 
> the old harness setup.
>
Can you elaborate more on this.   I am thinking you mean that once we 
move to Junit harness (using the adapters or otherwise) , it would be 
cleaner to know which environment it is run.  When we convert the test 
using the adapter, we do have to do some work to find out if the test 
should be run in client /server mode or not.   Is that right ?

Thanks,
Sunitha.