You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Kathey Marsden <km...@sbcglobal.net> on 2010/12/13 22:06:48 UTC
Mixed client sever and upgrade and compatibility testing with Junit
Tests
Normally I try to run the following tests using the Junit Tests on a
new release but have not done so for 10.7.
Since we only really have significant JUnit tests after 10.3 I usually
only test from 10.3 and higher. Analysis can be time consuming but
tends to find issues, and release notes needed. For example, for 10.5
we found, : DERBY-4122
<http://issues.apache.org/jira/browse/DERBY-4122>, DERBY-4127
<http://issues.apache.org/jira/browse/DERBY-4127>, DERBY-4128
<http://issues.apache.org/jira/browse/DERBY-4128>, DERBY-4135
<http://issues.apache.org/jira/browse/DERBY-4135>, plus missing release
notes for DERBY-2085 <http://issues.apache.org/jira/browse/DERBY-2085>
and DERBY-1062 <http://issues.apache.org/jira/browse/DERBY-1062>
Soft upgrade testing: Create database system/wombat with the old
version. Then using the old derbyTesting.jar and new product jars, run
suites.All. Then rerun with the old jars and make sure tests are clean.
Hard upgrade testing: Create database system/wombat with old version.
Connect with 10.5 with upgrade=true. Then run suites.all with 10.5
derbyTesting.jar and product jars.
Client/Server compatibility testing:
a) Put derbyclient.jar and derbyTesting.jar from the old version and the
other product jars from 10.5 in your classpath. Run tests and analyze
failures
b) Put derbyTesting.jar and server jars from the old version and
derbyclient.jar from the new version in your classpath. Run tests and
analyze failures.
Re: Mixed client sever and upgrade and compatibility testing with Junit Tests
Posted by "Dag H. Wanvik" <da...@oracle.com>.
Next installment:
> Hard upgrade testing: Create database system/wombat with old version
> (10.6.2.1). Connect with 10.7 with upgrade=true. Then run suites.all
> with 10.7 derbyTesting.jar and product jars.
This went fine, no errors.
Thanks,
Dag
Re: Mixed client sever and upgrade and compatibility testing with Junit Tests
Posted by "Dag H. Wanvik" <da...@oracle.com>.
Kathey Marsden <km...@sbcglobal.net> writes:
> Soft upgrade testing: Create database system/wombat with the old
> version. Then using the old derbyTesting.jar and new product jars,
> run suites.All.
> Then rerun with the old jars and make sure tests are
> clean.
I now did that second step above and it ran cleanly (I saved away
system/wombat from the soft upgrade run, deleted everything else,
reinstated system/wombat and ran with 10.6.2.1 jars).
Thanks,
Dag
Re: Mixed client sever and upgrade and compatibility testing with Junit Tests
Posted by "Dag H. Wanvik" <da...@ORACLE.COM>.
Kathey Marsden <km...@sbcglobal.net> writes:
> Thanks Dag. Roughly, how many tests failed do to the BOOLEAN issue?
> Does it block many tests?
> The soft upgrade testing is usually very useful for exposing
> compatibility bugs and issues that need release notes.
> It would be good to know to what extent that analysis was impacted and
> whether it will be practical to do soft upgrade testing in the future
> due to the noise generated by this issue.
Here is a summary of the soft upgrade errors.
28 errors, 99 failures, most to do with BOOLEAN.
Where error not specified below, there was some BOOLEAN related error,
cf the log at this location:
http://wiki.apache.org/db-derby/TenSevenOnePlatformTesting?action=AttachFile&do=view&target=softupgrade-run-derby.log.txt
The testGreekCharacters is presumably our improvement of client UTF-8
based database names. The LangScripts and TruncateTableTest errors
may merit some closer inspection, and possibly the
NetworkServerControlApiTest.
I'll be out until the New Year now, so feel free anyone to pick up
whetever is here ;-)
Thanks,
Dag
Summary - for details, refer to the above URL.
----------------------------------------------
org.apache.derbyTesting.functionTests.tests.derbynet.NetworkServerControlApiTest : 4 * AssertionFailedError
org.apache.derbyTesting.functionTests.tests.jdbcapi.AuthenticationTest: 4* AssertionFailedError (testGreekCharacters) "Connection should've been refused/failed"
org.apache.derbyTesting.functionTests.tests.jdbcapi.BlobClob4BlobTest: '6* AssertionFailedError (testNegativeTestDerby265?lob): "FAIL - can not access blob after implicit commit"
org.apache.derbyTesting.functionTests.tests.jdbcapi.DatabaseMetaDataTest
org.apache.derbyTesting.functionTests.tests.jdbcapi.DriverMgrAuthenticationTest: AssertionFailedError (testGreekCharacters): "Connection should've been refused/failed"
org.apache.derbyTesting.functionTests.tests.jdbcapi.JDBCHarnessJavaTest
org.apache.derbyTesting.functionTests.tests.jdbcapi.PoolDSAuthenticationTest: AssertionFailedError (testGreekCharacters): "Connection should've been refused/failed"
org.apache.derbyTesting.functionTests.tests.jdbcapi.SetTransactionIsolationTest: AssertionFailedError
org.apache.derbyTesting.functionTests.tests.jdbcapi.XADSAuthenticationTest: AssertionFailedError (testGreekCharacters): "Connection should've been refused/failed"
org.apache.derbyTesting.functionTests.tests.lang.CastingTest
org.apache.derbyTesting.functionTests.tests.lang.ErrorCodeTest: Unexpected row count expected:<161> but was:<164>
org.apache.derbyTesting.functionTests.tests.lang.LangScripts: logop, aggregate, cast, implicitConversions, orderby, schemas, valuesclause, LOB, unsupportedInInsaneBuilds: ComparisonFailure
org.apache.derbyTesting.functionTests.tests.lang.TriggerTest:
org.apache.derbyTesting.functionTests.tests.lang.TruncateTableTest: TABLE_THAT_DOES_NOT_EXIST
org.apache.derbyTesting.functionTests.tests.lang.UDTTest: vetDatatypeCount: expected:<21> but was:<20>
org.apache.derbyTesting.functionTests.tests.tools.SysinfoCPCheckTest: db2driver option removed
org.apache.derbyTesting.functionTests.tests.tools.ToolScripts
Re: Mixed client sever and upgrade and compatibility testing with
Junit Tests
Posted by Kathey Marsden <km...@sbcglobal.net>.
On 12/16/2010 5:50 AM, Dag H. Wanvik wrote:
>> https://issues.apache.org/jira/browse/DERBY-4946
> Right, I did not think of that as a bug initially, but reconsidering it
> I agree that is probably wrong behavior on Derby's part: metadata should
> reflect the present capabilities (in casu the soft upgraded state).
>
Thanks Dag. Roughly, how many tests failed do to the BOOLEAN issue?
Does it block many tests?
The soft upgrade testing is usually very useful for exposing
compatibility bugs and issues that need release notes.
It would be good to know to what extent that analysis was impacted and
whether it will be practical to do soft upgrade testing in the future
due to the noise generated by this issue.
Thanks
Kathey
Re: Mixed client sever and upgrade and compatibility testing with Junit Tests
Posted by "Dag H. Wanvik" <da...@oracle.com>.
Kathey Marsden <km...@sbcglobal.net> writes:
> I have not had a chance to look at the tests but I think the tests are
> clever but Derby DatabaseMetaData is not clever enough to know it
> should not return BOOLEAN in soft upgrade mode, so the tests are
> trying to use it because they think it is available when it is not.
> The derbyTesting.jar from 10.6 should run clean on a soft upgraded
> database using 10.7 product jars, just like any 10.6 application. Of
> course the new functionality in the 10.7 derbyTesting.jar should not
> work and so sensible that it has failures.
>
> I did verify the DatabaseMetaData getTypeInfo() bug and filed:
> https://issues.apache.org/jira/browse/DERBY-4946
Right, I did not think of that as a bug initially, but reconsidering it
I agree that is probably wrong behavior on Derby's part: metadata should
reflect the present capabilities (in casu the soft upgraded state).
Thanks,
Dag
Re: Mixed client sever and upgrade and compatibility testing with
Junit Tests
Posted by Kathey Marsden <km...@sbcglobal.net>.
On 12/15/2010 4:17 PM, Dag H. Wanvik wrote:
> Hi Kathey,
>
> thanks for helping me through this.
Thank you for running the tests!
> Right, well it seems the tests are not that clever generally. As I mention, using the 10.6 version of derbyTesting.jar saw more errors than
> the 10.7 one (in spite of the default db not having been hard upgraded),
> since the 10.7 version of derbyTesting is prepared to see
> BOOLEAN in metadata.
>
I have not had a chance to look at the tests but I think the tests are
clever but Derby DatabaseMetaData is not clever enough to know it should
not return BOOLEAN in soft upgrade mode, so the tests are trying to use
it because they think it is available when it is not. The
derbyTesting.jar from 10.6 should run clean on a soft upgraded database
using 10.7 product jars, just like any 10.6 application. Of course the
new functionality in the 10.7 derbyTesting.jar should not work and so
sensible that it has failures.
I did verify the DatabaseMetaData getTypeInfo() bug and filed:
https://issues.apache.org/jira/browse/DERBY-4946
Kathey
Re: Mixed client sever and upgrade and compatibility testing with Junit Tests
Posted by "Dag H. Wanvik" <da...@oracle.com>.
Hi Kathey,
thanks for helping me through this.
Kathey Marsden <km...@sbcglobal.net> writes:
> I have been thinking about your BOOLEAN failures and wonder if they
> should occur in soft upgrade mode. I think the DatabaseMetaData may
> be saying the type is available but it is not. I think it would not
> be unreasonable for an application (like the tests) to check if the
> data type is available and have conditional behavior based on
> that. Now that I type this it seems like this might be an old
> conversation, but can't fully remember.
Right, well it seems the tests are not that clever generally. As I
mention, using the 10.6 version of derbyTesting.jar saw more errors than
the 10.7 one (in spite of the default db not having been hard upgraded),
since the 10.7 version of derbyTesting is prepared to see
BOOLEAN in metadata.
Thanks,
Dag
Re: Mixed client sever and upgrade and compatibility testing with
Junit Tests
Posted by Kathey Marsden <km...@sbcglobal.net>.
On 12/14/2010 2:18 PM, Dag H. Wanvik wrote:
>
> Well,
> thats good, no new errors in soft upgrade more then!
>
I have been thinking about your BOOLEAN failures and wonder if they
should occur in soft upgrade mode. I think the DatabaseMetaData may be
saying the type is available but it is not. I think it would not be
unreasonable for an application (like the tests) to check if the data
type is available and have conditional behavior based on that. Now that
I type this it seems like this might be an old conversation, but can't
fully remember.
Thanks
Kathey
Re: Mixed client sever and upgrade and compatibility testing with Junit Tests
Posted by "Dag H. Wanvik" <da...@oracle.com>.
Knut Anders Hatlen <kn...@oracle.com> writes:
>> testBootingAnAlreadyBootedDatabase(org.apache.derbyTesting.functionTests.tests.store.ClassLoaderBootTest)java.lang.ExceptionInInitializerError
> [...]
>> Caused by: java.lang.SecurityException: sealing violation: package
>> org.apache.derby.iapi.services.sanity is sealed
>
> Could you be seeing DERBY-4669?
Indeed, thanks. I did have derbyTesting.jar before derby.jar. Well,
thats good, no new errors in soft upgrade more then!
Dag
Re: Mixed client sever and upgrade and compatibility testing with Junit Tests
Posted by Knut Anders Hatlen <kn...@oracle.com>.
dag.wanvik@oracle.com (Dag H. Wanvik) writes:
> Kathey Marsden <km...@sbcglobal.net> writes:
>
>> Soft upgrade testing: Create database system/wombat with the old
>> version. Then using the old derbyTesting.jar and new product jars,
>> run suites.All.
>
> I did this and saw two kinds of errors. The first is the BOOLEAN type upgrade
> requirement, expected, so OK.
>
> The second error I see is this (I ran with lib-debug). Not sure if this
> is expected or not?
>
> Dag
>
> 74) testBootingAnAlreadyBootedDatabase(org.apache.derbyTesting.functionTests.tests.store.ClassLoaderBootTest)java.lang.ExceptionInInitializerError
[...]
> Caused by: java.lang.SecurityException: sealing violation: package org.apache.derby.iapi.services.sanity is sealed
Could you be seeing DERBY-4669?
--
Knut Anders
Re: Mixed client sever and upgrade and compatibility testing with Junit Tests
Posted by "Dag H. Wanvik" <da...@oracle.com>.
dag.wanvik@oracle.com (Dag H. Wanvik) writes:
> Kathey Marsden <km...@sbcglobal.net> writes:
>
>> Soft upgrade testing: Create database system/wombat with the old
>> version. Then using the old derbyTesting.jar and new product jars,
>> run suites.All.
>
> I did this and saw two kinds of errors. The first is the BOOLEAN type upgrade
> requirement, expected, so OK.
To avoid confusion, I used the NEW derbyTesting.jar (hence the BOOLEAN errors).
Dag
Re: Mixed client sever and upgrade and compatibility testing with Junit Tests
Posted by "Dag H. Wanvik" <da...@oracle.com>.
I ran suitesAll in soft upgrade mode again, this time with insane jars
and the testing jar from 10.6.2.1.
This time I got more errors than when runnign with testing jar from
10.7.1.1, since many tests had been changed on account for the fact that
BOOLEAN is now supported. Other errors were also all explainable from
new behavior (tests had changed with the db software). The sealing error
was gone.
I am now running the 10.6 system/wombat with 10.6 jars to see if the
stint as a soft upgraded database left any side effects on the database.
Dag
Re: Mixed client sever and upgrade and compatibility testing with
Junit Tests
Posted by Kathey Marsden <km...@sbcglobal.net>.
On 12/14/2010 7:59 AM, Dag H. Wanvik wrote:
> Kathey Marsden<km...@sbcglobal.net> writes:
>
>> Soft upgrade testing: Create database system/wombat with the old
>> version. Then using the old derbyTesting.jar and new product jars,
>> run suites.All.
> I did this and saw two kinds of errors. The first is the BOOLEAN type upgrade
> requirement, expected, so OK.
>
> The second error I see is this (I ran with lib-debug). Not sure if this
> is expected or not?
>
Hi Dag,
Thank you for pursuing this testing. I have always run with insane jars
and could understand problems related to duplicate classes in the sane
jars from different releases because of the shared sanity classes, but
since for this particulary test, all the product jars are from the same
release and derbyTesting.jar doesn't have the sanity classes , I am
perplexed why you get the error. It would be interesting to understand
why it happens, but it would be equally fine to rerun with insane jars
where these classes are not involved.
Kathey
> Dagd
>
> 74) testBootingAnAlreadyBootedDatabase(org.apache.derbyTesting.functionTests.tests.store.ClassLoaderBootTest)java.lang.ExceptionInInitializerError
> at org.apache.derby.jdbc.EmbeddedDataSource.findDriver(EmbeddedDataSource.java:500)
> at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(EmbeddedDataSource.java:479)
> at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(EmbeddedDataSource.java:423)
> at org.apache.derbyTesting.functionTests.tests.store.ClassLoaderBootTest.testBootingAnAlreadyBootedDatabase(ClassLoaderBootTest.java:178)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:109)
> at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
> at junit.extensions.TestSetup.run(TestSetup.java:25)
> at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
> at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
> at junit.extensions.TestSetup.run(TestSetup.java:25)
> at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
> at junit.extensions.TestSetup.run(TestSetup.java:25)
> Caused by: java.lang.SecurityException: sealing violation: package org.apache.derby.iapi.services.sanity is sealed
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:227)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at org.apache.derbyTesting.functionTests.tests.store.ClassLoaderBootTest$DerbyURLClassLoader.loadClass(ClassLoaderBootTest.java:293)
> at org.apache.derbyTesting.functionTests.tests.store.ClassLoaderBootTest$DerbyURLClassLoader.loadClass(ClassLoaderBootTest.java:303)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at org.apache.derby.iapi.services.monitor.Monitor.startSystemModule(Monitor.java:369)
> at org.apache.derby.impl.services.monitor.BaseMonitor.runWithState(BaseMonitor.java:344)
> at org.apache.derby.impl.services.monitor.FileMonitor.<init>(FileMonitor.java:60)
> at org.apache.derby.iapi.services.monitor.Monitor.startMonitor(Monitor.java:289)
> at org.apache.derby.iapi.jdbc.JDBCBoot.boot(JDBCBoot.java:69)
> at org.apache.derby.jdbc.EmbeddedDriver.boot(EmbeddedDriver.java:199)
> at org.apache.derby.jdbc.EmbeddedDriver.<clinit>(EmbeddedDriver.java:96)
> ... 39 more
>
>
Re: Mixed client sever and upgrade and compatibility testing with Junit Tests
Posted by "Dag H. Wanvik" <da...@oracle.com>.
Kathey Marsden <km...@sbcglobal.net> writes:
> Soft upgrade testing: Create database system/wombat with the old
> version. Then using the old derbyTesting.jar and new product jars,
> run suites.All.
I did this and saw two kinds of errors. The first is the BOOLEAN type upgrade
requirement, expected, so OK.
The second error I see is this (I ran with lib-debug). Not sure if this
is expected or not?
Dag
74) testBootingAnAlreadyBootedDatabase(org.apache.derbyTesting.functionTests.tests.store.ClassLoaderBootTest)java.lang.ExceptionInInitializerError
at org.apache.derby.jdbc.EmbeddedDataSource.findDriver(EmbeddedDataSource.java:500)
at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(EmbeddedDataSource.java:479)
at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(EmbeddedDataSource.java:423)
at org.apache.derbyTesting.functionTests.tests.store.ClassLoaderBootTest.testBootingAnAlreadyBootedDatabase(ClassLoaderBootTest.java:178)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:109)
at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
at junit.extensions.TestSetup.run(TestSetup.java:25)
at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
at junit.extensions.TestSetup.run(TestSetup.java:25)
at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
at junit.extensions.TestSetup.run(TestSetup.java:25)
Caused by: java.lang.SecurityException: sealing violation: package org.apache.derby.iapi.services.sanity is sealed
at java.net.URLClassLoader.defineClass(URLClassLoader.java:227)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at org.apache.derbyTesting.functionTests.tests.store.ClassLoaderBootTest$DerbyURLClassLoader.loadClass(ClassLoaderBootTest.java:293)
at org.apache.derbyTesting.functionTests.tests.store.ClassLoaderBootTest$DerbyURLClassLoader.loadClass(ClassLoaderBootTest.java:303)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at org.apache.derby.iapi.services.monitor.Monitor.startSystemModule(Monitor.java:369)
at org.apache.derby.impl.services.monitor.BaseMonitor.runWithState(BaseMonitor.java:344)
at org.apache.derby.impl.services.monitor.FileMonitor.<init>(FileMonitor.java:60)
at org.apache.derby.iapi.services.monitor.Monitor.startMonitor(Monitor.java:289)
at org.apache.derby.iapi.jdbc.JDBCBoot.boot(JDBCBoot.java:69)
at org.apache.derby.jdbc.EmbeddedDriver.boot(EmbeddedDriver.java:199)
at org.apache.derby.jdbc.EmbeddedDriver.<clinit>(EmbeddedDriver.java:96)
... 39 more