10.11.1 checklist items

[Rick Hillegas <ri...@oracle.com>]

We've made good progress on tackling the release checklist items at 
http://wiki.apache.org/db-derby/TenElevenOneChecklist. There are still 
some outstanding tasks, though. If you plan to pick up one of the 
outstanding tasks, could you update the wiki accordingly? Would be good 
to decide what we want to do about the unassigned tasks.

Thanks,
-Rick

upgrade/compatibility testing done in the past.

[mike matrigali <mi...@gmail.com>]

>> Last time Kathey ran the upgrade/compatibility testing (run with an
>> older release's derbyTesting.jar with newer derby*.jar) she ran into a
>> lot of test failures because of expected changes, and I think no real
>> failures. I  don't intend to make the effort at this time, I do not
>> have the time, and I doubt Kathey has either.
> Can you help me understand what this test entails beyond what's in the
> upgrade and compatibility tests we run every night?

the upgrade and compatibility tests are very small and test the things
that someone has thought of and added specific tests for.  The goal of 
this upgrade/compatibility testing was to catch
compatibility problems of existing applications with new software.

I believe the intent is to use as large a portion of the junit tests 
from some past release and use it as a model of expected behavior in
that release.  Then you run those exact tests against a new software
release, with the assumption that diffs are possible upward 
compatibility problems.    I believe it was most successful in calling 
out possible issues that should be noted for release notes.  I do not
know if the db used was old db in soft upgrade mode, an old db hard
upgraded to current, or a new db.  All probably are interesting, but
would lean toward old db in soft upgrade mode being most interesting
as a model for an old application looking to upgrade the db software
but get same behavior.

The problem was that this testing in the past has generated a lot of
noise to bug/release note ratio.  It was by hand and don't think much
was done to make future testing easier.  I think it only in the past
was done starting from either most recent or 2 release back to current
release.  I believe it was on the order of weeks to interpret part
time.

I think knut's recent run tests that upgraded db's work with new 
software as the new software expects.

I think this other testing looks to see if new software acts as old
applications expect.

Neither does a complete job, but I think both are a good leverage of the
existing test framework to test different code paths for compatibility.

Re: 10.11.1 checklist items

[Rick Hillegas <ri...@oracle.com>]

Thanks, Myrna. Some comments inline...

On 8/18/14 10:37 AM, Myrna van Lunteren wrote:
> Hi Rick,
>
> Thanks for the reminder...
>
> I ran large data tests (and documented them on the platformtesting 
> page), so I added a checkmark for that; I also kicked off one long 
> running test, but will give final evaluation at the end of this week, 
> so I left that one angry.
Thanks!
>
> I do not (no longer) have the set-up/machines for IPV4/IVP6 testing 
> and apart from a problem many years ago when we first came out with 
> NetworkServer I don't think we've ever found problems with this testing.
I'd vote for removing this test from our checklist. I don't see much 
value in re-creating a lab for a test which never found any problems.
>
> Similarly, I think no recent cycle has done manual testing of recovery 
> after crashing engine; apart from whatever is in the functionTests 
> tests. I don't plan on doing it this time.
Right. We don't seem to have run this test for any feature release since 
at least 10.7.1. Does anyone recall what this test entails? Is there a 
systematic script for this test?
>
> Last time Kathey ran the upgrade/compatibility testing (run with an 
> older release's derbyTesting.jar with newer derby*.jar) she ran into a 
> lot of test failures because of expected changes, and I think no real 
> failures. I  don't intend to make the effort at this time, I do not 
> have the time, and I doubt Kathey has either.
Can you help me understand what this test entails beyond what's in the 
upgrade and compatibility tests we run every night?
>
> Perhaps we need an icon for things we don't intend to do for a 
> specific release? Or just remove them off the checklist for this release?
+1 to removing the ipv6/4 test from the checklist.

Thanks,
-Rick
>
> Myrna
>
>
> On Mon, Aug 18, 2014 at 9:15 AM, Rick Hillegas 
> <rick.hillegas@oracle.com <ma...@oracle.com>> wrote:
>
>     We've made good progress on tackling the release checklist items
>     at http://wiki.apache.org/db-derby/TenElevenOneChecklist. There
>     are still some outstanding tasks, though. If you plan to pick up
>     one of the outstanding tasks, could you update the wiki
>     accordingly? Would be good to decide what we want to do about the
>     unassigned tasks.
>
>     Thanks,
>     -Rick
>
>

Re: 10.11.1 checklist items

[Myrna van Lunteren <m....@gmail.com>]

Hi Rick,

Thanks for the reminder...

I ran large data tests (and documented them on the platformtesting page),
so I added a checkmark for that; I also kicked off one long running test,
but will give final evaluation at the end of this week, so I left that one
angry.

I do not (no longer) have the set-up/machines for IPV4/IVP6 testing and
apart from a problem many years ago when we first came out with
NetworkServer I don't think we've ever found problems with this testing.

Similarly, I think no recent cycle has done manual testing of recovery
after crashing engine; apart from whatever is in the functionTests tests. I
don't plan on doing it this time.

Last time Kathey ran the upgrade/compatibility testing (run with an older
release's derbyTesting.jar with newer derby*.jar) she ran into a lot of
test failures because of expected changes, and I think no real failures. I
don't intend to make the effort at this time, I do not have the time, and I
doubt Kathey has either.

Perhaps we need an icon for things we don't intend to do for a specific
release? Or just remove them off the checklist for this release?

Myrna


On Mon, Aug 18, 2014 at 9:15 AM, Rick Hillegas <ri...@oracle.com>
wrote:

> We've made good progress on tackling the release checklist items at
> http://wiki.apache.org/db-derby/TenElevenOneChecklist. There are still
> some outstanding tasks, though. If you plan to pick up one of the
> outstanding tasks, could you update the wiki accordingly? Would be good to
> decide what we want to do about the unassigned tasks.
>
> Thanks,
> -Rick
>