testing crash recovery

[Rick Hillegas <ri...@oracle.com>]

I would like to understand what additional testing of the release 
candidate should be performed in order to satisfy the checklist item 
called "Recovery after crashing engine". Various recovery tests are run 
as part of our platform release tests, but it I don't see any tests for 
crash recovery.

We do have a crash recovery test for encrypted databases 
(CryptoCrashRecovery), but that only runs on debug jars. So it has only 
been exercised by nightly/continuousIntegration tests on the 10.11 
branch, not on the release candidate itself.

I have some questions for the Store experts:

1) Does "Recovery after crashing engine" mean running 
CryptoCrashRecovery on the debug distribution of 10.11.1.1?

2) Does it entail some other, formal tests?

3) Or is it meant to cover some other ad-hoc tests which are run by hand 
and are not checked in under subversion?

Thanks,
-Rick

Re: testing crash recovery

[Myrna van Lunteren <m....@gmail.com>]

Hi,

I cannot check on the 10.7 release docs right now, I get 'Service
Unavailable' on wiki access, but I suspect I last did anything in this
area. As far as I remember, it was a manual approach of loading up some
randomly created database, kick off some process (I might have started one
of the system tests for this), then crashing the jvm (killing the jvm), and
then using ij, check the database (likely I just did some simple operations
(create, delete, select, update), and probably consistency checker).

Unless someone knows another approach that was done under this heading...

Myrna


On Wed, Aug 20, 2014 at 11:14 AM, Rick Hillegas <ri...@oracle.com>
wrote:

> I would like to understand what additional testing of the release
> candidate should be performed in order to satisfy the checklist item called
> "Recovery after crashing engine". Various recovery tests are run as part of
> our platform release tests, but it I don't see any tests for crash recovery.
>
> We do have a crash recovery test for encrypted databases
> (CryptoCrashRecovery), but that only runs on debug jars. So it has only
> been exercised by nightly/continuousIntegration tests on the 10.11 branch,
> not on the release candidate itself.
>
> I have some questions for the Store experts:
>
> 1) Does "Recovery after crashing engine" mean running CryptoCrashRecovery
> on the debug distribution of 10.11.1.1?
>
> 2) Does it entail some other, formal tests?
>
> 3) Or is it meant to cover some other ad-hoc tests which are run by hand
> and are not checked in under subversion?
>
> Thanks,
> -Rick
>

Re: testing crash recovery

[mike matrigali <mi...@gmail.com>]

I would say the short answer is that it was a short ad-hoc test
task.

I don't think we ever had anything very formal.  In the beginning
there were some store tests that did "crash recovery", by simply
doing a set of sql tests in order in ij and not exiting cleanly.
As each subsequent test was run on same db we would at least run through the
code path at boot time that looks at the log and recovers it.
More recently some utilities were added to make it easy to convert/write
new tests that exercise this path.  For instance OCRecoveryTest uses
these utilities to do some work in a forked jvm, exit that JVM, and
then verify results on reboot.

Exiting the JVM tests most of what I would consider crash recovery, lets
say 99% or more.  But the unflushed data is still in the computer cache. 
  So especially now that it is relatively easy to write
these kinds of tests in junit, time spent here would be best. Extreme 
crash recovery would actually detach the disk before data could be 
sync'd and turn off the machine.  I think we may have done something 
like this in early days and found some obvious bugs, but
would not expect much, as it is hard to look for correct results other
than a sucessful boot.
On 8/20/2014 11:14 AM, Rick Hillegas wrote:
> I would like to understand what additional testing of the release
> candidate should be performed in order to satisfy the checklist item
> called "Recovery after crashing engine". Various recovery tests are run
> as part of our platform release tests, but it I don't see any tests for
> crash recovery.
>
> We do have a crash recovery test for encrypted databases
> (CryptoCrashRecovery), but that only runs on debug jars. So it has only
> been exercised by nightly/continuousIntegration tests on the 10.11
> branch, not on the release candidate itself.
>
> I have some questions for the Store experts:
>
> 1) Does "Recovery after crashing engine" mean running
> CryptoCrashRecovery on the debug distribution of 10.11.1.1?
>
> 2) Does it entail some other, formal tests?
>
> 3) Or is it meant to cover some other ad-hoc tests which are run by hand
> and are not checked in under subversion?
>
> Thanks,
> -Rick
> .
>