Database corruption on crash during table creation

[David Van Couvering <da...@vancouvering.com>]

Hi, all.  I seem to have a repeatable issue where my database is getting
corrupted, and I can only assume it's happening when the table is getting
created, because I know Derby is great on handling crashes during normal
operation.

First of all - is it true that a Derby database can get corrupted if the VM
goes down in the middle of a DML like CREATE TABLE?

What I'm seeing is that upon restart, when I try to access the database, I
get the error "unable to create database FOO: the directory already
exists."  The directory contains what appears to be a Derby database, but
apparently not enough of one.

What I'm hoping for, is there a "forceCreate=true" kind of option when
opening a database that forces the creation of the database in the given
directory even if something is already there?

Otherwise it appears my only option is to detect this specific error
('directory already exists'), and then blow away what's there and try
again.  That's a chunk of work I'd rather not do, so I'm hoping you've
already addressed it (ahem :)).

Thanks,

David

-- 
David W. Van Couvering

http://www.linkedin.com/in/davidvc
http://davidvancouvering.blogspot.com
http://twitter.com/dcouvering

Re: Database corruption on crash during table creation

[David Van Couvering <da...@vancouvering.com>]

On Mon, Jul 5, 2010 at 11:52 PM, Kristian Waagan <Kr...@sun.com>wrote:

> On 06.07.10 08:23, David Van Couvering wrote:
>
>> Hi, all.  I seem to have a repeatable issue where my database is getting
>> corrupted, and I can only assume it's happening when the table is getting
>> created, because I know Derby is great on handling crashes during normal
>> operation.
>>
>> First of all - is it true that a Derby database can get corrupted if the
>> VM goes down in the middle of a DML like CREATE TABLE?
>>
>> What I'm seeing is that upon restart, when I try to access the database, I
>> get the error "unable to create database FOO: the directory already exists."
>>  The directory contains what appears to be a Derby database, but apparently
>> not enough of one.
>>
>
> Hi David,
>
> It would be great if you can create a Jira and attach the contents of the
> database directory (or maybe just the output from find). Including a
> runnable repro would be even better :)
>

I'll see what I can do - the error was reported by another developer and I
don't know if he kept his environment around.


> Also:
>  o Can the issue be reproduced using ij only?
>

No, this happens using JDBC


>  o Is this reproducible when only a single process is accessing/modifying
> the database?
>
>
Yes, this is single process, even single-threaded (I am funneling all Derby
requests to a given database through a single thread).


> I believe there has been a few reports of the error message you mentioned,
> but so far no repro has been available.
>
> Having a forceCreate option requires some thought, it would be unfortunate
> if Derby deleted (i.e. misuse or exploit) a bunch of user/system files. This
> wouldn't be such a big deal if everyone tightened the Java security policy
> for Derby, but I'm not sure that's the case...
>
> Yes, I can see this being an issue...

If you have clarity on what you want here, e.g. do you *want* a forceCreate,
I might consider doing it in the engine.  Of course I'd have to go through
legal to even be allowed to contribute as a Symantec employee :)

David

>
> Regards,
> --
> Kristian
>
>
>
>> What I'm hoping for, is there a "forceCreate=true" kind of option when
>> opening a database that forces the creation of the database in the given
>> directory even if something is already there?
>>
>> Otherwise it appears my only option is to detect this specific error
>> ('directory already exists'), and then blow away what's there and try again.
>>  That's a chunk of work I'd rather not do, so I'm hoping you've already
>> addressed it (ahem :)).
>>
>> Thanks,
>>
>> David
>>
>> --
>> David W. Van Couvering
>>
>> http://www.linkedin.com/in/davidvc
>> http://davidvancouvering.blogspot.com
>> http://twitter.com/dcouvering
>>
>
>


-- 
David W. Van Couvering

http://www.linkedin.com/in/davidvc
http://davidvancouvering.blogspot.com
http://twitter.com/dcouvering

Re: Database corruption on crash during table creation

[Kristian Waagan <Kr...@Sun.COM>]

On 06.07.10 08:23, David Van Couvering wrote:
> Hi, all.  I seem to have a repeatable issue where my database is 
> getting corrupted, and I can only assume it's happening when the table 
> is getting created, because I know Derby is great on handling crashes 
> during normal operation.
>
> First of all - is it true that a Derby database can get corrupted if 
> the VM goes down in the middle of a DML like CREATE TABLE?
>
> What I'm seeing is that upon restart, when I try to access the 
> database, I get the error "unable to create database FOO: the 
> directory already exists."  The directory contains what appears to be 
> a Derby database, but apparently not enough of one.

Hi David,

It would be great if you can create a Jira and attach the contents of 
the database directory (or maybe just the output from find). Including a 
runnable repro would be even better :)
Also:
   o Can the issue be reproduced using ij only?
   o Is this reproducible when only a single process is 
accessing/modifying the database?

I believe there has been a few reports of the error message you 
mentioned, but so far no repro has been available.

Having a forceCreate option requires some thought, it would be 
unfortunate if Derby deleted (i.e. misuse or exploit) a bunch of 
user/system files. This wouldn't be such a big deal if everyone 
tightened the Java security policy for Derby, but I'm not sure that's 
the case...


Regards,
-- 
Kristian

>
> What I'm hoping for, is there a "forceCreate=true" kind of option when 
> opening a database that forces the creation of the database in the 
> given directory even if something is already there?
>
> Otherwise it appears my only option is to detect this specific error 
> ('directory already exists'), and then blow away what's there and try 
> again.  That's a chunk of work I'd rather not do, so I'm hoping you've 
> already addressed it (ahem :)).
>
> Thanks,
>
> David
>
> -- 
> David W. Van Couvering
>
> http://www.linkedin.com/in/davidvc
> http://davidvancouvering.blogspot.com
> http://twitter.com/dcouvering

Re: Database corruption on crash during table creation

[Knut Anders Hatlen <Kn...@Sun.COM>]

On 07/ 6/10 08:23 AM, David Van Couvering wrote:
> Hi, all.  I seem to have a repeatable issue where my database is
> getting corrupted, and I can only assume it's happening when the table
> is getting created, because I know Derby is great on handling crashes
> during normal operation.
>
> First of all - is it true that a Derby database can get corrupted if
> the VM goes down in the middle of a DML like CREATE TABLE?

I hope not... :)

> What I'm seeing is that upon restart, when I try to access the
> database, I get the error "unable to create database FOO: the
> directory already exists."  The directory contains what appears to be
> a Derby database, but apparently not enough of one.

There is a report about a similar problem, probably caused by the
application being killed while the database was being created. See
DERBY-4589 and the possibly related DERBY-4098.

> What I'm hoping for, is there a "forceCreate=true" kind of option when
> opening a database that forces the creation of the database in the
> given directory even if something is already there? 
>
> Otherwise it appears my only option is to detect this specific error
> ('directory already exists'), and then blow away what's there and try
> again.  That's a chunk of work I'd rather not do, so I'm hoping you've
> already addressed it (ahem :)).

I'm afraid that this is a chunk of work that you'll have to do for now,
if this is an error you see frequently. May I suggest that when you go
ahead implementing it, you make the changes in the Derby engine and not
in your application, so that everyone benefits from it? :)

Thanks,

-- 
Knut Anders