New in-memory back end features suggested, rev 1 (was: Re: Derby in-memory back end - where to go next?)

[Kristian Waagan <Kr...@Sun.COM>]

Kristian Waagan wrote:
> Hello,
>
> In Derby 10.5 an in-memory back end, or storage engine, was included. 
> It stores all the data in main memory, with the exception of 
> derby.log. If this is news to you, and you want a quick intro to it, 
> see [1] and [2].
>
> I'm trying to gather some feedback on whether the current 
> implementation is found acceptable, or if there are additional 
> features people would like to see. I expect some wishes to emerge, and 
> I plan to record these on the wiki page [1]. The page can then be used 
> to guide further work in this area.

Hello all,

Thanks to everyone that chimed in on this thread earlier.
I have now updated the wiki page with some more information. It is very 
crude, I just wanted to get something out before I leave for two weeks 
of vacation.
Feel free to read through the section "Future features" and give your 
feedback on this thread.
Does the suggested features look reasonable?
Has something really important (for you) been left out?

When I get back, I think I'll start working on the new delete mechanism 
and maybe the monitoring. Besides from that, documentation must be added.
If time permits, or somebody else wants to put down some effort, maybe 
one or more of the remaining features can be added.
In addition to the wiki page, I suppose a separate Jira will be created 
for each feature. More detailed specifications will be attached to the 
individual Jiras.

The wiki page is at: http://wiki.apache.org/db-derby/InMemoryBackEndPrimer


Thanks,
-- 
Kristian

[ snip ]

Re: New in-memory back end features suggested, rev 1

[Rick Hillegas <Ri...@Sun.COM>]

Kristian Waagan wrote:
> Kristian Waagan wrote:
> ...
> The wiki page is at: 
> http://wiki.apache.org/db-derby/InMemoryBackEndPrimer
>
Thanks for pulling this spec together, Kristian. Some comments:

Proper Delete Mechanism

I agree that option (1) looks fine: the mechanism for deleting a 
database then looks like the mechanisms for creating and halting 
databases. I think it's ok to restrict this privilege to the DBO. For 
most use-cases that will probably be fine. There might be an oddball 
case of a System Administrator needing to delete an in-memory database 
without bouncing the VM. This sounds like something we could add when we 
add system privileges.


Automatic database persistence on JVM shutdown

I think we should say something about how the user experiences this feature:

1) It makes sense to me that snapshot-on-exit is an attribute of a 
database which is set when the database is initially booted. Can you 
change the setting and if so, who can do this and how?

2) It would be nice to say a few words about how you reboot one of these 
snapshots. Is this done via the restoreFrom connection attribute? Do you 
need to specify snapshot-on-exit when you reboot one of these in-memory 
databases or is it assumed that if you reboot one of these in-memory 
databases then you want the original exit behavior to continue in force?


Automatic database persistence on database shutdown

Similar questions about this feature. In addition, it becomes possible 
for a database to have two states: the in-memory version and the 
snapshot created when the database was shutdown.

3) If you reboot the database, do you get the version that's still in 
memory? In the meantime, someone could have booted the snapshot as a 
persistent database, fiddled with it, and re-written the snapshot.

4) Can you reboot the database with a restoreFrom setting? Does this 
override the version in memory or does this raise an error?

I think there's definitely an opportunity to share code between 
snapshot-on-exit and snapshot-on-shutdown. But they look like separate 
features to me. My gut feeling is that snapshot-on-shutdown is more 
important than snapshot-on-exit and has fewer problems. I think that 
most people who need to persist an in-memory database could write their 
application in such a way that they could gracefully park the database 
before bringing down the vm. The more I think about it, the more that 
shutdown-on-exit seems to refer to an undefined state of the database 
and cannot be guaranteed to work properly.

Thanks,
-Rick