Re: Startup time for Derby

[Andrew McIntyre <mc...@gmail.com>]

On 7/27/06, David Van Couvering <Da...@sun.com> wrote:
> I am here at OSCON and a user of Derby was complaining rather
> energetically to me at the cost of startup time for Derby.  He said this
> is a real problem for running unit tests, as this is compounded by
> running multiple tests, each one starting up a new database.

This is something that has come up in the past and I just had a random
thought regarding this.

The structure of a new, properly shutdown database has to be static
for a specific version of Derby, right? (ok, almost, see locale note
below)  So, what if we packaged a new database in a jar file,
derbynewdb.jar, along with its exact version info, and when a
connection URL with the create attribute is passed, the engine checks
that newdb.jar is available and that the version of the newdb jar file
in the classpath is identical. If so, the engine expands the new,
empty database from the jar into the local filesystem instead of
creating all of the system tables from scratch. If the version
mismatches or the newdb.jar isn't there (or is incomplete?), we create
a new db from scratch the regular way. A new db compressed into a jar
file is only 54K for 10.2, not so bad for (re)deployment.

Someone has to have thought of this before, so I'm wondering what the
issues are here. One problem that I can think of off-hand would be
that a database is created with the JVM's current locale as the
database locale. So it would be necessary to have a way to update the
database locale after the expansion of the new db from the jar file,
maybe a system procedure that needs to be called after the expansion
of the new database from the jar. Another is what if there is a
failure while uncompressing the db, what do you do on an attempt to
reconnect? There would need to be a way to verify that you're working
with a full deck of cards, as it were, but such a verification still
seems like it would be less heavy-weight than creating a new db from
scratch.

Anyway, the idea passed through my head, so I thought I'd throw it out there...

andrew

Re: Startup time for Derby

[Daniel John Debrunner <dj...@apache.org>]

Andrew McIntyre wrote:

> On 7/27/06, David Van Couvering <Da...@sun.com> wrote:
> 
>> I am here at OSCON and a user of Derby was complaining rather
>> energetically to me at the cost of startup time for Derby.  He said this
>> is a real problem for running unit tests, as this is compounded by
>> running multiple tests, each one starting up a new database.
> 
> 
> This is something that has come up in the past and I just had a random
> thought regarding this.
> 
> The structure of a new, properly shutdown database has to be static
> for a specific version of Derby, right? (ok, almost, see locale note
> below)  So, what if we packaged a new database in a jar file,
> derbynewdb.jar, along with its exact version info, and when a
> connection URL with the create attribute is passed, the engine checks
> that newdb.jar is available and that the version of the newdb jar file
> in the classpath is identical. If so, the engine expands the new,
> empty database from the jar into the local filesystem instead of
> creating all of the system tables from scratch. If the version
> mismatches or the newdb.jar isn't there (or is incomplete?), we create
> a new db from scratch the regular way. A new db compressed into a jar
> file is only 54K for 10.2, not so bad for (re)deployment.
> 
> Someone has to have thought of this before, so I'm wondering what the
> issues are here. One problem that I can think of off-hand would be
> that a database is created with the JVM's current locale as the
> database locale. So it would be necessary to have a way to update the
> database locale after the expansion of the new db from the jar file,
> maybe a system procedure that needs to be called after the expansion
> of the new database from the jar. Another is what if there is a
> failure while uncompressing the db, what do you do on an attempt to
> reconnect? There would need to be a way to verify that you're working
> with a full deck of cards, as it were, but such a verification still
> seems like it would be less heavy-weight than creating a new db from
> scratch.
> 
> Anyway, the idea passed through my head, so I thought I'd throw it out
> there...

I think a useful first step is the ability to create a database from
another database, regardless of its stored form. Restore does this
today, but is limited to reading files. If that was expanded to handle
other stored forms (through the StorageFactory) then the base underlying
technology would exist for what you want, and is more generalized to
allow applications to copy pre-populated (schema and data) databases.

However, to have a existing empty template database that works in all
situations may be more effort than it worth. Apart from the locale
issue, the owner of database objects becomes important with SQL
authorization which will become the default at some point in the future.
I'm concerned that more and more code will have to exist to patch up
this template database. The current Derby scheme of easy to create new
database was in part reaction to my experience with another database
system which used the template scheme.

I think one issue is that the switch to a log that syncs to disk on
every write has slowed database creation time and has slowed performance
in other areas (namely insertion of large BLOBs/CLOBs). Previously the
log was marked as don't sync during db create, now that flag has no
effect and so the database creation has to wait for every log write. I
think this is something that should be addressed.

Dan.

Re: Startup time for Derby

[David Van Couvering <Da...@Sun.COM>]

Oooh, I like this idea.  It would also be good to show people how to set 
up their own "model" databases with the tables they want, so they don't 
have to re-create all their own tables for each individual test.

The other people want for unit testing, of course, is a database that 
runs solely in memory.  It's a reason so many people love testing with HSQL.

David

Andrew McIntyre wrote:
> On 7/27/06, David Van Couvering <Da...@sun.com> wrote:
>> I am here at OSCON and a user of Derby was complaining rather
>> energetically to me at the cost of startup time for Derby.  He said this
>> is a real problem for running unit tests, as this is compounded by
>> running multiple tests, each one starting up a new database.
> 
> This is something that has come up in the past and I just had a random
> thought regarding this.
> 
> The structure of a new, properly shutdown database has to be static
> for a specific version of Derby, right? (ok, almost, see locale note
> below)  So, what if we packaged a new database in a jar file,
> derbynewdb.jar, along with its exact version info, and when a
> connection URL with the create attribute is passed, the engine checks
> that newdb.jar is available and that the version of the newdb jar file
> in the classpath is identical. If so, the engine expands the new,
> empty database from the jar into the local filesystem instead of
> creating all of the system tables from scratch. If the version
> mismatches or the newdb.jar isn't there (or is incomplete?), we create
> a new db from scratch the regular way. A new db compressed into a jar
> file is only 54K for 10.2, not so bad for (re)deployment.
> 
> Someone has to have thought of this before, so I'm wondering what the
> issues are here. One problem that I can think of off-hand would be
> that a database is created with the JVM's current locale as the
> database locale. So it would be necessary to have a way to update the
> database locale after the expansion of the new db from the jar file,
> maybe a system procedure that needs to be called after the expansion
> of the new database from the jar. Another is what if there is a
> failure while uncompressing the db, what do you do on an attempt to
> reconnect? There would need to be a way to verify that you're working
> with a full deck of cards, as it were, but such a verification still
> seems like it would be less heavy-weight than creating a new db from
> scratch.
> 
> Anyway, the idea passed through my head, so I thought I'd throw it out 
> there...
> 
> andrew