You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Stephen Fitch (JIRA)" <de...@db.apache.org> on 2005/10/25 19:33:55 UTC
[jira] Created: (DERBY-646) In-memory backend storage support
In-memory backend storage support
---------------------------------
Key: DERBY-646
URL: http://issues.apache.org/jira/browse/DERBY-646
Project: Derby
Type: New Feature
Components: Store
Environment: All
Reporter: Stephen Fitch
To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Francois Orsini <fr...@gmail.com>.
On 11/4/05, Øystein Grøvlen <Oy...@sun.com> wrote:
>
> >>>>> "SF" == Stephen Fitch <05...@acadiau.ca> writes:
>
> SF> I agree with Francois. Booting a database with a special client seems
> SF> needlessly complicated when it can be externally specified outside of
> SF> an application.
>
> I am not saying that a special client should be required. I just
> think it should be possible for an DBA to change database
> configuration without having to change the application.
>
> If I were to set up Derby in an client/server environment, I would be
> sceptical of allowing any client/application to boot and shut down
> databases. It should be possible to require special authorization to
> do such operations.
Yes and this usually what "Grant & Revoke" functionality enables you to do
at the "System Privileges" level (i.e. System / Database start-up / shutdown
operations).
[...]
>
> --
> Øystein
>
>
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "SF" == Stephen Fitch <05...@acadiau.ca> writes:
SF> I agree with Francois. Booting a database with a special client seems
SF> needlessly complicated when it can be externally specified outside of
SF> an application.
I am not saying that a special client should be required. I just
think it should be possible for an DBA to change database
configuration without having to change the application.
If I were to set up Derby in an client/server environment, I would be
sceptical of allowing any client/application to boot and shut down
databases. It should be possible to require special authorization to
do such operations.
SF> I understand where you're coming from now though Øystein.
SF> This all started because I was trying to get alternate StorageFactorys
SF> working in the network client/server setup. I was looking at it from
SF> the perspective that anyone could be connected and doing anything
SF> anywhere on the network server at anytime. So, to just assume what
SF> StorageFactory is already active is the one you want when you dont
SF> specify one may cause unpredictable results in this case.
At least in a production environment I do not think this is a common
scenario.
--
Øystein
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Stephen Fitch <05...@acadiau.ca>.
I agree with Francois. Booting a database with a special client seems
needlessly complicated when it can be externally specified outside of an
application.
I understand where you're coming from now though Øystein.
This all started because I was trying to get alternate StorageFactorys
working in the network client/server setup. I was looking at it from
the perspective that anyone could be connected and doing anything
anywhere on the network server at anytime. So, to just assume what
StorageFactory is already active is the one you want when you dont
specify one may cause unpredictable results in this case.
Stephen
Francois Orsini wrote:
> I agree with Oystein on the principle but at the same time we need to be
> consistent in the way we specify properties for a database before it is
> booted. (booting properties) - like when booting an encrypted database
> you would have to specify the 'bootPassword' property in the jdbc
> connection URL. If an application is well-written, it should *not*
> hardcode the connection URL but make it either a java property or have
> it defined as part of a Datasource so that the application does not get
> changed when the connection URL does.
>
> If we have a separate client to boot a database, IMHO it is going to
> cause confusion and we would end-up having 2 different ways of booting a
> database which I don't think it's right.
>
> Just my 0.02 cents...
>
> --francois
>
> On 11/3/05, *Øystein Grøvlen* <Oystein.Grovlen@sun.com
> <ma...@sun.com>> wrote:
>
> >>>>> "SF" == Stephen Fitch <051933f@acadiau.ca
> <ma...@acadiau.ca>> writes:
>
> SF> Øystein Grøvlen wrote:
> >> (Connecting without specifying StorageFactory
> >> should just use whatever has been booted.)
> >>
> SF> 3. If a StorageFactory exists and a connection
> is attempted WITHOUT
> SF> specifying a StorageFactory, and the StorageFactory in use
> is NOT the
> SF> default (DirStorageFactory), throw an Exception. I think
> this would
> SF> be preferred behavior because if you
> connected with just
> SF> jdbc:derby:dbname you would assume it would be using
> the default
> SF> StorageFactory.
>
> I do not agree on this. I would like to separate the concerns so that
> an application does not need to be rewritten to work with different
> storage factories. A special client can be used to boot the database
> with the desired StorageFactory.
>
> --
> Øystein
>
>
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Francois Orsini <fr...@gmail.com>.
I agree with Oystein on the principle but at the same time we need to be
consistent in the way we specify properties for a database before it is
booted. (booting properties) - like when booting an encrypted database you
would have to specify the 'bootPassword' property in the jdbc connection
URL. If an application is well-written, it should *not* hardcode the
connection URL but make it either a java property or have it defined as part
of a Datasource so that the application does not get changed when the
connection URL does.
If we have a separate client to boot a database, IMHO it is going to cause
confusion and we would end-up having 2 different ways of booting a database
which I don't think it's right.
Just my 0.02 cents...
--francois
On 11/3/05, Øystein Grøvlen <Oy...@sun.com> wrote:
>
> >>>>> "SF" == Stephen Fitch <05...@acadiau.ca> writes:
>
> SF> Øystein Grøvlen wrote:
> >> (Connecting without specifying StorageFactory
> >> should just use whatever has been booted.)
> >>
> SF> 3. If a StorageFactory exists and a connection is attempted WITHOUT
> SF> specifying a StorageFactory, and the StorageFactory in use is NOT the
> SF> default (DirStorageFactory), throw an Exception. I think this would
> SF> be preferred behavior because if you connected with just
> SF> jdbc:derby:dbname you would assume it would be using the default
> SF> StorageFactory.
>
> I do not agree on this. I would like to separate the concerns so that
> an application does not need to be rewritten to work with different
> storage factories. A special client can be used to boot the database
> with the desired StorageFactory.
>
> --
> Øystein
>
>
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "SF" == Stephen Fitch <05...@acadiau.ca> writes:
SF> Øystein Grøvlen wrote:
>> (Connecting without specifying StorageFactory
>> should just use whatever has been booted.)
>>
SF> 3. If a StorageFactory exists and a connection is attempted WITHOUT
SF> specifying a StorageFactory, and the StorageFactory in use is NOT the
SF> default (DirStorageFactory), throw an Exception. I think this would
SF> be preferred behavior because if you connected with just
SF> jdbc:derby:dbname you would assume it would be using the default
SF> StorageFactory.
I do not agree on this. I would like to separate the concerns so that
an application does not need to be rewritten to work with different
storage factories. A special client can be used to boot the database
with the desired StorageFactory.
--
Øystein
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Stephen Fitch <05...@acadiau.ca>.
Øystein Grøvlen wrote:
>>>>>>"SF" == Stephen Fitch <05...@acadiau.ca> writes:
>
>
> SF> Øystein Grøvlen wrote:
>
> >> What happens if one when connecting specifies another StorageFactory
>
> >> than the one currently in use?
>
> SF> Looks like Derby detects there is already a StorageFactory in use for
> SF> that directory and database name. It uses the existing one rather than
> SF> the specified one even if the specified one is different.
>
> I would have been good to get an exception if one explicitly specify
> another StorageFactory. (Connecting without specifying StorageFactory
> should just use whatever has been booted.)
>
I just ran a test with a clean version of the trunk code plus my new
in-memory code. With the jdbc:derby:[storagefactory]:dbname syntax,
derby will open the specified StorageFactory. This is probably because
it passes around memory:dbname as the database name. So when it checks
to see if there's a StorageFactory in use, it doesn't find it because
the database will be listed under alternatesubSubProtocol:dbname or just
dbname for the default StorageFactory.
If I create a database with the name "testSF" in /home/derby under
DirStorageFactory then another under a different WritableStorageFactory,
they will end up competing for the same directories. For example, my
in-memory implementation still needs to use the on disk tmp directory.
So how about this...
1. Only one StorageFactory for a given directory and database name can
be used at a time
2. If a StorageFactory exists and a connection is attempted specifying a
different StorageFactory, an exception will be thrown
> Øystein Grøvlen:
> (Connecting without specifying StorageFactory
> should just use whatever has been booted.)
>
3. If a StorageFactory exists and a connection is attempted WITHOUT
specifying a StorageFactory, and the StorageFactory in use is NOT the
default (DirStorageFactory), throw an Exception. I think this would be
preferred behavior because if you connected with just jdbc:derby:dbname
you would assume it would be using the default StorageFactory.
4. We can adopt an alternate method of specifying a StorageFactory with
an atrribute. This will make it usable on any driver. Attached is a
simple example patch that should do the job. With this patch, the
jdbc:derby:storagefactory:dbname;storageFactory=sfname syntax will
ignore sfname attribute if a storagefactory name is specified before the
database name.
Opinions?
Stephen Fitch
Acadia University
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "SF" == Stephen Fitch <05...@acadiau.ca> writes:
SF> Øystein Grøvlen wrote:
>> What happens if one when connecting specifies another StorageFactory
>> than the one currently in use?
SF> Looks like Derby detects there is already a StorageFactory in use for
SF> that directory and database name. It uses the existing one rather than
SF> the specified one even if the specified one is different.
I would have been good to get an exception if one explicitly specify
another StorageFactory. (Connecting without specifying StorageFactory
should just use whatever has been booted.)
--
Øystein
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Stephen Fitch <05...@acadiau.ca>.
Øystein Grøvlen wrote:
>>>>>>"SF" == Stephen Fitch <05...@acadiau.ca> writes:
>
>
> >> Will it be possible for different
> >> connections to use different StorageFactory within the same system?
>
> SF> My understanding is there's one StorageFactory per database, and all
> SF> connections to that database use that StorageFactory.
>
> What happens if one when connecting specifies another StorageFactory
> than the one currently in use?
Looks like Derby detects there is already a StorageFactory in use for
that directory and database name. It uses the existing one rather than
the specified one even if the specified one is different.
Stephen
>
> SF> I'd like to be able to import existing disk-based databases. So,
> SF> rather than specifying it when we create the database, we specify it
> SF> when booting the database.
>
> Good idea.
>
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "SF" == Stephen Fitch <05...@acadiau.ca> writes:
>> Will it be possible for different
>> connections to use different StorageFactory within the same system?
SF> My understanding is there's one StorageFactory per database, and all
SF> connections to that database use that StorageFactory.
What happens if one when connecting specifies another StorageFactory
than the one currently in use?
SF> I'd like to be able to import existing disk-based databases. So,
SF> rather than specifying it when we create the database, we specify it
SF> when booting the database.
Good idea.
--
Øystein
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Stephen Fitch <05...@acadiau.ca>.
Øystein Grøvlen wrote:
>>>>>>"SF" == Stephen Fitch <05...@acadiau.ca> writes:
>
>
> SF> Hi Mike,
> SF> the issue I'm having is I can't find a way to tell the network server
> SF> what StorageFactory to use from the network client driver.
>
>
> SF> On the embedded side of things, I just define a new subsubprotocol
> SF> when I start java with:
>
> SF> -Dderby.subSubProtocol.memory=org.apache.derby.impl.io.MemoryStorageFactory
> SF> (alternatively I can change some of the engine code and register it as
> SF> a persistent service which results in the same issue)
>
> I am a bit confused about why the use of StorageFactory is decided
> when opening a connection.
When you open a connection, derby attempts to boot the database (if it
exists) or creates a new one if specified. The StorageFactory needs to
be determined before any creation or access to the "StorageFile"s the
database is stored on.
>Will it be possible for different
> connections to use different StorageFactory within the same system?
My understanding is there's one StorageFactory per database, and all
connections to that database use that StorageFactory.
> I guess different StorageFactory within the same database is not
> possible.
Not really with the current design as far as I can tell :/.
I think the current design allows for a different StorageFactory
instance to be used for logs residing in a different directory than the
default. I'm planning to look at allowing the database to be in-memory
but logging to disk.
I'm also going to make my in-memory implementation automatically(?)
import a database from disk into memory if it already exists.
>In that case, why not make it a parameter when creating the
> database?
I'd like to be able to import existing disk-based databases. So, rather
than specifying it when we create the database, we specify it when
booting the database.
Something like:
jdbc:derby:test;storageFactory=memory
jdbc:derby://host:port/test;storageFactory=memory
It could coded dynamically like the current embedded client's syntax
(jdbc:derby:[storagefactory:]<database>;attributes). This way you can
specify whatever StorageFactory you want even if it's not part of derby.
Stephen Fitch
Acadia University
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
>>>>> "SF" == Stephen Fitch <05...@acadiau.ca> writes:
SF> Hi Mike,
SF> the issue I'm having is I can't find a way to tell the network server
SF> what StorageFactory to use from the network client driver.
SF> On the embedded side of things, I just define a new subsubprotocol
SF> when I start java with:
SF> -Dderby.subSubProtocol.memory=org.apache.derby.impl.io.MemoryStorageFactory
SF> (alternatively I can change some of the engine code and register it as
SF> a persistent service which results in the same issue)
I am a bit confused about why the use of StorageFactory is decided
when opening a connection. Will it be possible for different
connections to use different StorageFactory within the same system? I
guess different StorageFactory within the same database is not
possible. In that case, why not make it a parameter when creating the
database?
Generally, I would think it would be useful for a single
connection/database be able to use main-memory storage for some tables
and disk storage for others.
--
Øystein
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I figured I didn't understand the question. I just assumed you would
be configuring your derby build to only boot your in memory version,
though as you say it would be much more convenient to determine at
connection time. I will let the network server experts comment on how
best to get the information from the client to the server.
I have always wondered what the performance difference would be between
in memory vs. durability=test and finally with durability=test and the
page cache big enough to hold the whole database. I look forward to
your results.
Stephen Fitch wrote:
> Hi Mike,
>
> the issue I'm having is I can't find a way to tell the network server
> what StorageFactory to use from the network client driver.
>
> On the embedded side of things, I just define a new subsubprotocol when
> I start java with:
> -Dderby.subSubProtocol.memory=org.apache.derby.impl.io.MemoryStorageFactory
> (alternatively I can change some of the engine code and register it as a
> persistent service which results in the same issue)
>
> then tell it to use that StorageFactory when I connect to the database, ie:
>
> jdbc:derby:memory:dbname;attributes
>
> However, there does not appear to be an option to explicitly tell the
> network server which storagefactory to use.
>
> In the section entitled "Accessing the Network Server by using the
> network client driver" at
> http://db.apache.org/derby/docs/dev/adminguide/adminguide-single.html,
> it looks like the driver syntax does not support the above form of
> connection URL. There does not appear to be an attribute for specifying
> the storagefactory either.
>
> I am looking for confirmation that this is the case.
>
> If it is, I think the easiest way to add this functionality is another
> connection attribute. This should avoid any code changes to the drivers.
>
> -------
>
> As for my performance issues, I can tell the disk to turn off the write
> cache, which it promptly ignores :) (Sound familiar?
> http://weblogs.java.net/blog/davidvc/archive/2005/10/the_story_of_th.html)
>
> Good suggestions to try durability=test mode and make the page cache
> gigantic. I will test them versus my in-memory implementation.
> Thanks :)
>
> I did figure out what was giving me the unexpectedly poor performance
> increase while running in-memory. A little error on my part. I'm now
> getting 2-3x better performance than Dir4StorageFactory :). Though this
> number is entirely dependant on RAM speed, OS and disk speed so it will
> vary between machines. My testbed was Gentoo Linux, kernel 2.6.13, JDK
> 1.4.2.08, 7200 RPM ATA hard drive w/8MB cache, 768MB of DDR266 RAM and
> an AMD 2400+ CPU.
>
> Thanks for your advice,
>
>
>
>
> Stephen Fitch
> Acadia University
>
> Mike Matrigali wrote:
>
>> I am not sure I understand your question about network client/server and
>> Storage factory. The Network layer is way above the StorageFactory
>> layer, so any change you make that makes an embedded client run in
>> memory will automatically also be used if you access it using the
>> network server interfaces (the in memory portion will always be on the
>> server side of the client/server interface).
>>
>> Your numbers don't surprise me if your disk is not really supporting
>> synchronous I/O for log and data. It may also be interesting to compare
>> your implementation numbers with running the existing server under
>> durability=test mode, especially if you ever figure out how to get your
>> disk to allow you to sync the log file.
>>
>> The areas that are going to cause Derby to go slow:
>> o sync of log file at commit time (write cache enabled disk disables
>> this sync)
>> o sync of data files at checkpoint time (write cache disables this sync)
>> o sync of data files when db cache gets full (write cache disables, for
>> fair comparison you should make cache as big as database if you want
>> to compare with your in memory implementation)
>> o read of uncached data from disk into cache - I guess this never
>> happens in your implementation as all data has to come about from
>> updates.
>>
>> Stephen Fitch wrote:
>>
>>> Hi Norbert,
>>>
>>> I have it working in the trunk embedded client in gentoo linux under
>>> jdk 1.4.2.08 but it should theoretically run under jdk 1.3.1 and any
>>> recent release of derby. However, as far as I can tell the network
>>> client/server doesn't support alternate implementations of
>>> StorageFactory. I'm still trying to track down if this is the case.
>>> If it is, another connection URL attribute may have to be added for
>>> in-memory to work on the network side of things.
>>>
>>> I'm having issues trying to test the performance as well since I'm
>>> competing with derby's page cache, java.nio's and my hard drive write
>>> cache (which I can't seem to turn off).
>>>
>>> *VERY* preliminary numbers indicate a 10-20% performance increase on
>>> inserts which is well below my expectations. I'm going to code up
>>> some tests for other operations today though.
>>>
>>> If anyone's interested in trying the code out or going over it to
>>> look for areas of improvement it would be much appreciated.
>>>
>>> I suppose the best way to distro it would to email a svn diff patch
>>> and source code for the new classes to the mailing list? I'm not
>>> looking to get it added to svn though, as it still needs work.
>>>
>>> Regards,
>>>
>>>
>>>
>>>
>>> Stephen Fitch
>>> Acadia University
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Toth-Gati Norbert wrote:
>>>
>>>> Hi Stephen,
>>>>
>>>> So now this means the support for in-memory storage is completed.
>>>> I will give it a try. But glad you finally got to the end. Good job!
>>>>
>>>> Regards,
>>>> Norbert
>>>>
>>>> ----- Original Message ----- From: "Stephen Fitch (JIRA)"
>>>> <de...@db.apache.org>
>>>> To: <de...@db.apache.org>
>>>> Sent: Tuesday, October 25, 2005 8:33 PM
>>>> Subject: [jira] Created: (DERBY-646) In-memory backend storage support
>>>>
>>>>
>>>>> In-memory backend storage support
>>>>> ---------------------------------
>>>>>
>>>>> Key: DERBY-646
>>>>> URL: http://issues.apache.org/jira/browse/DERBY-646
>>>>> Project: Derby
>>>>> Type: New Feature
>>>>> Components: Store
>>>>> Environment: All
>>>>> Reporter: Stephen Fitch
>>>>>
>>>>>
>>>>> To allow creation and modification of databases in-memory without
>>>>> requiring disk access or space to store the database.
>>>>>
>>>>> --
>>>>> This message is automatically generated by JIRA.
>>>>> -
>>>>> If you think it was sent incorrectly contact one of the
>>>>> administrators:
>>>>> http://issues.apache.org/jira/secure/Administrators.jspa
>>>>> -
>>>>> For more information on JIRA, see:
>>>>> http://www.atlassian.com/software/jira
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Stephen Fitch <05...@acadiau.ca>.
Hi Mike,
the issue I'm having is I can't find a way to tell the network server
what StorageFactory to use from the network client driver.
On the embedded side of things, I just define a new subsubprotocol when
I start java with:
-Dderby.subSubProtocol.memory=org.apache.derby.impl.io.MemoryStorageFactory
(alternatively I can change some of the engine code and register it as a
persistent service which results in the same issue)
then tell it to use that StorageFactory when I connect to the database, ie:
jdbc:derby:memory:dbname;attributes
However, there does not appear to be an option to explicitly tell the
network server which storagefactory to use.
In the section entitled "Accessing the Network Server by using the
network client driver" at
http://db.apache.org/derby/docs/dev/adminguide/adminguide-single.html,
it looks like the driver syntax does not support the above form of
connection URL. There does not appear to be an attribute for specifying
the storagefactory either.
I am looking for confirmation that this is the case.
If it is, I think the easiest way to add this functionality is another
connection attribute. This should avoid any code changes to the drivers.
-------
As for my performance issues, I can tell the disk to turn off the write
cache, which it promptly ignores :) (Sound familiar?
http://weblogs.java.net/blog/davidvc/archive/2005/10/the_story_of_th.html)
Good suggestions to try durability=test mode and make the page cache
gigantic. I will test them versus my in-memory implementation.
Thanks :)
I did figure out what was giving me the unexpectedly poor performance
increase while running in-memory. A little error on my part. I'm now
getting 2-3x better performance than Dir4StorageFactory :). Though this
number is entirely dependant on RAM speed, OS and disk speed so it will
vary between machines. My testbed was Gentoo Linux, kernel 2.6.13, JDK
1.4.2.08, 7200 RPM ATA hard drive w/8MB cache, 768MB of DDR266 RAM and
an AMD 2400+ CPU.
Thanks for your advice,
Stephen Fitch
Acadia University
Mike Matrigali wrote:
> I am not sure I understand your question about network client/server and
> Storage factory. The Network layer is way above the StorageFactory
> layer, so any change you make that makes an embedded client run in
> memory will automatically also be used if you access it using the
> network server interfaces (the in memory portion will always be on the
> server side of the client/server interface).
>
> Your numbers don't surprise me if your disk is not really supporting
> synchronous I/O for log and data. It may also be interesting to compare
> your implementation numbers with running the existing server under
> durability=test mode, especially if you ever figure out how to get your
> disk to allow you to sync the log file.
>
> The areas that are going to cause Derby to go slow:
> o sync of log file at commit time (write cache enabled disk disables
> this sync)
> o sync of data files at checkpoint time (write cache disables this sync)
> o sync of data files when db cache gets full (write cache disables, for
> fair comparison you should make cache as big as database if you want
> to compare with your in memory implementation)
> o read of uncached data from disk into cache - I guess this never
> happens in your implementation as all data has to come about from updates.
>
> Stephen Fitch wrote:
>
>> Hi Norbert,
>>
>> I have it working in the trunk embedded client in gentoo linux under
>> jdk 1.4.2.08 but it should theoretically run under jdk 1.3.1 and any
>> recent release of derby. However, as far as I can tell the network
>> client/server doesn't support alternate implementations of
>> StorageFactory. I'm still trying to track down if this is the case.
>> If it is, another connection URL attribute may have to be added for
>> in-memory to work on the network side of things.
>>
>> I'm having issues trying to test the performance as well since I'm
>> competing with derby's page cache, java.nio's and my hard drive write
>> cache (which I can't seem to turn off).
>>
>> *VERY* preliminary numbers indicate a 10-20% performance increase on
>> inserts which is well below my expectations. I'm going to code up
>> some tests for other operations today though.
>>
>> If anyone's interested in trying the code out or going over it to look
>> for areas of improvement it would be much appreciated.
>>
>> I suppose the best way to distro it would to email a svn diff patch
>> and source code for the new classes to the mailing list? I'm not
>> looking to get it added to svn though, as it still needs work.
>>
>> Regards,
>>
>>
>>
>>
>> Stephen Fitch
>> Acadia University
>>
>>
>>
>>
>>
>>
>>
>>
>> Toth-Gati Norbert wrote:
>>
>>> Hi Stephen,
>>>
>>> So now this means the support for in-memory storage is completed.
>>> I will give it a try. But glad you finally got to the end. Good job!
>>>
>>> Regards,
>>> Norbert
>>>
>>> ----- Original Message ----- From: "Stephen Fitch (JIRA)"
>>> <de...@db.apache.org>
>>> To: <de...@db.apache.org>
>>> Sent: Tuesday, October 25, 2005 8:33 PM
>>> Subject: [jira] Created: (DERBY-646) In-memory backend storage support
>>>
>>>
>>>> In-memory backend storage support
>>>> ---------------------------------
>>>>
>>>> Key: DERBY-646
>>>> URL: http://issues.apache.org/jira/browse/DERBY-646
>>>> Project: Derby
>>>> Type: New Feature
>>>> Components: Store
>>>> Environment: All
>>>> Reporter: Stephen Fitch
>>>>
>>>>
>>>> To allow creation and modification of databases in-memory without
>>>> requiring disk access or space to store the database.
>>>>
>>>> --
>>>> This message is automatically generated by JIRA.
>>>> -
>>>> If you think it was sent incorrectly contact one of the administrators:
>>>> http://issues.apache.org/jira/secure/Administrators.jspa
>>>> -
>>>> For more information on JIRA, see:
>>>> http://www.atlassian.com/software/jira
>>>>
>>>
>>>
>>
>>
>
>
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I am not sure I understand your question about network client/server and
Storage factory. The Network layer is way above the StorageFactory
layer, so any change you make that makes an embedded client run in
memory will automatically also be used if you access it using the
network server interfaces (the in memory portion will always be on the
server side of the client/server interface).
Your numbers don't surprise me if your disk is not really supporting
synchronous I/O for log and data. It may also be interesting to compare
your implementation numbers with running the existing server under
durability=test mode, especially if you ever figure out how to get your
disk to allow you to sync the log file.
The areas that are going to cause Derby to go slow:
o sync of log file at commit time (write cache enabled disk disables
this sync)
o sync of data files at checkpoint time (write cache disables this sync)
o sync of data files when db cache gets full (write cache disables, for
fair comparison you should make cache as big as database if you want
to compare with your in memory implementation)
o read of uncached data from disk into cache - I guess this never
happens in your implementation as all data has to come about from updates.
Stephen Fitch wrote:
> Hi Norbert,
>
> I have it working in the trunk embedded client in gentoo linux under jdk
> 1.4.2.08 but it should theoretically run under jdk 1.3.1 and any recent
> release of derby. However, as far as I can tell the network
> client/server doesn't support alternate implementations of
> StorageFactory. I'm still trying to track down if this is the case. If
> it is, another connection URL attribute may have to be added for
> in-memory to work on the network side of things.
>
> I'm having issues trying to test the performance as well since I'm
> competing with derby's page cache, java.nio's and my hard drive write
> cache (which I can't seem to turn off).
>
> *VERY* preliminary numbers indicate a 10-20% performance increase on
> inserts which is well below my expectations. I'm going to code up some
> tests for other operations today though.
>
> If anyone's interested in trying the code out or going over it to look
> for areas of improvement it would be much appreciated.
>
> I suppose the best way to distro it would to email a svn diff patch and
> source code for the new classes to the mailing list? I'm not looking to
> get it added to svn though, as it still needs work.
>
> Regards,
>
>
>
>
> Stephen Fitch
> Acadia University
>
>
>
>
>
>
>
>
> Toth-Gati Norbert wrote:
>
>> Hi Stephen,
>>
>> So now this means the support for in-memory storage is completed.
>> I will give it a try. But glad you finally got to the end. Good job!
>>
>> Regards,
>> Norbert
>>
>> ----- Original Message ----- From: "Stephen Fitch (JIRA)"
>> <de...@db.apache.org>
>> To: <de...@db.apache.org>
>> Sent: Tuesday, October 25, 2005 8:33 PM
>> Subject: [jira] Created: (DERBY-646) In-memory backend storage support
>>
>>
>>> In-memory backend storage support
>>> ---------------------------------
>>>
>>> Key: DERBY-646
>>> URL: http://issues.apache.org/jira/browse/DERBY-646
>>> Project: Derby
>>> Type: New Feature
>>> Components: Store
>>> Environment: All
>>> Reporter: Stephen Fitch
>>>
>>>
>>> To allow creation and modification of databases in-memory without
>>> requiring disk access or space to store the database.
>>>
>>> --
>>> This message is automatically generated by JIRA.
>>> -
>>> If you think it was sent incorrectly contact one of the administrators:
>>> http://issues.apache.org/jira/secure/Administrators.jspa
>>> -
>>> For more information on JIRA, see:
>>> http://www.atlassian.com/software/jira
>>>
>>
>>
>
>
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Andrew McIntyre <mc...@gmail.com>.
On 10/26/05, Satheesh Bandaram <sa...@sourcery.org> wrote:
> I wonder if marking it "not for inclusion in ASF projects" might prevent
> others in the community to continue enhancing the patch. (to be later
> included in ASF) Would marking it "Grant license to ASF for inclusion in
> ASF works" with a comment that the patch is not ready for contribution
> be better, if the contributor chooses to?
There's not a clear definition of the two options on the JIRA page,
but the way I read it is that when you choose 'don't grant' on the
JIRA attachment page, you are not licensing your code to Apache under
the ASL and therefore it cannot be checked into Subversion or be
redistributed as part of an official Apache project.
I don't see it as precluding collaborative work on any code, but any
contributions you make to the other person's code are covered by the
copyright on the original code, which has not been licensed to you.
Which means at the end of the day, the person could take your
suggestions and go home without licensing the final product to Apache,
an unlikely but unfortunately possible outcome.
So, thinking about it, marking it as 'licensed to the ASF' is
certainly a better option for anyone who genuinely wants to contribute
the code to the project when they are done.
andrew
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Satheesh Bandaram <sa...@Sourcery.Org>.
I wonder if marking it "not for inclusion in ASF projects" might prevent
others in the community to continue enhancing the patch. (to be later
included in ASF) Would marking it "Grant license to ASF for inclusion in
ASF works" with a comment that the patch is not ready for contribution
be better, if the contributor chooses to?
Satheesh
Andrew McIntyre wrote:
>>
>>
>You could do that, but a better option is probably to attach it to the
>JIRA entry you created for the entry. You can mark it "not for
>inclusion in ASF projects" if you don't think this version is ready to
>be committed.
>
>andrew
>
>
>
>
>
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Andrew McIntyre <mc...@gmail.com>.
On 10/26/05, Stephen Fitch <05...@acadiau.ca> wrote:
>
> I suppose the best way to distro it would to email a svn diff patch and
> source code for the new classes to the mailing list? I'm not looking to
> get it added to svn though, as it still needs work.
>
You could do that, but a better option is probably to attach it to the
JIRA entry you created for the entry. You can mark it "not for
inclusion in ASF projects" if you don't think this version is ready to
be committed.
andrew
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Stephen Fitch <05...@acadiau.ca>.
the code has been posted at
http://issues.apache.org/jira/browse/DERBY-646 :)
Stephen Fitch
Stephen Fitch wrote:
> Hi Norbert,
>
> I have it working in the trunk embedded client in gentoo linux under jdk
> 1.4.2.08 but it should theoretically run under jdk 1.3.1 and any recent
> release of derby. However, as far as I can tell the network
> client/server doesn't support alternate implementations of
> StorageFactory. I'm still trying to track down if this is the case. If
> it is, another connection URL attribute may have to be added for
> in-memory to work on the network side of things.
>
> I'm having issues trying to test the performance as well since I'm
> competing with derby's page cache, java.nio's and my hard drive write
> cache (which I can't seem to turn off).
>
> *VERY* preliminary numbers indicate a 10-20% performance increase on
> inserts which is well below my expectations. I'm going to code up some
> tests for other operations today though.
>
> If anyone's interested in trying the code out or going over it to look
> for areas of improvement it would be much appreciated.
>
> I suppose the best way to distro it would to email a svn diff patch and
> source code for the new classes to the mailing list? I'm not looking to
> get it added to svn though, as it still needs work.
>
> Regards,
>
>
>
>
> Stephen Fitch
> Acadia University
>
>
>
>
>
>
>
>
> Toth-Gati Norbert wrote:
>
>> Hi Stephen,
>>
>> So now this means the support for in-memory storage is completed.
>> I will give it a try. But glad you finally got to the end. Good job!
>>
>> Regards,
>> Norbert
>>
>> ----- Original Message ----- From: "Stephen Fitch (JIRA)"
>> <de...@db.apache.org>
>> To: <de...@db.apache.org>
>> Sent: Tuesday, October 25, 2005 8:33 PM
>> Subject: [jira] Created: (DERBY-646) In-memory backend storage support
>>
>>
>>> In-memory backend storage support
>>> ---------------------------------
>>>
>>> Key: DERBY-646
>>> URL: http://issues.apache.org/jira/browse/DERBY-646
>>> Project: Derby
>>> Type: New Feature
>>> Components: Store
>>> Environment: All
>>> Reporter: Stephen Fitch
>>>
>>>
>>> To allow creation and modification of databases in-memory without
>>> requiring disk access or space to store the database.
>>>
>>> --
>>> This message is automatically generated by JIRA.
>>> -
>>> If you think it was sent incorrectly contact one of the administrators:
>>> http://issues.apache.org/jira/secure/Administrators.jspa
>>> -
>>> For more information on JIRA, see:
>>> http://www.atlassian.com/software/jira
>>>
>>
>>
>
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Satheesh Bandaram <sa...@Sourcery.Org>.
Looks like you are making good progress! I would suggest attaching a SVN
diff patch to Jira entry. Jira provides a way to attach files and will
also allow you grant license to ASF, if you choose to.
Satheesh
Stephen Fitch wrote:
>
> I suppose the best way to distro it would to email a svn diff patch
> and source code for the new classes to the mailing list? I'm not
> looking to get it added to svn though, as it still needs work.
>
> Regards,
>
>
>
>
> Stephen Fitch
> Acadia University
>
>
>
>
>
>
>
>
> Toth-Gati Norbert wrote:
>
>> Hi Stephen,
>>
>> So now this means the support for in-memory storage is completed.
>> I will give it a try. But glad you finally got to the end. Good job!
>>
>> Regards,
>> Norbert
>>
>> ----- Original Message ----- From: "Stephen Fitch (JIRA)"
>> <de...@db.apache.org>
>> To: <de...@db.apache.org>
>> Sent: Tuesday, October 25, 2005 8:33 PM
>> Subject: [jira] Created: (DERBY-646) In-memory backend storage support
>>
>>
>>> In-memory backend storage support
>>> ---------------------------------
>>>
>>> Key: DERBY-646
>>> URL: http://issues.apache.org/jira/browse/DERBY-646
>>> Project: Derby
>>> Type: New Feature
>>> Components: Store
>>> Environment: All
>>> Reporter: Stephen Fitch
>>>
>>>
>>> To allow creation and modification of databases in-memory without
>>> requiring disk access or space to store the database.
>>>
>>> --
>>> This message is automatically generated by JIRA.
>>> -
>>> If you think it was sent incorrectly contact one of the administrators:
>>> http://issues.apache.org/jira/secure/Administrators.jspa
>>> -
>>> For more information on JIRA, see:
>>> http://www.atlassian.com/software/jira
>>>
>>
>>
>
>
>
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Stephen Fitch <05...@acadiau.ca>.
Hi Norbert,
I have it working in the trunk embedded client in gentoo linux under jdk
1.4.2.08 but it should theoretically run under jdk 1.3.1 and any recent
release of derby. However, as far as I can tell the network
client/server doesn't support alternate implementations of
StorageFactory. I'm still trying to track down if this is the case. If
it is, another connection URL attribute may have to be added for
in-memory to work on the network side of things.
I'm having issues trying to test the performance as well since I'm
competing with derby's page cache, java.nio's and my hard drive write
cache (which I can't seem to turn off).
*VERY* preliminary numbers indicate a 10-20% performance increase on
inserts which is well below my expectations. I'm going to code up some
tests for other operations today though.
If anyone's interested in trying the code out or going over it to look
for areas of improvement it would be much appreciated.
I suppose the best way to distro it would to email a svn diff patch and
source code for the new classes to the mailing list? I'm not looking to
get it added to svn though, as it still needs work.
Regards,
Stephen Fitch
Acadia University
Toth-Gati Norbert wrote:
> Hi Stephen,
>
> So now this means the support for in-memory storage is completed.
> I will give it a try. But glad you finally got to the end. Good job!
>
> Regards,
> Norbert
>
> ----- Original Message ----- From: "Stephen Fitch (JIRA)"
> <de...@db.apache.org>
> To: <de...@db.apache.org>
> Sent: Tuesday, October 25, 2005 8:33 PM
> Subject: [jira] Created: (DERBY-646) In-memory backend storage support
>
>
>> In-memory backend storage support
>> ---------------------------------
>>
>> Key: DERBY-646
>> URL: http://issues.apache.org/jira/browse/DERBY-646
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Environment: All
>> Reporter: Stephen Fitch
>>
>>
>> To allow creation and modification of databases in-memory without
>> requiring disk access or space to store the database.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> If you think it was sent incorrectly contact one of the administrators:
>> http://issues.apache.org/jira/secure/Administrators.jspa
>> -
>> For more information on JIRA, see:
>> http://www.atlassian.com/software/jira
>>
>
>
Re: [jira] Created: (DERBY-646) In-memory backend storage support
Posted by Toth-Gati Norbert <no...@gmail.com>.
Hi Stephen,
So now this means the support for in-memory storage is completed.
I will give it a try. But glad you finally got to the end. Good job!
Regards,
Norbert
----- Original Message -----
From: "Stephen Fitch (JIRA)" <de...@db.apache.org>
To: <de...@db.apache.org>
Sent: Tuesday, October 25, 2005 8:33 PM
Subject: [jira] Created: (DERBY-646) In-memory backend storage support
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
>
>
> To allow creation and modification of databases in-memory without
> requiring disk access or space to store the database.
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the administrators:
> http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:
> http://www.atlassian.com/software/jira
>
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Adam Wozniak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675046#action_12675046 ]
Adam Wozniak commented on DERBY-646:
------------------------------------
Till 2009-02-22 I'm out of office. In urgent matters call my mobile (+48 604 277 044) or use adam.wozniak@capgemini.com address. Regards, Adam Woźniak
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Cheng Che Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674106#action_12674106 ]
Cheng Che Chen commented on DERBY-646:
--------------------------------------
Thanks for the tip on the "extraDBMSclasses.properties" file. I'm now able to build the JAR files properly with my classes.
At this point, I have JUnit test cases for the basic building blocks of my implementation. They run successfully stand-alone, but I have yet to figure out how to integrate them into Derby's test framework.
I have also done some manual functional testing of my in-memory storage factory using examples from the Derby tutorial (http://db.apache.org/derby/papers/DerbyTut/index.html).
I have not modified any existing Derby tests to use the new in-memory storage factory, as I have yet to figure out how Derby's test framework works.
If people are willing to have a look at what I have so far, then I will post the patch; otherwise, please wait until I have plug things into the Derby test framework.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Stephen Fitch (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=all ]
Stephen Fitch updated DERBY-646:
--------------------------------
Attachment: in-memory.tar
svn.diff
I currently have an in-memory implementation of the storage factory interface "working" with the embedded client. It requires a very minimal change to one existing class and the addition of 6 more.
It's easily plugged into the embeded client with -Dderby.subSubProtocol.memory=org.apache.derby.impl.io.MemoryStorageFactory, but I'm still trying to work out how to get it going with the network server. (use jdbc:derby:memory:<database;attributes> as connection url)
MemoryStorageFactory - In-memory implementation of StorageFactory
MemoryStorageFile extends InputStreamFile - wraps most calls to another class with a hashtable keyed by full filepath names
MemoryStorageRandomAccessFile - extends java.io.RandomAccessFile and overrides its' read, write and native methods.
DynamicByteArrayIOStream - Readable, writable dynamically accessible byte array. This was done because I didnt want to have to implement the DataInput and DataOutput interfaces specified by the StorageRandomAccessFile interface (which would have been just copying them from java.io.RandomAccessFile). It is by no means an optimal implementation, but it works.
DynamicByteArrayIOStreamFile - the class above extended with file attributes (ex: do I exist, am I a directory).
MemoryMetadata - generalized location for StorageFile paths and DynamicByteArrayIOStreamFile objects. Also
The naming convention is debatable and as this is still a work in progress can be changed.
Any and all suggestions welcome :)
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: in-memory.tar, svn.diff
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Waagan updated DERBY-646:
----------------------------------
Attachment: derby-646-performance_comparison_1a.txt
'derby-646-performance_comparison_1a.txt' is my previous post as a text file. The formatting here is a little better...
Cheng wrote:
-----
So the connection string looks like "jdbc:derby:memory:MyTestDb" or "jdbc:derby://localhost:1527/memory:MyTestDb".
-----
Regarding the client/server URL, should we change the client driver to understand the following URL instead?
"jdbc:derby:memory://localhost:1527/MyTestDb"
It looks cleaner to me.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Cheng Che Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12673850#action_12673850 ]
Cheng Che Chen commented on DERBY-646:
--------------------------------------
Hi folks,
I've implemented the interfaces org.apache.derby.io.{StorageFile,StorageRandomAccessFile,WritableStorageFactory} such that all file system operations are emulated entirely in memory, producing no file or directory on disk. I'm more than happy to contribute the implementation.
However, I have a build problem for which I hope to get some help. To start, for simplicity, I added my source files to "trunk/java/engine/org/apache/derby/impl/io" -- so that I didn't have to create/modify any "build.xml" files. I then tried to build the sources and the JAR files. The sources built with no problem -- all of my classes appeared in "trunk/classes/org/apache/derby/impl/io". However, for some reason, the JAR build process did not put my classes into "derby.jar". As a result, I had to set the JAVA class path to "trunk/classes" during testing.
I'd appreciate any help resolving this build problem.
Thanks.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=all ]
Kristian Waagan updated DERBY-646:
----------------------------------
Assignee: (was: Stephen Fitch)
Derby Info: [Release Note Needed] (was: [Patch Available, Release Note Needed])
I tried to apply this patch. It applied (some chunks needed to be offset), but it did not compile.
I'm clearing the patch available flag, as the patch is clearly not ready for commit.
There seems to be some interest for this feature, so I *might* spend some time to wash the patch to the level where it can be applied and tested. However, some more work from the original contributer would be greatly appreciated :) For instance, documenting new classes would be very helpful (as requested earlier by Dan).
Note that I'm also unassigning the original contributer for now, since it has been approx 8 months since the last activity.
Feel free to reclaim the issue if you plan working on it!
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Adrian Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680170#action_12680170 ]
Adrian Cole commented on DERBY-646:
-----------------------------------
JBoss Cache would consider switching onto Derby for some internal work, if this feature is implemented. As of now, our tests take 2 minutes longer with embedded derby then hsqldb due to file i/o.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (DERBY-646) In-memory backend storage support
Posted by Mike Matrigali <mi...@sbcglobal.net>.
In the meantime you should try out the current system using the
durability=test flag, for your unit testing needs. It was implemented
specifically for this application from other user requests. It does
no sychronous I/O so effectively uses the OS's memory cache.
James Synge (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12369704 ]
>
> James Synge commented on DERBY-646:
> -----------------------------------
>
> I'm with Mike in wanting to use this for unit testing purposes. My group has a "fake" db implementation that provides an in-memory "database" (not sql), but it is becoming a pain to maintain it as we try to take more advantage of SQL features, so Stephen's solution sounds great.
>
>
>
>>In-memory backend storage support
>>---------------------------------
>>
>> Key: DERBY-646
>> URL: http://issues.apache.org/jira/browse/DERBY-646
>> Project: Derby
>> Type: New Feature
>> Components: Store
>> Environment: All
>> Reporter: Stephen Fitch
>> Assignee: Stephen Fitch
>> Attachments: svn.diff
>>
>>To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
>
>
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "James Synge (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12369704 ]
James Synge commented on DERBY-646:
-----------------------------------
I'm with Mike in wanting to use this for unit testing purposes. My group has a "fake" db implementation that provides an in-memory "database" (not sql), but it is becoming a pain to maintain it as we try to take more advantage of SQL features, so Stephen's solution sounds great.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Stephen Fitch
> Attachments: svn.diff
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "David Van Couvering (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=all ]
David Van Couvering updated DERBY-646:
--------------------------------------
Derby Info: [Patch Available, Release Note Needed]
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assigned To: Stephen Fitch
> Attachments: svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "David Van Couvering (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12426368 ]
David Van Couvering commented on DERBY-646:
-------------------------------------------
Are separate tests needed for this feature. Isn't this a configuration option, e.g. couldn't we run the storemats suite with the system configured to use this store module?
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assigned To: Stephen Fitch
> Attachments: svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Dave Syer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12513869 ]
Dave Syer commented on DERBY-646:
---------------------------------
java.lang.SecurityException: sealing violation: can't seal package org.apache.derby.impl.io: already loaded
at java.net.URLClassLoader.defineClass(URLClassLoader.java:235)
at java.net.URLClassLoader.access$100(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
at java.net.URLClassLoader.access$100(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:164)
at test.EmeddedMemoryTests.setUp(EmeddedMemoryTests.java:41)
at junit.framework.TestCase.runBare(TestCase.java:125)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Waagan closed DERBY-646.
---------------------------------
Fix Version/s: 10.5.1.1
Resolution: Fixed
Closing issue.
Further improvements and additions should be tracked under separate Jiras.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.5.1.1
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Fix For: 10.5.1.1
>
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Updated: (DERBY-646) In-memory backend storage support
Posted by Kristian Waagan <Kr...@Sun.COM>.
David Van Couvering wrote:
> Great to see you working on this, Kristian. Regarding the CLA, we went
> through this a while ago, and yes, it is sufficient.
Thanks for the clarification.
Just want to point out that I am not actively working on this issue, so
anyone else that want to pick it up is welcome to do so :)
I just felt it had been abandoned for too long, and wanted to give it a
little "kick behind" since there seems to be interest for the feature in
the (user) community.
--
Kristian
>
> David
>
> Kristian Waagan (JIRA) wrote:
>> [ http://issues.apache.org/jira/browse/DERBY-646?page=all ]
>>
>> Kristian Waagan updated DERBY-646:
>> ----------------------------------
>>
>> Attachment: derby-646-1a-raw-compiles.diff
>> derby-646-1a-raw-compiles.stat
>>
>> 'derby-646-1a-raw-compiles.diff' is a minor fix of the original patch
>> 'svn.diff'. The patch now applies to trunk and compiles. No other
>> clean-ups/changes have been done, but I had to rewrite a little bit to
>> solve circular compile dependencies.
>>
>> As far as I can tell, the basic functionality is working. I was able
>> to run with the memory storage backend, which creates a database
>> directory with lock files and a tmp directory. Tried running a few
>> tests with a "hacked" JDBCClient implementation, and the tests passed.
>> Had some problems with warnings regarding dual boot (missing
>> shutdown/clean-up?).
>>
>> I think a little more clean-up should be done first (implement a
>> missing method - getURL, remove commented out code, add some more
>> comments), and then the patch can be reviewed for functional/design
>> improvements.
>> We might want some tests written specifically for the memory backend,
>> and then run the existing tests with the memory backend as well.
>>
>> Stephen Fitch is listed at
>> http://people.apache.org/~jim/committers.html under "Unlisted CLAs".
>> Is this enough to allow us to safely bring this code into the repository?
>>
>>> In-memory backend storage support
>>> ---------------------------------
>>>
>>> Key: DERBY-646
>>> URL: http://issues.apache.org/jira/browse/DERBY-646
>>> Project: Derby
>>> Issue Type: New Feature
>>> Components: Store
>>> Environment: All
>>> Reporter: Stephen Fitch
>>> Attachments: derby-646-1a-raw-compiles.diff,
>>> derby-646-1a-raw-compiles.stat, svn.diff
>>>
>>>
>>> To allow creation and modification of databases in-memory without
>>> requiring disk access or space to store the database.
>>
Re: [jira] Updated: (DERBY-646) In-memory backend storage support
Posted by David Van Couvering <da...@vancouvering.com>.
Great to see you working on this, Kristian. Regarding the CLA, we went
through this a while ago, and yes, it is sufficient.
David
Kristian Waagan (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-646?page=all ]
>
> Kristian Waagan updated DERBY-646:
> ----------------------------------
>
> Attachment: derby-646-1a-raw-compiles.diff
> derby-646-1a-raw-compiles.stat
>
> 'derby-646-1a-raw-compiles.diff' is a minor fix of the original patch 'svn.diff'. The patch now applies to trunk and compiles. No other clean-ups/changes have been done, but I had to rewrite a little bit to solve circular compile dependencies.
>
> As far as I can tell, the basic functionality is working. I was able to run with the memory storage backend, which creates a database directory with lock files and a tmp directory. Tried running a few tests with a "hacked" JDBCClient implementation, and the tests passed. Had some problems with warnings regarding dual boot (missing shutdown/clean-up?).
>
> I think a little more clean-up should be done first (implement a missing method - getURL, remove commented out code, add some more comments), and then the patch can be reviewed for functional/design improvements.
> We might want some tests written specifically for the memory backend, and then run the existing tests with the memory backend as well.
>
> Stephen Fitch is listed at http://people.apache.org/~jim/committers.html under "Unlisted CLAs". Is this enough to allow us to safely bring this code into the repository?
>
>> In-memory backend storage support
>> ---------------------------------
>>
>> Key: DERBY-646
>> URL: http://issues.apache.org/jira/browse/DERBY-646
>> Project: Derby
>> Issue Type: New Feature
>> Components: Store
>> Environment: All
>> Reporter: Stephen Fitch
>> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>>
>>
>> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
>
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=all ]
Kristian Waagan updated DERBY-646:
----------------------------------
Attachment: derby-646-1a-raw-compiles.diff
derby-646-1a-raw-compiles.stat
'derby-646-1a-raw-compiles.diff' is a minor fix of the original patch 'svn.diff'. The patch now applies to trunk and compiles. No other clean-ups/changes have been done, but I had to rewrite a little bit to solve circular compile dependencies.
As far as I can tell, the basic functionality is working. I was able to run with the memory storage backend, which creates a database directory with lock files and a tmp directory. Tried running a few tests with a "hacked" JDBCClient implementation, and the tests passed. Had some problems with warnings regarding dual boot (missing shutdown/clean-up?).
I think a little more clean-up should be done first (implement a missing method - getURL, remove commented out code, add some more comments), and then the patch can be reviewed for functional/design improvements.
We might want some tests written specifically for the memory backend, and then run the existing tests with the memory backend as well.
Stephen Fitch is listed at http://people.apache.org/~jim/committers.html under "Unlisted CLAs". Is this enough to allow us to safely bring this code into the repository?
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671849#action_12671849 ]
Kristian Waagan commented on DERBY-646:
---------------------------------------
I had a quick look at this again, looking at how an in-memory backend could be wired into Derby.
>From what I can see, the easiest way is to add a single entry to BaseMonitor (and the new constant for the subSubProtocol):
storageFactories.put( PersistentService.VFMEM, "org.apache.derby.impl.io.VFMemoryStorageFactory");
If VFMemoryStorageFactory (or whatever the class will be called) behaves in the expected way, everything should "just work". The solution won't be optimal. The primary goal is to offer a configuration that doesn't leave any trace in the form of files and directories. Performance may be somewhat better, but the memory usage will be high. Not only will all the data be stored in memory, parts of it will be stored in multiple copies in memory.
Note that even though the configuration will implement PersistentService, data won't be persisted beyond the life-time of the JVM running Derby. If this causes any problems with rest of the code, remains to be seen.
A new service could be implemented as well, but I think this will require a lot more effort (feel free to comment on this if you have more information).
I'll play some more with the idea and post my findings.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674134#action_12674134 ]
Kristian Waagan commented on DERBY-646:
---------------------------------------
Cheng,
As it happens, I have a working prototype as well. It closely follows the approach taken by Stephen Fitch in the original patch.
If possible, I think we should somehow try to coordinate our work. I'd like to spend a few more days cleaning up the initial code contribution, as it is pretty fresh.
Regarding testing, the easiest way to do it is to override the storage factory for PersistentService.DIRECTORY in BaseMonitor. Then you can just run the tests as normal, either by using JUnit directly or by running ant junitreport.
Currently, I'm getting 24 failures and 66 errors when running junitreport. Somes of these are to be expected, as some tests explicitly tries to access files in the database directory. I also note problems in general with encryption, backup and restoreFrom (in the url) etc, but I suspect these tests require the database files to exist as well.
Also, some tests will just fail because all the data is stored in memory, for instance 'ant junit-lowmem'.
Lastly, the test framework has a decorator to delete a database and/or a directory. Since it is working against the file system only, it doesn't manage to delete an in-memory database.
Note that bugs in the prototype tend to hide if they are triggered during boot. If things go wrong, I recommend to check derby.log before starting debugging.
I'll post my code (and the test results) in a day or two, and we should consider how to best cooperate on this issue.
Regards,
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Rick Hillegas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683074#action_12683074 ]
Rick Hillegas commented on DERBY-646:
-------------------------------------
I don't think that we can say that this is a finished, full-fledged feature. However, I would like to see the release notes mention that this is an experimental feature which we'd like users to test-drive. We will need some kind of primer (perhaps a wiki page) and it would be great if the release notes could direct users to that primer for more information.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Rick Hillegas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676783#action_12676783 ]
Rick Hillegas commented on DERBY-646:
-------------------------------------
For the first time in the history of the Derby community, two complete implementations of a significant feature have surfaced almost simultaneously. I'm afraid we have to pick one. I hope that the other contributor does not feel snubbed. Thanks to Kristian and Cheng for these two great submissions.
I have taken a quick look at the patches and at the commentary on this JIRA. I prefer Kristian's implementation for the following reasons:
1) The coding style is slightly easier for me to understand.
2) According to the performance data which Kristian posted, his implementation is slightly faster.
I hope that Cheng's ICLA arrives soon so that we can mine his patch for ideas that can be layered on top of Kristian's work. For instance,
A) Cheng's solution lets you snapshot the database to disk at shutdown time. I think some users will find this useful. I understand that we could simulate this behavior using backup and restore, but I would not discount the appeal of this automatic behavior. Perhaps the api here could be an additional subprotocol.
B) Cheng's solution has a finalizer method. This seems particularly useful for delivering (A) in the face of a system crash. But winddown logic seems generally useful for a complicated program like an rdbms.
C) Cheng provides some additional tests. Some of these tests may be implementation-agnostic and worth running against Kristian's implementation. It would be helpful if Cheng could describe these tests in greater detail so that we can decide whether to incorporate them.
Thanks again to both Kristian and Cheng.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688272#action_12688272 ]
Kristian Waagan commented on DERBY-646:
---------------------------------------
Primer at http://wiki.apache.org/db-derby/InMemoryBackEndPrimer
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Stephen Fitch (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=all ]
Stephen Fitch updated DERBY-646:
--------------------------------
Attachment: (was: in-memory.tar)
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Stephen Fitch
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681285#action_12681285 ]
Kristian Waagan commented on DERBY-646:
---------------------------------------
Thanks for the testing, Knut Anders.
Seems like I messed up some of the path-handling logic (absolute vs relative) in the patch I committed. I'm looking into the problem now.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Fix For: 10.5.0.0
>
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676400#action_12676400 ]
Kathey Marsden commented on DERBY-646:
--------------------------------------
Kristian asked:
>We now have two candidate solutions. How do we proceed?
I haven't looked at the code but I guess the first thing we need to consider is legal. Is Cheng Che Chen willing to file an ICLA?
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674172#action_12674172 ]
Kristian Waagan commented on DERBY-646:
---------------------------------------
Yes, me too. However, since I was interested in learning a bit more about Derby and because I didn't know the state of the existing patch, I started from scratch.
It is my impression that the existing patch isn't "production ready".
I think it would be cool to compare our approaches. Maybe we can draw from both of them and create an even better solution!
The only thing my prototype leaves on disk, is derby.log. I think this can be easily changed, but often you would want to consult the log anyway.
The goal of my implementation is to avoid creating files/dirs on disk, which for some people is attractive for running tests. I believe there are also performance benefits, especially for update operations, at the cost of increased memory usage of course. The database creation time should also be shorter, but I haven't obtained any numbers on this yet.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845754#action_12845754 ]
Knut Anders Hatlen commented on DERBY-646:
------------------------------------------
It seems like the fix for this issue went into 10.5. Could we close it now?
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.5.1.1
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Cheng Che Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Che Chen updated DERBY-646:
---------------------------------
Attachment: (was: derby-646-20090222.diff)
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Stephen Fitch (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=all ]
Stephen Fitch updated DERBY-646:
--------------------------------
Attachment: (was: svn.diff)
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Stephen Fitch
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676716#action_12676716 ]
Kristian Waagan commented on DERBY-646:
---------------------------------------
Here are some results from running 'ant junitreport' with patch 2a
(vfmem).
The prefix 'org.apache.derbyTesting.' has been removed for brevity.
The first number is the number of tests run with the vfmem back end,
the second number is the number of tests run with a clean trunk.
Note that I changes the memory back end to be the default back end in
this run (modified the entry for DIRECTORY storage factory in BaseMonitor), and
I also changed DropDatabaseSetup to delete the in-memory database.
Suites run without failures:
junit.EnvTest (9/9)
functionTests.tests.jdbcapi.AutoloadTest (3/3)
functionTests.tests.jdbcapi.JDBCDriversEmbeddedTest (8/8)
functionTests.tests.jdbcapi.JDBCDriversClientTest (4/4)
functionTests.tests.jdbcapi.JDBCDriversAllTest (12/12)
functionTests.tests.tools._Suite (87/87)
functionTests.tests.demo._Suite (2/2)
functionTests.tests.engine._Suite (11/11)
functionTests.suites.EncryptionSuite (7/7)
functionTests.tests.jdbc4._Suite (896/898) // One test added later
functionTests.tests.management._Suite (100/100)
The follwing tests had failures and/or errors:
functionTests.tests.derbynet._Suite (t=122/f=1/e=0)
(f) ShutDownDBWhenNSShutsDownTest.testDatabasesShutDownWhenNSShutdown
functionTests.tests.lang._Suite (t=1473/f=4/e=2)
(f) DatabaseClassLoadingTest.testCreateDatabaseJar
(f) DatabaseClassLoadingTest.testDatabaseInJar
(f) ConnectTest.testDBName
(f) ConnectTest.testDBName
(e) DatabaseClassLoadingTest.testHackedJarReplacedClass
(e) DBInJarTest.testConnectDBInJar
functionTests.tests.jdbcapi._Suite (t=6301/f=2/e=0)
(f) PoolXADSCreateShutdownDBTest.testXADS
(f) PoolXADSCreateShutdownDBTest.testXADS
functionTests.tests.store._Suite (t=98/f=4/e=19)
(f) BootAllTest.testSettingBootAllPropertyWithHomePropertySet
(f) OSReadOnlyTest.testOSReadOnly
(f) OSReadOnlyTest.testOSReadOnly
(f) BackupRestoreTest.testDerby3875
o Ok, test is trying to correct file on disk.
(e) OfflineBackupTest.testCreateFromRestoreFrom
6x (e) EncryptionKeyAESTest
6x (e) EncryptionKeyBlowfishTest
6x (e) EncryptionKeyDESTest
unitTests.junit._Suite
(crashes due to ant/junit problem, will rerun later)
(e) AccessControlException
functionTests.tests.upgradeTests._Suite (t=420/f=12/e=42)
(needs investigation)
functionTests.tests.memory._Suite
o Ok, fails because it sets the heap size really low and tries
to insert data that is bigger than the heap.
To sum up, I'm currently getting 23 failures and 65 errors.
The time it takes to run the suites is around 45 minutes.
I believe the 'ant junitreport' target is roughly equal to running
suites.All.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681257#action_12681257 ]
Knut Anders Hatlen commented on DERBY-646:
------------------------------------------
I came across one issue when I experimented with the in-memory backend. When I exit from ij, I see this exception:
Exception in thread "main" java.lang.NullPointerException
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(BaseDataFileFactory.java:2684)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.removeStubs(BaseDataFileFactory.java:1591)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.stop(BaseDataFileFactory.java:499)
at org.apache.derby.impl.services.monitor.TopService.stop(TopService.java:405)
at org.apache.derby.impl.services.monitor.TopService.shutdown(TopService.java:349)
at org.apache.derby.impl.services.monitor.BaseMonitor.shutdown(BaseMonitor.java:239)
at org.apache.derby.impl.services.monitor.BaseMonitor.shutdown(BaseMonitor.java:205)
at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:231)
at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:119)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:207)
at org.apache.derby.impl.tools.ij.utilMain.cleanupGo(utilMain.java:413)
at org.apache.derby.impl.tools.ij.utilMain.go(utilMain.java:246)
at org.apache.derby.impl.tools.ij.Main.go(Main.java:210)
at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:177)
at org.apache.derby.impl.tools.ij.Main.main(Main.java:73)
at org.apache.derby.tools.ij.main(ij.java:59)
To reproduce, execute the following commands in ij:
connect 'jdbc:derby:in-memory:mydb;create=true';
exit;
Other than that, it seems to be working fine.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Fix For: 10.5.0.0
>
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Waagan updated DERBY-646:
----------------------------------
Attachment: derby-646-2b-vfmem_first_rev.diff
Committed patch 'derby-646-2b-vfmem_first_rev.diff' to trunk with revision 752114.
I expect more work will be done under various sub-tasks of this issue.
It would be nice to get early feedback from users and developers willing to try out the feature under development.
Note that I have set the subSubProtocol to be 'in-memory' based on the feedback on DERBY-4084, but this may change.
The in-memory back end is working in my environment, but there may be problems on other platforms. For instance, is the case-insensitive mode of Windows a problem?
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "David Sims (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674279#action_12674279 ]
David Sims commented on DERBY-646:
----------------------------------
I'm following this thread with great interest. Many thanks to you guys for working on this.
> The only thing my prototype leaves on disk, is derby.log. I think this can be easily
> changed, but often you would want to consult the log anyway.
For what it's worth, in my situation, the option of having absolutely nothing written to disk is a huge benefit.
Thank you for your work,
David
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Waagan updated DERBY-646:
----------------------------------
Attachment: derby-646-2a-vfmem_first_rev.diff
derby-646-2a-vfmem_first_rev.stat
The patch 'derby-646-2a-vfmem_first_rev.diff' is the first revision of an in-memory backend for Derby.
The implementation takes the same general approach as the original patch contributed by Stephen Fitch, but it is implemented from scratch.
This was a choice I made because the state of the original patch was unknown, and I wanted to learn more about the internals of Derby in this area.
Since this is the first revision, there are limitations;
o The maximum "file size" is 256 MB, 512 MB or 2 GB depending on the page size used.
The system tries to make the blocks used to store data the same size as the pages used by Derby. When the page size is "indeterminable", 4 KB is used by default.
If you hit the 256 MB limit, you have two options; set the page size to 32 KB, or modify BlockedByteArray.
I think that either the max size should be configurable via properties, or the BlockedByteArray should be extended with a better growth functionality. This is basically a tradeoff between memory usage for smaller files, and CPU usage (copying bytes). As an alternative, I think one can avoid both excessive memory overhead and byte copying by adding complexity :)
o The in-memory backend is not enabled for the client driver.
I haven't investigated this, but if you want to test with the client driver, replace the storage factory class for PersistentStorage.DIRECTORY in BaseMonitor with the VFMemoryStoreageFactory.
This will make the in-memory backend the default backend, and *none* of you databases will be persisted.
I think some handling code in the client driver has to be added to make the in-memory backend available there.
o There is no easy way to delete a database, except for restarting the JVM.
If this is a problem for your usage, there is a purgeDatabase-method in the storage factory (a database is currently named by an absolute file path in the storage factory).
[How to test]
1) Download the patch (2a), apply it and build Derby.
2) Connect to a database using the following prefix in the JDBC connection url: "jdbc:derby:mem" (remember to create the database the first time you connect to it).
WARNING: I have only tested this on OpenSolaris. It would be nice if someone running Windows could take the patch for a test-spin.
I'm sure there are more limitations and problems, but I'd like to get this out in the community as soon as possible. I have managed to run most of the existing regression tests with the patch, so it should be usable.
Feedback is welcome.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Stephen Fitch (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=all ]
Stephen Fitch updated DERBY-646:
--------------------------------
Attachment: svn.diff
Implementation rewritten to be more modular. Requires no changes to existing code to work. Define property derby.subSubProtocol.memory=org.apache.derby.impl.io.MemoryStorageFactory and use connection url jdbc:derby:memory:databasename
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Stephen Fitch
> Attachments: svn.diff
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683062#action_12683062 ]
Kristian Waagan commented on DERBY-646:
---------------------------------------
The core functionality is in 10.5 already.
I'd like to get the following into 10.5 as well (DERBY-4085):
a) Add the functional test written by Cheng.
b) Enable the new tests as part of a suite (i.e. suites.All and as the ant junit-core target).
New features will be logged as separate Jiras and added to the trunk.
I do plan to write a Wiki-page documenting the feature to some extent. And then I hope users will give us feedback so that we can consider adding more features for the next Derby feature release.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Fix For: 10.5.0.0
>
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Cheng Che Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676902#action_12676902 ]
Cheng Che Chen commented on DERBY-646:
--------------------------------------
I have just emailed scanned images of signed ICLA to secretary@apache.org, as instructed by Kathey Marsden.
Regarding Rick Hillegas' question about the tests included in my patch, there are three sets of unit tests and one functional test.
The unit tests verify the correctness of my internal data structures. They are probably not useful to other implementations, unless, for example, they have similar properties to be verified.
The functional test implements an application in both JAVA and SQL and cross-checks the computed results. It can indeed be applied to any implementation of storage factory by simply changing the database name for the SQL portion of the test object. The functional test consists of the following files: "GenMog.java", "SampMog.java", "ClusMog.java", "ClusMogSQL.java", and "MogTest.java" -- the last file is the JUnit test object.
However, I have some doubt about whether I have configured/used Derby's test framework properly in "MogTest.java". This file started out as a stand-alone JUnit test. When I was running it stand-alone, it would incur no disk activity. After I converted the file to run under Derby's test framework, the disk activity LED on my computer would go wild every time I ran it. I wonder if I have unwittingly been testing the default storage factory with the converted test.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Cheng Che Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674113#action_12674113 ]
Cheng Che Chen commented on DERBY-646:
--------------------------------------
BTW, there appears to be a discrepancy between code in the constructor of (StorageFactoryService) and documentation for StorageFactory.newStorageFile(String path).
In the constructor of (StorageFactoryService), there is this block of code:
rootStorageFactory = getStorageFactoryInstance( true, null, null, null);
if( home != null)
{
StorageFile rootDir = rootStorageFactory.newStorageFile( null);
rootDir.mkdirs();
}
Now, according to the javadoc for StorageFactory.newStorageFile(String path), when the parameter (path) is null, we should return the database directory. But in the above code block, the parameter representing the database directory passed into the initialization of the storage factory, is null.
According to the documentation for StorageFactory.init(), when the database name parameter is null, we will only work with the home directory. So I believe the author of the above code block intended (rootDir) to represent the home directory in this case (i.e., when the database parameter is null). This behavior, however, is not specified in the documentation for StorageFactory.newStorageFile(String path).
I noticed this discrepancy during manual testing because I was getting (NullPointerException) at the call to rootDir.mkdirs() with my storage factory implementation. I worked around the problem by having my implementation of newStorageFile(String path) return the home directory when the internal database parameter is null, in the case when the input (path) parameter is null.
In any event, there appears to be a discrepancy between code and documentation, unless I have misinterpreted the code and/or the documentation.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (DERBY-646) In-memory backend storage support
Posted by Daniel John Debrunner <dj...@apache.org>.
David Van Couvering wrote:
> I guess it's not a regression or bug, so not a release note, but I
> didn't know how else to indicate that documentation was needed, given
> that it's a new feature...
A new Jira entry indicating documentation is needed. :-)
Dan.
Re: [jira] Commented: (DERBY-646) In-memory backend storage support
Posted by David Van Couvering <Da...@Sun.COM>.
I guess it's not a regression or bug, so not a release note, but I
didn't know how else to indicate that documentation was needed, given
that it's a new feature...
David
Daniel John Debrunner wrote:
> David Van Couvering wrote:
>
>> I see your point about new classes added to the public API needing
>> javadoc, that seems reasonable.
>>
>> By release, do you mean a Derby release.
>
> I missed an 'e'. You checked the 'Release Note Needed' flag, why would a
> release note be needed?
>
> Dan.
>
>
Re: [jira] Commented: (DERBY-646) In-memory backend storage support
Posted by Daniel John Debrunner <dj...@apache.org>.
David Van Couvering wrote:
> I see your point about new classes added to the public API needing
> javadoc, that seems reasonable.
>
> By release, do you mean a Derby release.
I missed an 'e'. You checked the 'Release Note Needed' flag, why would a
release note be needed?
Dan.
Re: [jira] Commented: (DERBY-646) In-memory backend storage support
Posted by David Van Couvering <Da...@Sun.COM>.
I see your point about new classes added to the public API needing
javadoc, that seems reasonable.
By release, do you mean a Derby release or an ICLA? Assuming the
latter, I thought the ICLA issue had been cleared up. The ICLA has been
on file for some time, search for Stephen Fitch in
http://people.apache.org/~jim/committers.html.
David
Daniel John Debrunner (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12426372 ]
>
> Daniel John Debrunner commented on DERBY-646:
> ---------------------------------------------
>
> Probably no new tests are required but some changes to ensure the feature is tested, such as the ability to run tests with this configuration and suites that use the configuration.
> As in my earlier comments, I don't think this patch is ready to be committed, my concern is the new classes added to the publi api, with no coments or documentation as to what they are for.
> Why would a release not be needed?
>
>> In-memory backend storage support
>> ---------------------------------
>>
>> Key: DERBY-646
>> URL: http://issues.apache.org/jira/browse/DERBY-646
>> Project: Derby
>> Issue Type: New Feature
>> Components: Store
>> Environment: All
>> Reporter: Stephen Fitch
>> Assigned To: Stephen Fitch
>> Attachments: svn.diff
>>
>>
>> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
>
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12426372 ]
Daniel John Debrunner commented on DERBY-646:
---------------------------------------------
Probably no new tests are required but some changes to ensure the feature is tested, such as the ability to run tests with this configuration and suites that use the configuration.
As in my earlier comments, I don't think this patch is ready to be committed, my concern is the new classes added to the publi api, with no coments or documentation as to what they are for.
Why would a release not be needed?
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assigned To: Stephen Fitch
> Attachments: svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Cheng Che Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Che Chen updated DERBY-646:
---------------------------------
Attachment: derby-646-20090222.diff
derby-646-20090222.stat
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "David Sims (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674339#action_12674339 ]
David Sims commented on DERBY-646:
----------------------------------
Thanks, Kristian.
In my situation, I would use Derby as a strictly embedded database, and I don't want my users to even be confused by the presence of a derby.log file, however empty it may be. The experience with my users tells me that it's best just to have absolutely nothing. I can deal with the usual SQLException errors that come along.
However you and the others might feel that should best be configured in Derby, I leave up to you all. I would just like to chime in and request that there be a way to configure things so that absolutely nothing is on disk, including derby.log -- however that might work out.
Thanks,
David
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681638#action_12681638 ]
Kristian Waagan commented on DERBY-646:
---------------------------------------
Cheng, thank you for your continued interest :)
Since the code branch is cut on the coming Monday, there isn't much more that will be done for the in-memory solution for the upcoming 10.5 release, except testing and stabilization. This leaves plenty of time for an opportunity if you want to contribute later on.
Regarding testing, I think that is a very good idea. I have already started extracting the Mog test from your patch, and it runs fine with the current code.
The unit tests aren't as easy to adopt, as they test somewhat different characteristics / functionality (for instance peek, and mark/reset) and rely partly on internal variables etc. However, it shouldn't be hard to use your work to write similar tests for the current implementation.
I'm sorry I'm charging on without waiting for more feedback from you and other interested parties. The reason is merely my wish to get something committed in time for 10.5. This will allow interested users to more easily test out the feature and give the community valuable feedback before the next feature release.
I'm restating my goal for the feature:
A basic, working in-memory storage back end that can be used in scenarios where durability isn't required;
o basic, it won't have functionality like save on shutdown / JVM exit, or automatic restore on boot.
o working, but not tested to the same degree as the rest of the Derby code.
o undocumented (in the user docs).
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Fix For: 10.5.0.0
>
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Dave Syer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12512711 ]
Dave Syer commented on DERBY-646:
---------------------------------
I would like to test this storage strategy, but I don't know how to set it up. I have compiled the code against 10.2.1.6, and now I want to obtain a connection (ideally through a DataSource). This is what I tried
System.setProperty("derby.subSubProtocol.memory",
"org.apache.derby.impl.io.MemoryStorageFactory");
EmbeddedDataSource ds = new EmbeddedDataSource();
ds.setDatabaseName("memory:test");
ds.setCreateDatabase("create");
Connection connection = ds.getConnection();
It fails with
java.sql.SQLException: Failed to start database 'memory:test', see the next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
...
There is no "next exception" in the stack trace.
I also tried:
System.setProperty("derby.subSubProtocol.memory",
"org.apache.derby.impl.io.MemoryStorageFactory");
DriverManager.registerDriver(new org.apache.derby.jdbc.EmbeddedDriver());
Connection connection = DriverManager.getConnection("jdbc:derby:memory:test")
which fails the same way.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Cheng Che Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674441#action_12674441 ]
Cheng Che Chen commented on DERBY-646:
--------------------------------------
I believe "derby.log" is Derby's execution log, and operations on that log would probably not be going through the database storage layer. A run-time option to disable execution logging should be a separate work item (please file a ticket) and not be included in the resolution of this issue, in my opinion.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Cheng Che Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Che Chen updated DERBY-646:
---------------------------------
Attachment: derby-646-20090222.diff
derby-646-20090222.stat
Hi folks,
Attached (derby-646-20090222.stat, derby-646-20090222.diff) is my implementation of the interfaces org.apache.derby.io.{StorageFile,StorageRandomAccessFile,WritableStorageFactory} such that all file system operations are emulated entirely in memory, producing no file or directory on disk.
A brief description of the content:
- java\engine\org\apache\derby\impl\io\memory
This directory contains the source files for the implementation of the database storage factory interfaces.
- java\testing\org\apache\derbyTesting\functionTests\tests\memoryIO
This directory contains the source files for the JUnit tests. It consists of three sets of fairly extensive unit tests of the basic building blocks of my implementation. They started out as five sets of stand-alone JUnit tests and were converted to conform to Derby's JUnit test framework. In the process of conversion, two sets of stand-alone tests were dropped because they use on-disk operations to verify the correctness of in-memory operations, and Derby's test framework does not allow arbitrary operations on disk, as far as I can tell. These unit tests run only in the embedded mode of Derby's test framework, because they make no DB connections.
In addition, there is one functional test consisting of a simple but non-trivial clustering application. It is non-trivial because clustering is an important application in many engineering and scientific disciplines; it is simple in that it computes clustering of numerical data in one dimensional space -- the simplest case. The application implements the same algorithm in both JAVA (computed by the JUnit test case object itself) and SQL (computed by Derby using the in-memory storage factory) and cross-checks the computed results, within some floating-point tolerance. The SQL portion of the application computes quite a few CREATE, INSERT, DELETE, SELECT, JOIN, GROUP BY, ORDER BY, DROP; and so should exercise the storage layer to a good extent. This functional test runs in both the embedded mode and the client mode of Derby's test framework.
The sub-sub-protocol associated with this in-memory storage factory is named "memory". So the connection string looks like "jdbc:derby:memory:MyTestDb" or "jdbc:derby://localhost:1527/memory:MyTestDb".
-----
HELP!!! I've gotten the tests compiled properly with "ant all". The compiled class files show up under "classes\org\apache\derbyTesting\functionTests\tests\memoryIO". But again I'm running into the problem of not knowing how to get them included in the "derbyTesting.jar" file.
Nor did I figure out how to have them automatically executed with "ant junit-all", "ant junitreport", etc. For now, this is what I do after "ant all" to execute the tests on Windows:
java -cp .\tools\java\junit.jar;.\classes junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.memoryIO._Suite
-----
Existing files modified by this patch:
- java\engine\org\apache\derby\impl\build.xml
tools\jar\extraDBMSclasses.properties
These two files were modified so that my implementation would be compiled and included in Derby's JAR files.
- java\engine\org\apache\derby\iapi\services\monitor\PersistentService.java
java\engine\org\apache\derby\impl\services\monitor\BaseMonitor.java
These two files were modified only because I did not feel like typing "-Dderby.subSubProtocol.memory=org.apache.derby.impl.io.memory.MemoryStorageFactory" for every run. I simply added a mapping from the "memory" service to the "org.apache.derby.impl.io.memory.MemoryStorageFactory" class in BaseMonitor.
- java\testing\build.xml
This file was modified so that my tests would be compiled into class files under "classes\org\apache\derbyTesting\functionTests\tests\memoryIO".
-----
Thanks.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Cheng Che Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676557#action_12676557 ]
Cheng Che Chen commented on DERBY-646:
--------------------------------------
I believe I have included the Apache license header in all the files and had checked the box granting rights to Apache when I submitted those files. According to the comment by Andrew McIntyre on 16/Mar/07 10:32 AM in this thread, that should be sufficient, no? To have an ICLA on file, I'll have to fill it out and mail it to Apache -- this will take some time. Let me know if this is truly required. Thanks.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Stefan Arentz (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12437075 ]
Stefan Arentz commented on DERBY-646:
-------------------------------------
Has anything happened with this patch? I don't see any comments from the original submitter.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assigned To: Stephen Fitch
> Attachments: svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Myrna van Lunteren (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682899#action_12682899 ]
Myrna van Lunteren commented on DERBY-646:
------------------------------------------
So the initial commits are in 10.5. How can we proceed with this?
New JIRA's for subsequent work?
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Fix For: 10.5.0.0
>
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Waagan updated DERBY-646:
----------------------------------
Affects Version/s: 10.5.0.0
Fix Version/s: 10.5.0.0
Indicating that the first iteration of this feature is targeted for 10.5.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Fix For: 10.5.0.0
>
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12673865#action_12673865 ]
Kristian Waagan commented on DERBY-646:
---------------------------------------
If your classes aren't referenced by any other classes used by Derby, they won't be included in the jar file(s).
Assuming you are setting properties to wire in your implementation, you probably need to add your "root class(es)" to 'tools/jar/extraDBMSclasses.properties'.
As you can see, there are many other classes in there. These are used as roots to determine which classes to include in the jars.
Hope this helps,
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674692#action_12674692 ]
Kathey Marsden commented on DERBY-646:
--------------------------------------
As Kristian mentioned, you can disable error logging by using derby.stream.error.field or derby.stream.error.method. See
http://db.apache.org/derby/docs/10.4/tuning/rtunproper35028.html
http://db.apache.org/derby/docs/10.4/tuning/rtunproper33027.html
So we probably don't need to file a separate issue for that.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Andrew McIntyre (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12412180 ]
Andrew McIntyre commented on DERBY-646:
---------------------------------------
The patch is attached to this issue as a diff file. You need to get a local copy of the Derby source, then download the patch from JIRA (generally, right-click -> save as... on the attachment 'svn.diff'), then patch your local copy using 'patch -p0 < svn.diff' or your IDE, if it supports patching. I know Eclipse will apply patches generated by CVS and SVN. You will then need to build a local copy of Derby with the patch applied to try out the patch.
For help checking out the Derby source code from Subversion, see: http://db.apache.org/derby/derby_downloads.html#Derby+source+code
For help building, see BUILDING.txt at the top of the source tree and/or http://svn.apache.org/repos/asf/db/derby/code/trunk/BUILDING.txt
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Stephen Fitch
> Attachments: svn.diff
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (DERBY-646) In-memory backend storage
support
Posted by "Cheng Che Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674106#action_12674106 ]
chatom edited comment on DERBY-646 at 2/16/09 10:38 PM:
----------------------------------------------------------------
Thanks for the tip on the "extraDBMSclasses.properties" file. I'm now able to build the JAR files properly with my classes.
At this point, I have JUnit test cases for the basic building blocks of my implementation. They run successfully stand-alone, but I have yet to figure out how to integrate them into Derby's test framework.
I have also done some manual functional testing of my in-memory storage factory using examples from the Derby tutorial (http://db.apache.org/derby/papers/DerbyTut/index.html).
I have not modified any existing Derby tests to use the new in-memory storage factory, as I have yet to figure out how Derby's test framework works.
All source files for this implementation are contained in a new sub-directory "java/engine/org/apache/derby/impl/io/memory". There is no change to any public API nor to any existing JAVA source file. One line was added to "java/engine/org/apache/derby/impl/build.xml", and one line was added to "tools/jar/extraDBMSclasses.properties", for the new source files to build.
If people are willing to have a look at what I have so far, then I will post the patch; otherwise, please wait until I have plugged things into the Derby test framework.
was (Author: chatom):
Thanks for the tip on the "extraDBMSclasses.properties" file. I'm now able to build the JAR files properly with my classes.
At this point, I have JUnit test cases for the basic building blocks of my implementation. They run successfully stand-alone, but I have yet to figure out how to integrate them into Derby's test framework.
I have also done some manual functional testing of my in-memory storage factory using examples from the Derby tutorial (http://db.apache.org/derby/papers/DerbyTut/index.html).
I have not modified any existing Derby tests to use the new in-memory storage factory, as I have yet to figure out how Derby's test framework works.
If people are willing to have a look at what I have so far, then I will post the patch; otherwise, please wait until I have plugged things into the Derby test framework.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Dag H. Wanvik" <Da...@Sun.COM>.
"Myrna van Lunteren (JIRA)" <ji...@apache.org> writes:
> Although this issue has had a number of votes, I have to say, it
> currently doesn't look like it'll make it for 10.3 - it appears
> there has not been any testing of the patch by prospecting users nor
> did anyone sign up to drive this to completion.
>
> If you are interested in this, feel free to give the patch a try and
> help driving this into the next release after 10.3.
There is also a recent contribution in the form of an uncommitted
patch, which may interest you:
http://issues.apache.org/jira/browse/DERBY-2798
"A new approach for main-memory database"
I have not looked at it yet, so I can't vouch for it. Apparently it is
still proof-of-concept code.
Dag
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Myrna van Lunteren (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12502804 ]
Myrna van Lunteren commented on DERBY-646:
------------------------------------------
Although this issue has had a number of votes, I have to say, it currently doesn't look like it'll make it for 10.3 - it appears there has not been any testing of the patch by prospecting users nor did anyone sign up to drive this to completion.
If you are interested in this, feel free to give the patch a try and help driving this into the next release after 10.3.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Cheng Che Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Che Chen updated DERBY-646:
---------------------------------
Attachment: (was: derby-646-20090222.stat)
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680180#action_12680180 ]
Kristian Waagan commented on DERBY-646:
---------------------------------------
I'm postponing the first commit until tomorrow, mostly because I've run out of time and want to write a overview of the code for the commit message.
As soon as the patch is in, users are welcome to test it out.
Note that the first revision won't have the features save on shutdown and restore on boot.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Rick Hillegas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676785#action_12676785 ]
Rick Hillegas commented on DERBY-646:
-------------------------------------
I also would like to see us deliver an in-memory implementation in 10.5. I recommend that we check in one of these implementations but treat it as experimental code, not supported by official user documentation. A writeup of how to use the feature can appear on our wiki. After we release 10.5, we can solicit user feedback and adjust the api accordingly. Then in 10.6 we can expose this feature as solid and official with supporting user documentation. I think that this feature is important enough that we should consider spinning a 10.6 release a couple months after 10.5 if our users provide fast feedback.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12678800#action_12678800 ]
Kristian Waagan commented on DERBY-646:
---------------------------------------
According to the 10.5 release page on the Wiki, the 10.5 branch will be
created 2008-03-16. This leaves very little time to work on the
in-memory back end. I'm describing my plan for this user-requested
feature below. Most of the elements of the plan originates from the
comments on this Jira issue.
** Patch status
The community has three patches available;
A) derby-646-20090222.diff
B) derby-646-2a-vfmem_first_rev.diff
C) svn.diff
Patch C is the original patch from Stephen Fitch. It was uploaded in
March 2006. It is writing the database log to disk. The general state is
not very well known, but I believe it is not quite "production ready"
int it's current form. The patch demonstrates the overall approach,
which is also used by the two other patches. I can't speak for A, but B
is more or less a re-implementation of C. This follows from the
interfaces that have to be implemented.
Stephen Fitch has a CLA on file.
Patch A is from Cheng Che Chen, which has written a solution of his own.
The solution passes basic ad-hoc testing, and has been successfully used
in several of the performance tests in the Derby code repository.
This solution seems to be the most feature rich one, and it also has a
functional test.
Cheng Che Chen has a CLA on file.
Patch B is from Kristian Waagan. The test has passed ad-hoc testing, and
has been successfully used in several of the performance tests in the
Derby code repository. The main difference from patch A, is that it
seems to perform better on CMT machines.
The solution is able to run most of suites.All without failures.
Kristian Waagan has a CLA on file.
** Plan
1) Commit patch B.
(reasons: CMT performance, size, readability)
2) Work on the committed code, addressing the TODOs.
3) Incorporate tests from patch A and some additional tests.
(additional tests: backup in-mem then restore/re-boot with directory
(default) storage engine, simple boot/create test)
4) Incorporate features from patch A.
(persist to disk on exit/shutdown, restore from disk on boot,
finalizer, more?)
5) Figure out what to do on OOME (if anything).
6) General testing and improvements.
7) Documentation.
I hope to address at least steps 1-3 before the branch is cut.
It is not clear to me whether an in-memory back end will be a documented
and supported feature of 10.5. I like the idea of providing it as-is
without any documentation in 10.5, and then solidify it in 10.6.
In the meantime, interested users can try it out and give us feedback,
and the community can continue to test and improve the feature.
Unless someone objects, I will commit the patch on Monday.
I would also like to invite those who plan to contribute actively on the
feature to let the community know. Feel free to flesh out the details of
the plan.
Finally, a great thank you to both Stephen Fitch and Cheng Che Chen for
contributing their work to the Derby community! I hope continued
contribution is of interest :)
I would be very happy if Cheng could spare some time to get some of his
more advanced features into the Derby code line.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Mike Perham (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12359484 ]
Mike Perham commented on DERBY-646:
-----------------------------------
I would love to see Derby in-memory only support for unit testing purposes. This would solve a major problem of ours.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Stephen Fitch
> Attachments: in-memory.tar, svn.diff
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (DERBY-646) In-memory backend storage
support
Posted by "Cheng Che Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675713#action_12675713 ]
chatom edited comment on DERBY-646 at 2/22/09 3:09 PM:
---------------------------------------------------------------
Hi folks,
Attached (derby-646-20090222.stat, derby-646-20090222.diff) is my implementation of the interfaces org.apache.derby.io.{StorageFile,StorageRandomAccessFile,WritableStorageFactory} such that all file system operations are emulated entirely in memory, producing no file or directory on disk.
A brief description of the content:
- java\engine\org\apache\derby\impl\io\memory
This directory contains the source files for the implementation of the database storage factory interfaces.
- java\testing\org\apache\derbyTesting\functionTests\tests\memoryIO
This directory contains the source files for the JUnit tests. It consists of three sets of fairly extensive unit tests of the basic building blocks of my implementation. They started out as five sets of stand-alone JUnit tests and were converted to conform to Derby's JUnit test framework. In the process of conversion, two sets of stand-alone tests were dropped because they use on-disk operations to verify the correctness of in-memory operations, and Derby's test framework does not allow arbitrary operations on disk, as far as I can tell. These unit tests run only in the embedded mode of Derby's test framework, because they make no DB connections.
In addition, there is one functional test consisting of a simple but non-trivial clustering application. It is non-trivial because clustering is an important application in many engineering and scientific disciplines; it is simple in that it computes clustering of numerical data in one dimensional space -- the simplest case. The application implements the same algorithm in both JAVA (computed by the JUnit test case object itself) and SQL (computed by Derby using the in-memory storage factory) and cross-checks the computed results, within some floating-point tolerance. The SQL portion of the application computes quite a few CREATE, INSERT, DELETE, SELECT, JOIN, GROUP BY, ORDER BY, DROP; and so should exercise the storage layer to a good extent. This functional test runs in both the embedded mode and the client mode of Derby's test framework.
The sub-sub-protocol associated with this in-memory storage factory is named "memory". So the connection string looks like "jdbc:derby:memory:MyTestDb" or "jdbc:derby://localhost:1527/memory:MyTestDb".
-----
HELP!!! I've gotten the tests compiled properly with "ant all". The compiled class files show up under "classes\org\apache\derbyTesting\functionTests\tests\memoryIO". But again I'm running into the problem of not knowing how to get them included in the "derbyTesting.jar" file.
Nor did I figure out how to have them automatically executed with "ant junit-all", "ant junitreport", etc. For now, this is what I do after "ant all" to execute the tests on Windows:
java -cp .\tools\java\junit.jar;.\classes junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.memoryIO._Suite
(BTW, when I execute the above command, the disk activity LED on my computer goes wild during the functional test. That doesn't happen with the stand-alone version of the same functional test (before conversion to run under Derby's test framework). Perhaps I have mis-configured something in the test object itself such that the above command does not actually test the in-memory storage factory?)
-----
Existing files modified by this patch:
- java\engine\org\apache\derby\impl\build.xml
tools\jar\extraDBMSclasses.properties
These two files were modified so that my implementation would be compiled and included in Derby's JAR files.
- java\engine\org\apache\derby\iapi\services\monitor\PersistentService.java
java\engine\org\apache\derby\impl\services\monitor\BaseMonitor.java
These two files were modified only because I did not feel like typing "-Dderby.subSubProtocol.memory=org.apache.derby.impl.io.memory.MemoryStorageFactory" for every run. I simply added a mapping from the "memory" service to the "org.apache.derby.impl.io.memory.MemoryStorageFactory" class in BaseMonitor.
- java\testing\build.xml
This file was modified so that my tests would be compiled into class files under "classes\org\apache\derbyTesting\functionTests\tests\memoryIO".
-----
Thanks.
was (Author: chatom):
Hi folks,
Attached (derby-646-20090222.stat, derby-646-20090222.diff) is my implementation of the interfaces org.apache.derby.io.{StorageFile,StorageRandomAccessFile,WritableStorageFactory} such that all file system operations are emulated entirely in memory, producing no file or directory on disk.
A brief description of the content:
- java\engine\org\apache\derby\impl\io\memory
This directory contains the source files for the implementation of the database storage factory interfaces.
- java\testing\org\apache\derbyTesting\functionTests\tests\memoryIO
This directory contains the source files for the JUnit tests. It consists of three sets of fairly extensive unit tests of the basic building blocks of my implementation. They started out as five sets of stand-alone JUnit tests and were converted to conform to Derby's JUnit test framework. In the process of conversion, two sets of stand-alone tests were dropped because they use on-disk operations to verify the correctness of in-memory operations, and Derby's test framework does not allow arbitrary operations on disk, as far as I can tell. These unit tests run only in the embedded mode of Derby's test framework, because they make no DB connections.
In addition, there is one functional test consisting of a simple but non-trivial clustering application. It is non-trivial because clustering is an important application in many engineering and scientific disciplines; it is simple in that it computes clustering of numerical data in one dimensional space -- the simplest case. The application implements the same algorithm in both JAVA (computed by the JUnit test case object itself) and SQL (computed by Derby using the in-memory storage factory) and cross-checks the computed results, within some floating-point tolerance. The SQL portion of the application computes quite a few CREATE, INSERT, DELETE, SELECT, JOIN, GROUP BY, ORDER BY, DROP; and so should exercise the storage layer to a good extent. This functional test runs in both the embedded mode and the client mode of Derby's test framework.
The sub-sub-protocol associated with this in-memory storage factory is named "memory". So the connection string looks like "jdbc:derby:memory:MyTestDb" or "jdbc:derby://localhost:1527/memory:MyTestDb".
-----
HELP!!! I've gotten the tests compiled properly with "ant all". The compiled class files show up under "classes\org\apache\derbyTesting\functionTests\tests\memoryIO". But again I'm running into the problem of not knowing how to get them included in the "derbyTesting.jar" file.
Nor did I figure out how to have them automatically executed with "ant junit-all", "ant junitreport", etc. For now, this is what I do after "ant all" to execute the tests on Windows:
java -cp .\tools\java\junit.jar;.\classes junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.memoryIO._Suite
-----
Existing files modified by this patch:
- java\engine\org\apache\derby\impl\build.xml
tools\jar\extraDBMSclasses.properties
These two files were modified so that my implementation would be compiled and included in Derby's JAR files.
- java\engine\org\apache\derby\iapi\services\monitor\PersistentService.java
java\engine\org\apache\derby\impl\services\monitor\BaseMonitor.java
These two files were modified only because I did not feel like typing "-Dderby.subSubProtocol.memory=org.apache.derby.impl.io.memory.MemoryStorageFactory" for every run. I simply added a mapping from the "memory" service to the "org.apache.derby.impl.io.memory.MemoryStorageFactory" class in BaseMonitor.
- java\testing\build.xml
This file was modified so that my tests would be compiled into class files under "classes\org\apache\derbyTesting\functionTests\tests\memoryIO".
-----
Thanks.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689121#action_12689121 ]
Kristian Waagan commented on DERBY-646:
---------------------------------------
Thanks for testing the new feature, Knut Magne :)
I'm currently working on a patch, where I'm considering using java.io.File. That should take care of any other OS that has a different style for paths as well.
Feel free to log a bug in Jira. If you don't, I will tomorrow.
Regarding memory footprint, you have to balance the memory overhead against performance. Setting the page cache to the minimum size will degrade performance in many configurations.
The worst case I saw in my simple testing, was a drop from 28361 (pageCacheSize=1000) tps to 4059 (pageCacheSize=40) tps. This was a single record update test using a page size of 32 K.
I'm not quite sure how you would go about discarding the log records, but you do need the UNDO log records to support transactions. Doing something in this area should be possible.
I don't know the log system well enough to say something more, but maybe someone else does?
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Dyre Tjeldvoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dyre Tjeldvoll updated DERBY-646:
---------------------------------
This issue seems to use either 'existing application impact' or 'release note needed' incorrectly, according to the description at
http://db.apache.org/derby/DerbyBugGuidelines.html#Set+appropriate+special+%22flags%22
I plan to remove the flags shortly unless there are additional comments.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Ashwin Jayaprakash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12514237 ]
Ashwin Jayaprakash commented on DERBY-646:
------------------------------------------
The Derby JAR file is sealed - and this means that all the Derby packages specified in the JAR file's Manifest are sealed and you cannot add a new Class that shares the Package names to the Classpath.
Since there are classes in org.apache.derby.impl.io Package in the Derby "sealed" JAR, you cannot append a Class like org.apache.derby.impl.io.PureMemStore that resides outside the JAR to the ClassPath. Unless I'm completely wrong, you will have to use the Source code and add your files there and then recreate the JAR file.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "V.Narayanan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676561#action_12676561 ]
V.Narayanan commented on DERBY-646:
-----------------------------------
can u please assign the issue to your name? Also for major contributions a icla is generally required and this looks like a major feature. my two paisa!
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Ashwin Jayaprakash (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12412140 ]
Ashwin Jayaprakash commented on DERBY-646:
------------------------------------------
Hi,
I would like to try this patch. But, I don't see any JAR or Java files attached. Could you pls tell me where I can download this patch?
Thanks,
Ashwin.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Stephen Fitch
> Attachments: svn.diff
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683063#action_12683063 ]
Kathey Marsden commented on DERBY-646:
--------------------------------------
I think that until this feature is documented and fully supported, it shouldn't be resolved and be included in the release notes. The separate Jiras could be subtasks of this one and then the parent issue resolved for 10.6. Just my opinion.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Fix For: 10.5.0.0
>
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Cheng Che Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674165#action_12674165 ]
Cheng Che Chen commented on DERBY-646:
--------------------------------------
Sometime back I played a bit with the original solution posted in this thread. I remember that it produced some files and directories on disk although it was supposed to be an in-memory storage factory. But if it works and suits the tasks at hand, we should let it be.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Francois Orsini (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681006#action_12681006 ]
Francois Orsini commented on DERBY-646:
---------------------------------------
+1. Having these (hooks) even as an experiment could be very useful while the in-memory feature is being tested.
I would probably wait after 10.5 because allowing for database deletion, but that is just me - just because there needs probably more time to investigate it and its possible side effects.
Maybe worth re-enforcing somewhere (notes if any) that the database is locked and its activity blocked while its estimated space usage is computed.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Fix For: 10.5.0.0
>
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Knut Magne Solem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689154#action_12689154 ]
Knut Magne Solem commented on DERBY-646:
----------------------------------------
You need the log for undo, but for committed transactions I can't see why you need to keep the log records for a non-durable in-memory database. It could be hard to do this because you are doing in-memory stuff on this level. I tried to just discard writing log records to virtualFiles, but rollback failed because it was flushing the log and reading undo records from the storage. So we need to keep some log in virtualFiles to get undo working :)
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=comments#action_12369719 ]
Daniel John Debrunner commented on DERBY-646:
---------------------------------------------
I think it would be great to have an in-memory db support like this and thanks for working on it Stephen,
but there are a few issues to resolve first.
We would need an ICLA on file at apache from Stephen Fitch before this could be committed into Derby.
A quick look at the patch showed a couple of extra classes in the org.apache.derby.io package which is the published api for
the virtual file system, classes such as VirtualFileData and VirtualFS. I can't tell what the purpose of these classes are since there
are no comments for them, thus it makes it impossible for anyone to implement them without guessing what the requirements are.
Also seemed to be no tests provided with the patch, so how would a reviewer know that this works or not?
Copyrights on all the files seem to have incorrect dates, and file names.
The wiki has information on what is required:
http://wiki.apache.org/db-derby/DerbyContributorChecklist
http://wiki.apache.org/db-derby/DerbyCommitProcess
http://wiki.apache.org/db-derby/PatchAdvice
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Stephen Fitch
> Attachments: svn.diff
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675052#action_12675052 ]
Kristian Waagan commented on DERBY-646:
---------------------------------------
Forgot to say, but if you want to spend time on some of the known weaker spots of the patch, search the diff for "TODO".
Also, I'm wondering if it would be wise or not to ditch the self-made path handling and rely on java.io.File instead.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (DERBY-646) In-memory backend storage support
Posted by "Stephen Fitch (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=all ]
Stephen Fitch reassigned DERBY-646:
-----------------------------------
Assign To: Stephen Fitch
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Stephen Fitch
> Attachments: in-memory.tar, svn.diff
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Mike Matrigali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Matrigali updated DERBY-646:
---------------------------------
As to how to proceed, first is there an icla for cheng's contribution?
Is it possible for either of these patches to run through the entire derby test suite, in in-memory mode?
how does either compare to "test mode" with the page cache set as big as expected in-memory
use?
Kristian, or any other committer, can you take a look at both implementations and comment
on any
strength's/weeknesses from both. As you suggest it would be nice if we could get a combined
agreed implementation from both.
If goal is really the fast approaching 10.5, then it would be nice to have some sort of upper level user documentation. How to use it, what are limits, is all functionality supported. What happens in case it runs out of memory, or internal limits.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=all ]
Kristian Waagan reassigned DERBY-646:
-------------------------------------
Assignee: (was: Kristian Waagan)
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Knut Magne Solem (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688889#action_12688889 ]
Knut Magne Solem commented on DERBY-646:
----------------------------------------
Good to see something is happening in this field! Out of curiosity I tried this out on Windows XP, but it failed in embedded mode when creating an in-memory database.
The error was this assert:
ERROR XJ001: Java exception: 'ASSERT FAILED serviceName = memory:C:\Documents and Settings\user\workspace\derby\MyDbTest;storageFactory.getCanonicalName() = C:\Documents and Settings\user\workspace\derby\MyDbTest: org.apache.derby.shared.common.sanity.AssertFailure'.
To make it work I altered line 310 in VFMemoryStorageFactory.normalizePath() to allow windows-style paths. I changed it to this: "} else if (dir.charAt(0) != getSeparator() && !dir.toUpperCase().matches("[a-zA-Z]{1}:{1}\\\\{1}.*")){"
Appart from that i worked great.
To reduce memory footprint I guess you can reduce the cache to a minimum and maybe discard flushed log records instead of writing them to memory?
Knut Magne,
(previous user knutmas)
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Andrew McIntyre (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12481708 ]
Andrew McIntyre commented on DERBY-646:
---------------------------------------
"Stephen Fitch is listed at http://people.apache.org/~jim/committers.html under "Unlisted CLAs". Is this enough to allow us to safely bring this code into the repository?"
The answer to this question is yes, sorry I missed it when it was posed several months ago. The 'Grant license to ASF' box was also checked when the original code was submitted, so anyone who wants to work on getting this code complete enough to be committed should feel free to do so.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674300#action_12674300 ]
Kristian Waagan commented on DERBY-646:
---------------------------------------
Thank you for the feedback, David.
> For what it's worth, in my situation, the option of having absolutely nothing written to disk is a huge benefit.
What do you want to happen with derby.log, which may contain critical information about errors happening in Derby?
Note that the file by default will be overwritten when you restart Derby, so it shouldn't eat up your disk space. Since very little is written to it under normal operation, it shouldn't degrade performance. If you fire up lots of instances in different directories (without setting derby.system.home), you do get 'derby.log' files lying around though.
If you want to discard the information, I think you have the possibility to do that yourself using one of the derby.stream.error properties. To make it even easier, we could add a stream class to the Derby engine jar file that simply discards the information (if specifying null causes the information to be printed to std err).
My personal choice would be to not discard error information by default though, and I think it should be a conscientious choice made by the user / developer, but feel free to disagree :)
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Ashwin Jayaprakash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12514277 ]
Ashwin Jayaprakash commented on DERBY-646:
------------------------------------------
You can un-JAR the file to a directory, paste your compiled Classes into that folder and use this "exploded" directory with the original classes and your changes in the Classpath instead of having the JAR in the ClassPath.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (DERBY-646) In-memory backend storage
support
Posted by "Cheng Che Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674106#action_12674106 ]
chatom edited comment on DERBY-646 at 2/16/09 10:29 PM:
----------------------------------------------------------------
Thanks for the tip on the "extraDBMSclasses.properties" file. I'm now able to build the JAR files properly with my classes.
At this point, I have JUnit test cases for the basic building blocks of my implementation. They run successfully stand-alone, but I have yet to figure out how to integrate them into Derby's test framework.
I have also done some manual functional testing of my in-memory storage factory using examples from the Derby tutorial (http://db.apache.org/derby/papers/DerbyTut/index.html).
I have not modified any existing Derby tests to use the new in-memory storage factory, as I have yet to figure out how Derby's test framework works.
If people are willing to have a look at what I have so far, then I will post the patch; otherwise, please wait until I have plugged things into the Derby test framework.
was (Author: chatom):
Thanks for the tip on the "extraDBMSclasses.properties" file. I'm now able to build the JAR files properly with my classes.
At this point, I have JUnit test cases for the basic building blocks of my implementation. They run successfully stand-alone, but I have yet to figure out how to integrate them into Derby's test framework.
I have also done some manual functional testing of my in-memory storage factory using examples from the Derby tutorial (http://db.apache.org/derby/papers/DerbyTut/index.html).
I have not modified any existing Derby tests to use the new in-memory storage factory, as I have yet to figure out how Derby's test framework works.
If people are willing to have a look at what I have so far, then I will post the patch; otherwise, please wait until I have plug things into the Derby test framework.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Cliff Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12502724 ]
Cliff Evans commented on DERBY-646:
-----------------------------------
I think people are under valuing this request as a useful tool in production code. I'm currently using H2 to implement a multi-threaded matching engine which takes a series of events and matches aggregated information held in an in-memory database in real time. The datasets never get too large but the matching can be quite complex. Being able to use the expressive power of SQL without the overhead of pesisting the information to disk makes the matching process very fast while providing all of the thread synchronisation required in a tried and tested package. My only problem with H2 is the granularity of its locking. Table or nothing. This causes blocking as the tables are updated which an in-memory, row-locked version of Derby would reduce considerably. We're talking manna from heaven here.
Are there any plans to include this in the main release?
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-646?page=all ]
Kristian Waagan reassigned DERBY-646:
-------------------------------------
Assignee: Kristian Waagan
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: http://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assigned To: Kristian Waagan
> Attachments: svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Waagan updated DERBY-646:
----------------------------------
Attachment: derby-646-3a-jmx_experiment.diff
Patch 3a is a JMX experiment (very poorly documented, not very well tested, contains bad code). It provides two JMX attributes and one operation; getDatabases, estimateSpaceUsage, and estimateSpaceUsage(String).
You can obtain a list of the databases stored in memory, get a space usage estimate for all the databases or for a single database.
Is this something we should consider adding?
We could also allow for deleting databases through JMX, but then we have to do authentication (plus authorization?).
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Fix For: 10.5.0.0
>
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Dave Syer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12514254 ]
Dave Syer commented on DERBY-646:
---------------------------------
Is there an "unsealed" derby.jar available? I guess I just lost interest in testing this very interesting looking feature. I hope someone else can do it.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Myrna van Lunteren (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Myrna van Lunteren updated DERBY-646:
-------------------------------------
Fix Version/s: (was: 10.5.0.0)
Following Kathey's recommendation, taking off fix in 10.5 - so this won't be in the 10.5 fixed bugs list. It's still on the list of new functionality. We need to somehow point out that this is the first stage, though. Is it ok to say it's 'experimental' or is that too scary?
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Dyre Tjeldvoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dyre Tjeldvoll updated DERBY-646:
---------------------------------
Derby Info: (was: [Release Note Needed])
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Cheng Che Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681623#action_12681623 ]
Cheng Che Chen commented on DERBY-646:
--------------------------------------
Kristian, you might want to adapt the functional test that I had submitted for your own in-memory storage factory. That way, you can have your changes automatically tested to some extent after every build.
If I understand correctly, you tested your initial implementation by manually changing the default storage factory to point to your implementation and then running the Derby unit tests.
As for helping out with developing this feature further, I'll see what I can do. In the coming weeks, I don't see much free time to devote to this. So perhaps for the release after the imminent upcoming one, there'll be an opportunity.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Affects Versions: 10.5.0.0
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Fix For: 10.5.0.0
>
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-2b-vfmem_first_rev.diff, derby-646-3a-jmx_experiment.diff, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676729#action_12676729 ]
Kathey Marsden commented on DERBY-646:
--------------------------------------
If you have access to a scanner, you can email your scanned ICLA to secretary@apache.org (see http://www.apache.org/licenses/#clas).
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Dave Syer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12513868 ]
Dave Syer commented on DERBY-646:
---------------------------------
I trie dit with 10.3.1.2 in case that was the problem. Had to patch the patch slightly to implement two methods that were unimplemented abstract (guessed the implementation, but did get far enough to test it).
Now I get a more helpful stack trace (but maybe this is another problem?):
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676272#action_12676272 ]
Kristian Waagan commented on DERBY-646:
---------------------------------------
We now have two candidate solutions. How do we proceed?
I would very much like to get an in-memory back end included in the upcoming
10.5 release. In my opinion this should be pretty safe, as almost no existing
code is modified and the user has to take an active step to use the in-memory
back end.
To start mapping the performance characteristics of the two solutions, I ran
some of the performance tests we have. Note that these are based on only a few
runs, and a run time of 60 seconds only (pluss 30 seconds warmup). The numbers
are average tps (transactions per second).
There is some variation in the tests, so where the numbers differ only by a very
small percentage, I think we can assume the performance is comparable for now.
Hardware #1: 32 hardware execution threads, 1200 MHz.
sr_update sr_select bank_tx
1 thread
memory 1963 3059 823
vfmem 2307 3753 878
8 threads
memory 7662 14321 2163
vfmem 12692 17695 2294
32 threads
memory 7357 13984 2076
vfmem 12219 16546 2144
Hardware #2: Dual core Opteron, 2.4 GHz
sr_update sr_select bank_tx
1 thread
memory 12257 20295 5123
vfmem 12288 20657 5189
4 threads
memory 15064 32117 6120
vfmem 15857 32629 6115
8 threads
memory 16682 31257 5767
vfmem 16357 31361 5775
Hardware #3: 2xDual core Opteron, 2.8 GHz
sr_update sr_select bank_tx
1 thread
memory 13548 22184 5805
vfmem 13705 21770 5765
4 threads
memory 29019 57861 10443
vfmem 29727 58053 10479
8 threads
memory 26348 46580 9372
vfmem 26676 46805 9347
16 threads
memory 24500 44249 8976
vfmem 24883 43832 9014
Hardware #2: Dual core Opteron, 2.4 GHz, sr_update, page cache size
40 1000 15000
1 thread
memory 9932 12234 19207
vfmem 10075 12223 18995
8 threads
memory 12898 16540 26393
vfmem 13148 16662 28017
Hardware #2: Dual core Opteron, 2.4 GHz, sr_update, page size
Page cache size | 40 | | 1000 |
4 KB 32 KB 4 KB 32 KB
1 thread
memory 9902 2902 12369 20168
vfmem 10342 3247 12059 19801
8 threads
memory 12926 3599 16523 27333
vfmem 13261 4059 16592 28361
As far as I can see from these simple performance tests, the only configuration where the two solutions differs significantly, is when running on a CMT (chip multi threading) machine. Of the loads I have tested, this is most clearly visible with update load.
I have not investigated this further.
Another thing one could look into, is the memory usage of the two solutions.
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (DERBY-646) In-memory backend storage support
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kristian Waagan reassigned DERBY-646:
-------------------------------------
Assignee: Kristian Waagan
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Assignee: Kristian Waagan
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-20090222.diff, derby-646-20090222.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, derby-646-performance_comparison_1a.txt, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-646) In-memory backend storage support
Posted by "Adam Wozniak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Wozniak updated DERBY-646:
-------------------------------
Comment: was deleted
> In-memory backend storage support
> ---------------------------------
>
> Key: DERBY-646
> URL: https://issues.apache.org/jira/browse/DERBY-646
> Project: Derby
> Issue Type: New Feature
> Components: Store
> Environment: All
> Reporter: Stephen Fitch
> Attachments: derby-646-1a-raw-compiles.diff, derby-646-1a-raw-compiles.stat, derby-646-2a-vfmem_first_rev.diff, derby-646-2a-vfmem_first_rev.stat, svn.diff
>
>
> To allow creation and modification of databases in-memory without requiring disk access or space to store the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.