You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Dibyendu Majumdar (JIRA)" <ji...@apache.org> on 2008/01/24 23:58:43 UTC
[jira] Created: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Implement a Pluggable Storage Engine Architecture in Derby
----------------------------------------------------------
Key: DERBY-3351
URL: https://issues.apache.org/jira/browse/DERBY-3351
Project: Derby
Issue Type: New Feature
Components: Services, SQL, Store
Reporter: Dibyendu Majumdar
Assignee: Dibyendu Majumdar
My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could be the lead to a pluggble storage engine architecture similar to that of MySQL.
I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Francois Orsini (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628139#action_12628139 ]
Francois Orsini commented on DERBY-3351:
----------------------------------------
Adding the link below as a note as it would be great to have a pluggable storage cookbook for Derby.
http://forge.mysql.com/wiki/MySQL_Internals_Custom_Engine
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Dibyendu Majumdar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12563644#action_12563644 ]
Dibyendu Majumdar commented on DERBY-3351:
------------------------------------------
I was troubled by the fact that the Monitor implementation looks for org/apache/derby/modules.properties in the classpath, and the module.properties file that is found contains modules that are a mix from Language level and Store level modules. Therefore, by default, it will try to always load these. In a situation where the Store is replaced, or a different Store is used, this may not be desirable.
The references to the services.io package is problematic for similar reasons. A Store replacement may have its own method of managing serialization, so it would be better if the Monitor package was blind to the type of serialization mechanism used. One option is to have a common Class/Object registry that is used by both monitor and the io services, rather than creating a dependency between the two.
I have another question - the StorageFactoryService contains calls to java.io package as well as the storageFactory abstraction. Shouldn't all IO be delegated to storageFactory components? It seems that even if a memory based storageFactory is used, the StorageFactoryService will still require some file IO.
Thank you for answering my questions.
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Dibyendu Majumdar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12563990#action_12563990 ]
Dibyendu Majumdar commented on DERBY-3351:
------------------------------------------
Dan & Jørgen,
Many thanks for the info on how the monitor system works.
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12563875#action_12563875 ]
Jørgen Løland commented on DERBY-3351:
--------------------------------------
Hi Dabyendu,
I have recently added an alternative implementation (i.e. a new module) of the Database interface. The new module is called SlaveDatabase and can be found in trunk but not in the official derby releases yet. I'll use the Database service as an example to shed some light on how the monitor works.
There are now two implementations of the Database interface in modules.properties[1], and when the Monitor is asked to boot a database, it has to chose which of these modules to use. The Monitor knows from modules.properties that there are two possible implementations, and asks the first one (seemingly random order) if it should be used by calling it's canSupport method. The method call either returns true, in which case Monitor decides to go for that module, or it returns false which makes the Monitor repeat the process with the next module. Thus, by making the canSupport methods of the different modules of a service mutually exclusive, you have full control of which module is chosen by Monitor when it boots a service.
If you haven't read it already, I suggest you have a look at the class javadoc in org.apache.derby.iapi.services.monitor.Monitor.
Hope this helps.
[1]
derby.module.database=org.apache.derby.impl.db.BasicDatabase and
derby.module.database.slave=org.apache.derby.impl.db.SlaveDatabase
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Dibyendu Majumdar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12563307#action_12563307 ]
Dibyendu Majumdar commented on DERBY-3351:
------------------------------------------
> And the upshot of this is that today all the modules in the store are pluggable. :-) One can implement a different store under Derby today!
It makes my life easier that Derby was engineered to have this separation of layers.
Do you know if anyone has actually implemented a different Store for Derby? It would greatly help to know what issues they faced.
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Dibyendu Majumdar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12604637#action_12604637 ]
Dibyendu Majumdar commented on DERBY-3351:
------------------------------------------
Hi Rick,
I am still working on this, although I haven't had much progress for the past couple of months. I am changing my job at present and have been tied up because of that. Hope to get back to this very soon.
Thanks
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Dibyendu Majumdar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12562798#action_12562798 ]
Dibyendu Majumdar commented on DERBY-3351:
------------------------------------------
Hi Dan,
Thanks for the input. I agree that it is important to define the role of the Store at a high level.
In my view, the role of the Store should be to perform:
Transaction logging
Transaction Management
Table & Index creation & maintenance
Locking
Buffer Cache management
IO
Many of these modules can be reusable in their own right.
But the Store boundary still makes sense to me, because of following reasons:
Firstly there is a natural distinction between the language (SQL) and the engine (Store). I suppose this distinction must have driven the original design.
Secondly some of the Store modules are likely to be highly interdependent. For instance, it is possible to imagine a storage engine that supports multi-version concurrency. Such an engine would probably have a different implementation of the transaction manager and the lock manager. I think that making transaction management re-usable is particularly hard because there is so much dependency on it - how logging is done, how recovery is done, how locking is done.
Was the original intention that Store should be usable on its own right? That is the other side of the coin, in the sense that the Store boundary naturally maps to a non-SQL dbms.
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Dibyendu Majumdar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12563331#action_12563331 ]
Dibyendu Majumdar commented on DERBY-3351:
------------------------------------------
I am looking at the services.monitor package, and have a few questions:
1. The Monitor component appears to be a general purpose "IoC container". Yet, there is a hard-coded reference to a modules.properties file. This file appears to contain a number of module definitions, which are outside the services.monitor package.
Why is the rationale for a hard-coded reference to a module.properties file?
2. The Monitor class appears to depend upon functionality provided by services.io package, as it provides an interface to instantiate classes by format id.
This functionality seems to be unrelated to the purpose of monitor package, unless I am missing something here. Would it be better if knowledge of how to create instances was encapsulated in the services.io package, which is responsible for the serialization format and related functions?
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Daniel John Debrunner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12563597#action_12563597 ]
Daniel John Debrunner commented on DERBY-3351:
----------------------------------------------
The monitor is an IoC container, but it's probably too specific to Derby to be truly general purpose and probably more general purpose than Derby needs.
Modules.properties contains the list of modules that can be used by the system, since it's the input to the monitor and the monitor can handle any module, there is no requirement for the modules to be in the services.monitor package. The monitor is code that manages the lifetime of modules in a Derby system.
Not sure what issue you are seeing with modules.properties being hard-coded, it's the defined way that modules make themselves known to the system. There can actually be multiple modules.properties, each with the same name but a different location, typically as a resource in a jar file.
The mapping of format identifiers to instances may make more sense in the services.io package. The only issue would be ensuring that when the monitor (derby's system) is shutdown that any cached mapping of format identifier to class is cleaned up. That's the only portion that is actually happening in the monitor, and since the cache has the lifetime of the monitor it might make sense to leave it there.
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Daniel John Debrunner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12562769#action_12562769 ]
Daniel John Debrunner commented on DERBY-3351:
----------------------------------------------
Some thinking at a high level would be useful before diving into which packages are important.
What's the role of the store module? Currently row storage, transaction locking, transaction durability, ordering, transaction management & indexing (and maybe others) are all in the store code in derby. Is a different pluggable store going to provide all of that functionality, or would it be good to make some of the features re-useable across different stores?
E.g. transaction managerment should probably be pulled above the store.
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (DERBY-3351) Implement a Pluggable
Storage Engine Architecture in Derby
Posted by "Jørgen Løland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12563875#action_12563875 ]
jorgenlo edited comment on DERBY-3351 at 1/29/08 11:38 PM:
----------------------------------------------------------------
Hi Dibyendu,
I have recently added an alternative implementation (i.e. a new module) of the Database interface. The new module is called SlaveDatabase and can be found in trunk but not in the official derby releases yet. I'll use the Database service as an example to shed some light on how the monitor works.
There are now two implementations of the Database interface in modules.properties[1], and when the Monitor is asked to boot a database, it has to chose which of these modules to use. The Monitor knows from modules.properties that there are two possible implementations, and asks the first one (seemingly random order) if it should be used by calling it's canSupport method. The method call either returns true, in which case Monitor decides to go for that module, or it returns false which makes the Monitor repeat the process with the next module. Thus, by making the canSupport methods of the different modules of a service mutually exclusive, you have full control of which module is chosen by Monitor when it boots a service.
If you haven't read it already, I suggest you have a look at the class javadoc in org.apache.derby.iapi.services.monitor.Monitor.
Hope this helps.
[1]
derby.module.database=org.apache.derby.impl.db.BasicDatabase and
derby.module.database.slave=org.apache.derby.impl.db.SlaveDatabase
was (Author: jorgenlo):
Hi Dabyendu,
I have recently added an alternative implementation (i.e. a new module) of the Database interface. The new module is called SlaveDatabase and can be found in trunk but not in the official derby releases yet. I'll use the Database service as an example to shed some light on how the monitor works.
There are now two implementations of the Database interface in modules.properties[1], and when the Monitor is asked to boot a database, it has to chose which of these modules to use. The Monitor knows from modules.properties that there are two possible implementations, and asks the first one (seemingly random order) if it should be used by calling it's canSupport method. The method call either returns true, in which case Monitor decides to go for that module, or it returns false which makes the Monitor repeat the process with the next module. Thus, by making the canSupport methods of the different modules of a service mutually exclusive, you have full control of which module is chosen by Monitor when it boots a service.
If you haven't read it already, I suggest you have a look at the class javadoc in org.apache.derby.iapi.services.monitor.Monitor.
Hope this helps.
[1]
derby.module.database=org.apache.derby.impl.db.BasicDatabase and
derby.module.database.slave=org.apache.derby.impl.db.SlaveDatabase
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Dibyendu Majumdar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12562741#action_12562741 ]
Dibyendu Majumdar commented on DERBY-3351:
------------------------------------------
Regarding Storage Engine Boundary:
Obviously org.apache.derby.iapi.store.* and org.apache.derby.impl.store.* fall within the Store boundary.
There are a number of packages in org.apache.derby.iapi.services that the store depends upon.
Should these packages be part of the Store API - to be used by SQL and other layers?
I am thinking of io, context, monitor, sanity, locks, etc.
The dictionary is not referenced by the Store it seems, but the Store does know about Row types.
Should at least part of the dictionary be within the Store boundary?
Types should probably also be within the Store boundary.
I scanned the Store unit tests and extracted the import statements to see which packages are used.
Given below is the list.
org.apache.derby.catalog.UUID;
org.apache.derby.iapi.error.ExceptionSeverity;
org.apache.derby.iapi.error.StandardException;
org.apache.derby.iapi.reference.Attribute;
org.apache.derby.iapi.reference.EngineType;
org.apache.derby.iapi.reference.Property;
org.apache.derby.iapi.reference.SQLState;
org.apache.derby.iapi.services.context.ContextManager;
org.apache.derby.iapi.services.context.ContextService;
org.apache.derby.iapi.services.daemon.DaemonService;
org.apache.derby.iapi.services.i18n.MessageService;
org.apache.derby.iapi.services.io.DynamicByteArrayOutputStream;
org.apache.derby.iapi.services.io.FormatIdOutputStream;
org.apache.derby.iapi.services.io.FormatIdUtil;
org.apache.derby.iapi.services.io.Formatable;
org.apache.derby.iapi.services.io.FormatableBitSet;
org.apache.derby.iapi.services.io.LimitObjectInput;
org.apache.derby.iapi.services.io.Storable;
org.apache.derby.iapi.services.io.StoredFormatIds;
org.apache.derby.iapi.services.locks.*;
org.apache.derby.iapi.services.monitor.ModuleFactory;
org.apache.derby.iapi.services.monitor.Monitor;
org.apache.derby.iapi.services.property.PropertyUtil;
org.apache.derby.iapi.services.sanity.SanityManager;
org.apache.derby.iapi.services.stream.HeaderPrintWriter;
org.apache.derby.iapi.services.uuid.UUIDFactory;
org.apache.derby.iapi.store.access.*;
org.apache.derby.iapi.store.access.conglomerate.LogicalUndo;
org.apache.derby.iapi.store.access.conglomerate.TransactionManager;
org.apache.derby.iapi.store.access.xa.*;
org.apache.derby.iapi.store.raw.*;
org.apache.derby.iapi.store.raw.data.RawContainerHandle;
org.apache.derby.iapi.store.raw.log.LogInstant;
org.apache.derby.iapi.store.raw.xact.RawTransaction;
org.apache.derby.iapi.types.CloneableObject;
org.apache.derby.iapi.types.DataValueDescriptor;
org.apache.derby.iapi.types.Orderable;
org.apache.derby.iapi.types.RowLocation;
org.apache.derby.iapi.types.SQLChar;
org.apache.derby.iapi.types.SQLInteger;
org.apache.derby.iapi.types.SQLLongint;
org.apache.derby.iapi.types.StringDataValue;
org.apache.derby.iapi.util.ByteArray;
org.apache.derby.impl.store.access.btree.BTree;
org.apache.derby.impl.store.access.btree.index.*;
org.apache.derby.impl.store.access.conglomerate.*;
org.apache.derby.impl.store.access.conglomerate.TemplateRow;
org.apache.derby.impl.store.access.heap.*;
org.apache.derby.impl.store.raw.data.*;
org.apache.derby.impl.store.raw.log.*;
org.apache.derby.impl.store.raw.log.LogCounter;
org.apache.derby.io.StorageRandomAccessFile;
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Daniel John Debrunner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12562880#action_12562880 ]
Daniel John Debrunner commented on DERBY-3351:
----------------------------------------------
> Was the original intention that Store should be usable on its own right?
Yes.
And the upshot of this is that today all the modules in the store are pluggable. :-) One can implement a different store under Derby today!
The api is not as clean as one written from scratch (the current state has morphed from a clean api to a not-so clean one over 10+ years of development and the closer integration of the sql/store layer as it was realized that there was no demand for a stand alone store).
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Rick Hillegas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12604034#action_12604034 ]
Rick Hillegas commented on DERBY-3351:
--------------------------------------
Hello, Dibyendu. Are you still working on this project and, if so, do you have anything to share with the community or any further advice which we can provide? Thanks-Rick
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Dibyendu Majumdar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12563302#action_12563302 ]
Dibyendu Majumdar commented on DERBY-3351:
------------------------------------------
> I discussed this on the mailing last about a year ago, I'll look for the links.
I assume you are referring to DERBY-2490.
I think that in order to have a "clean API", we would need to look closely at dependencies that the Store has on the rest of Derby, as well as the dependencies that exist between the higher language layer and Store.
The transaction management area is obviously one of those areas - thank you for pointing that out.
At present, I am simply gathering information and trying to understand how best to go about with this, so any pointers are useful. It may well be that this overall task will end up having several sub-tasks, one of which will be DERBY-2490.
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Dibyendu Majumdar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12562759#action_12562759 ]
Dibyendu Majumdar commented on DERBY-3351:
------------------------------------------
Thinking more about it - packages in services probably shouldn't be in the Store boundary. But we need a way to ensure that the bits that the Store depends upon are separated somehow from the bits that are needed solely for the higher SQL layer. In other words, these bits will be come part of the contract between the Store and Derby - so that Derby will have to guarantee the presence of these APIs to Store.
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Daniel John Debrunner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12563648#action_12563648 ]
Daniel John Debrunner commented on DERBY-3351:
----------------------------------------------
> Therefore, by default, it will try to always load these. In a situation where the Store is replaced, or a different Store is used, this may not be desirable.
No, that's not a problem. The monitor doesn't just load modules from modules.properties and use them, it selects the module implementation that is suitable for the given environment by:
- seeing what the current JDK level is and if a module implementation supports it
- seeing what classes a module implementation requires and if they exist
- if the module implements ModuleSupportable and if so asking the module if it can support the current environment.
As an example, modules.properties today contains three JDBC implementations, JSR169, JDBC 3 and JDBC 4, having multiple exist is not an issue, the monitor selects the correct one.
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Dibyendu Majumdar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dibyendu Majumdar updated DERBY-3351:
-------------------------------------
Description:
My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
was:
My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could be the lead to a pluggble storage engine architecture similar to that of MySQL.
I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3351) Implement a Pluggable Storage Engine
Architecture in Derby
Posted by "Daniel John Debrunner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12562887#action_12562887 ]
Daniel John Debrunner commented on DERBY-3351:
----------------------------------------------
Just to add providing a clean pluggable api is good project, but potentially a very big project. It might be best to attack the various modules one at a time, as far as possible. E.g. a better transaction management system would be a good first start. Some o-o based that allowed clients (e.g. the jdbc and sql) layers to register events at commit and rollback time. Currently there is too much code that adds extra logic to the commit path (in multiple places). I discussed this on the mailing last about a year ago, I'll look for the links. The general idea would be:
- transaction management - keeping track of which transactions are active - may be independent of the store
- transaction durability - function of the store
- transaction api for sql & jdbc layer (event based callback scheme) -may be independent of the store (function of transaction management)
> Implement a Pluggable Storage Engine Architecture in Derby
> ----------------------------------------------------------
>
> Key: DERBY-3351
> URL: https://issues.apache.org/jira/browse/DERBY-3351
> Project: Derby
> Issue Type: New Feature
> Components: Services, SQL, Store
> Reporter: Dibyendu Majumdar
> Assignee: Dibyendu Majumdar
>
> My aim is to create a pluggable storage engine architecture for Derby, so that the default store implementation can be replaced with alternative storage engines. I have created my own storage engine which I would like to use with Derby's SQL layer, so that is a motivation. But I also think that this will benefit the community, and could lead to a pluggable storage engine architecture similar to that of MySQL.
> I am not yet sure where the storage engine boundary should lie. I would welcome input in this area.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.