[jira] [Commented] (CONNECTORS-286) Get ManifoldCF to run on top of a key/value store like Voldemort, for potential massive scalability improvements and speed gains

["Karl Wright (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CONNECTORS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171178#comment-13171178 ] 

Karl Wright commented on CONNECTORS-286:
----------------------------------------

Work on a database java API, with a key/value back-end implementation, has been going on for some six weeks now and is nearly complete.  It may be a valuable first step to convert ManifoldCF to this layer, and create standard SQL database implementations of it as a replacement for the current ManifoldCF implementation layer, to see how that works.  The functionality needed by the database in order to participate is limited to the following:

- transaction support
- ability to read tables sequentially
- ability to read through a table index in order, with specific conditions on that index read

Everything else is synthesized by the Warthog API layer.  This should make the job of porting to different databases much much easier - if it works.

Code as it currently stands is in branches/CONNECTORS-286/warthog.

                
> Get ManifoldCF to run on top of a key/value store like Voldemort, for potential massive scalability improvements and speed gains
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CONNECTORS-286
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-286
>             Project: ManifoldCF
>          Issue Type: New Feature
>          Components: Framework core
>            Reporter: Karl Wright
>            Assignee: Karl Wright
>             Fix For: ManifoldCF next
>
>
> ManifoldCF's reliance on a relational database limits its throughput and scalability.  I am now convinced it is possible to build all the structures we need within a distributed key-value store like Voldemort, which has the nice side effect of permitting massive scaling.  I envision there will be several layers to this project, some of which may have broader utility in the open-source community at large:
> (1) An atomic serialization layer, which adds serialization capabilities to an non-transactional substrate;
> (2) A transaction layer, which uses atomic serialization to build a notion of light transactions;
> (3) A table and index layer, which defines SQL-like concepts of tables and btree indexes on top of the transaction layer, via a Java API;
> (4) A generic "database abstraction" layer, which is capable of representing both standard SQL databases as well as this NoSQL variant, so that ManifoldCF can support both models.
> This is obviously a major development task, and as such is not envisioned to be completed by the next standard release.  Work will indeed need to be done in a branch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira