[jira] [Commented] (CONNECTORS-286) Get ManifoldCF to run on top of a key/value store like Voldemort, for potential massive scalability improvements and speed gains

["Karl Wright (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CONNECTORS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178695#comment-13178695 ] 

Karl Wright commented on CONNECTORS-286:
----------------------------------------

Using the Warthog API as the standard ManifoldCF way of dealing with databases may not be practical, for the following reasons.
- A significant amount of the actual functionality of Warthog comes from java methods you supply to it.  This is incompatible fundamentally with using a standard database to do the same thing, because there are bound to be situations where the two implementations disagree.
- A full database implementation under Warthog entails using the database for table storage and index access (ordered) with conditions applied to the index.  Warthog would do the rest.  But it is conceivable that this would not perform as well as native database queries.
- It is not clear how to construct a cache key in Warthog, so caching database results will require some thought.  Caching at the interface to the underlying database is not practical at all, because only partial resultsets will be read from many of the queries.
- It's not even clear (yet) whether critical functionality is missing from Warthog that will be needed to implement ManifoldCF.

Nevertheless, the next step is to try to create an implementation of Warthog where WHTableStore, WHTable, and WHIndex are implemented by an underlying relational database.  The difficulty in this, as stated above, occurs because the index (for example) is defined in terms of a WHComparator for each column being indexed, which is opaque Java code. Instead of merely performing the comparison, the code must, in addition, be in accordance with what the database is doing, AND also be capable of assisting in the generation of SQL code.  Special SQL-consistent WHComparator implementations are therefore going to be necessary, which also implement another interface (SQLInspectable?).  The WHIndex implementation can therefore use them to do what it needs, and complain if somebody tries to use incompatible comparator implementations.

Thus, each implementation of the Warthog API consists of:
- Implementations of WHTableStore and WHTable and WHIndex
- A body of comparators, filters, etc. that implement data types consistent with the SQL database
 

                
> Get ManifoldCF to run on top of a key/value store like Voldemort, for potential massive scalability improvements and speed gains
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CONNECTORS-286
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-286
>             Project: ManifoldCF
>          Issue Type: New Feature
>          Components: Framework core
>            Reporter: Karl Wright
>            Assignee: Karl Wright
>             Fix For: ManifoldCF next
>
>
> ManifoldCF's reliance on a relational database limits its throughput and scalability.  I am now convinced it is possible to build all the structures we need within a distributed key-value store like Voldemort, which has the nice side effect of permitting massive scaling.  I envision there will be several layers to this project, some of which may have broader utility in the open-source community at large:
> (1) An atomic serialization layer, which adds serialization capabilities to an non-transactional substrate;
> (2) A transaction layer, which uses atomic serialization to build a notion of light transactions;
> (3) A table and index layer, which defines SQL-like concepts of tables and btree indexes on top of the transaction layer, via a Java API;
> (4) A generic "database abstraction" layer, which is capable of representing both standard SQL databases as well as this NoSQL variant, so that ManifoldCF can support both models.
> This is obviously a major development task, and as such is not envisioned to be completed by the next standard release.  Work will indeed need to be done in a branch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira