[jira] [Commented] (S2GRAPH-1) Add Redis storage(Key/Value storage) engine for S2Graph

["Hyunsung Jo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/S2GRAPH-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15232029#comment-15232029 ] 

Hyunsung Jo commented on S2GRAPH-1:
-----------------------------------

The [Redis data model document | https://docs.google.com/presentation/d/1iAWApMInn4Ae8Y5P7D8Xkj9rjh_qJXObQHEVAMdqc9A/edit#slide=id.gf6049552c_0_0] is updated with the latest design changes.
 
Some of the key changes include:
* Snapshot Edge is stored as key value (opposed to sorted set) since it will not have multiple values.
* An Index Edge value consists of a Qualifier, non-indexed properties, and the length of Qualifier.
  - In HBase, A Qualifier of an Index Edge includes information such as index prop. values, target vertex ids, and op. codes. Since Redis is not a column-oriented storage, the same information is prepended to the value with an additional timestamp for default timestamp ordering. For lack of a better term, we call it Qualifier :).
  - Length of the Qualifier is appended at the end of the value.
  - Edge degree is handled by ZCARD.
* Vertex Edge updates are managed with an index field at the end of the value. 

We also have a couple of issues that should be discussed: 
* Table not supported: Originally, we considered keeping a virtual Table information in the key since Redis Cluster does not support multiple tables. However, performance issues when dropping all records in a table was a concern so decided not worry about Tables too much during POC stage.
* ```incrementCounts``` is not yet implemented: This operation is used exclusively by S2Counter. With the current Redis data model, incrementCounts seemed like too much complication. But, I think we can overcome this by utilizing the original S2Counter design which also used Redis.

> Add Redis storage(Key/Value storage) engine for S2Graph
> -------------------------------------------------------
>
>                 Key: S2GRAPH-1
>                 URL: https://issues.apache.org/jira/browse/S2GRAPH-1
>             Project: S2Graph
>          Issue Type: Task
>            Reporter: Junki Kim
>            Assignee: Junki Kim
>              Labels: features
>
> h2. Background
> The purpose of S2Graph is to support all features of edge/vertex operations  with HBase. It is also to orchestrate with HBase’s key features such as scale out / fault tolerant / highly available / etc. However, if there is no engineer to support stably in managing Hadoop cluster, it is very difficult to operate and maintain HBase and Hadoop clusters. Some companies, which already have another key/value storages to serve their services, may want to use S2Graph and also maintain their existing key/value storage environment. With S2Graph’s persistent storage, we would have a better support in operating another key/value storage. This is because S2Graph already has a strong feature, which can support multiple storage engines.
> What is best key/value storage engine starting to support S2Graph?
> Can companies maintain their existing key/value storage engine and use S2Graph at the same time? Yes, they can.
> Most of companies already have cache layers using Redis or Memcache. It is ok for the companies to start with one of these storages to support another key/value storage engine. It is because Redis is already suitable to support HA and fault tolerant features. Therefore, they do not need to replace this existing key/value storage engine.
> h2. Requirements
> - Support S2Graph storage engine’s interface
> -- Insert/Delete/Increment/etc.
> -- Query
> - Support serializer/deserializer to make edge/vertex data suit to key/value storage’s schema ( e.g. byte array encoding/decoding )
> - Support idempotent feature (this is a similar feature to S2Graph)
> - Design saving schema for edge(IndexEdge, SnapshotEdge) / vertex
> - Support query builder for S2Graph’s getEdges syntax
> - etc.
> I would like to invite everyone to openly discuss about this. I will also make a to-do list before I carry this issue in depth. Your feedback is appreciated :)
> P.S.
> I couldn't fill some information about this issue, e.g. affected version, fix version, component, etc. Please help me to fill out these mandatory fields. 
> In addition, this issue is umbrella issue for real subsequent tasks, so I would like to make this epic issue. Can anyone help me out? I am new to create apache jira issue :)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)