You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@carbondata.apache.org by "Ajith S (JIRA)" <ji...@apache.org> on 2018/08/04 02:57:00 UTC

[jira] [Created] (CARBONDATA-2824) Distributed CarbonStore

Ajith S created CARBONDATA-2824:
-----------------------------------

             Summary: Distributed CarbonStore
                 Key: CARBONDATA-2824
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-2824
             Project: CarbonData
          Issue Type: New Feature
            Reporter: Ajith S
            Assignee: Ajith S


Currently the CarbonStore is very tightly coupled with FileSystem interface and which runs in process JVM like in spark. We can instead make CarbonStore run as a separate service which can be accessed via network/rpc. So as a Followup of CARBONDATA-2688 (CarbonStore Java API and REST API) we can make carbon store distributed 

This has some advantages. 
1. Distributed CarbonStore can support parallel scanning i.e multiple tasks can start scanning data parallely, which may have a higher parallelism factor than compute layer 
2. Distributed CarbonStore can support index service to multiple apps like (spark/ flink/ presto), such that index will be shared to save resource 
3. Distributed CarbonStore  resource consumption is isolated from application and easily scalable to support higher workloads 
4. As a future improvement, Distributed CarbonStore  can implement a query cache since it has independent resources 

Distributed CarbonStore will have 2 main deployment parts: 
Cluster of remote carbon store service 
SDK which acts as a client for communication with store 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)