[jira] [Commented] (CARBONDATA-3296) Support incremental dataload to datamap and other mv datamap enhancements

["Akash R Nilugal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CARBONDATA-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771755#comment-16771755 ] 

Akash R Nilugal commented on CARBONDATA-3296:
---------------------------------------------

can track the issue discussion at [http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-Support-Incremental-load-in-datamap-and-other-MV-datamap-enhancement-tt75160.html]

 

Design Document can be found at [https://docs.google.com/document/d/13XgEBUIqaAKdrlQftebr5BNOplL3u9qxuFe-IJUB3cM/edit#heading=h.h311u6t3pve9]

> Support incremental dataload to datamap and other mv datamap enhancements
> -------------------------------------------------------------------------
>
>                 Key: CARBONDATA-3296
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-3296
>             Project: CarbonData
>          Issue Type: Bug
>            Reporter: Akash R Nilugal
>            Priority: Major
>
> Currently in carbondata we have datamaps like preaggregate, lucene, bloom, 
> mv and we have 
> lazy and non-lazy methods to load data to datamaps. But lazy load is not 
> allowed for datamaps 
> like preagg, lucene, bloom.but, it is allowed for mv datamap. In lazy load 
> of mv datamap, for 
> every rebuild(load to datamap) we load the complete data of main table and 
> overwrite the existing 
> segment in datamap based on datamap query. 
> This is very costly in terms of performance and we also need to support the 
> lazy and non-lazy load 
> for all the datamaps. This can help in reduce the actual dataload time to 
> main table and whenever 
> user wants, he can do the lazy load for the datamaps present for that table. 
> Basically we need not overwrite the existing data every time we load to 
> datamap, we need to increment 
> the data in new segments similar to main table. This will help to get 
> better performance. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)