[jira] [Commented] (CARBONDATA-3296) Support incremental dataload to datamap and other mv datamap enhancements

["Akash R Nilugal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CARBONDATA-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16833601#comment-16833601 ] 

Akash R Nilugal commented on CARBONDATA-3296:
---------------------------------------------

Design document is updated at https://docs.google.com/document/d/1AACOYmBpwwNdHjJLOub0utSc6JCBMZn8VL5CvZ9hygA/edit

> Support incremental dataload to datamap and other mv datamap enhancements
> -------------------------------------------------------------------------
>
>                 Key: CARBONDATA-3296
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-3296
>             Project: CarbonData
>          Issue Type: Bug
>            Reporter: Akash R Nilugal
>            Priority: Major
>
> Currently in carbondata we have datamaps like preaggregate, lucene, bloom, 
> mv and we have 
> lazy and non-lazy methods to load data to datamaps. But lazy load is not 
> allowed for datamaps 
> like preagg, lucene, bloom.but, it is allowed for mv datamap. In lazy load 
> of mv datamap, for 
> every rebuild(load to datamap) we load the complete data of main table and 
> overwrite the existing 
> segment in datamap based on datamap query. 
> This is very costly in terms of performance and we also need to support the 
> lazy and non-lazy load 
> for all the datamaps. This can help in reduce the actual dataload time to 
> main table and whenever 
> user wants, he can do the lazy load for the datamaps present for that table. 
> Basically we need not overwrite the existing data every time we load to 
> datamap, we need to increment 
> the data in new segments similar to main table. This will help to get 
> better performance. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)