[GitHub] [carbondata] kumarvishal09 commented on issue #3179: [CARBONDATA-3338] Support Incremental DataLoad for MV Datamap[with single parent table]

[GitBox <gi...@apache.org>]

kumarvishal09 commented on issue #3179: [CARBONDATA-3338] Support Incremental DataLoad for MV Datamap[with single parent table]
URL: https://github.com/apache/carbondata/pull/3179#issuecomment-489635271
 
 
   
   
   
   
   
   @Indhumathi27 
   After from few minor comments LGTM
   
   I have few optmization which we can consider in future pr's
   
   1. In case of below scenario we can avoid reloading the MV
   Maintable segments:0,1,2
   MV: 0 => 0,1,2
   Now after maintable compaction it is reload the 0.1 segment of maintable to MV, this we can avoid by changing the mapping {0,1,2}=>{0.1}
   
   2. Suppose I have mv select user, sum(column1)......from <maintable>
   and each segment of mv has same user and number of records is around 10M per segments in MV
   So if we are compacting 10 segments it will have 100M records, in this case we can fire self query on MV and aggregate the records , so after compaction it will reduce the records to 10M and it will improve the query performance.
   Some of the aggregator we have to change like count to sum during compaction
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services