[jira] [Created] (CARBONDATA-4187) Performance Issue with Materialized views - increased loading time due to full refresh

["Sushant Sammanwar (Jira)" <ji...@apache.org>]

Sushant Sammanwar created CARBONDATA-4187:
---------------------------------------------

             Summary: Performance Issue with Materialized views - increased loading time due to full refresh
                 Key: CARBONDATA-4187
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-4187
             Project: CarbonData
          Issue Type: Bug
          Components: core
    Affects Versions: 2.1.0
            Reporter: Sushant Sammanwar


Hi Team ,

We have been doing a POC by using Carbon 2.1.0 and created a wrapper code around carbon and deployed it as docker container.
Concurrent data loading is happening in many tables.
Our objective if get optimal performance for aggregated queries and using materialized views .
Our observation is after creating MVs data loading is slow and not able to keep-up the pace of incoming data .
Process is also consuming a lot of memory when MVs are created .
Data is received in continuous manner and MVs are refreshed which is resulting in increased load time.
Ideally MVs should only perform incremental refresh as it doesnot require to calculate old data again.
But it seems the full refresh is causing high memory usages and increased loading time.

Testing involved loading data without MVs for 6 hrs , then creating MVs and load data again for 4 hours.
Loading time with MVs increased there creating backlog of data ( loaded only 1/5 th no. of rows than expected).

Below are major bottlenecks observed :
1. High Memory consumption after creating MVs
2. MVs doing a full refresh

Please find attached details of testing with list of tables.
Below is definition of table :

create table if not exists fact_365_1_eutrancell_1 (ts timestamp, metric STRING, tags_id STRING, value DOUBLE, epoch bigint) partitioned by (ts2 timestamp) STORED AS carbondata TBLPROPERTIES ('SORT_COLUMNS'='metric')

Below is definition of MV :

create materialized view if not exists fact_365_1_eutrancell_1_hour as select tags_id ,metric,timeseries(ts,'hour') as ts,sum(value),avg(value),min(value),max(value) from fact_365_1_eutrancell_1 group by metric, tags_id, timeseries(ts,'hour')

Can you suggest why MV creation is slowing down the ingestion so much and what can be done to improve ?
Is there any way to have incremental refresh of MV - refresh only that hour for which we are loading the data ?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)