You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by Shreelekhya Gampa <sh...@gmail.com> on 2022/03/17 16:09:10 UTC
[DISCUSSION] Incremental Dataload of Average aggregate in MV
Hi everyone,
Currently, MV in carbon is supported as incremental or full refresh load
based on the type of query. Whenever MV is created with Average aggregate,
a full refresh is done meaning it reloads the whole MV for any newly added
segments. This will slow down the loading. With an incremental data load of
Average, only the segments that are newly added can be loaded to the MV.
With the Average function, incremental loading doesn't work directly as the
average of avg(col) in MV won't give correct results. To achieve
proper results we need the sum and count of the columns.
Following is the link to the design document. Please let me know
your thoughts about the same.
https://docs.google.com/document/d/1kPEMCX50FLZcmyzm6kcIQtUH9KXWDIqh-Hco7NkTp80/edit?usp=sharing
Thanks,
Shreelekhya Gampa
Re: [DISCUSSION] Incremental Dataloadof Average aggregate in MV
Posted by Shreelekhya Gampa <sh...@gmail.com>.
Thank you for your inputs.
I have handled the comments and updated the document accordingly.
On Wed, Mar 23, 2022 at 9:31 AM Kunal Kapoor <ku...@gmail.com>
wrote:
> Hi Shreelekhya,
> +1 for this feature,
>
> I could not understand how you would be able to handle compatibility, for
> the old tables the SQL doesn't have to be rewritten, while for new tables
> it has to be. So how will you decide this?
> Same scenario for old table load
>
> Also, Need to benchmark the impact on query performance for a large number
> of segments
>
> On Tue, Mar 22, 2022 at 10:28 AM Indhumathi <indhumuthumurugesh@apache.org
> >
> wrote:
>
> > Hello Shreelekhya,
> >
> > Please find the comments inline in the design document link shared.
> >
> > Regards,
> > Indhumathi M
> >
> > On 2022/03/17 16:09:10 Shreelekhya Gampa wrote:
> > > Hi everyone,
> > >
> > > Currently, MV in carbon is supported as incremental or full refresh
> load
> > > based on the type of query. Whenever MV is created with Average
> > aggregate,
> > > a full refresh is done meaning it reloads the whole MV for any newly
> > added
> > > segments. This will slow down the loading. With an incremental data
> load
> > of
> > > Average, only the segments that are newly added can be loaded to the
> MV.
> > > With the Average function, incremental loading doesn't work directly as
> > the
> > > average of avg(col) in MV won't give correct results. To achieve
> > > proper results we need the sum and count of the columns.
> > >
> > > Following is the link to the design document. Please let me know
> > > your thoughts about the same.
> > >
> >
> https://docs.google.com/document/d/1kPEMCX50FLZcmyzm6kcIQtUH9KXWDIqh-Hco7NkTp80/edit?usp=sharing
> > >
> > >
> > > Thanks,
> > > Shreelekhya Gampa
> > >
> >
>
Re: [DISCUSSION] Incremental Dataloadof Average aggregate in MV
Posted by Akash r <ak...@gmail.com>.
Hi Shreelekhya,
+1 for the feature. It was pending for a long time, thanks for working on
it.
I have added comments in the doc, please check
Regards,
Akash R Nilugal
On Wed, Mar 23, 2022 at 9:31 AM Kunal Kapoor <ku...@gmail.com>
wrote:
> Hi Shreelekhya,
> +1 for this feature,
>
> I could not understand how you would be able to handle compatibility, for
> the old tables the SQL doesn't have to be rewritten, while for new tables
> it has to be. So how will you decide this?
> Same scenario for old table load
>
> Also, Need to benchmark the impact on query performance for a large number
> of segments
>
> On Tue, Mar 22, 2022 at 10:28 AM Indhumathi <indhumuthumurugesh@apache.org
> >
> wrote:
>
> > Hello Shreelekhya,
> >
> > Please find the comments inline in the design document link shared.
> >
> > Regards,
> > Indhumathi M
> >
> > On 2022/03/17 16:09:10 Shreelekhya Gampa wrote:
> > > Hi everyone,
> > >
> > > Currently, MV in carbon is supported as incremental or full refresh
> load
> > > based on the type of query. Whenever MV is created with Average
> > aggregate,
> > > a full refresh is done meaning it reloads the whole MV for any newly
> > added
> > > segments. This will slow down the loading. With an incremental data
> load
> > of
> > > Average, only the segments that are newly added can be loaded to the
> MV.
> > > With the Average function, incremental loading doesn't work directly as
> > the
> > > average of avg(col) in MV won't give correct results. To achieve
> > > proper results we need the sum and count of the columns.
> > >
> > > Following is the link to the design document. Please let me know
> > > your thoughts about the same.
> > >
> >
> https://docs.google.com/document/d/1kPEMCX50FLZcmyzm6kcIQtUH9KXWDIqh-Hco7NkTp80/edit?usp=sharing
> > >
> > >
> > > Thanks,
> > > Shreelekhya Gampa
> > >
> >
>
Re: [DISCUSSION] Incremental Dataloadof Average aggregate in MV
Posted by Kunal Kapoor <ku...@gmail.com>.
Hi Shreelekhya,
+1 for this feature,
I could not understand how you would be able to handle compatibility, for
the old tables the SQL doesn't have to be rewritten, while for new tables
it has to be. So how will you decide this?
Same scenario for old table load
Also, Need to benchmark the impact on query performance for a large number
of segments
On Tue, Mar 22, 2022 at 10:28 AM Indhumathi <in...@apache.org>
wrote:
> Hello Shreelekhya,
>
> Please find the comments inline in the design document link shared.
>
> Regards,
> Indhumathi M
>
> On 2022/03/17 16:09:10 Shreelekhya Gampa wrote:
> > Hi everyone,
> >
> > Currently, MV in carbon is supported as incremental or full refresh load
> > based on the type of query. Whenever MV is created with Average
> aggregate,
> > a full refresh is done meaning it reloads the whole MV for any newly
> added
> > segments. This will slow down the loading. With an incremental data load
> of
> > Average, only the segments that are newly added can be loaded to the MV.
> > With the Average function, incremental loading doesn't work directly as
> the
> > average of avg(col) in MV won't give correct results. To achieve
> > proper results we need the sum and count of the columns.
> >
> > Following is the link to the design document. Please let me know
> > your thoughts about the same.
> >
> https://docs.google.com/document/d/1kPEMCX50FLZcmyzm6kcIQtUH9KXWDIqh-Hco7NkTp80/edit?usp=sharing
> >
> >
> > Thanks,
> > Shreelekhya Gampa
> >
>
Re: [DISCUSSION] Incremental Dataloadof Average aggregate in MV
Posted by Indhumathi <in...@apache.org>.
Hello Shreelekhya,
Please find the comments inline in the design document link shared.
Regards,
Indhumathi M
On 2022/03/17 16:09:10 Shreelekhya Gampa wrote:
> Hi everyone,
>
> Currently, MV in carbon is supported as incremental or full refresh load
> based on the type of query. Whenever MV is created with Average aggregate,
> a full refresh is done meaning it reloads the whole MV for any newly added
> segments. This will slow down the loading. With an incremental data load of
> Average, only the segments that are newly added can be loaded to the MV.
> With the Average function, incremental loading doesn't work directly as the
> average of avg(col) in MV won't give correct results. To achieve
> proper results we need the sum and count of the columns.
>
> Following is the link to the design document. Please let me know
> your thoughts about the same.
> https://docs.google.com/document/d/1kPEMCX50FLZcmyzm6kcIQtUH9KXWDIqh-Hco7NkTp80/edit?usp=sharing
>
>
> Thanks,
> Shreelekhya Gampa
>