You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Abhilash L L <ab...@infoworks.io> on 2016/01/28 12:07:49 UTC
Hot swapping cube post build
We have a use case where we want to rebuild the cube with an updated data
set without downtime on requests
Lets say we have cube C1.
We get some new data and we rebuild the cube.
Lets call this C2 with the new data. (Assume no change to cube structure)
When C2 is building, we want C1 to be still serving requests.
Once C2 is done building, we hot swap C1 with C2
This way there is no downtime on the requests (even if it is, its very less)
Another problem is, since it has to use the same hive table names and
schema as C1, we can recreate the tables (external) pointing to the data
for C2
We cannot use the incremental cube data addition since as of now its hard
to figure out of the change set.
What is the best way to achieve this ?
Assumption:
Since we cannot have two cubes with same name under same project, we need
two different cubes.
Regards,
Abhilash
Re: Hot swapping cube post build
Posted by Luke Han <lu...@gmail.com>.
Hi Abhilash,
Any data model change will require rebuild so far.
Our practices is to clone existing cube metadata as new one and add
extra measure/dimension or make changes, then build it.
Once new one ready, disable old one.
Hope this will help you.
Thanks.
Luke
Best Regards!
---------------------
Luke Han
On Mon, Feb 1, 2016 at 4:21 PM, Abhilash L L <ab...@infoworks.io> wrote:
> Sure, we will try out refresh
>
>
> What do you suggest in case there are some changes in the schema / extra
> measures etc
>
> Regards,
> Abhilash
>
> On Sat, Jan 30, 2016 at 12:28 PM, Li Yang <li...@apache.org> wrote:
>
> > Assuming the cube definition does not change, all you need is "refresh"
> an
> > existing cube segment. The old cube segment will continue serving until
> the
> > new build is complete. No down time during the whole process.
> >
> > Try "refresh"
> >
> >
> >
> > On Friday, January 29, 2016, hongbin ma <ma...@apache.org> wrote:
> >
> > > have you ever checked out the "refresh" function for cubes?
> > >
> > > On Thu, Jan 28, 2016 at 7:07 PM, Abhilash L L <abhilash@infoworks.io
> > > <javascript:;>> wrote:
> > >
> > > > We have a use case where we want to rebuild the cube with an updated
> > data
> > > > set without downtime on requests
> > > >
> > > > Lets say we have cube C1.
> > > > We get some new data and we rebuild the cube.
> > > > Lets call this C2 with the new data. (Assume no change to cube
> > structure)
> > > >
> > > > When C2 is building, we want C1 to be still serving requests.
> > > > Once C2 is done building, we hot swap C1 with C2
> > > >
> > > > This way there is no downtime on the requests (even if it is, its
> very
> > > > less)
> > > >
> > > > Another problem is, since it has to use the same hive table names and
> > > > schema as C1, we can recreate the tables (external) pointing to the
> > data
> > > > for C2
> > > >
> > > > We cannot use the incremental cube data addition since as of now its
> > > hard
> > > > to figure out of the change set.
> > > >
> > > > What is the best way to achieve this ?
> > > >
> > > > Assumption:
> > > > Since we cannot have two cubes with same name under same project, we
> > need
> > > > two different cubes.
> > > >
> > > >
> > > > Regards,
> > > > Abhilash
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > *Bin Mahone | 马洪宾*
> > > Apache Kylin: http://kylin.io
> > > Github: https://github.com/binmahone
> > >
> >
>
Re: Hot swapping cube post build
Posted by Abhilash L L <ab...@infoworks.io>.
Sure, we will try out refresh
What do you suggest in case there are some changes in the schema / extra
measures etc
Regards,
Abhilash
On Sat, Jan 30, 2016 at 12:28 PM, Li Yang <li...@apache.org> wrote:
> Assuming the cube definition does not change, all you need is "refresh" an
> existing cube segment. The old cube segment will continue serving until the
> new build is complete. No down time during the whole process.
>
> Try "refresh"
>
>
>
> On Friday, January 29, 2016, hongbin ma <ma...@apache.org> wrote:
>
> > have you ever checked out the "refresh" function for cubes?
> >
> > On Thu, Jan 28, 2016 at 7:07 PM, Abhilash L L <abhilash@infoworks.io
> > <javascript:;>> wrote:
> >
> > > We have a use case where we want to rebuild the cube with an updated
> data
> > > set without downtime on requests
> > >
> > > Lets say we have cube C1.
> > > We get some new data and we rebuild the cube.
> > > Lets call this C2 with the new data. (Assume no change to cube
> structure)
> > >
> > > When C2 is building, we want C1 to be still serving requests.
> > > Once C2 is done building, we hot swap C1 with C2
> > >
> > > This way there is no downtime on the requests (even if it is, its very
> > > less)
> > >
> > > Another problem is, since it has to use the same hive table names and
> > > schema as C1, we can recreate the tables (external) pointing to the
> data
> > > for C2
> > >
> > > We cannot use the incremental cube data addition since as of now its
> > hard
> > > to figure out of the change set.
> > >
> > > What is the best way to achieve this ?
> > >
> > > Assumption:
> > > Since we cannot have two cubes with same name under same project, we
> need
> > > two different cubes.
> > >
> > >
> > > Regards,
> > > Abhilash
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > *Bin Mahone | 马洪宾*
> > Apache Kylin: http://kylin.io
> > Github: https://github.com/binmahone
> >
>
Re: Hot swapping cube post build
Posted by Li Yang <li...@apache.org>.
Assuming the cube definition does not change, all you need is "refresh" an
existing cube segment. The old cube segment will continue serving until the
new build is complete. No down time during the whole process.
Try "refresh"
On Friday, January 29, 2016, hongbin ma <ma...@apache.org> wrote:
> have you ever checked out the "refresh" function for cubes?
>
> On Thu, Jan 28, 2016 at 7:07 PM, Abhilash L L <abhilash@infoworks.io
> <javascript:;>> wrote:
>
> > We have a use case where we want to rebuild the cube with an updated data
> > set without downtime on requests
> >
> > Lets say we have cube C1.
> > We get some new data and we rebuild the cube.
> > Lets call this C2 with the new data. (Assume no change to cube structure)
> >
> > When C2 is building, we want C1 to be still serving requests.
> > Once C2 is done building, we hot swap C1 with C2
> >
> > This way there is no downtime on the requests (even if it is, its very
> > less)
> >
> > Another problem is, since it has to use the same hive table names and
> > schema as C1, we can recreate the tables (external) pointing to the data
> > for C2
> >
> > We cannot use the incremental cube data addition since as of now its
> hard
> > to figure out of the change set.
> >
> > What is the best way to achieve this ?
> >
> > Assumption:
> > Since we cannot have two cubes with same name under same project, we need
> > two different cubes.
> >
> >
> > Regards,
> > Abhilash
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>
Re: Hot swapping cube post build
Posted by hongbin ma <ma...@apache.org>.
have you ever checked out the "refresh" function for cubes?
On Thu, Jan 28, 2016 at 7:07 PM, Abhilash L L <ab...@infoworks.io> wrote:
> We have a use case where we want to rebuild the cube with an updated data
> set without downtime on requests
>
> Lets say we have cube C1.
> We get some new data and we rebuild the cube.
> Lets call this C2 with the new data. (Assume no change to cube structure)
>
> When C2 is building, we want C1 to be still serving requests.
> Once C2 is done building, we hot swap C1 with C2
>
> This way there is no downtime on the requests (even if it is, its very
> less)
>
> Another problem is, since it has to use the same hive table names and
> schema as C1, we can recreate the tables (external) pointing to the data
> for C2
>
> We cannot use the incremental cube data addition since as of now its hard
> to figure out of the change set.
>
> What is the best way to achieve this ?
>
> Assumption:
> Since we cannot have two cubes with same name under same project, we need
> two different cubes.
>
>
> Regards,
> Abhilash
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone