You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Li Yang <li...@apache.org> on 2015/08/29 03:42:45 UTC
what is CubeDesc.retentionRange ?
Saw a strange field on CubeDesc called "retentionRange". It duplicates with
the same name property on CubeInstance, and got a comment saying -- "this
field will not be serialized ,can be deserialized to front javascript"
What is this for? I got a feeling something is not done in the correct way.
Re: what is CubeDesc.retentionRange ?
Posted by Luke Han <lu...@gmail.com>.
Thanks Jason.
More comments about CubeDesc and CubeInstance:
1. All human being inputs, relative to source data, data model, settings,
are go to CubeDesc, as metadata.
2. All machine (kylin application) generated runtime information, including
dictionary, segments, hbase htable, statistics, are go to CubeInstance, as
runtime metadata.
More further, the "copy" action will only need copy cube definition
persistent in CubeDesc and create new one with different UUID and Name.
Hope that make sense for these two different metadata types:)
Best Regards!
---------------------
Luke Han
On Tue, Sep 15, 2015 at 5:07 PM, Jian Zhong <he...@gmail.com> wrote:
> Hi all,
>
> As discussed, we have agreed to migrate "retentionRange" and "
> autoMergeTimeRanges" from CubeInstance to CubeDesc
>
> It's done now on both 1.x-staging and 2.x-staging.
>
> Please be informed.
>
> 1.x-staging commit:
>
> https://github.com/apache/incubator-kylin/commit/e0f5e3983b127649419cbe33c27e647b8e84029f
>
> 2.x-staging commit
>
> https://github.com/apache/incubator-kylin/commit/c083e874079c51a44a580946a7161083d0937d34
>
> On Sun, Sep 6, 2015 at 4:59 PM, Li Yang <li...@apache.org> wrote:
>
> > Have to agree with the dynamic/static separation. "retentionRange" and
> > "autoMergeTimeRanges" are indeed static info and thus belong to the type
> > not the instance. If CubeDesc grow too big, it can split inside.
> >
> > On Wed, Sep 2, 2015 at 9:51 PM, Luke Han <lu...@gmail.com> wrote:
> >
> > > IMO, retentionRange and autoMergeRanges are "metadata" just like
> > > "filter" conditions, it describe how a cube to pull and maintain a data
> > > range,
> > >
> > > As our design pattern, cube desc contains static information and cube
> > > instance
> > > contains runtime information, then I would like to say those two
> > properties
> > > should
> > > belong to cube desc.
> > >
> > > Thanks.
> > >
> > >
> > >
> > > Best Regards!
> > > ---------------------
> > >
> > > Luke Han
> > >
> > > On Wed, Sep 2, 2015 at 5:50 PM, hongbin ma <ma...@apache.org>
> wrote:
> > >
> > > > I'm still not sure it's a good design. Why do we choose to complicate
> > two
> > > > classes? There's no harm to add more configuration entries to cube
> > desc.
> > > >
> > > > The criteria to choose between cube instance and cube desc should be
> > > > dynamic/static informations, retentionRange and autoMergeTimeRanges
> > looks
> > > > like static informations.
> > > >
> > > > On Wed, Sep 2, 2015 at 5:40 PM, Li Yang <li...@apache.org> wrote:
> > > >
> > > > > A class should do what it designed to do *only*. For me, CubeDesc
> > > > defines a
> > > > > cube, including dimensions, measures, aggregation policies, and
> > that's
> > > > it.
> > > > > With these, it is already a complicated big class. I don't want it
> > grow
> > > > > further bigger.
> > > > >
> > > > > We can discuss certain metadata belongs to cube descriptor or not.
> > > > > "autoMergeTimeRanges"
> > > > > sounds somewhat qualified. "retentionRange" I'm not very sure. But
> in
> > > > > general, I hope things that are not very related go to
> CubeInstance.
> > > > >
> > > > >
> > > > > On Tue, Sep 1, 2015 at 7:23 PM, hongbin ma <ma...@apache.org>
> > > wrote:
> > > > >
> > > > > > I'm wondering why retentionRange has to be a property on
> > > cubeintance?
> > > > > > the same for autoMergeTimeRanges.
> > > > > >
> > > > > > It would be easier for metadata manipulation if all of such
> > > > > configurations
> > > > > > reside in cubedesc,
> > > > > > so will reduce our frontend efforts.
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > *Bin Mahone | 马洪宾*
> > > > > > Apache Kylin: http://kylin.io
> > > > > > Github: https://github.com/binmahone
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > *Bin Mahone | 马洪宾*
> > > > Apache Kylin: http://kylin.io
> > > > Github: https://github.com/binmahone
> > > >
> > >
> >
>
Re: what is CubeDesc.retentionRange ?
Posted by Jian Zhong <he...@gmail.com>.
Hi all,
As discussed, we have agreed to migrate "retentionRange" and "
autoMergeTimeRanges" from CubeInstance to CubeDesc
It's done now on both 1.x-staging and 2.x-staging.
Please be informed.
1.x-staging commit:
https://github.com/apache/incubator-kylin/commit/e0f5e3983b127649419cbe33c27e647b8e84029f
2.x-staging commit
https://github.com/apache/incubator-kylin/commit/c083e874079c51a44a580946a7161083d0937d34
On Sun, Sep 6, 2015 at 4:59 PM, Li Yang <li...@apache.org> wrote:
> Have to agree with the dynamic/static separation. "retentionRange" and
> "autoMergeTimeRanges" are indeed static info and thus belong to the type
> not the instance. If CubeDesc grow too big, it can split inside.
>
> On Wed, Sep 2, 2015 at 9:51 PM, Luke Han <lu...@gmail.com> wrote:
>
> > IMO, retentionRange and autoMergeRanges are "metadata" just like
> > "filter" conditions, it describe how a cube to pull and maintain a data
> > range,
> >
> > As our design pattern, cube desc contains static information and cube
> > instance
> > contains runtime information, then I would like to say those two
> properties
> > should
> > belong to cube desc.
> >
> > Thanks.
> >
> >
> >
> > Best Regards!
> > ---------------------
> >
> > Luke Han
> >
> > On Wed, Sep 2, 2015 at 5:50 PM, hongbin ma <ma...@apache.org> wrote:
> >
> > > I'm still not sure it's a good design. Why do we choose to complicate
> two
> > > classes? There's no harm to add more configuration entries to cube
> desc.
> > >
> > > The criteria to choose between cube instance and cube desc should be
> > > dynamic/static informations, retentionRange and autoMergeTimeRanges
> looks
> > > like static informations.
> > >
> > > On Wed, Sep 2, 2015 at 5:40 PM, Li Yang <li...@apache.org> wrote:
> > >
> > > > A class should do what it designed to do *only*. For me, CubeDesc
> > > defines a
> > > > cube, including dimensions, measures, aggregation policies, and
> that's
> > > it.
> > > > With these, it is already a complicated big class. I don't want it
> grow
> > > > further bigger.
> > > >
> > > > We can discuss certain metadata belongs to cube descriptor or not.
> > > > "autoMergeTimeRanges"
> > > > sounds somewhat qualified. "retentionRange" I'm not very sure. But in
> > > > general, I hope things that are not very related go to CubeInstance.
> > > >
> > > >
> > > > On Tue, Sep 1, 2015 at 7:23 PM, hongbin ma <ma...@apache.org>
> > wrote:
> > > >
> > > > > I'm wondering why retentionRange has to be a property on
> > cubeintance?
> > > > > the same for autoMergeTimeRanges.
> > > > >
> > > > > It would be easier for metadata manipulation if all of such
> > > > configurations
> > > > > reside in cubedesc,
> > > > > so will reduce our frontend efforts.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > *Bin Mahone | 马洪宾*
> > > > > Apache Kylin: http://kylin.io
> > > > > Github: https://github.com/binmahone
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > *Bin Mahone | 马洪宾*
> > > Apache Kylin: http://kylin.io
> > > Github: https://github.com/binmahone
> > >
> >
>
Re: what is CubeDesc.retentionRange ?
Posted by Li Yang <li...@apache.org>.
Have to agree with the dynamic/static separation. "retentionRange" and
"autoMergeTimeRanges" are indeed static info and thus belong to the type
not the instance. If CubeDesc grow too big, it can split inside.
On Wed, Sep 2, 2015 at 9:51 PM, Luke Han <lu...@gmail.com> wrote:
> IMO, retentionRange and autoMergeRanges are "metadata" just like
> "filter" conditions, it describe how a cube to pull and maintain a data
> range,
>
> As our design pattern, cube desc contains static information and cube
> instance
> contains runtime information, then I would like to say those two properties
> should
> belong to cube desc.
>
> Thanks.
>
>
>
> Best Regards!
> ---------------------
>
> Luke Han
>
> On Wed, Sep 2, 2015 at 5:50 PM, hongbin ma <ma...@apache.org> wrote:
>
> > I'm still not sure it's a good design. Why do we choose to complicate two
> > classes? There's no harm to add more configuration entries to cube desc.
> >
> > The criteria to choose between cube instance and cube desc should be
> > dynamic/static informations, retentionRange and autoMergeTimeRanges looks
> > like static informations.
> >
> > On Wed, Sep 2, 2015 at 5:40 PM, Li Yang <li...@apache.org> wrote:
> >
> > > A class should do what it designed to do *only*. For me, CubeDesc
> > defines a
> > > cube, including dimensions, measures, aggregation policies, and that's
> > it.
> > > With these, it is already a complicated big class. I don't want it grow
> > > further bigger.
> > >
> > > We can discuss certain metadata belongs to cube descriptor or not.
> > > "autoMergeTimeRanges"
> > > sounds somewhat qualified. "retentionRange" I'm not very sure. But in
> > > general, I hope things that are not very related go to CubeInstance.
> > >
> > >
> > > On Tue, Sep 1, 2015 at 7:23 PM, hongbin ma <ma...@apache.org>
> wrote:
> > >
> > > > I'm wondering why retentionRange has to be a property on
> cubeintance?
> > > > the same for autoMergeTimeRanges.
> > > >
> > > > It would be easier for metadata manipulation if all of such
> > > configurations
> > > > reside in cubedesc,
> > > > so will reduce our frontend efforts.
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > *Bin Mahone | 马洪宾*
> > > > Apache Kylin: http://kylin.io
> > > > Github: https://github.com/binmahone
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > *Bin Mahone | 马洪宾*
> > Apache Kylin: http://kylin.io
> > Github: https://github.com/binmahone
> >
>
Re: what is CubeDesc.retentionRange ?
Posted by Luke Han <lu...@gmail.com>.
IMO, retentionRange and autoMergeRanges are "metadata" just like
"filter" conditions, it describe how a cube to pull and maintain a data
range,
As our design pattern, cube desc contains static information and cube
instance
contains runtime information, then I would like to say those two properties
should
belong to cube desc.
Thanks.
Best Regards!
---------------------
Luke Han
On Wed, Sep 2, 2015 at 5:50 PM, hongbin ma <ma...@apache.org> wrote:
> I'm still not sure it's a good design. Why do we choose to complicate two
> classes? There's no harm to add more configuration entries to cube desc.
>
> The criteria to choose between cube instance and cube desc should be
> dynamic/static informations, retentionRange and autoMergeTimeRanges looks
> like static informations.
>
> On Wed, Sep 2, 2015 at 5:40 PM, Li Yang <li...@apache.org> wrote:
>
> > A class should do what it designed to do *only*. For me, CubeDesc
> defines a
> > cube, including dimensions, measures, aggregation policies, and that's
> it.
> > With these, it is already a complicated big class. I don't want it grow
> > further bigger.
> >
> > We can discuss certain metadata belongs to cube descriptor or not.
> > "autoMergeTimeRanges"
> > sounds somewhat qualified. "retentionRange" I'm not very sure. But in
> > general, I hope things that are not very related go to CubeInstance.
> >
> >
> > On Tue, Sep 1, 2015 at 7:23 PM, hongbin ma <ma...@apache.org> wrote:
> >
> > > I'm wondering why retentionRange has to be a property on cubeintance?
> > > the same for autoMergeTimeRanges.
> > >
> > > It would be easier for metadata manipulation if all of such
> > configurations
> > > reside in cubedesc,
> > > so will reduce our frontend efforts.
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > *Bin Mahone | 马洪宾*
> > > Apache Kylin: http://kylin.io
> > > Github: https://github.com/binmahone
> > >
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>
Re: what is CubeDesc.retentionRange ?
Posted by hongbin ma <ma...@apache.org>.
I'm still not sure it's a good design. Why do we choose to complicate two
classes? There's no harm to add more configuration entries to cube desc.
The criteria to choose between cube instance and cube desc should be
dynamic/static informations, retentionRange and autoMergeTimeRanges looks
like static informations.
On Wed, Sep 2, 2015 at 5:40 PM, Li Yang <li...@apache.org> wrote:
> A class should do what it designed to do *only*. For me, CubeDesc defines a
> cube, including dimensions, measures, aggregation policies, and that's it.
> With these, it is already a complicated big class. I don't want it grow
> further bigger.
>
> We can discuss certain metadata belongs to cube descriptor or not.
> "autoMergeTimeRanges"
> sounds somewhat qualified. "retentionRange" I'm not very sure. But in
> general, I hope things that are not very related go to CubeInstance.
>
>
> On Tue, Sep 1, 2015 at 7:23 PM, hongbin ma <ma...@apache.org> wrote:
>
> > I'm wondering why retentionRange has to be a property on cubeintance?
> > the same for autoMergeTimeRanges.
> >
> > It would be easier for metadata manipulation if all of such
> configurations
> > reside in cubedesc,
> > so will reduce our frontend efforts.
> >
> >
> >
> > --
> > Regards,
> >
> > *Bin Mahone | 马洪宾*
> > Apache Kylin: http://kylin.io
> > Github: https://github.com/binmahone
> >
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: what is CubeDesc.retentionRange ?
Posted by Li Yang <li...@apache.org>.
A class should do what it designed to do *only*. For me, CubeDesc defines a
cube, including dimensions, measures, aggregation policies, and that's it.
With these, it is already a complicated big class. I don't want it grow
further bigger.
We can discuss certain metadata belongs to cube descriptor or not.
"autoMergeTimeRanges"
sounds somewhat qualified. "retentionRange" I'm not very sure. But in
general, I hope things that are not very related go to CubeInstance.
On Tue, Sep 1, 2015 at 7:23 PM, hongbin ma <ma...@apache.org> wrote:
> I'm wondering why retentionRange has to be a property on cubeintance?
> the same for autoMergeTimeRanges.
>
> It would be easier for metadata manipulation if all of such configurations
> reside in cubedesc,
> so will reduce our frontend efforts.
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>
Re: what is CubeDesc.retentionRange ?
Posted by hongbin ma <ma...@apache.org>.
I'm wondering why retentionRange has to be a property on cubeintance?
the same for autoMergeTimeRanges.
It would be easier for metadata manipulation if all of such configurations
reside in cubedesc,
so will reduce our frontend efforts.
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: what is CubeDesc.retentionRange ?
Posted by Li Yang <li...@apache.org>.
I see... frontend should pass both CubeDesc and CubeInstance then.
Duplicate property on CubeDesc just to pass to frontend does not make sense.
On Mon, Aug 31, 2015 at 2:25 PM, Shi, Shaofeng <sh...@ebay.com> wrote:
> ³retentionRange" is an attribution defined on CubeInstance; While as Kylin
> cube wizard doesn¹t write CubeInstance directly (it only has access on
> CubeDesc); In order to allow user specify the value from front-end, this
> property was duplicated on CubeDesc; Yes you¹re right, as properties will
> be defined at CubeInstance level, the cube wizard need the capability to
> write CubeInstance directly; JIRA is created for it:
>
>
> https://issues.apache.org/jira/browse/KYLIN-981
>
>
> On 8/29/15, 9:42 AM, "Li Yang" <li...@apache.org> wrote:
>
> >Saw a strange field on CubeDesc called "retentionRange". It duplicates
> >with
> >the same name property on CubeInstance, and got a comment saying -- "this
> >field will not be serialized ,can be deserialized to front javascript"
> >
> >What is this for? I got a feeling something is not done in the correct
> >way.
>
>
Re: what is CubeDesc.retentionRange ?
Posted by "Shi, Shaofeng" <sh...@ebay.com>.
³retentionRange" is an attribution defined on CubeInstance; While as Kylin
cube wizard doesn¹t write CubeInstance directly (it only has access on
CubeDesc); In order to allow user specify the value from front-end, this
property was duplicated on CubeDesc; Yes you¹re right, as properties will
be defined at CubeInstance level, the cube wizard need the capability to
write CubeInstance directly; JIRA is created for it:
https://issues.apache.org/jira/browse/KYLIN-981
On 8/29/15, 9:42 AM, "Li Yang" <li...@apache.org> wrote:
>Saw a strange field on CubeDesc called "retentionRange". It duplicates
>with
>the same name property on CubeInstance, and got a comment saying -- "this
>field will not be serialized ,can be deserialized to front javascript"
>
>What is this for? I got a feeling something is not done in the correct
>way.