You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by "13802880779@139.com" <13...@139.com> on 2015/08/28 03:20:18 UTC
one question, how kylin store data in hbase?
for example，if i have 3 calculation item in cube, will kylin store each dimention of calculations in 3 rows or only 1 row(and split it when query)?
中国移动广东有限公司 网管中心 梁猛
13802880779@139.com
Re: one question, how kylin store data in hbase?
Posted by "Shi, Shaofeng" <sh...@ebay.com>.
hi meng, what do you mean by “calculation item”? I didn’t get the question;
On 8/28/15, 9:20 AM, "13802880779@139.com" <13...@139.com> wrote:
>for example，if i have 3 calculation item in cube, will kylin store each
>dimention of calculations in 3 rows or only 1 row(and split it when
>query)?
>
>
>
>
>中国移动广东有限公司 网管中心 梁猛
>13802880779@139.com
Re: one question, how kylin store data in hbase?
Posted by Sarnath <st...@gmail.com>.
Hi Steve,
Thanks for this email. This is precisely what I wanted to know earlier. I
only got vague answers. Your illustration really helped. Thanks a lot.
Best,
Sarnath
On Aug 28, 2015 8:03 AM, "steven zhang" <st...@outlook.com> wrote:
> Hi Meng,
>
> To my understanding:
>
> kylin using RowKey:=cuboid+dimensions RowValue:=measures(name+value)
>
> For example
>
> select class_id,sum(score) as total_score from student_score group by
> class_id
>
> 1.kylin will generate SQLDigest(include parameters) when execute sql
>
> 2.If the dimension index of class_id is 4 so the cuboid is 00010000 = 16
> ,cuboid length = 8 bytes “00100000" is the last byte and the rest of
> rowkey store the dimension value of class_id
>
> 3.How many rows is affected by numbers of dimensions . if dimensions
> numbers = 5 so the combination is 2^5 = 32
>
> 4.Hbase scan range (startRowKey and stopRowKey). if group by
> dimension1,dimension2 (dimension1 index=7 , dimension2 index=5) so
> startRowKey = 0000000 96 00 stopRowKey = 0000000 96 FF0
> and the measure value is after group by dimension1,dimension2
> (high—>low) . calculate the item such as SUM(Price),Min(Price) etc.
>
> Best regards
>
> Steven Zhang
>
>
> > On Aug 28, 2015, at 9:20 AM, 13802880779@139.com wrote:
> >
> > for example，if i have 3 calculation item in cube, will kylin store each
> dimention of calculations in 3 rows or only 1 row(and split it when query)?
> >
> >
> >
> >
> > 中国移动广东有限公司 网管中心 梁猛
> > 13802880779@139.com
>
>
Re: one question, how kylin store data in hbase?
Posted by Luke Han <lu...@gmail.com>.
Hi meng,
you could refer to this deck (P19) as a reference:
http://pan.baidu.com/disk/home#path=%252FEvents%252FStrata%252BHadoop%2520World%25202015%2520London
<http://pan.baidu.com/disk/home#path=%252FEvents%252FStrata%252BHadoop%2520World%25202015%2520London>
Thanks.
Best Regards!
---------------------
Luke Han
On Fri, Aug 28, 2015 at 11:38 AM, Shi, Shaofeng <sh...@ebay.com> wrote:
> In most of the cases it only uses 1 column to persist the measure values;
> But sometimes it will split to multiple columns; You can get the
> information in Cube JSON file, in “hbase_mapping” part;
>
>
> On 8/28/15, 11:17 AM, "steven zhang" <st...@outlook.com> wrote:
>
> >Hi Meng,
> >
> >kylin have not use multiple columns store multiple measures just only F1:M
> >so kylin using "rowkey:dim1+dim2, value:sum1+","+count2+",”+sum3” as you
> >list
> >
> >Best regards
> >
> >Steven Zhang
> >
> >> On Aug 28, 2015, at 10:59 AM, 13802880779@139.com wrote:
> >>
> >> sorry about my poor english, what i means is:
> >>
> >> example: dim1,dim2,sum1,count2,sum3
> >>
> >> will kylin store the aggr result like:
> >> rowkey:dim1+dim2,value:sum1
> >> rowkey:dim1+dim2,value:count2
> >> rowkey:dim1+dim2,value:sum3
> >>
> >> or:
> >> rowkey:dim1+dim2, value:sum1+","+count2+","+sum3
> >>
> >> the second way seem to save a lot of storage because it only store
> >>rowkey once;
> >>
> >>
> >>
> >> 中国移动广东有限公司 网管中心 梁猛
> >> 13802880779@139.com
> >>
> >> 发件人： steven zhang
> >> 发送时间： 2015-08-28 10:33
> >> 收件人： dev
> >> 主题： Re: one question, how kylin store data in hbase?
> >> Hi Meng,
> >>
> >> To my understanding:
> >>
> >> kylin using RowKey:=cuboid+dimensions RowValue:=measures(name+value)
> >>
> >> For example
> >> select class_id,sum(score) as total_score from student_score group by
> >>class_id
> >>
> >> 1.kylin will generate SQLDigest(include parameters) when execute sql
> >>
> >> 2.If the dimension index of class_id is 4 so the cuboid is 00010000 =
> >>16 ,cuboid length = 8 bytes “00100000" is the last byte and the rest of
> >>rowkey store the dimension value of class_id
> >>
> >> 3.How many rows is affected by numbers of dimensions . if dimensions
> >>numbers = 5 so the combination is 2^5 = 32
> >>
> >> 4.Hbase scan range (startRowKey and stopRowKey). if group by
> >>dimension1,dimension2 (dimension1 index=7 , dimension2 index=5) so
> >>startRowKey = 0000000 96 00 stopRowKey = 0000000 96 FF0
> >> and the measure value is after group by dimension1,dimension2
> >>(high—>low) . calculate the item such as SUM(Price),Min(Price) etc.
> >>
> >> Best regards
> >>
> >> Steven Zhang
> >>
> >>
> >>> On Aug 28, 2015, at 9:20 AM, 13802880779@139.com wrote:
> >>>
> >>> for example，if i have 3 calculation item in cube, will kylin store
> >>>each dimention of calculations in 3 rows or only 1 row(and split it
> >>>when query)?
> >>>
> >>>
> >>>
> >>>
> >>> 中国移动广东有限公司 网管中心 梁猛
> >>> 13802880779@139.com
> >>
> >
>
>
Re: one question, how kylin store data in hbase?
Posted by "Shi, Shaofeng" <sh...@ebay.com>.
In most of the cases it only uses 1 column to persist the measure values;
But sometimes it will split to multiple columns; You can get the
information in Cube JSON file, in “hbase_mapping” part;
On 8/28/15, 11:17 AM, "steven zhang" <st...@outlook.com> wrote:
>Hi Meng,
>
>kylin have not use multiple columns store multiple measures just only F1:M
>so kylin using "rowkey:dim1+dim2, value:sum1+","+count2+",”+sum3” as you
>list
>
>Best regards
>
>Steven Zhang
>
>> On Aug 28, 2015, at 10:59 AM, 13802880779@139.com wrote:
>>
>> sorry about my poor english, what i means is:
>>
>> example: dim1,dim2,sum1,count2,sum3
>>
>> will kylin store the aggr result like:
>> rowkey:dim1+dim2,value:sum1
>> rowkey:dim1+dim2,value:count2
>> rowkey:dim1+dim2,value:sum3
>>
>> or:
>> rowkey:dim1+dim2, value:sum1+","+count2+","+sum3
>>
>> the second way seem to save a lot of storage because it only store
>>rowkey once;
>>
>>
>>
>> 中国移动广东有限公司 网管中心 梁猛
>> 13802880779@139.com
>>
>> 发件人： steven zhang
>> 发送时间： 2015-08-28 10:33
>> 收件人： dev
>> 主题： Re: one question, how kylin store data in hbase?
>> Hi Meng,
>>
>> To my understanding:
>>
>> kylin using RowKey:=cuboid+dimensions RowValue:=measures(name+value)
>>
>> For example
>> select class_id,sum(score) as total_score from student_score group by
>>class_id
>>
>> 1.kylin will generate SQLDigest(include parameters) when execute sql
>>
>> 2.If the dimension index of class_id is 4 so the cuboid is 00010000 =
>>16 ,cuboid length = 8 bytes “00100000" is the last byte and the rest of
>>rowkey store the dimension value of class_id
>>
>> 3.How many rows is affected by numbers of dimensions . if dimensions
>>numbers = 5 so the combination is 2^5 = 32
>>
>> 4.Hbase scan range (startRowKey and stopRowKey). if group by
>>dimension1,dimension2 (dimension1 index=7 , dimension2 index=5) so
>>startRowKey = 0000000 96 00 stopRowKey = 0000000 96 FF0
>> and the measure value is after group by dimension1,dimension2
>>(high―>low) . calculate the item such as SUM(Price),Min(Price) etc.
>>
>> Best regards
>>
>> Steven Zhang
>>
>>
>>> On Aug 28, 2015, at 9:20 AM, 13802880779@139.com wrote:
>>>
>>> for example，if i have 3 calculation item in cube, will kylin store
>>>each dimention of calculations in 3 rows or only 1 row(and split it
>>>when query)?
>>>
>>>
>>>
>>>
>>> 中国移动广东有限公司 网管中心 梁猛
>>> 13802880779@139.com
>>
>
Re: one question, how kylin store data in hbase?
Posted by steven zhang <st...@outlook.com>.
Hi Meng,
kylin have not use multiple columns store multiple measures just only F1:M
so kylin using "rowkey:dim1+dim2, value:sum1+","+count2+",”+sum3” as you list
Best regards
Steven Zhang
> On Aug 28, 2015, at 10:59 AM, 13802880779@139.com wrote:
>
> sorry about my poor english, what i means is:
>
> example: dim1,dim2,sum1,count2,sum3
>
> will kylin store the aggr result like:
> rowkey:dim1+dim2,value:sum1
> rowkey:dim1+dim2,value:count2
> rowkey:dim1+dim2,value:sum3
>
> or:
> rowkey:dim1+dim2, value:sum1+","+count2+","+sum3
>
> the second way seem to save a lot of storage because it only store rowkey once;
>
>
>
> 中国移动广东有限公司 网管中心 梁猛
> 13802880779@139.com
>
> 发件人： steven zhang
> 发送时间： 2015-08-28 10:33
> 收件人： dev
> 主题： Re: one question, how kylin store data in hbase?
> Hi Meng,
>
> To my understanding:
>
> kylin using RowKey:=cuboid+dimensions RowValue:=measures(name+value)
>
> For example
> select class_id,sum(score) as total_score from student_score group by class_id
>
> 1.kylin will generate SQLDigest(include parameters) when execute sql
>
> 2.If the dimension index of class_id is 4 so the cuboid is 00010000 = 16 ,cuboid length = 8 bytes “00100000" is the last byte and the rest of rowkey store the dimension value of class_id
>
> 3.How many rows is affected by numbers of dimensions . if dimensions numbers = 5 so the combination is 2^5 = 32
>
> 4.Hbase scan range (startRowKey and stopRowKey). if group by dimension1,dimension2 (dimension1 index=7 , dimension2 index=5) so startRowKey = 0000000 96 00 stopRowKey = 0000000 96 FF0
> and the measure value is after group by dimension1,dimension2 (high—>low) . calculate the item such as SUM(Price),Min(Price) etc.
>
> Best regards
>
> Steven Zhang
>
>
>> On Aug 28, 2015, at 9:20 AM, 13802880779@139.com wrote:
>>
>> for example，if i have 3 calculation item in cube, will kylin store each dimention of calculations in 3 rows or only 1 row(and split it when query)?
>>
>>
>>
>>
>> 中国移动广东有限公司 网管中心 梁猛
>> 13802880779@139.com
>
回复: Re: one question, how kylin store data in hbase?
Posted by "13802880779@139.com" <13...@139.com>.
sorry about my poor english, what i means is:
example: dim1,dim2,sum1,count2,sum3
will kylin store the aggr result like:
rowkey:dim1+dim2,value:sum1
rowkey:dim1+dim2,value:count2
rowkey:dim1+dim2,value:sum3
or:
rowkey:dim1+dim2, value:sum1+","+count2+","+sum3
the second way seem to save a lot of storage because it only store rowkey once;
中国移动广东有限公司 网管中心 梁猛
13802880779@139.com
发件人： steven zhang
发送时间： 2015-08-28 10:33
收件人： dev
主题： Re: one question, how kylin store data in hbase?
Hi Meng,
To my understanding:
kylin using RowKey:=cuboid+dimensions RowValue:=measures(name+value)
For example
select class_id,sum(score) as total_score from student_score group by class_id
1.kylin will generate SQLDigest(include parameters) when execute sql
2.If the dimension index of class_id is 4 so the cuboid is 00010000 = 16 ,cuboid length = 8 bytes “00100000" is the last byte and the rest of rowkey store the dimension value of class_id
3.How many rows is affected by numbers of dimensions . if dimensions numbers = 5 so the combination is 2^5 = 32
4.Hbase scan range (startRowKey and stopRowKey). if group by dimension1,dimension2 (dimension1 index=7 , dimension2 index=5) so startRowKey = 0000000 96 00 stopRowKey = 0000000 96 FF0
and the measure value is after group by dimension1,dimension2 (high—>low) . calculate the item such as SUM(Price),Min(Price) etc.
Best regards
Steven Zhang
> On Aug 28, 2015, at 9:20 AM, 13802880779@139.com wrote:
>
> for example，if i have 3 calculation item in cube, will kylin store each dimention of calculations in 3 rows or only 1 row(and split it when query)?
>
>
>
>
> 中国移动广东有限公司 网管中心 梁猛
> 13802880779@139.com
Re: one question, how kylin store data in hbase?
Posted by steven zhang <st...@outlook.com>.
Hi Meng,
To my understanding:
kylin using RowKey:=cuboid+dimensions RowValue:=measures(name+value)
For example
select class_id,sum(score) as total_score from student_score group by class_id
1.kylin will generate SQLDigest(include parameters) when execute sql
2.If the dimension index of class_id is 4 so the cuboid is 00010000 = 16 ,cuboid length = 8 bytes “00100000" is the last byte and the rest of rowkey store the dimension value of class_id
3.How many rows is affected by numbers of dimensions . if dimensions numbers = 5 so the combination is 2^5 = 32
4.Hbase scan range (startRowKey and stopRowKey). if group by dimension1,dimension2 (dimension1 index=7 , dimension2 index=5) so startRowKey = 0000000 96 00 stopRowKey = 0000000 96 FF0
and the measure value is after group by dimension1,dimension2 (high—>low) . calculate the item such as SUM(Price),Min(Price) etc.
Best regards
Steven Zhang
> On Aug 28, 2015, at 9:20 AM, 13802880779@139.com wrote:
>
> for example，if i have 3 calculation item in cube, will kylin store each dimention of calculations in 3 rows or only 1 row(and split it when query)?
>
>
>
>
> 中国移动广东有限公司 网管中心 梁猛
> 13802880779@139.com