You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by "Zhang, Zhong" <zz...@cardlytics.com> on 2016/01/04 17:30:42 UTC
cut size for hbase region
Hi All,
Happy new year!
Kylin provides three options for cut size. Please see the following:
# The cut size for hbase region, in GB.
# E.g, for cube whose capacity be marked as "SMALL", split region per 10GB by default
kylin.hbase.region.cut.small=10
kylin.hbase.region.cut.medium=20
kylin.hbase.region.cut.large=100
I choose cube size as small to build the cube and the following is one of the HTable I got.
HTable: KYLIN_O03ZWB4DK9
* Region Count: 979
* Size: 5.75 TB
* Start Time: 2011-12-31 00:00:00
* End Time: 2014-05-01 01:00:00
So the size of the HTable is 5.75TB and there are 979 regions in total?
Let's do a little bit math. 979*10GB (since split region per 10GB when cube size is
marked as small) definitely does not equal 5.75TB. Do I understand correctly?
Best regards
Zhong
Re: cut size for hbase region
Posted by Li Yang <li...@apache.org>.
Given v1.1 code. Just from the code, the only guess I could make is the
"kylin.hbase.region.count.max" in kylin.properties is really more than 1000.
To confirm, we need to see the reducer log of step "Calculate HTable Region
Splits" if it's still there...
On Tue, Jan 5, 2016 at 1:38 PM, Zhang, Zhong <zz...@cardlytics.com> wrote:
> Hongbin,
>
> It's 1.1. Shaofeng pointed that it's a bug in the configuration in 1.1.
>
> Thanks,
> Zhong
>
> -----Original Message-----
> From: hongbin ma [mailto:mahongbin@apache.org]
> Sent: Monday, January 04, 2016 9:01 PM
> To: dev@kylin.apache.org
> Subject: Re: cut size for hbase region
>
> btw, what is the version you're using? in theory region count of a single
> segment will not exceed 500 by default, we want to check that
>
> On Tue, Jan 5, 2016 at 9:40 AM, hongbin ma <ma...@apache.org> wrote:
>
> > the cutting is based on estimation. due to hbase compression and
> > encoding, the estimation might be not very accurate. one recent ticket
> > on this is
> > https://issues.apache.org/jira/browse/KYLIN-1237
> >
> > On Tue, Jan 5, 2016 at 12:30 AM, Zhang, Zhong <zz...@cardlytics.com>
> > wrote:
> >
> >> Hi All,
> >>
> >> Happy new year!
> >>
> >> Kylin provides three options for cut size. Please see the following:
> >>
> >> # The cut size for hbase region, in GB.
> >> # E.g, for cube whose capacity be marked as "SMALL", split region per
> >> 10GB by default
> >> kylin.hbase.region.cut.small=10
> >> kylin.hbase.region.cut.medium=20
> >> kylin.hbase.region.cut.large=100
> >>
> >> I choose cube size as small to build the cube and the following is
> >> one of the HTable I got.
> >> HTable: KYLIN_O03ZWB4DK9
> >>
> >> * Region Count: 979
> >> * Size: 5.75 TB
> >> * Start Time: 2011-12-31 00:00:00
> >> * End Time: 2014-05-01 01:00:00
> >> So the size of the HTable is 5.75TB and there are 979 regions in total?
> >>
> >> Let's do a little bit math. 979*10GB (since split region per 10GB
> >> when cube size is marked as small) definitely does not equal 5.75TB.
> >> Do I understand correctly?
> >>
> >> Best regards
> >> Zhong
> >>
> >>
> >
> >
> > --
> > Regards,
> >
> > *Bin Mahone | 马洪宾*
> > Apache Kylin: http://kylin.io
> > Github: https://github.com/binmahone
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>
RE: cut size for hbase region
Posted by "Zhang, Zhong" <zz...@cardlytics.com>.
Hongbin,
It's 1.1. Shaofeng pointed that it's a bug in the configuration in 1.1.
Thanks,
Zhong
-----Original Message-----
From: hongbin ma [mailto:mahongbin@apache.org]
Sent: Monday, January 04, 2016 9:01 PM
To: dev@kylin.apache.org
Subject: Re: cut size for hbase region
btw, what is the version you're using? in theory region count of a single segment will not exceed 500 by default, we want to check that
On Tue, Jan 5, 2016 at 9:40 AM, hongbin ma <ma...@apache.org> wrote:
> the cutting is based on estimation. due to hbase compression and
> encoding, the estimation might be not very accurate. one recent ticket
> on this is
> https://issues.apache.org/jira/browse/KYLIN-1237
>
> On Tue, Jan 5, 2016 at 12:30 AM, Zhang, Zhong <zz...@cardlytics.com>
> wrote:
>
>> Hi All,
>>
>> Happy new year!
>>
>> Kylin provides three options for cut size. Please see the following:
>>
>> # The cut size for hbase region, in GB.
>> # E.g, for cube whose capacity be marked as "SMALL", split region per
>> 10GB by default
>> kylin.hbase.region.cut.small=10
>> kylin.hbase.region.cut.medium=20
>> kylin.hbase.region.cut.large=100
>>
>> I choose cube size as small to build the cube and the following is
>> one of the HTable I got.
>> HTable: KYLIN_O03ZWB4DK9
>>
>> * Region Count: 979
>> * Size: 5.75 TB
>> * Start Time: 2011-12-31 00:00:00
>> * End Time: 2014-05-01 01:00:00
>> So the size of the HTable is 5.75TB and there are 979 regions in total?
>>
>> Let's do a little bit math. 979*10GB (since split region per 10GB
>> when cube size is marked as small) definitely does not equal 5.75TB.
>> Do I understand correctly?
>>
>> Best regards
>> Zhong
>>
>>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: cut size for hbase region
Posted by hongbin ma <ma...@apache.org>.
btw, what is the version you're using? in theory region count of a single
segment will not exceed 500 by default, we want to check that
On Tue, Jan 5, 2016 at 9:40 AM, hongbin ma <ma...@apache.org> wrote:
> the cutting is based on estimation. due to hbase compression and encoding,
> the estimation might be not very accurate. one recent ticket on this is
> https://issues.apache.org/jira/browse/KYLIN-1237
>
> On Tue, Jan 5, 2016 at 12:30 AM, Zhang, Zhong <zz...@cardlytics.com>
> wrote:
>
>> Hi All,
>>
>> Happy new year!
>>
>> Kylin provides three options for cut size. Please see the following:
>>
>> # The cut size for hbase region, in GB.
>> # E.g, for cube whose capacity be marked as "SMALL", split region per
>> 10GB by default
>> kylin.hbase.region.cut.small=10
>> kylin.hbase.region.cut.medium=20
>> kylin.hbase.region.cut.large=100
>>
>> I choose cube size as small to build the cube and the following is one of
>> the HTable I got.
>> HTable: KYLIN_O03ZWB4DK9
>>
>> * Region Count: 979
>> * Size: 5.75 TB
>> * Start Time: 2011-12-31 00:00:00
>> * End Time: 2014-05-01 01:00:00
>> So the size of the HTable is 5.75TB and there are 979 regions in total?
>>
>> Let's do a little bit math. 979*10GB (since split region per 10GB when
>> cube size is
>> marked as small) definitely does not equal 5.75TB. Do I understand
>> correctly?
>>
>> Best regards
>> Zhong
>>
>>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: cut size for hbase region
Posted by hongbin ma <ma...@apache.org>.
the cutting is based on estimation. due to hbase compression and encoding,
the estimation might be not very accurate. one recent ticket on this is
https://issues.apache.org/jira/browse/KYLIN-1237
On Tue, Jan 5, 2016 at 12:30 AM, Zhang, Zhong <zz...@cardlytics.com> wrote:
> Hi All,
>
> Happy new year!
>
> Kylin provides three options for cut size. Please see the following:
>
> # The cut size for hbase region, in GB.
> # E.g, for cube whose capacity be marked as "SMALL", split region per 10GB
> by default
> kylin.hbase.region.cut.small=10
> kylin.hbase.region.cut.medium=20
> kylin.hbase.region.cut.large=100
>
> I choose cube size as small to build the cube and the following is one of
> the HTable I got.
> HTable: KYLIN_O03ZWB4DK9
>
> * Region Count: 979
> * Size: 5.75 TB
> * Start Time: 2011-12-31 00:00:00
> * End Time: 2014-05-01 01:00:00
> So the size of the HTable is 5.75TB and there are 979 regions in total?
>
> Let's do a little bit math. 979*10GB (since split region per 10GB when
> cube size is
> marked as small) definitely does not equal 5.75TB. Do I understand
> correctly?
>
> Best regards
> Zhong
>
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone