You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by "Zhang, Zhong" <zz...@cardlytics.com> on 2016/01/04 18:24:13 UTC
configuration for HBase region count
Hi All,
Happy new year!
The following rows are copied from the default Kylin
configuration file:
# HBase min and max region count
kylin.hbase.region.count.min=1
kylin.hbase.region.count.min=500
I guess the 500 should be max?
When I built my first cube, I did not change anything
in the configuration file which means we set the
minimum region count twice. And the following is
one of the HTable I got:
HTable: KYLIN_O03ZWB4DK9
* Region Count: 979
* Size: 5.75 TB
* Start Time: 2011-12-31 00:00:00
* End Time: 2014-05-01 01:00:00
The region count is 979 which is definitely larger than
500. Does it mean that the setting actually did not work?
Best regards,
Zhong
Re: configuration for HBase region count
Posted by Li Yang <li...@apache.org>.
Or another possibility maybe you were at a lower version when the cube is
firstly built, then you upgraded to 1.1?
On Thu, Jan 7, 2016 at 3:42 PM, Li Yang <li...@apache.org> wrote:
> From code, it only makes sense that "kylin.hbase.region.count.max" was
> set to 1000 or higher in your env. Because the default hard coded max is
> 500.
>
> On Tue, Jan 5, 2016 at 10:52 PM, Zhang, Zhong <zz...@cardlytics.com>
> wrote:
>
>> Hi Li,
>>
>>
>>
>> It’s 1.1.1.
>>
>>
>>
>> Best regards,
>>
>> Zhong
>>
>>
>>
>> *From:* Li Yang [mailto:liyang@apache.org]
>> *Sent:* Tuesday, January 05, 2016 4:56 AM
>> *To:* user@kylin.apache.org
>> *Subject:* Re: configuration for HBase region count
>>
>>
>>
>> Or if the exact version of Kylin is provided, we can looked into the code
>> and deduce the wrong.
>>
>>
>>
>> On Tue, Jan 5, 2016 at 10:32 AM, ShaoFeng Shi <sh...@apache.org>
>> wrote:
>>
>> Zhong,
>>
>> I guess you're using Kylin v1.1; For the two "kylin.hbase.region.count.min"
>> in kylin.properties, that was a bug and was fixed in v1.2 release, please
>> check https://issues.apache.org/jira/browse/KYLIN-1098 Please manually
>> change the second to "kylin.hbase.region.count.max", or upgrade to
>> latest v1.2;
>>
>>
>>
>> Regarding the region count issue, it is based on estimation and the
>> accuracy would be impacted a little by compaction; but from 500 to 900 this
>> is abnormal; we need some logs to analysis, if you can find the reducer
>> stdout log of "Calculate HTable Region Splits" step, that would be
>> helpful; The log looks like:
>>
>>
>>
>> Log Type: stdout
>>
>> Log Length: 1067
>>
>> 3.406: [GC [PSYoungGen: 290816K->15832K(338944K)] 290816K->15912K(1112064K), 0.0331610 secs] [Times: user=0.19 sys=0.01, real=0.03 secs]
>>
>> 1 regions
>>
>> 1 GB per region
>>
>>
>>
>> Thanks;
>>
>>
>>
>>
>>
>> 2016-01-05 1:24 GMT+08:00 Zhang, Zhong <zz...@cardlytics.com>:
>>
>> Hi All,
>>
>>
>>
>> Happy new year!
>>
>>
>>
>> The following rows are copied from the default Kylin
>>
>> configuration file:
>>
>> # HBase min and max region count
>>
>> kylin.hbase.region.count.min=1
>>
>> kylin.hbase.region.count.min=500
>>
>> I guess the 500 should be max?
>>
>>
>>
>> When I built my first cube, I did not change anything
>>
>> in the configuration file which means we set the
>>
>> minimum region count twice. And the following is
>>
>> one of the HTable I got:
>>
>> *HTable:* KYLIN_O03ZWB4DK9
>>
>> - Region Count: 979
>> - Size: 5.75 TB
>> - Start Time: 2011-12-31 00:00:00
>> - End Time: 2014-05-01 01:00:00
>>
>> The region count is 979 which is definitely larger than
>>
>> 500. Does it mean that the setting actually did not work?
>>
>>
>>
>> Best regards,
>>
>> Zhong
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Best regards,
>>
>>
>>
>> Shaofeng Shi
>>
>>
>>
>>
>>
>
>
Re: configuration for HBase region count
Posted by Li Yang <li...@apache.org>.
>From code, it only makes sense that "kylin.hbase.region.count.max" was set
to 1000 or higher in your env. Because the default hard coded max is 500.
On Tue, Jan 5, 2016 at 10:52 PM, Zhang, Zhong <zz...@cardlytics.com> wrote:
> Hi Li,
>
>
>
> It’s 1.1.1.
>
>
>
> Best regards,
>
> Zhong
>
>
>
> *From:* Li Yang [mailto:liyang@apache.org]
> *Sent:* Tuesday, January 05, 2016 4:56 AM
> *To:* user@kylin.apache.org
> *Subject:* Re: configuration for HBase region count
>
>
>
> Or if the exact version of Kylin is provided, we can looked into the code
> and deduce the wrong.
>
>
>
> On Tue, Jan 5, 2016 at 10:32 AM, ShaoFeng Shi <sh...@apache.org>
> wrote:
>
> Zhong,
>
> I guess you're using Kylin v1.1; For the two "kylin.hbase.region.count.min"
> in kylin.properties, that was a bug and was fixed in v1.2 release, please
> check https://issues.apache.org/jira/browse/KYLIN-1098 Please manually
> change the second to "kylin.hbase.region.count.max", or upgrade to latest
> v1.2;
>
>
>
> Regarding the region count issue, it is based on estimation and the
> accuracy would be impacted a little by compaction; but from 500 to 900 this
> is abnormal; we need some logs to analysis, if you can find the reducer
> stdout log of "Calculate HTable Region Splits" step, that would be helpful;
> The log looks like:
>
>
>
> Log Type: stdout
>
> Log Length: 1067
>
> 3.406: [GC [PSYoungGen: 290816K->15832K(338944K)] 290816K->15912K(1112064K), 0.0331610 secs] [Times: user=0.19 sys=0.01, real=0.03 secs]
>
> 1 regions
>
> 1 GB per region
>
>
>
> Thanks;
>
>
>
>
>
> 2016-01-05 1:24 GMT+08:00 Zhang, Zhong <zz...@cardlytics.com>:
>
> Hi All,
>
>
>
> Happy new year!
>
>
>
> The following rows are copied from the default Kylin
>
> configuration file:
>
> # HBase min and max region count
>
> kylin.hbase.region.count.min=1
>
> kylin.hbase.region.count.min=500
>
> I guess the 500 should be max?
>
>
>
> When I built my first cube, I did not change anything
>
> in the configuration file which means we set the
>
> minimum region count twice. And the following is
>
> one of the HTable I got:
>
> *HTable:* KYLIN_O03ZWB4DK9
>
> - Region Count: 979
> - Size: 5.75 TB
> - Start Time: 2011-12-31 00:00:00
> - End Time: 2014-05-01 01:00:00
>
> The region count is 979 which is definitely larger than
>
> 500. Does it mean that the setting actually did not work?
>
>
>
> Best regards,
>
> Zhong
>
>
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi
>
>
>
>
>
RE: configuration for HBase region count
Posted by "Zhang, Zhong" <zz...@cardlytics.com>.
Hi Li,
It’s 1.1.1.
Best regards,
Zhong
From: Li Yang [mailto:liyang@apache.org]
Sent: Tuesday, January 05, 2016 4:56 AM
To: user@kylin.apache.org
Subject: Re: configuration for HBase region count
Or if the exact version of Kylin is provided, we can looked into the code and deduce the wrong.
On Tue, Jan 5, 2016 at 10:32 AM, ShaoFeng Shi <sh...@apache.org>> wrote:
Zhong,
I guess you're using Kylin v1.1; For the two "kylin.hbase.region.count.min" in kylin.properties, that was a bug and was fixed in v1.2 release, please check https://issues.apache.org/jira/browse/KYLIN-1098 Please manually change the second to "kylin.hbase.region.count.max", or upgrade to latest v1.2;
Regarding the region count issue, it is based on estimation and the accuracy would be impacted a little by compaction; but from 500 to 900 this is abnormal; we need some logs to analysis, if you can find the reducer stdout log of "Calculate HTable Region Splits" step, that would be helpful; The log looks like:
Log Type: stdout
Log Length: 1067
3.406: [GC [PSYoungGen: 290816K->15832K(338944K)] 290816K->15912K(1112064K), 0.0331610 secs] [Times: user=0.19 sys=0.01, real=0.03 secs]
1 regions
1 GB per region
Thanks;
2016-01-05 1:24 GMT+08:00 Zhang, Zhong <zz...@cardlytics.com>>:
Hi All,
Happy new year!
The following rows are copied from the default Kylin
configuration file:
# HBase min and max region count
kylin.hbase.region.count.min=1
kylin.hbase.region.count.min=500
I guess the 500 should be max?
When I built my first cube, I did not change anything
in the configuration file which means we set the
minimum region count twice. And the following is
one of the HTable I got:
HTable: KYLIN_O03ZWB4DK9
* Region Count: 979
* Size: 5.75 TB
* Start Time: 2011-12-31 00:00:00
* End Time: 2014-05-01 01:00:00
The region count is 979 which is definitely larger than
500. Does it mean that the setting actually did not work?
Best regards,
Zhong
--
Best regards,
Shaofeng Shi
Re: configuration for HBase region count
Posted by Li Yang <li...@apache.org>.
Or if the exact version of Kylin is provided, we can looked into the code
and deduce the wrong.
On Tue, Jan 5, 2016 at 10:32 AM, ShaoFeng Shi <sh...@apache.org>
wrote:
> Zhong,
> I guess you're using Kylin v1.1; For the two "kylin.hbase.region.count.min"
> in kylin.properties, that was a bug and was fixed in v1.2 release, please
> check https://issues.apache.org/jira/browse/KYLIN-1098 Please manually
> change the second to "kylin.hbase.region.count.max", or upgrade to latest
> v1.2;
>
> Regarding the region count issue, it is based on estimation and the
> accuracy would be impacted a little by compaction; but from 500 to 900 this
> is abnormal; we need some logs to analysis, if you can find the reducer
> stdout log of "Calculate HTable Region Splits" step, that would be helpful;
> The log looks like:
>
> Log Type: stdout
>
> Log Length: 1067
>
> 3.406: [GC [PSYoungGen: 290816K->15832K(338944K)] 290816K->15912K(1112064K), 0.0331610 secs] [Times: user=0.19 sys=0.01, real=0.03 secs]
> 1 regions
>
> 1 GB per region
>
> Thanks;
>
>
> 2016-01-05 1:24 GMT+08:00 Zhang, Zhong <zz...@cardlytics.com>:
>
>> Hi All,
>>
>>
>>
>> Happy new year!
>>
>>
>>
>> The following rows are copied from the default Kylin
>>
>> configuration file:
>>
>> # HBase min and max region count
>>
>> kylin.hbase.region.count.min=1
>>
>> kylin.hbase.region.count.min=500
>>
>> I guess the 500 should be max?
>>
>>
>>
>> When I built my first cube, I did not change anything
>>
>> in the configuration file which means we set the
>>
>> minimum region count twice. And the following is
>>
>> one of the HTable I got:
>>
>> *HTable:* KYLIN_O03ZWB4DK9
>>
>> - Region Count: 979
>> - Size: 5.75 TB
>> - Start Time: 2011-12-31 00:00:00
>> - End Time: 2014-05-01 01:00:00
>>
>> The region count is 979 which is definitely larger than
>>
>> 500. Does it mean that the setting actually did not work?
>>
>>
>>
>> Best regards,
>>
>> Zhong
>>
>>
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi
>
>
Re: configuration for HBase region count
Posted by ShaoFeng Shi <sh...@apache.org>.
Zhong,
I guess you're using Kylin v1.1; For the two "kylin.hbase.region.count.min"
in kylin.properties, that was a bug and was fixed in v1.2 release, please
check https://issues.apache.org/jira/browse/KYLIN-1098 Please manually
change the second to "kylin.hbase.region.count.max", or upgrade to latest
v1.2;
Regarding the region count issue, it is based on estimation and the
accuracy would be impacted a little by compaction; but from 500 to 900 this
is abnormal; we need some logs to analysis, if you can find the reducer
stdout log of "Calculate HTable Region Splits" step, that would be helpful;
The log looks like:
Log Type: stdout
Log Length: 1067
3.406: [GC [PSYoungGen: 290816K->15832K(338944K)]
290816K->15912K(1112064K), 0.0331610 secs] [Times: user=0.19 sys=0.01,
real=0.03 secs]
1 regions
1 GB per region
Thanks;
2016-01-05 1:24 GMT+08:00 Zhang, Zhong <zz...@cardlytics.com>:
> Hi All,
>
>
>
> Happy new year!
>
>
>
> The following rows are copied from the default Kylin
>
> configuration file:
>
> # HBase min and max region count
>
> kylin.hbase.region.count.min=1
>
> kylin.hbase.region.count.min=500
>
> I guess the 500 should be max?
>
>
>
> When I built my first cube, I did not change anything
>
> in the configuration file which means we set the
>
> minimum region count twice. And the following is
>
> one of the HTable I got:
>
> *HTable:* KYLIN_O03ZWB4DK9
>
> - Region Count: 979
> - Size: 5.75 TB
> - Start Time: 2011-12-31 00:00:00
> - End Time: 2014-05-01 01:00:00
>
> The region count is 979 which is definitely larger than
>
> 500. Does it mean that the setting actually did not work?
>
>
>
> Best regards,
>
> Zhong
>
>
>
--
Best regards,
Shaofeng Shi