You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by ro...@stratebi.com on 2018/03/07 14:50:03 UTC
Kylin, EMR and HBase timeout
Hi,
We have a Kylin 2.2 installation that uses an EMR cluster, so it uses HBase
on S3 as storage. It works well usually, but sometimes building process get
error at step Build Dimension Dictionary, as
hbase.ipc.callTimeoutException.
In order to solve it, I changed rpc configuration as indicated in
http://kylin.apache.org/docs23/install/kylin_aws_emr.html . I have changed,
the following properties on hbase-site.xml for master and all slaves.
<property>
<name>hbase.rpc.timeout</name>
<value>3600000</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>3600000</value>
</property>
Then I restarted HBase Master and HBase Regions Servers. I checked
However I still, getting the same error. Looks like that Kylin uses
different value for these properties.
I also changed the following kylin.properties:
kylin.metadata.hbase-rpc-timeout=3600000
kylin.metadata.hbase-client-scanner-timeout-period=3600000
I restarted Kyling, but still getting same issue.
Someone could help me?
Thanks in advance!
Roberto Tardío Olmos
Senior Big Data & Business Intelligence Consultant
Avenida de Brasil, 17, Planta 16.28020 Madrid
Fijo: 91.788.34.10
http://bigdata.stratebi.com/
http://www.stratebi.com
Re: Kylin, EMR and HBase timeout
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Roberto, you're welcome; and thanks for the update!
2018-03-22 6:41 GMT+08:00 <ro...@stratebi.com>:
> Hi ShaoFeng,
>
>
>
> We realized that the change of these parameters has no effect until
> version 2.3, because we use the 2.2.
>
>
>
> Although later we intend to update to 2.3, we have solved compiling
> version 2.2 with a cherry pick of the code of 2.3 so that the parameters
> are established according to the values of Kylin properties.
>
>
>
> https://github.com/apache/kylin/commit/8e8d71b99a340831512f7ca34cbc00
> 98a190f1f6
>
>
>
> Therefore we have solved these timeout issues using Kylin with HBase on S3.
>
>
>
> In any case, thank you very much for your help.
>
>
>
> Best regards!
>
>
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* jueves, 8 de marzo de 2018 13:43
>
> *To:* user <us...@kylin.apache.org>
> *Subject:* Re: Kylin, EMR and HBase timeout
>
>
>
> Hi Roberto,
>
>
>
> Oh, I just saw you had already tried that. From the code level, I do say
> it was set to the configuration when initiating a connection to HBase. Not
> sure whether it was respected by the server or not.
>
>
>
> Besides, these two parameters are very basic config, which cannot be set
> at Cube level.
>
>
>
> 2018-03-08 19:29 GMT+08:00 <ro...@stratebi.com>:
>
> Hi,
>
>
>
> Could I override the following properties at cube definition?
>
>
>
> kylin.metadata.hbase-rpc-timeout=3600000
>
> kylin.metadata.hbase-client-scanner-timeout-period=3600000
>
>
>
> I would like to check if Kylin is using these values.
>
>
>
> I restarted kylin after modifying kylin.properties and also reload config.
> But looks like Kylins are not taking in to account these properties values.
>
>
>
> Kind Regards,
>
>
>
> *From:* roberto.tardio@stratebi.com [mailto:roberto.tardio@stratebi.com]
> *Sent:* jueves, 8 de marzo de 2018 11:33
> *To:* user@kylin.apache.org
> *Subject:* RE: Kylin, EMR and HBase timeout
>
>
>
> Hi,
>
>
>
> Thanks ShaoFeng!
>
>
>
> As a said on first email, I changed kylin.metadata.hbase-rpc-timeout=3600000
> in kylin.properties and restarted Kylin. However looks like that Kylin is
> not using this value, as the error log show “org.apache.hadoop.hbase.ipc.CallTimeoutException:
> Call id=4280915, waitTime=5001, *operationTimeout=5000*”
>
>
>
> About HBase Health:
>
>
>
> I executed command hbase hbck and looks like ok except the followinng:
>
>
>
> 1 inconsistencies detected.
>
> Status: INCONSISTENT
>
>
>
> Could this inconsistence affect HBase stability?I’m not an expert on HBase
> J.
>
>
>
> Best Regards,
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org
> <sh...@apache.org>]
> *Sent:* jueves, 8 de marzo de 2018 10:05
> *To:* user <us...@kylin.apache.org>
> *Subject:* Re: Kylin, EMR and HBase timeout
>
>
>
> followed the previous, the value is in milliseconds, the default is:
>
>
>
> *kylin.metadata.hbase-rpc-timeout=*5000
>
>
>
> 2018-03-08 17:03 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>
> Hi Roberto,
>
>
>
> Is the HBase cluster in health status? Is there any error/warning in
> region server's log? Suggest to take a look at the server side.
>
>
>
> The long timeout 3600000 value is to allow HBase transfer the Cube (HFile)
> to S3. It is not beautiful, but without it we may fail to build.
>
>
>
> The error you faced is when accessing Kylin metadata, the timeout value is
> overwritten to 5 seconds (with parameter "
> *kylin.metadata.hbase-rpc-timeout*"). You can customize it in
> kylin.properties if want.
>
>
>
>
>
>
>
> 2018-03-07 22:50 GMT+08:00 <ro...@stratebi.com>:
>
> Hi,
>
>
>
> We have a Kylin 2.2 installation that uses an EMR cluster, so it uses
> HBase on S3 as storage. It works well usually, but sometimes building
> process get error at step “Build Dimension Dictionary”, as hbase.ipc.
> callTimeoutException.
>
>
>
>
>
> In order to solve it, I changed rpc configuration as indicated in
> http://kylin.apache.org/docs23/install/kylin_aws_emr.html . I have
> changed, the following properties on hbase-site.xml for master and all
> slaves.
>
>
>
> <property>
>
> <name>hbase.rpc.timeout</name>
>
> <value>3600000</value>
>
> </property>
>
>
>
> <property>
>
> <name>hbase.client.scanner.timeout.period</name>
>
> <value>3600000</value>
>
> </property>
>
>
>
> Then I restarted HBase Master and HBase Regions Servers. I checked
>
>
>
>
>
> However I still, getting the same error. Looks like that Kylin uses
> different value for these properties.
>
>
>
> I also changed the following kylin.properties:
>
>
>
> kylin.metadata.hbase-rpc-timeout=3600000
>
> kylin.metadata.hbase-client-scanner-timeout-period=3600000
>
>
>
> I restarted Kyling, but still getting same issue.
>
>
>
> Someone could help me?
>
>
>
> Thanks in advance!
>
> *Roberto Tardío Olmos*
>
> *Senior Big Data & Business Intelligence Consultant*
>
> Avenida de Brasil, 17
> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
> Planta 16.28020 Madrid
>
> Fijo: 91.788.34.10
>
>
> [image:
> http://www.stratebi.com/image/layout_set_logo?img_id=21615&t=1486381163544]
>
>
>
> http://bigdata.stratebi.com/
>
>
>
> http://www.stratebi.com
>
>
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
--
Best regards,
Shaofeng Shi 史少锋
RE: Kylin, EMR and HBase timeout
Posted by ro...@stratebi.com.
Hi ShaoFeng,
We realized that the change of these parameters has no effect until version 2.3, because we use the 2.2.
Although later we intend to update to 2.3, we have solved compiling version 2.2 with a cherry pick of the code of 2.3 so that the parameters are established according to the values of Kylin properties.
https://github.com/apache/kylin/commit/8e8d71b99a340831512f7ca34cbc0098a190f1f6
Therefore we have solved these timeout issues using Kylin with HBase on S3.
In any case, thank you very much for your help.
Best regards!
From: ShaoFeng Shi [mailto:shaofengshi@apache.org]
Sent: jueves, 8 de marzo de 2018 13:43
To: user <us...@kylin.apache.org>
Subject: Re: Kylin, EMR and HBase timeout
Hi Roberto,
Oh, I just saw you had already tried that. From the code level, I do say it was set to the configuration when initiating a connection to HBase. Not sure whether it was respected by the server or not.
Besides, these two parameters are very basic config, which cannot be set at Cube level.
2018-03-08 19:29 GMT+08:00 <roberto.tardio@stratebi.com <ma...@stratebi.com> >:
Hi,
Could I override the following properties at cube definition?
kylin.metadata.hbase-rpc-timeout=3600000
kylin.metadata.hbase-client-scanner-timeout-period=3600000
I would like to check if Kylin is using these values.
I restarted kylin after modifying kylin.properties and also reload config. But looks like Kylins are not taking in to account these properties values.
Kind Regards,
From: roberto.tardio@stratebi.com <ma...@stratebi.com> [mailto:roberto.tardio@stratebi.com <ma...@stratebi.com> ]
Sent: jueves, 8 de marzo de 2018 11:33
To: user@kylin.apache.org <ma...@kylin.apache.org>
Subject: RE: Kylin, EMR and HBase timeout
Hi,
Thanks ShaoFeng!
As a said on first email, I changed kylin.metadata.hbase-rpc-timeout=3600000 in kylin.properties and restarted Kylin. However looks like that Kylin is not using this value, as the error log show “org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=4280915, waitTime=5001, operationTimeout=5000”
About HBase Health:
I executed command hbase hbck and looks like ok except the followinng:
1 inconsistencies detected.
Status: INCONSISTENT
Could this inconsistence affect HBase stability?I’m not an expert on HBase J.
Best Regards,
From: ShaoFeng Shi [mailto:shaofengshi@apache.org]
Sent: jueves, 8 de marzo de 2018 10:05
To: user <user@kylin.apache.org <ma...@kylin.apache.org> >
Subject: Re: Kylin, EMR and HBase timeout
followed the previous, the value is in milliseconds, the default is:
kylin.metadata.hbase-rpc-timeout=5000
2018-03-08 17:03 GMT+08:00 ShaoFeng Shi <shaofengshi@apache.org <ma...@apache.org> >:
Hi Roberto,
Is the HBase cluster in health status? Is there any error/warning in region server's log? Suggest to take a look at the server side.
The long timeout 3600000 value is to allow HBase transfer the Cube (HFile) to S3. It is not beautiful, but without it we may fail to build.
The error you faced is when accessing Kylin metadata, the timeout value is overwritten to 5 seconds (with parameter "kylin.metadata.hbase-rpc-timeout"). You can customize it in kylin.properties if want.
2018-03-07 22:50 GMT+08:00 <roberto.tardio@stratebi.com <ma...@stratebi.com> >:
Hi,
We have a Kylin 2.2 installation that uses an EMR cluster, so it uses HBase on S3 as storage. It works well usually, but sometimes building process get error at step “Build Dimension Dictionary”, as hbase.ipc.callTimeoutException.
In order to solve it, I changed rpc configuration as indicated in http://kylin.apache.org/docs23/install/kylin_aws_emr.html . I have changed, the following properties on hbase-site.xml for master and all slaves.
<property>
<name>hbase.rpc.timeout</name>
<value>3600000</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>3600000</value>
</property>
Then I restarted HBase Master and HBase Regions Servers. I checked
However I still, getting the same error. Looks like that Kylin uses different value for these properties.
I also changed the following kylin.properties:
kylin.metadata.hbase-rpc-timeout=3600000
kylin.metadata.hbase-client-scanner-timeout-period=3600000
I restarted Kyling, but still getting same issue.
Someone could help me?
Thanks in advance!
Roberto Tardío Olmos
Senior Big Data & Business Intelligence Consultant
Avenida de Brasil, 17 <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g> , Planta 16.28020 Madrid
Fijo: 91.788.34.10
http://bigdata.stratebi.com/
http://www.stratebi.com
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
Re: Kylin, EMR and HBase timeout
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Roberto,
Oh, I just saw you had already tried that. From the code level, I do say it
was set to the configuration when initiating a connection to HBase. Not
sure whether it was respected by the server or not.
Besides, these two parameters are very basic config, which cannot be set at
Cube level.
2018-03-08 19:29 GMT+08:00 <ro...@stratebi.com>:
> Hi,
>
>
>
> Could I override the following properties at cube definition?
>
>
>
> kylin.metadata.hbase-rpc-timeout=3600000
>
> kylin.metadata.hbase-client-scanner-timeout-period=3600000
>
>
>
> I would like to check if Kylin is using these values.
>
>
>
> I restarted kylin after modifying kylin.properties and also reload config.
> But looks like Kylins are not taking in to account these properties values.
>
>
>
> Kind Regards,
>
>
>
> *From:* roberto.tardio@stratebi.com [mailto:roberto.tardio@stratebi.com]
> *Sent:* jueves, 8 de marzo de 2018 11:33
> *To:* user@kylin.apache.org
> *Subject:* RE: Kylin, EMR and HBase timeout
>
>
>
> Hi,
>
>
>
> Thanks ShaoFeng!
>
>
>
> As a said on first email, I changed kylin.metadata.hbase-rpc-timeout=3600000
> in kylin.properties and restarted Kylin. However looks like that Kylin is
> not using this value, as the error log show “org.apache.hadoop.hbase.ipc.CallTimeoutException:
> Call id=4280915, waitTime=5001, *operationTimeout=5000*”
>
>
>
> About HBase Health:
>
>
>
> I executed command hbase hbck and looks like ok except the followinng:
>
>
>
> 1 inconsistencies detected.
>
> Status: INCONSISTENT
>
>
>
> Could this inconsistence affect HBase stability?I’m not an expert on HBase
> J.
>
>
>
> Best Regards,
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org
> <sh...@apache.org>]
> *Sent:* jueves, 8 de marzo de 2018 10:05
> *To:* user <us...@kylin.apache.org>
> *Subject:* Re: Kylin, EMR and HBase timeout
>
>
>
> followed the previous, the value is in milliseconds, the default is:
>
>
>
> *kylin.metadata.hbase-rpc-timeout=*5000
>
>
>
> 2018-03-08 17:03 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>
> Hi Roberto,
>
>
>
> Is the HBase cluster in health status? Is there any error/warning in
> region server's log? Suggest to take a look at the server side.
>
>
>
> The long timeout 3600000 value is to allow HBase transfer the Cube (HFile)
> to S3. It is not beautiful, but without it we may fail to build.
>
>
>
> The error you faced is when accessing Kylin metadata, the timeout value is
> overwritten to 5 seconds (with parameter "
> *kylin.metadata.hbase-rpc-timeout*"). You can customize it in
> kylin.properties if want.
>
>
>
>
>
>
>
> 2018-03-07 22:50 GMT+08:00 <ro...@stratebi.com>:
>
> Hi,
>
>
>
> We have a Kylin 2.2 installation that uses an EMR cluster, so it uses
> HBase on S3 as storage. It works well usually, but sometimes building
> process get error at step “Build Dimension Dictionary”, as hbase.ipc.
> callTimeoutException.
>
>
>
>
>
> In order to solve it, I changed rpc configuration as indicated in
> http://kylin.apache.org/docs23/install/kylin_aws_emr.html . I have
> changed, the following properties on hbase-site.xml for master and all
> slaves.
>
>
>
> <property>
>
> <name>hbase.rpc.timeout</name>
>
> <value>3600000</value>
>
> </property>
>
>
>
> <property>
>
> <name>hbase.client.scanner.timeout.period</name>
>
> <value>3600000</value>
>
> </property>
>
>
>
> Then I restarted HBase Master and HBase Regions Servers. I checked
>
>
>
>
>
> However I still, getting the same error. Looks like that Kylin uses
> different value for these properties.
>
>
>
> I also changed the following kylin.properties:
>
>
>
> kylin.metadata.hbase-rpc-timeout=3600000
>
> kylin.metadata.hbase-client-scanner-timeout-period=3600000
>
>
>
> I restarted Kyling, but still getting same issue.
>
>
>
> Someone could help me?
>
>
>
> Thanks in advance!
>
> *Roberto Tardío Olmos*
>
> *Senior Big Data & Business Intelligence Consultant*
>
> Avenida de Brasil, 17
> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
> Planta 16.28020 Madrid
>
> Fijo: 91.788.34.10
>
>
> [image:
> http://www.stratebi.com/image/layout_set_logo?img_id=21615&t=1486381163544]
>
>
>
> http://bigdata.stratebi.com/
>
>
>
> http://www.stratebi.com
>
>
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
--
Best regards,
Shaofeng Shi 史少锋
RE: Kylin, EMR and HBase timeout
Posted by ro...@stratebi.com.
Hi,
Could I override the following properties at cube definition?
kylin.metadata.hbase-rpc-timeout=3600000
kylin.metadata.hbase-client-scanner-timeout-period=3600000
I would like to check if Kylin is using these values.
I restarted kylin after modifying kylin.properties and also reload config. But looks like Kylins are not taking in to account these properties values.
Kind Regards,
From: roberto.tardio@stratebi.com [mailto:roberto.tardio@stratebi.com]
Sent: jueves, 8 de marzo de 2018 11:33
To: user@kylin.apache.org
Subject: RE: Kylin, EMR and HBase timeout
Hi,
Thanks ShaoFeng!
As a said on first email, I changed kylin.metadata.hbase-rpc-timeout=3600000 in kylin.properties and restarted Kylin. However looks like that Kylin is not using this value, as the error log show “org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=4280915, waitTime=5001, operationTimeout=5000”
About HBase Health:
I executed command hbase hbck and looks like ok except the followinng:
1 inconsistencies detected.
Status: INCONSISTENT
Could this inconsistence affect HBase stability?I’m not an expert on HBase J.
Best Regards,
From: ShaoFeng Shi [mailto:shaofengshi@apache.org]
Sent: jueves, 8 de marzo de 2018 10:05
To: user <user@kylin.apache.org <ma...@kylin.apache.org> >
Subject: Re: Kylin, EMR and HBase timeout
followed the previous, the value is in milliseconds, the default is:
kylin.metadata.hbase-rpc-timeout=5000
2018-03-08 17:03 GMT+08:00 ShaoFeng Shi <shaofengshi@apache.org <ma...@apache.org> >:
Hi Roberto,
Is the HBase cluster in health status? Is there any error/warning in region server's log? Suggest to take a look at the server side.
The long timeout 3600000 value is to allow HBase transfer the Cube (HFile) to S3. It is not beautiful, but without it we may fail to build.
The error you faced is when accessing Kylin metadata, the timeout value is overwritten to 5 seconds (with parameter "kylin.metadata.hbase-rpc-timeout"). You can customize it in kylin.properties if want.
2018-03-07 22:50 GMT+08:00 <roberto.tardio@stratebi.com <ma...@stratebi.com> >:
Hi,
We have a Kylin 2.2 installation that uses an EMR cluster, so it uses HBase on S3 as storage. It works well usually, but sometimes building process get error at step “Build Dimension Dictionary”, as hbase.ipc.callTimeoutException.
In order to solve it, I changed rpc configuration as indicated in http://kylin.apache.org/docs23/install/kylin_aws_emr.html . I have changed, the following properties on hbase-site.xml for master and all slaves.
<property>
<name>hbase.rpc.timeout</name>
<value>3600000</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>3600000</value>
</property>
Then I restarted HBase Master and HBase Regions Servers. I checked
However I still, getting the same error. Looks like that Kylin uses different value for these properties.
I also changed the following kylin.properties:
kylin.metadata.hbase-rpc-timeout=3600000
kylin.metadata.hbase-client-scanner-timeout-period=3600000
I restarted Kyling, but still getting same issue.
Someone could help me?
Thanks in advance!
Roberto Tardío Olmos
Senior Big Data & Business Intelligence Consultant
Avenida de Brasil, 17 <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g> , Planta 16.28020 Madrid
Fijo: 91.788.34.10
http://bigdata.stratebi.com/
http://www.stratebi.com
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
RE: Kylin, EMR and HBase timeout
Posted by ro...@stratebi.com.
Hi,
Thanks ShaoFeng!
As a said on first email, I changed kylin.metadata.hbase-rpc-timeout=3600000 in kylin.properties and restarted Kylin. However looks like that Kylin is not using this value, as the error log show “org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=4280915, waitTime=5001, operationTimeout=5000”
About HBase Health:
I executed command hbase hbck and looks like ok except the followinng:
1 inconsistencies detected.
Status: INCONSISTENT
Could this inconsistence affect HBase stability?I’m not an expert on HBase J.
Best Regards,
From: ShaoFeng Shi [mailto:shaofengshi@apache.org]
Sent: jueves, 8 de marzo de 2018 10:05
To: user <us...@kylin.apache.org>
Subject: Re: Kylin, EMR and HBase timeout
followed the previous, the value is in milliseconds, the default is:
kylin.metadata.hbase-rpc-timeout=5000
2018-03-08 17:03 GMT+08:00 ShaoFeng Shi <shaofengshi@apache.org <ma...@apache.org> >:
Hi Roberto,
Is the HBase cluster in health status? Is there any error/warning in region server's log? Suggest to take a look at the server side.
The long timeout 3600000 value is to allow HBase transfer the Cube (HFile) to S3. It is not beautiful, but without it we may fail to build.
The error you faced is when accessing Kylin metadata, the timeout value is overwritten to 5 seconds (with parameter "kylin.metadata.hbase-rpc-timeout"). You can customize it in kylin.properties if want.
2018-03-07 22:50 GMT+08:00 <roberto.tardio@stratebi.com <ma...@stratebi.com> >:
Hi,
We have a Kylin 2.2 installation that uses an EMR cluster, so it uses HBase on S3 as storage. It works well usually, but sometimes building process get error at step “Build Dimension Dictionary”, as hbase.ipc.callTimeoutException.
In order to solve it, I changed rpc configuration as indicated in http://kylin.apache.org/docs23/install/kylin_aws_emr.html . I have changed, the following properties on hbase-site.xml for master and all slaves.
<property>
<name>hbase.rpc.timeout</name>
<value>3600000</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>3600000</value>
</property>
Then I restarted HBase Master and HBase Regions Servers. I checked
However I still, getting the same error. Looks like that Kylin uses different value for these properties.
I also changed the following kylin.properties:
kylin.metadata.hbase-rpc-timeout=3600000
kylin.metadata.hbase-client-scanner-timeout-period=3600000
I restarted Kyling, but still getting same issue.
Someone could help me?
Thanks in advance!
Roberto Tardío Olmos
Senior Big Data & Business Intelligence Consultant
Avenida de Brasil, 17 <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g> , Planta 16.28020 Madrid
Fijo: 91.788.34.10
http://bigdata.stratebi.com/
http://www.stratebi.com
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
Re: Kylin, EMR and HBase timeout
Posted by ShaoFeng Shi <sh...@apache.org>.
followed the previous, the value is in milliseconds, the default is:
kylin.metadata.hbase-rpc-timeout=5000
2018-03-08 17:03 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> Hi Roberto,
>
> Is the HBase cluster in health status? Is there any error/warning in
> region server's log? Suggest to take a look at the server side.
>
> The long timeout 3600000 value is to allow HBase transfer the Cube (HFile)
> to S3. It is not beautiful, but without it we may fail to build.
>
> The error you faced is when accessing Kylin metadata, the timeout value is
> overwritten to 5 seconds (with parameter "kylin.metadata.hbase-rpc-timeout
> "). You can customize it in kylin.properties if want.
>
>
>
> 2018-03-07 22:50 GMT+08:00 <ro...@stratebi.com>:
>
>> Hi,
>>
>>
>>
>> We have a Kylin 2.2 installation that uses an EMR cluster, so it uses
>> HBase on S3 as storage. It works well usually, but sometimes building
>> process get error at step “Build Dimension Dictionary”, as
>> hbase.ipc.callTimeoutException.
>>
>>
>>
>>
>>
>> In order to solve it, I changed rpc configuration as indicated in
>> http://kylin.apache.org/docs23/install/kylin_aws_emr.html . I have
>> changed, the following properties on hbase-site.xml for master and all
>> slaves.
>>
>>
>>
>> <property>
>>
>> <name>hbase.rpc.timeout</name>
>>
>> <value>3600000</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>hbase.client.scanner.timeout.period</name>
>>
>> <value>3600000</value>
>>
>> </property>
>>
>>
>>
>> Then I restarted HBase Master and HBase Regions Servers. I checked
>>
>>
>>
>>
>>
>> However I still, getting the same error. Looks like that Kylin uses
>> different value for these properties.
>>
>>
>>
>> I also changed the following kylin.properties:
>>
>>
>>
>> kylin.metadata.hbase-rpc-timeout=3600000
>>
>> kylin.metadata.hbase-client-scanner-timeout-period=3600000
>>
>>
>>
>> I restarted Kyling, but still getting same issue.
>>
>>
>>
>> Someone could help me?
>>
>>
>>
>> Thanks in advance!
>>
>> *Roberto Tardío Olmos*
>>
>> *Senior Big Data & Business Intelligence Consultant*
>>
>> Avenida de Brasil, 17
>> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
>> Planta 16.28020 Madrid
>>
>> Fijo: 91.788.34.10
>>
>>
>> [image:
>> http://www.stratebi.com/image/layout_set_logo?img_id=21615&t=1486381163544]
>>
>>
>>
>> http://bigdata.stratebi.com/
>>
>>
>>
>> http://www.stratebi.com
>>
>>
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Kylin, EMR and HBase timeout
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Roberto,
Is the HBase cluster in health status? Is there any error/warning in region
server's log? Suggest to take a look at the server side.
The long timeout 3600000 value is to allow HBase transfer the Cube (HFile)
to S3. It is not beautiful, but without it we may fail to build.
The error you faced is when accessing Kylin metadata, the timeout value is
overwritten to 5 seconds (with parameter "kylin.metadata.hbase-rpc-timeout
"). You can customize it in kylin.properties if want.
2018-03-07 22:50 GMT+08:00 <ro...@stratebi.com>:
> Hi,
>
>
>
> We have a Kylin 2.2 installation that uses an EMR cluster, so it uses
> HBase on S3 as storage. It works well usually, but sometimes building
> process get error at step “Build Dimension Dictionary”, as hbase.ipc.
> callTimeoutException.
>
>
>
>
>
> In order to solve it, I changed rpc configuration as indicated in
> http://kylin.apache.org/docs23/install/kylin_aws_emr.html . I have
> changed, the following properties on hbase-site.xml for master and all
> slaves.
>
>
>
> <property>
>
> <name>hbase.rpc.timeout</name>
>
> <value>3600000</value>
>
> </property>
>
>
>
> <property>
>
> <name>hbase.client.scanner.timeout.period</name>
>
> <value>3600000</value>
>
> </property>
>
>
>
> Then I restarted HBase Master and HBase Regions Servers. I checked
>
>
>
>
>
> However I still, getting the same error. Looks like that Kylin uses
> different value for these properties.
>
>
>
> I also changed the following kylin.properties:
>
>
>
> kylin.metadata.hbase-rpc-timeout=3600000
>
> kylin.metadata.hbase-client-scanner-timeout-period=3600000
>
>
>
> I restarted Kyling, but still getting same issue.
>
>
>
> Someone could help me?
>
>
>
> Thanks in advance!
>
> *Roberto Tardío Olmos*
>
> *Senior Big Data & Business Intelligence Consultant*
>
> Avenida de Brasil, 17
> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
> Planta 16.28020 Madrid
>
> Fijo: 91.788.34.10
>
>
> [image:
> http://www.stratebi.com/image/layout_set_logo?img_id=21615&t=1486381163544]
>
>
>
> http://bigdata.stratebi.com/
>
>
>
> http://www.stratebi.com
>
>
>
--
Best regards,
Shaofeng Shi 史少锋