You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Johnson <it...@163.com> on 2019/10/10 07:10:10 UTC
kylin构建cube过程中，连接hbase超时问题
kylin：2.6.2，HBase 1.2.0-cdh5.15.0
The error occurred in #3 Step Name: Extract Fact Table Distinct Columns
error log：
java.lang.RuntimeException: org.apache.kylin.job.exception.PersistentException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Thu Oct 10 14:13:09 GMT+08:00 2019, RpcRetryingCaller{globalStartTime=1570687984759, pause=100, retries=1}, java.io.IOException: Call to tx-dn41.data/10.14.243.51:60020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=22841735, waitTime=5001, operationTimeout=5000 expired.
at org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:174)
at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:118)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
Caused by: org.apache.kylin.job.exception.PersistentException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Thu Oct 10 14:13:09 GMT+08:00 2019, RpcRetryingCaller{globalStartTime=1570687984759, pause=100, retries=1}, java.io.IOException: Call to tx-dn99.data/10.14.293.58:60020 failed on local exception:org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=22841735, waitTime=5001, operationTimeout=5000 expired.
at org.apache.kylin.job.dao.ExecutableDao.getJobOutput(ExecutableDao.java:407)
at org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:169)
... 8 more
Then，I found that this hbase node was doing FULL GC at this moment，GC lasts a long time。Are there any parameters that can modify the timeout，org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=22841735, waitTime=5001。
Re:kylin构建cube过程中，连接hbase超时问题
Posted by Johnson <it...@163.com>.
I modified hbase-site.xml：
<property>
<name>hbase.rpc.timeout</name>
<value>600000</value>
</property>
<property>
<name>hbase.client.operation.timeout</name>
<value>600000</value>
</property>
在 2019-10-10 15:10:10，"Johnson" <it...@163.com> 写道：
kylin：2.6.2，HBase 1.2.0-cdh5.15.0
The error occurred in #3 Step Name: Extract Fact Table Distinct Columns
error log：
java.lang.RuntimeException: org.apache.kylin.job.exception.PersistentException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Thu Oct 10 14:13:09 GMT+08:00 2019, RpcRetryingCaller{globalStartTime=1570687984759, pause=100, retries=1}, java.io.IOException: Call to tx-dn41.data/10.14.243.51:60020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=22841735, waitTime=5001, operationTimeout=5000 expired.
at org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:174)
at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:118)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
Caused by: org.apache.kylin.job.exception.PersistentException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Thu Oct 10 14:13:09 GMT+08:00 2019, RpcRetryingCaller{globalStartTime=1570687984759, pause=100, retries=1}, java.io.IOException: Call to tx-dn99.data/10.14.293.58:60020 failed on local exception:org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=22841735, waitTime=5001, operationTimeout=5000 expired.
at org.apache.kylin.job.dao.ExecutableDao.getJobOutput(ExecutableDao.java:407)
at org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:169)
... 8 more
Then，I found that this hbase node was doing FULL GC at this moment，GC lasts a long time。Are there any parameters that can modify the timeout，org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=22841735, waitTime=5001。
Re:Re: kylin构建cube过程中，连接hbase超时问题
Posted by Johnson <it...@163.com>.
Hi Yaqian：
Thank you very much，I will try。
在 2019-10-10 16:07:30，"Yaqian Zhang" <Ya...@126.com> 写道：
Hi Johnson:
You can set the parameters “kylin.metadata.hbase-rpc-timeout” to modify the timeout, the default value is 5000ms.
在 2019年10月10日，15:10，Johnson <it...@163.com> 写道：
kylin：2.6.2，HBase 1.2.0-cdh5.15.0
The error occurred in #3 Step Name: Extract Fact Table Distinct Columns
error log：
java.lang.RuntimeException: org.apache.kylin.job.exception.PersistentException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Thu Oct 10 14:13:09 GMT+08:00 2019, RpcRetryingCaller{globalStartTime=1570687984759, pause=100, retries=1}, java.io.IOException: Call to tx-dn41.data/10.14.243.51:60020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=22841735, waitTime=5001, operationTimeout=5000 expired.
at org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:174)
at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:118)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
Caused by: org.apache.kylin.job.exception.PersistentException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Thu Oct 10 14:13:09 GMT+08:00 2019, RpcRetryingCaller{globalStartTime=1570687984759, pause=100, retries=1}, java.io.IOException: Call to tx-dn99.data/10.14.293.58:60020 failed on local exception:org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=22841735, waitTime=5001, operationTimeout=5000 expired.
at org.apache.kylin.job.dao.ExecutableDao.getJobOutput(ExecutableDao.java:407)
at org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:169)
... 8 more
Then，I found that this hbase node was doing FULL GC at this moment，GC lasts a long time。Are there any parameters that can modify the timeout，org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=22841735, waitTime=5001。
Re: kylin构建cube过程中，连接hbase超时问题
Posted by Yaqian Zhang <Ya...@126.com>.
Hi Johnson:
You can set the parameters “kylin.metadata.hbase-rpc-timeout” to modify the timeout, the default value is 5000ms.
> 在 2019年10月10日，15:10，Johnson <it...@163.com> 写道：
>
> kylin：2.6.2，HBase 1.2.0-cdh5.15.0
> The error occurred in #3 Step Name: Extract Fact Table Distinct Columns
> error log：
> java.lang.RuntimeException: org.apache.kylin.job.exception.PersistentException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
> Thu Oct 10 14:13:09 GMT+08:00 2019, RpcRetryingCaller{globalStartTime=1570687984759, pause=100, retries=1}, java.io.IOException: Call to tx-dn41.data/10.14.243.51:60020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=22841735, waitTime=5001, operationTimeout=5000 expired.
> at org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:174)
> at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:118)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
> Caused by: org.apache.kylin.job.exception.PersistentException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
> Thu Oct 10 14:13:09 GMT+08:00 2019, RpcRetryingCaller{globalStartTime=1570687984759, pause=100, retries=1}, java.io.IOException: Call to tx-dn99.data/10.14.293.58:60020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=22841735, waitTime=5001, operationTimeout=5000 expired.
>
> at org.apache.kylin.job.dao.ExecutableDao.getJobOutput(ExecutableDao.java:407)
> at org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:169)
> ... 8 more
>
> Then，I found that this hbase node was doing FULL GC at this moment，GC lasts a long time。Are there any parameters that can modify the timeout，org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=22841735, waitTime=5001。
>
>
>