You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Guangxu Cheng (Jira)" <ji...@apache.org> on 2020/08/21 03:06:00 UTC
[jira] [Created] (KYLIN-4711) Change default value to 3 for
kylin.metadata.hbase-client-retries-number
Guangxu Cheng created KYLIN-4711:
------------------------------------
Summary: Change default value to 3 for kylin.metadata.hbase-client-retries-number
Key: KYLIN-4711
URL: https://issues.apache.org/jira/browse/KYLIN-4711
Project: Kylin
Issue Type: Improvement
Affects Versions: v3.1.0
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
```shell
java.lang.RuntimeException: org.apache.kylin.job.exception.PersistentException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Thu Aug 20 21:06:01 GMT+08:00 2020, RpcRetryingCaller
{globalStartTime=1597928761253, pause=1000, retries=1}
, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region kylin_production_metadata,/execute_output/3adc92f2-edcd-2705-5a9c-ad0afe4a0808-01,1594348337103.48b9e5e9c3c7891750236fcec84b38d5. is not online on xxx.xxx.xxx.xxx,16031,1558009276096
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3033)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1110)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2064)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33857)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2189)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
on xxx.xxx.xxx.xxx,16031,1558009276096
at org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:174)
at org.apache.kylin.job.execution.AbstractExecutable.getOutput(AbstractExecutable.java:450)
at org.apache.kylin.job.execution.AbstractExecutable.isDiscarded(AbstractExecutable.java:561)
at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:165)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:191)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:191)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
Recently, our build job failed occasionally. After analysis, it was found that the reason for the failure was due to abnormal access to the MetaStore. We use HBase as MetaStore.
When accessing HBase, the client will cache the region information of the table in the client. When the region was moved, client will not actively update the information in the cache. So the client will receive a NotServingRegionException, the client will update the cache information when retrying. But the number of retries in kylin is 1, which means that the clinet will not try again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)