[jira] [Closed] (KYLIN-4711) Change default value to 3 for kylin.metadata.hbase-client-retries-number

["Xiaoxiang Yu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KYLIN-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiaoxiang Yu closed KYLIN-4711.
-------------------------------

Released at kylin 3.1.2

> Change default value to 3 for kylin.metadata.hbase-client-retries-number
> ------------------------------------------------------------------------
>
>                 Key: KYLIN-4711
>                 URL: https://issues.apache.org/jira/browse/KYLIN-4711
>             Project: Kylin
>          Issue Type: Improvement
>    Affects Versions: v3.1.0
>            Reporter: Guangxu Cheng
>            Assignee: Guangxu Cheng
>            Priority: Major
>             Fix For: v3.1.2
>
>
> ```shell
>  java.lang.RuntimeException: org.apache.kylin.job.exception.PersistentException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
>  Thu Aug 20 21:06:01 GMT+08:00 2020, RpcRetryingCaller
> {globalStartTime=1597928761253, pause=1000, retries=1}
> , org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region kylin_production_metadata,/execute_output/3adc92f2-edcd-2705-5a9c-ad0afe4a0808-01,1594348337103.48b9e5e9c3c7891750236fcec84b38d5. is not online on xxx.xxx.xxx.xxx,16031,1558009276096
>  at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3033)
>  at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1110)
>  at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2064)
>  at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33857)
>  at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2189)
>  at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
>  at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
>  at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
>  at java.lang.Thread.run(Thread.java:745)
>  on xxx.xxx.xxx.xxx,16031,1558009276096
>  at org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:174)
>  at org.apache.kylin.job.execution.AbstractExecutable.getOutput(AbstractExecutable.java:450)
>  at org.apache.kylin.job.execution.AbstractExecutable.isDiscarded(AbstractExecutable.java:561)
>  at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:165)
>  at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:191)
>  at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
>  at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:191)
>  at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
>  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
>  ```
>  Recently, our build job failed occasionally. After analysis, it was found that the reason for the failure was due to abnormal access to the MetaStore. We use HBase as MetaStore. 
>  When accessing HBase, the client will cache the region information of the table in the client. When the region was moved, client will not actively update the information in the cache. So the client will receive a NotServingRegionException, the client will update the cache information when retrying. But the number of retries in kylin is 1, which means that the clinet will not try again.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)