You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by 旭鹤 <xu...@dianjia.io> on 2018/04/24 12:00:21 UTC
回复 kylin help
设置的队列大小是500 发生问题的时候，都在等待。
看下这段能不能看出来什么
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4ca122b1 rejected from java.util.concurrent.ThreadPoolExecutor@466b92cd[Shutting down, pool size = 31, active threads = 31, queued tasks = 0, completed tasks = 52306] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1795) at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC.runEPRange(CubeHBaseEndpointRPC.java:205) at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC.access$000(CubeHBaseEndpointRPC.java:69) at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(CubeHBaseEndpointRPC.java:186) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)2018-04-24 19:11:29,755 DEBUG [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] common.KylinConfig:278 : KYLIN_CONF property was not set, will seek KYLIN_HOME env variable2018-04-24 19:11:29,756 ERROR [pool-10-thread-283] v2.CubeHBaseEndpointRPC:340 : <sub-thread for Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f GTScanRequest 2257581>Error when visiting cubes by endpointjava.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4ca122b1 rejected from java.util.concurrent.ThreadPoolExecutor@466b92cd[Shutting down, pool size = 31, active threads = 31, queued tasks = 0, completed tasks = 52306] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1795) at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC.runEPRange(CubeHBaseEndpointRPC.java:205) at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC.access$000(CubeHBaseEndpointRPC.java:69) at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(CubeHBaseEndpointRPC.java:186) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)2018-04-24 19:11:29,756 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] gtrecord.SequentialCubeTupleIterator:73 : Using Iterators.concat to merge segment results2018-04-24 19:11:29,756 DEBUG [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] enumerator.OLAPEnumerator:120 : return TupleIterator...2018-04-24 19:11:29,756 ERROR [Query efb08523-db3b-434b-b4f7-35ad
env variable2018-04-24 19:11:29,724 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] gtrecord.CubeSegmentScanner:56 : Init CubeSegmentScanner for segment 20180420000000_201804210000002018-04-24 19:11:29,725 DEBUG [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] v2.CubeHBaseRPC:315 : hbase.rpc.timeout = 60000 ms, use 54000 ms as timeout for coprocessor2018-04-24 19:11:29,725 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] v2.CubeHBaseEndpointRPC:152 : Serialized scanRequestBytes 2778 bytes, rawScanBytesString 88 bytes2018-04-24 19:11:29,726 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] v2.CubeHBaseEndpointRPC:154 : The scan 589bd9fa for segment view_dm_brand_dc_storage_sale_dm_model_v2_cube[20180420000000_20180421000000] is as below with 1 separate raw scans, shard part of start/end key is set to 02018-04-24 19:11:29,726 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] v2.CubeHBaseRPC:288 : Visiting hbase table KYLIN_X0VZKW9TP4: cuboid require post aggregation, from 56 to 63 Start: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x3F\x1E\x04\x28\x00\x00\x00\x00 (\x00\x00\x00\x00\x00\x00\x00\x00\x00?\x1E\x04(\x00\x00\x00\x00) Stop: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x3F\x1E\x04\x28\x00\xFF\xFF\xFF\x00 (\x00\x00\x00\x00\x00\x00\x00\x00\x00?\x1E\x04(\x00\xFF\xFF\xFF\x00) Fuzzy key counts: 1. Fuzzy keys : \x00\x00\x00\x00\x00\x00\x00\x00\x00\x3F\x1E\x04\x28\x00\x00\x00\x00 \x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x01\x01;2018-04-24 19:11:29,727 DEBUG [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] v2.CubeHBaseEndpointRPC:159 : Submitting rpc to 3 shards starting from shard 2, scan range count 12018-04-24 19:11:29,727 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] common.KylinConfig:319 : Loading kylin-defaults.properties from file:/opt/apache-kylin-2.3.1-bin/tomcat/webapps/kylin/WEB-INF/lib/kylin-core-common-2.3.1.jar!/kylin-defaults.properties2018-04-24 19:11:29,727 ERROR [pool-10-thread-283] v2.CubeHBaseEndpointRPC:340 : <sub-thread for Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f GTScanRequest 7b499b30>Error when visiting cubes by endpointjava.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4ca122b1 rejected from java.util.concurrent.ThreadPoolExecutor@466b92cd[Shutting down, pool size = 31, active threads = 31, queued tasks = 0, completed tasks = 52306] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
v2.CubeHBaseRPC:315 : hbase.rpc.timeout = 60000 ms, use 54000 ms as timeout for coprocessor
协处理器处理超时
------------------------------------------------------------------发件人：旭鹤 <xu...@dianjia.io>发送时间：2018年4月24日(星期二) 18:54收件人：Ma Gang <mg...@163.com>主 题：回复：回复：kylin help
这个里有很多在等待
------------------------------------------------------------------发件人：Ma Gang <mg...@163.com>发送时间：2018年4月24日(星期二) 18:25收件人：旭鹤 <xu...@dianjia.io>主 题：Re:回复：kylin help
That may be the cause, you should monitor the hbase metrics properly(rpc queue size etc.)
At 2018-04-24 17:56:28, "旭鹤" <xu...@dianjia.io> wrote:
Hi，我们刚刚调了hbase 的 hbase.ipc.server.read.threadpool.size 由原来的10 调至1000 目前还在测试当中你看看，有没有相关的联系。
------------------------------------------------------------------发件人：Ma Gang <mg...@163.com>发送时间：2018年4月24日(星期二) 17:49收件人：user <us...@kylin.apache.org>; 旭鹤 <xu...@dianjia.io>主 题：Re:kylin help
Hi,Currently you can set the configuration: "kylin.storage.partition.max-scan-bytes", to limit the scan bytes for each region to prevent possible OOM in region server, by default it is set to 3GB, and the configuration can be set in cube level. In our private Kylin env, we add two new configuration to limit the max return bytes and max return rows for each region to do more exact resource control, if community thinks that make sense, I can contribute the patch.
在 2018-04-24 16:22:50，"旭鹤" <xu...@dianjia.io> 写道：
当并发量上来的时候 很多请求超时没响应。三台query 机中只有某一台出现问题，但是另外两台也无法查询 请问是不是协处理器的问题，协处理 有什么需要配置的么？报错信息：kylin.log:Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: Coprocessor passed deadline! Maybe server is overloaded
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.checkDeadline(CubeVisitService.java:225)
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:259)
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:5555)
kylin.log- at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7931)
kylin.log- at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1969)
kylin.log- at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1951)
kylin.log- at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652)
kylin.log- at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)
kylin.log- at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
kylin.log- at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)
--
kylin.log- at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:107)
kylin.log- at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56)
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.visitCube(CubeVisitProtos.java:5616)
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$2.call(CubeHBaseEndpointRPC.java:237)
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$2.call(CubeHBaseEndpointRPC.java:206)
kylin.log- at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1800)
kylin.log- at java.util.concurrent.FutureTask.run(FutureTask.java:266)
kylin.log- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
kylin.log- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
kylin.log- ... 1 more
kylin.log:Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException): org.apache.hadoop.hbase.DoNotRetryIOException: Coprocessor passed deadline! Maybe server is overloaded
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.checkDeadline(CubeVisitService.java:225)
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:259)
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:5555)
kylin.log- at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7931)
kylin.log- at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1969)
kylin.log- at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1951)
kylin.log- at org.apache.hadoop.hbas
Re:回复 kylin help
Posted by Ma Gang <mg...@163.com>.
According to log, the coprocessor executor service is in 'Shutting down' state, so all coprocessor requests are rejected. Is this log happens when you shutdown the Kylin server? From code I cannot see anywhere that coprocessor executor will be shut down, except in the system's shutdown hook.
You should monitor the hbase region server metrics, this looks like region server issue.
在 2018-04-24 20:00:21，"旭鹤" <xu...@dianjia.io> 写道：
设置的队列大小是500
发生问题的时候，都在等待。
看下这段能不能看出来什么
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4ca122b1 rejected from java.util.concurrent.ThreadPoolExecutor@466b92cd[Shutting down, pool size = 31, active threads = 31, queued tasks = 0, completed tasks = 52306]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1795)
at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC.runEPRange(CubeHBaseEndpointRPC.java:205)
at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC.access$000(CubeHBaseEndpointRPC.java:69)
at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(CubeHBaseEndpointRPC.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-04-24 19:11:29,755 DEBUG [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] common.KylinConfig:278 : KYLIN_CONF property was not set, will seek KYLIN_HOME env variable
2018-04-24 19:11:29,756 ERROR [pool-10-thread-283] v2.CubeHBaseEndpointRPC:340 : <sub-thread for Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f GTScanRequest 2257581>Error when visiting cubes by endpoint
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4ca122b1 rejected from java.util.concurrent.ThreadPoolExecutor@466b92cd[Shutting down, pool size = 31, active threads = 31, queued tasks = 0, completed tasks = 52306]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1795)
at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC.runEPRange(CubeHBaseEndpointRPC.java:205)
at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC.access$000(CubeHBaseEndpointRPC.java:69)
at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(CubeHBaseEndpointRPC.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-04-24 19:11:29,756 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] gtrecord.SequentialCubeTupleIterator:73 : Using Iterators.concat to merge segment results
2018-04-24 19:11:29,756 DEBUG [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] enumerator.OLAPEnumerator:120 : return TupleIterator...
2018-04-24 19:11:29,756 ERROR [Query efb08523-db3b-434b-b4f7-35ad
env variable
2018-04-24 19:11:29,724 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] gtrecord.CubeSegmentScanner:56 : Init CubeSegmentScanner for segment 20180420000000_20180421000000
2018-04-24 19:11:29,725 DEBUG [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] v2.CubeHBaseRPC:315 : hbase.rpc.timeout = 60000 ms, use 54000 ms as timeout for coprocessor
2018-04-24 19:11:29,725 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] v2.CubeHBaseEndpointRPC:152 : Serialized scanRequestBytes 2778 bytes, rawScanBytesString 88 bytes
2018-04-24 19:11:29,726 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] v2.CubeHBaseEndpointRPC:154 : The scan 589bd9fa for segment view_dm_brand_dc_storage_sale_dm_model_v2_cube[20180420000000_20180421000000] is as below with 1 separate raw scans, shard part of start/end key is set to 0
2018-04-24 19:11:29,726 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] v2.CubeHBaseRPC:288 : Visiting hbase table KYLIN_X0VZKW9TP4: cuboid require post aggregation, from 56 to 63 Start: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x3F\x1E\x04\x28\x00\x00\x00\x00 (\x00\x00\x00\x00\x00\x00\x00\x00\x00?\x1E\x04(\x00\x00\x00\x00) Stop: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x3F\x1E\x04\x28\x00\xFF\xFF\xFF\x00 (\x00\x00\x00\x00\x00\x00\x00\x00\x00?\x1E\x04(\x00\xFF\xFF\xFF\x00) Fuzzy key counts: 1. Fuzzy keys : \x00\x00\x00\x00\x00\x00\x00\x00\x00\x3F\x1E\x04\x28\x00\x00\x00\x00 \x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x01\x01;
2018-04-24 19:11:29,727 DEBUG [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] v2.CubeHBaseEndpointRPC:159 : Submitting rpc to 3 shards starting from shard 2, scan range count 1
2018-04-24 19:11:29,727 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49] common.KylinConfig:319 : Loading kylin-defaults.properties from file:/opt/apache-kylin-2.3.1-bin/tomcat/webapps/kylin/WEB-INF/lib/kylin-core-common-2.3.1.jar!/kylin-defaults.properties
2018-04-24 19:11:29,727 ERROR [pool-10-thread-283] v2.CubeHBaseEndpointRPC:340 : <sub-thread for Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f GTScanRequest 7b499b30>Error when visiting cubes by endpoint
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4ca122b1 rejected from java.util.concurrent.ThreadPoolExecutor@466b92cd[Shutting down, pool size = 31, active threads = 31, queued tasks = 0, completed tasks = 52306]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
v2.CubeHBaseRPC:315 : hbase.rpc.timeout = 60000 ms, use 54000 ms as timeout for coprocessor
协处理器处理超时
------------------------------------------------------------------
发件人：旭鹤 <xu...@dianjia.io>
发送时间：2018年4月24日(星期二) 18:54
收件人：Ma Gang <mg...@163.com>
主 题：回复：回复：kylin help
这个里有很多在等待
------------------------------------------------------------------
发件人：Ma Gang <mg...@163.com>
发送时间：2018年4月24日(星期二) 18:25
收件人：旭鹤 <xu...@dianjia.io>
主 题：Re:回复：kylin help
That may be the cause, you should monitor the hbase metrics properly(rpc queue size etc.)
At 2018-04-24 17:56:28, "旭鹤" <xu...@dianjia.io> wrote:
Hi，
我们刚刚调了hbase 的 hbase.ipc.server.read.threadpool.size
由原来的10 调至1000 目前还在测试当中
你看看，有没有相关的联系。
------------------------------------------------------------------
发件人：Ma Gang <mg...@163.com>
发送时间：2018年4月24日(星期二) 17:49
收件人：user <us...@kylin.apache.org>; 旭鹤 <xu...@dianjia.io>
主 题：Re:kylin help
Hi,
Currently you can set the configuration: "kylin.storage.partition.max-scan-bytes", to limit the scan bytes for each region to prevent possible OOM in region server, by default it is set to 3GB, and the configuration can be set in cube level.
In our private Kylin env, we add two new configuration to limit the max return bytes and max return rows for each region to do more exact resource control, if community thinks that make sense, I can contribute the patch.
在 2018-04-24 16:22:50，"旭鹤" <xu...@dianjia.io> 写道：
当并发量上来的时候 很多请求超时没响应。三台query 机中只有某一台出现问题，但是另外两台也无法查询
请问是不是协处理器的问题，协处理 有什么需要配置的么？
报错信息：
kylin.log:Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: Coprocessor passed deadline! Maybe server is overloaded
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.checkDeadline(CubeVisitService.java:225)
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:259)
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:5555)
kylin.log- at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7931)
kylin.log- at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1969)
kylin.log- at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1951)
kylin.log- at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652)
kylin.log- at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)
kylin.log- at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
kylin.log- at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)
--
kylin.log- at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:107)
kylin.log- at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56)
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.visitCube(CubeVisitProtos.java:5616)
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$2.call(CubeHBaseEndpointRPC.java:237)
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$2.call(CubeHBaseEndpointRPC.java:206)
kylin.log- at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1800)
kylin.log- at java.util.concurrent.FutureTask.run(FutureTask.java:266)
kylin.log- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
kylin.log- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
kylin.log- ... 1 more
kylin.log:Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException): org.apache.hadoop.hbase.DoNotRetryIOException: Coprocessor passed deadline! Maybe server is overloaded
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.checkDeadline(CubeVisitService.java:225)
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:259)
kylin.log- at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:5555)
kylin.log- at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7931)
kylin.log- at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1969)
kylin.log- at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1951)
kylin.log- at org.apache.hadoop.hbas