You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by 张 明宝 <zh...@outlook.com> on 2018/08/29 11:26:36 UTC
Coprocessor passed deadline! Maybe server is overloaded issue
Hello,
I built a cube with over 10 million source records successfully, and I could
use the cube to do some SQL query. However, when I want to do fuzzy query, I
have to face such errors:
##################################################################################################################
org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: Coprocessor passed deadline! Maybe server is overloaded at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.checkDeadline(CubeVisitService.java:225) at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:259) at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:5555) at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7873) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2008) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1990) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:748) while executing SQL: "select distinct dns, count(distinct devid) as num_report from dns_table where datetime ='2018-08-16' group by dns order by num_report desc LIMIT 50000"
##################################################################################################################
In kylin source code, this error is throwed by this method .
##############################################################
private void checkDeadline(long deadline) throws DoNotRetryIOException {
if (System.currentTimeMillis() > deadline) {
logger.info("Deadline has passed, abort now!");
throw new DoNotRetryIOException("Coprocessor passed deadline! Maybe server is overloaded");
}
}
#############################################################
I want to set the deadline-value longer ,but i don't know how to change it and what negative effects it will brings。
Looking forward to your reply and help.
Best wishes,
Mingbao Zhang
答复: 答复: Coprocessor passed deadline! Maybe server is overloaded issue
Posted by 张 明宝 <zh...@outlook.com>.
Hi ShaoFeng:
I will try all the methods you mentioned. thanks for your suggestions,
Best wishes,
MingBao Zhang,
________________________________
发件人: ShaoFeng Shi <sh...@apache.org>
发送时间: 2018年8月29日 20:26
收件人: user
主题: Re: 答复: Coprocessor passed deadline! Maybe server is overloaded issue
Hi mingbao,
Your query is not a regular OLAP query. The "URL" cardinality is 20 million, and there is no other filtering or limit condition exception the date, so its result set is still very big. Besides, you use the bitmap for the distinct "deviceid", the bitmap is memory-consuming which takes much memory than other measures. Totally it would take a lot of memory. That is a big burden to IO and HBase region server.
Here are some suggestions:
1) if you only care about certain URLs which has many devices, you can consider to use the Top N measure;
2) the "deviceid" is used in count distinct, you can remove it from dimension so the cube can be smaller
3) try to normalize the URL to shorter length;
3) add more dimension and filter condition to the cube/query, so the resultset won't be that big.
2018-08-30 10:10 GMT+08:00 张 明宝 <zh...@outlook.com>>:
Hi ShaoFeng:
Thanks for your reply ! here are some fetures of data and cube。
My sql is:
select distinct url, count(distinct devid) as num_report from url_table where datetime ='2018-08-16' group by urlorder by num_report desc
my data feature is like this:
columns count count distinct
data-lenth type
datetime 100 million 300 10 string
devid 100 million
1000 10 string
url 100 million
20 million 0 - 500 string
And this is my Cube configuration:
[cid:5334574f-6671-4937-9b8a-c3485d47a522]
[cid:cab704a3-4131-4591-97f2-d9cdd7bcfabc]
[cid:c0aaac54-7005-44a4-bde0-74f52df5eaec]
________________________________
发件人: ShaoFeng Shi <sh...@apache.org>>
发送时间: 2018年8月29日 18:47
收件人: user
主题: Re: Coprocessor passed deadline! Maybe server is overloaded issue
Hi Mingbao,
The timeout is 300 seconds, which is not configurable now. It can be externalized as a configuration if needed.
The purpose of timeout is to prevent from too long waiting and resource occupation. As Kylin's query should be returned in the second level, this threshold is long enough.
You need to investigate for the root cause. For example, whether the HBase cluster is in a healthy state, what's the SQL query and whether it causes a bunch of scan and post-aggregation in HBase, etc.
If you can share more information about the query and the Cube design, that would help in the investigation.
2018-08-29 19:26 GMT+08:00 张 明宝 <zh...@outlook.com>>:
Hello,
I built a cube with over 10 million source records successfully, and I could
use the cube to do some SQL query. However, when I want to do fuzzy query, I
have to face such errors:
##################################################################################################################
org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: Coprocessor passed deadline! Maybe server is overloaded at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.checkDeadline(CubeVisitService.java:225) at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:259) at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:5555) at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7873) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2008) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1990) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188) at org.apache.hadoop.hbase.ipc.Ca<http://org.apache.hadoop.hbase.ipc.Ca>llRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:748) while executing SQL: "select distinct dns, count(distinct devid) as num_report from dns_table where datetime ='2018-08-16' group by dns order by num_report desc LIMIT 50000"
##################################################################################################################
In kylin source code, this error is throwed by this method .
##############################################################
private void checkDeadline(long deadline) throws DoNotRetryIOException {
if (System.currentTimeMillis() > deadline) {
logger.info<http://logger.info>("Deadline has passed, abort now!");
throw new DoNotRetryIOException("Coprocessor passed deadline! Maybe server is overloaded");
}
}
#############################################################
I want to set the deadline-value longer ,but i don't know how to change it and what negative effects it will brings。
Looking forward to your reply and help.
Best wishes,
Mingbao Zhang
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
Re: 答复: Coprocessor passed deadline! Maybe server is overloaded issue
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi mingbao,
Your query is not a regular OLAP query. The "URL" cardinality is 20
million, and there is no other filtering or limit condition exception the
date, so its result set is still very big. Besides, you use the bitmap for
the distinct "deviceid", the bitmap is memory-consuming which takes much
memory than other measures. Totally it would take a lot of memory. That is
a big burden to IO and HBase region server.
Here are some suggestions:
1) if you only care about certain URLs which has many devices, you can
consider to use the Top N measure;
2) the "deviceid" is used in count distinct, you can remove it from
dimension so the cube can be smaller
3) try to normalize the URL to shorter length;
3) add more dimension and filter condition to the cube/query, so the
resultset won't be that big.
2018-08-30 10:10 GMT+08:00 张 明宝 <zh...@outlook.com>:
> Hi ShaoFeng:
> Thanks for your reply ! here are some fetures of data and cube。
>
>
>
> My sql is:
>
> select distinct url, count(distinct devid) as num_report from url_table
> where datetime ='2018-08-16' group by urlorder by num_report desc
>
>
> my data feature is like this:
>
> columns count count distinct
> data-lenth type
> datetime 100 million 300 10 string
> devid 100 million
> 1000 10 string
> url 100 million
> 20 million 0 - 500 string
>
> And this is my Cube configuration:
>
>
>
>
>
>
>
>
> ------------------------------
> *发件人:* ShaoFeng Shi <sh...@apache.org>
> *发送时间:* 2018年8月29日 18:47
> *收件人:* user
> *主题:* Re: Coprocessor passed deadline! Maybe server is overloaded issue
>
> Hi Mingbao,
>
> The timeout is 300 seconds, which is not configurable now. It can be
> externalized as a configuration if needed.
>
> The purpose of timeout is to prevent from too long waiting and resource
> occupation. As Kylin's query should be returned in the second level, this
> threshold is long enough.
>
> You need to investigate for the root cause. For example, whether the HBase
> cluster is in a healthy state, what's the SQL query and whether it causes a
> bunch of scan and post-aggregation in HBase, etc.
>
> If you can share more information about the query and the Cube design,
> that would help in the investigation.
>
> 2018-08-29 19:26 GMT+08:00 张 明宝 <zh...@outlook.com>:
>
> Hello,
>
>
> I built a cube with over 10 million source records successfully, and I could
> use the cube to do some SQL query. However, when I want to do fuzzy query, I
> have to face such errors:
>
>
> ############################################################
> ######################################################
> org.apache.hadoop.hbase.DoNotRetryIOException:
> org.apache.hadoop.hbase.DoNotRetryIOException: Coprocessor passed
> deadline! Maybe server is overloaded at org.apache.kylin.storage.hbase
> .cube.v2.coprocessor.endpoint.CubeVisitService.checkDeadline(CubeVisitService.java:225)
> at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.
> CubeVisitService.visitCube(CubeVisitService.java:259) at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.
> generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:5555)
> at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7873)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServi
> ceOnRegion(RSRpcServices.java:2008) at org.apache.hadoop.hbase.region
> server.RSRpcServices.execService(RSRpcServices.java:1990) at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$
> ClientService$2.callBlockingMethod(ClientProtos.java:33652) at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188) at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at
> java.lang.Thread.run(Thread.java:748) while executing SQL: "select
> distinct dns, count(distinct devid) as num_report from dns_table where
> datetime ='2018-08-16' group by dns order by num_report desc LIMIT 50000"
> ############################################################
> ######################################################
> In kylin source code, this error is throwed by this method .
>
> ##############################################################
> private void checkDeadline(long deadline) throws DoNotRetryIOException {
> if (System.currentTimeMillis() > deadline) {
> logger.info("Deadline has passed, abort now!");
> throw new DoNotRetryIOException("Coprocessor passed deadline!
> Maybe server is overloaded");
> }
> }
> #############################################################
>
> I want to set the deadline-value longer ,but i don't know how to change it and
> what negative effects it will brings。
>
> Looking forward to your reply and help.
>
>
> Best wishes,
> Mingbao Zhang
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
Best regards,
Shaofeng Shi 史少锋
答复: Coprocessor passed deadline! Maybe server is overloaded issue
Posted by 张 明宝 <zh...@outlook.com>.
Hi ShaoFeng:
Thanks for your reply ! here are some fetures of data and cube。
My sql is:
select distinct url, count(distinct devid) as num_report from url_table where datetime ='2018-08-16' group by urlorder by num_report desc
my data feature is like this:
columns count count distinct
data-lenth type
datetime 100 million 300 10 string
devid 100 million
1000 10 string
url 100 million
20 million 0 - 500 string
And this is my Cube configuration:
[cid:5334574f-6671-4937-9b8a-c3485d47a522]
[cid:cab704a3-4131-4591-97f2-d9cdd7bcfabc]
[cid:c0aaac54-7005-44a4-bde0-74f52df5eaec]
________________________________
发件人: ShaoFeng Shi <sh...@apache.org>
发送时间: 2018年8月29日 18:47
收件人: user
主题: Re: Coprocessor passed deadline! Maybe server is overloaded issue
Hi Mingbao,
The timeout is 300 seconds, which is not configurable now. It can be externalized as a configuration if needed.
The purpose of timeout is to prevent from too long waiting and resource occupation. As Kylin's query should be returned in the second level, this threshold is long enough.
You need to investigate for the root cause. For example, whether the HBase cluster is in a healthy state, what's the SQL query and whether it causes a bunch of scan and post-aggregation in HBase, etc.
If you can share more information about the query and the Cube design, that would help in the investigation.
2018-08-29 19:26 GMT+08:00 张 明宝 <zh...@outlook.com>>:
Hello,
I built a cube with over 10 million source records successfully, and I could
use the cube to do some SQL query. However, when I want to do fuzzy query, I
have to face such errors:
##################################################################################################################
org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: Coprocessor passed deadline! Maybe server is overloaded at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.checkDeadline(CubeVisitService.java:225) at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:259) at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:5555) at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7873) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2008) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1990) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:748) while executing SQL: "select distinct dns, count(distinct devid) as num_report from dns_table where datetime ='2018-08-16' group by dns order by num_report desc LIMIT 50000"
##################################################################################################################
In kylin source code, this error is throwed by this method .
##############################################################
private void checkDeadline(long deadline) throws DoNotRetryIOException {
if (System.currentTimeMillis() > deadline) {
logger.info<http://logger.info>("Deadline has passed, abort now!");
throw new DoNotRetryIOException("Coprocessor passed deadline! Maybe server is overloaded");
}
}
#############################################################
I want to set the deadline-value longer ,but i don't know how to change it and what negative effects it will brings。
Looking forward to your reply and help.
Best wishes,
Mingbao Zhang
--
Best regards,
Shaofeng Shi 史少锋
Re: Coprocessor passed deadline! Maybe server is overloaded issue
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Mingbao,
The timeout is 300 seconds, which is not configurable now. It can be
externalized as a configuration if needed.
The purpose of timeout is to prevent from too long waiting and resource
occupation. As Kylin's query should be returned in the second level, this
threshold is long enough.
You need to investigate for the root cause. For example, whether the HBase
cluster is in a healthy state, what's the SQL query and whether it causes a
bunch of scan and post-aggregation in HBase, etc.
If you can share more information about the query and the Cube design, that
would help in the investigation.
2018-08-29 19:26 GMT+08:00 张 明宝 <zh...@outlook.com>:
> Hello,
>
>
> I built a cube with over 10 million source records successfully, and I could
> use the cube to do some SQL query. However, when I want to do fuzzy query, I
> have to face such errors:
>
>
> ############################################################
> ######################################################
> org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException:
> Coprocessor passed deadline! Maybe server is overloaded at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.
> endpoint.CubeVisitService.checkDeadline(CubeVisitService.java:225) at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.
> endpoint.CubeVisitService.visitCube(CubeVisitService.java:259) at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.
> CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:5555) at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7873)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2008)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> execService(RSRpcServices.java:1990) at org.apache.hadoop.hbase.
> protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188) at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at
> java.lang.Thread.run(Thread.java:748) while executing SQL: "select
> distinct dns, count(distinct devid) as num_report from dns_table where
> datetime ='2018-08-16' group by dns order by num_report desc LIMIT 50000"
> ############################################################
> ######################################################
> In kylin source code, this error is throwed by this method .
>
> ##############################################################
> private void checkDeadline(long deadline) throws DoNotRetryIOException {
> if (System.currentTimeMillis() > deadline) {
> logger.info("Deadline has passed, abort now!");
> throw new DoNotRetryIOException("Coprocessor passed deadline!
> Maybe server is overloaded");
> }
> }
> #############################################################
>
> I want to set the deadline-value longer ,but i don't know how to change it and
> what negative effects it will brings。
>
> Looking forward to your reply and help.
>
>
> Best wishes,
> Mingbao Zhang
>
--
Best regards,
Shaofeng Shi 史少锋