You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/12/03 13:33:09 UTC
[GitHub] [incubator-doris] caiconghui opened a new issue #5021: [Proposal] Use thrift protocol instead of http protocol for master to send heartbeat to other fe to escape unstable heartbeat response
caiconghui opened a new issue #5021:
URL: https://github.com/apache/incubator-doris/issues/5021
The heartbeat send by master to other fe is usually read timeout now,
many log like the following:
2020-12-03 21:21:41,328 WARN 105 [Util.getResultForUrl():361] failed to get result from url: http://xxxxxxxxx:port/api/bootstrap?cluster_id=1xxxxx&token=8456a755-xxxx-4ef9-a460-1547d23fc810. Read timed out
2020-12-03 21:21:41,329 WARN 32 [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: FRONTEND, status: BAD, msg: got exception, name: xxxxxxxxx_9010_1602645254326, queryPort: 0, rpcPort: 0, replayedJournalId: 0
So, I want to use thrift protocol instead of http protocol for master to send heartbeat to other fe to escape unstable heartbeat response, just like broker and be heartbeat
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on issue #5021: [Proposal] Use thrift protocol instead of http protocol for master to send heartbeat to other fe to escape unstable heartbeat response
Posted by GitBox <gi...@apache.org>.
caiconghui commented on issue #5021:
URL: https://github.com/apache/incubator-doris/issues/5021#issuecomment-745080077
> +1. But I think we still need to deep into this issue: Why http heartbeat will timeout.
@kangkaisen
netty use reactor model, some connection would be binded to one NioEventLoop thread, if some business logic wasting to much time, it will affect other requeset, and now doris process all business logic in io thread(NioEventLoop)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on issue #5021: [Proposal] Use thrift protocol instead of http protocol for master to send heartbeat to other fe to escape unstable heartbeat response
Posted by GitBox <gi...@apache.org>.
caiconghui commented on issue #5021:
URL: https://github.com/apache/incubator-doris/issues/5021#issuecomment-739790072
Some times I found that http response return slow is not because business logic need to cost lots if time but the http framework constructed by netty. I have made an experiment.
```
while (true) {
long start = System.currentTimeMillis();
TestHeartBeat.getResultForUrl(url2, null, connection_timeout, read_timeout);
long cost = System.currentTimeMillis() - start;
String strDateFormat = "yyyy-MM-dd HH:mm:ss";
SimpleDateFormat sdf = new SimpleDateFormat(strDateFormat);
System.out.println( sdf.format(new Date(start)) + " get result2 cost " + cost + "ms");
Thread.sleep(5000);
}
```
olap_tool.sh read_timeout=10000 (timeout here i set it to 10s)
client log
```
2020-12-07 17:15:51 get result2 cost 77ms
2020-12-07 17:15:57 get result2 cost 8ms
2020-12-07 17:16:02 get result2 cost 6ms
2020-12-07 17:16:07 get result2 cost 3254ms
2020-12-07 17:16:15 get result2 cost 6ms
2020-12-07 17:16:20 get result2 cost 7ms
2020-12-07 17:16:25 get result2 cost 7ms
2020-12-07 17:16:30 get result2 cost 3326ms
2020-12-07 17:16:38 get result2 cost 6ms
2020-12-07 17:16:43 get result2 cost 6ms
2020-12-07 17:16:48 get result2 cost 7ms
2020-12-07 17:16:53 get result2 cost 4186ms
2020-12-07 17:17:02 get result2 cost 7ms
2020-12-07 17:17:07 get result2 cost 6ms
2020-12-07 17:17:12 get result2 cost 5ms
```
server log
```
2020-12-07 17:15:51,981 INFO 148 [RestBaseAction.handleRequest():52] receive http request. url=/metrics?test_url
2020-12-07 17:15:57,015 INFO 78 [RestBaseAction.handleRequest():52] receive http request. url=/metrics?test_url
2020-12-07 17:16:02,023 INFO 92 [RestBaseAction.handleRequest():52] receive http request. url=/metrics?test_url
2020-12-07 17:16:10,276 INFO 100 [RestBaseAction.handleRequest():52] receive http request. url=/metrics?test_url
2020-12-07 17:16:15,285 INFO 115 [RestBaseAction.handleRequest():52] receive http request. url=/metrics?test_url
2020-12-07 17:16:20,293 INFO 126 [RestBaseAction.handleRequest():52] receive http request. url=/metrics?test_url
2020-12-07 17:16:25,300 INFO 138 [RestBaseAction.handleRequest():52] receive http request. url=/metrics?test_url
2020-12-07 17:16:33,627 INFO 152 [RestBaseAction.handleRequest():52] receive http request. url=/metrics?test_url
2020-12-07 17:16:38,635 INFO 86 [RestBaseAction.handleRequest():52] receive http request. url=/metrics?test_url
2020-12-07 17:16:43,641 INFO 100 [RestBaseAction.handleRequest():52] receive http request. url=/metrics?test_url
2020-12-07 17:16:48,649 INFO 108 [RestBaseAction.handleRequest():52] receive http request. url=/metrics?test_url
2020-12-07 17:16:57,835 INFO 119 [RestBaseAction.handleRequest():52] receive http request. url=/metrics?test_url
2020-12-07 17:17:02,842 INFO 138 [RestBaseAction.handleRequest():52] receive http request. url=/metrics?test_url
2020-12-07 17:17:07,849 INFO 150 [RestBaseAction.handleRequest():52] receive http request. url=/metrics?test_url
2020-12-07 17:17:12,855 INFO 80 [RestBaseAction.handleRequest():52] receive http request. url=/metrics?test_url
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #5021: [Proposal] Use thrift protocol instead of http protocol for master to send heartbeat to other fe to escape unstable heartbeat response
Posted by GitBox <gi...@apache.org>.
morningman closed issue #5021:
URL: https://github.com/apache/incubator-doris/issues/5021
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui edited a comment on issue #5021: [Proposal] Use thrift protocol instead of http protocol for master to send heartbeat to other fe to escape unstable heartbeat response
Posted by GitBox <gi...@apache.org>.
caiconghui edited a comment on issue #5021:
URL: https://github.com/apache/incubator-doris/issues/5021#issuecomment-745080077
> +1. But I think we still need to deep into this issue: Why http heartbeat will timeout.
@kangkaisen
netty use reactor model, some connection would be binded to one NioEventLoop thread, if some business logic wasting to much time, it will affect other requeset, and now doris process all business logic in io thread(NioEventLoop)
some http url may cause slow reponse, like /api/show_proc?path=/transactions/29327, this url sometimes may get result slowly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on issue #5021: [Proposal] Use thrift protocol instead of http protocol for master to send heartbeat to other fe to escape unstable heartbeat response
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on issue #5021:
URL: https://github.com/apache/incubator-doris/issues/5021#issuecomment-745089638
@caiconghui I seeļ¼ Thank you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui edited a comment on issue #5021: [Proposal] Use thrift protocol instead of http protocol for master to send heartbeat to other fe to escape unstable heartbeat response
Posted by GitBox <gi...@apache.org>.
caiconghui edited a comment on issue #5021:
URL: https://github.com/apache/incubator-doris/issues/5021#issuecomment-745080077
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui edited a comment on issue #5021: [Proposal] Use thrift protocol instead of http protocol for master to send heartbeat to other fe to escape unstable heartbeat response
Posted by GitBox <gi...@apache.org>.
caiconghui edited a comment on issue #5021:
URL: https://github.com/apache/incubator-doris/issues/5021#issuecomment-745080077
> +1. But I think we still need to deep into this issue: Why http heartbeat will timeout.
@kangkaisen
netty use reactor model, some connection would be binded to one NioEventLoop thread, if some business logic wasting to much time, it will affect other request, and now doris process all business logic in io thread(NioEventLoop)
some http url may cause slow response, like /api/show_proc?path=/transactions/29327, this url sometimes may get result slowly.
here can be one of the cause that why http request received by handler delayed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on issue #5021: [Proposal] Use thrift protocol instead of http protocol for master to send heartbeat to other fe to escape unstable heartbeat response
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on issue #5021:
URL: https://github.com/apache/incubator-doris/issues/5021#issuecomment-739911041
+1. But I think we still need to deep into this issue: Why http heartbeat will timeout.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on issue #5021: [Proposal] Use thrift protocol instead of http protocol for master to send heartbeat to other fe to escape unstable heartbeat response
Posted by GitBox <gi...@apache.org>.
caiconghui commented on issue #5021:
URL: https://github.com/apache/incubator-doris/issues/5021#issuecomment-739791003
@kangkaisen @morningman
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org