You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@servicecomb.apache.org by GitBox <gi...@apache.org> on 2020/08/14 08:44:37 UTC
[GitHub] [servicecomb-java-chassis] MockingJayWong opened a new issue #1919: 微服务调用超时探究
MockingJayWong opened a new issue #1919:
URL: https://github.com/apache/servicecomb-java-chassis/issues/1919
### 微服务调用超时探究
> 本服务A目前调用服务B同一接口偶现超时情况, 通过定位metric日志和access日志，目前发现两种场景。
* 本服务内报超时, 但是对方的access.log无记录日志。对应的metric日志描述如下，getConnect阶段就失效了。检查了client端连接池的配置，单个ip:port连接数最大限制配置的是50，而且查看对应的连接数情况，未找出连接过多而无法获取连接的异常嫌疑。___这块的原因还有什么可能性？___
```
os:
cpu:
all usage: 0.60% all idle: 99.40% process: 3.55%
net:
send(Bps) recv(Bps) send(pps) recv(pps) interface
5 5 0 0 lo
111.120K 42.174K 268 336 eth0
vertx:
instances:
name eventLoopContext-created
registry 0
transport 0
monitor-center 0
config-center 0
transport:
client.endpoints:
remote connectCount disconnectCount queue connections send(Bps) receive(Bps)
(summary) 76 48 0 99 364 12.579K
server.endpoints:
listen connectCount disconnectCount rejectByLimit connections send(Bps) receive(Bps)
0.0.0.0:18080 30 20 0 71 17.956K 561
(summary) 30 20 0 71 17.956K 561
---省略----
details:
rest.200:
rest.400:
rest.490:
服务B.schema.对应接口:
prepare : 0.008/0.008 handlersReq : 0.217/0.217 cFiltersReq: 0.022/0.022 sendReq : 0.000/0.000
getConnect : 0.000/0.000 writeBuf : 0.000/0.000 waitResp : 0.000/0.000 wakeConsumer: 0.001/0.001
cFiltersResp: 0.000/0.000 handlersResp: 2.740/2.740
```
* 本服务报超时, 但是对方服务的metric.log, access.log中表现正常
```
access.log duration显示2ms:
[20200810 14:01:46.707][I][tloop-thread-19][accesslog.42] -10.4.161.25||"POST 对应接口 HTTP/1.1"||200||2||76||10.11.88.79:18080||10/08/2020:14:01:46 +0800||5f30e2ca44821e32||-||-||-
metrics.log:
对应接口:_ prepare: 0.009/0.009 queue : 0.060/0.060 filtersReq : 0.035/0.035 handlersReq: 0.045/0.045_ execute: 1.453/1.453 handlersResp: 0.011/0.011 filtersResp: 0.030/0.030 sendResp : 0.012/0.012_
```
请教下两种情况下的超时分析思路，谢谢。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] MockingJayWong commented on issue #1919: 微服务调用超时探究
Posted by GitBox <gi...@apache.org>.
MockingJayWong commented on issue #1919:
URL: https://github.com/apache/servicecomb-java-chassis/issues/1919#issuecomment-673964623
@liubao68
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] MockingJayWong commented on issue #1919: 微服务调用超时探究
Posted by GitBox <gi...@apache.org>.
MockingJayWong commented on issue #1919:
URL: https://github.com/apache/servicecomb-java-chassis/issues/1919#issuecomment-674628618
目前超时时间这块配置的是500ms.
我这里打印的是两个metrics.log， 第一个只打了consumer的，第二个只打了对面接口provider的。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] 907474646 removed a comment on issue #1919: 微服务调用超时探究
Posted by GitBox <gi...@apache.org>.
907474646 removed a comment on issue #1919:
URL: https://github.com/apache/servicecomb-java-chassis/issues/1919#issuecomment-674491531
本服务超时，对方服务正常，大概率是对方服务业务处理超时了。对方那边不会有错误日志的，因为对方服务没有处理超时的概念，只有请求方有做超时处理
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] liubao68 commented on issue #1919: 微服务调用超时探究
Posted by GitBox <gi...@apache.org>.
liubao68 commented on issue #1919:
URL: https://github.com/apache/servicecomb-java-chassis/issues/1919#issuecomment-674626305
你的超时时间是否设置的很短？
（1）看时间是 handlersResp: 2.740/2.740 比较耗时。 这个时间的含义是client已经收到响应，然后通过 HttpClientFilters 处理响应耗时。
（2）这个 metrics 代表的是 producer 执行时间， execute 是业务执行时间，可能包含访问其他 client、数据库等逻辑。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] 907474646 commented on issue #1919: 微服务调用超时探究
Posted by GitBox <gi...@apache.org>.
907474646 commented on issue #1919:
URL: https://github.com/apache/servicecomb-java-chassis/issues/1919#issuecomment-674491531
本服务超时，对方服务正常，大概率是对方服务业务处理超时了。对方那边不会有错误日志的，因为对方服务没有处理超时的概念，只有请求方有做超时处理
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org