You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by GitBox <gi...@apache.org> on 2021/06/25 12:45:18 UTC
[GitHub] [incubator-brpc] wuzhiguocarter opened a new issue #1448: 上游请求下游耗时异常问题排查（下游服务本身耗时正常）
wuzhiguocarter opened a new issue #1448:
URL: https://github.com/apache/incubator-brpc/issues/1448
遇到一个真实的线上问题，A服务耗时自身看起来（日志和监控99分位不到10ms）没有问题，A的上游服务B调用A耗时异常大（从B服务监控看耗时99分位在50ms左右），做了如下的排查尝试：
（1） A/B服务cpu.idle都在90%以上很空闲
（2）查A/B服务的锁争用分析也不明显（0.001s/10s）
（3）单独压测A服务，A耗时无异常
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] wuzhiguocarter commented on issue #1448: 上游请求下游耗时异常问题排查（下游服务本身耗时正常）
Posted by GitBox <gi...@apache.org>.
wuzhiguocarter commented on issue #1448:
URL: https://github.com/apache/incubator-brpc/issues/1448#issuecomment-1021903165
> 所以更新到新的版本后就没有这个问题了吗 @wuzhiguocarter . 谢谢.
#1177 按照这个issue的方案修改了brpc源码来修复的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] wuzhiguocarter commented on issue #1448: 上游请求下游耗时异常问题排查（下游服务本身耗时正常）
Posted by GitBox <gi...@apache.org>.
wuzhiguocarter commented on issue #1448:
URL: https://github.com/apache/incubator-brpc/issues/1448#issuecomment-868474760
A服务是纯brpc server
B服务server端使用的thrift server，请求A服务使用的是brpc client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] xiaokai-wang commented on issue #1448: 上游请求下游耗时异常问题排查（下游服务本身耗时正常）
Posted by GitBox <gi...@apache.org>.
xiaokai-wang commented on issue #1448:
URL: https://github.com/apache/incubator-brpc/issues/1448#issuecomment-911478972
所以更新到新的版本后就没有这个问题了吗 @wuzhiguocarter . 谢谢.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] wuzhiguocarter commented on issue #1448: 上游请求下游耗时异常问题排查（下游服务本身耗时正常）
Posted by GitBox <gi...@apache.org>.
wuzhiguocarter commented on issue #1448:
URL: https://github.com/apache/incubator-brpc/issues/1448#issuecomment-893475202
根因和 https://github.com/apache/incubator-brpc/issues/1177 一致
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] chenzhangyi commented on issue #1448: 上游请求下游耗时异常问题排查（下游服务本身耗时正常）
Posted by GitBox <gi...@apache.org>.
chenzhangyi commented on issue #1448:
URL: https://github.com/apache/incubator-brpc/issues/1448#issuecomment-869391217
可以用 ss 命令看下连接上的 queue size。 如果queue size比较大的话。 这个问题可能和
https://github.com/apache/incubator-brpc/pull/1193
这个pr要解决的问题是类似的.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] cdjingit commented on issue #1448: 上游请求下游耗时异常问题排查（下游服务本身耗时正常）
Posted by GitBox <gi...@apache.org>.
cdjingit commented on issue #1448:
URL: https://github.com/apache/incubator-brpc/issues/1448#issuecomment-874712358
@wuzhiguocarter 看看最新的代码能解决你们碰到的问题吗？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] wuzhiguocarter commented on issue #1448: 上游请求下游耗时异常问题排查（下游服务本身耗时正常）
Posted by GitBox <gi...@apache.org>.
wuzhiguocarter commented on issue #1448:
URL: https://github.com/apache/incubator-brpc/issues/1448#issuecomment-893475957
> @wuzhiguocarter 看看最新的代码能解决你们碰到的问题吗？
感谢各位抽空帮忙！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] lorinlee commented on issue #1448: 上游请求下游耗时异常问题排查（下游服务本身耗时正常）
Posted by GitBox <gi...@apache.org>.
lorinlee commented on issue #1448:
URL: https://github.com/apache/incubator-brpc/issues/1448#issuecomment-869033642
Hi @wuzhiguocarter ，请问延迟高，是只测了B调用A服务这一小块的延迟吧？没有包含任何B的业务逻辑，只有rpc调用的开销，需要先排除B业务逻辑的开销哈。
看有单独对A服务做压测，压测客户端是使用了brpc吗？压测期间在客户端看到的延迟是否正常？
A服务耗时测的是哪部分的耗时，是否包括收到消息后的排队等待时间？B调用A的延迟avg和pct50是否也显著高于A服务本身的延迟？
网络调用链路其实是比较长的，延迟高可能是客户端机器负载不稳定，可能是网络不稳定，也可能是某些badcase导致长尾，情况很多。对于延迟高的情况，建议增加一些指标，或者通过抓网络包，先定位到哪个阶段慢，再具体排查是什么原因导致的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] wuzhiguocarter commented on issue #1448: 上游请求下游耗时异常问题排查（下游服务本身耗时正常）
Posted by GitBox <gi...@apache.org>.
wuzhiguocarter commented on issue #1448:
URL: https://github.com/apache/incubator-brpc/issues/1448#issuecomment-869296344
1. 排除了负载不稳定
2. 排除网络不稳定
3. 排除badcase导致的长尾
4. 压测客户端也是brpc client
5. B调用A的延迟avg和pct50是否也显著高于A服务本身的延迟
6. 抓包后发现ack_rtt > 20ms有很多，ping测试B->A rtt在0.1ms一下
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] wuzhiguocarter commented on issue #1448: 上游请求下游耗时异常问题排查（下游服务本身耗时正常）
Posted by GitBox <gi...@apache.org>.
wuzhiguocarter commented on issue #1448:
URL: https://github.com/apache/incubator-brpc/issues/1448#issuecomment-868507105
查看brpc框架源码，Socket::Create方法也禁用了nagling
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] wuzhiguocarter closed issue #1448: 上游请求下游耗时异常问题排查（下游服务本身耗时正常）
Posted by GitBox <gi...@apache.org>.
wuzhiguocarter closed issue #1448:
URL: https://github.com/apache/incubator-brpc/issues/1448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org