You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by GitBox <gi...@apache.org> on 2019/08/14 07:05:19 UTC

[GitHub] [incubator-brpc] lwmonster opened a new issue #887: 大量请求被堵住，进到服务时已经超时

lwmonster opened a new issue #887: 大量请求被堵住，进到服务时已经超时
URL: https://github.com/apache/incubator-brpc/issues/887
 
 
   对我们的服务进行压测发现一个现象：
   **qps 达到 2000 多的时候，cpu 利用率只有 50%，不会再升高了。此时从客户端看，有大量超时，超时时间是 70ms。**
   ![image](https://user-images.githubusercontent.com/4264580/62999836-34b88c80-bea2-11e9-9f91-855cb26321ae.png)
   ![image](https://user-images.githubusercontent.com/4264580/63000043-c3c5a480-bea2-11e9-8dd6-c5ce90a70c42.png)
   从上图可以看到，有超过一半的请求都超时了。
   
   
   ## 首先排查系统资源问题，感觉并不存在 io-bound 问题：
   ![image](https://user-images.githubusercontent.com/4264580/62999714-dbe8f400-bea1-11e9-8b7b-9e4e08b1fcc5.png)
   ![image](https://user-images.githubusercontent.com/4264580/62999736-eacfa680-bea1-11e9-8cd4-7db62d68e5fd.png)
   ![image](https://user-images.githubusercontent.com/4264580/62999744-ee632d80-bea1-11e9-950a-279d57d9dbee.png)
   ![image](https://user-images.githubusercontent.com/4264580/62999748-f15e1e00-bea1-11e9-8195-f20be5ad46d4.png)
   
   ## 然后查看 rpcz：
   ![image](https://user-images.githubusercontent.com/4264580/63000149-0a1b0380-bea3-11e9-9ace-1d90a9666f69.png)
   ![image](https://user-images.githubusercontent.com/4264580/63000152-0c7d5d80-bea3-11e9-8310-da05e7617e92.png)
   从上图可以看出，服务端的实际处理时间很短，感觉请求都堵在了网络上，所以怀疑网络带宽不足，接下来测试网络带宽。
   
   ## 网络带宽测试
   ![image](https://user-images.githubusercontent.com/4264580/63000625-31260500-bea4-11e9-9992-548f1ad89f02.png)
   可以看到万兆网卡的带宽也没有问题。
   
   ## contention profiling
   ![image](https://user-images.githubusercontent.com/4264580/63000690-5a469580-bea4-11e9-9413-dd07d55cd641.png)
   又做了一下 contention profiling ，也没感觉有啥问题。
   
   思路到这里卡住了，现在看到的现象是：从客户端发起请求，到 brpc Received request 这段时间比较长，各位神牛帮忙看看？ 
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org