You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by "sinomiko (via GitHub)" <gi...@apache.org> on 2023/11/23 11:01:18 UTC

[I] 在大流量下,消息大,下游hang住情况下,内存暴涨 (brpc)

sinomiko opened a new issue, #2457:
URL: https://github.com/apache/brpc/issues/2457

   **Describe the bug (描述bug)**
   如标题:
   k8s部署,使用jemalloc 5.3.0,在突发大流量下2000qps, 下游hang住后,服务会内存突然上涨并直到oom。
   
   流量下降后,concurrency 最大800,恢复到2,查看了下socket iobuf,也会恢复正常。
   
   看了下brpc有些使用的是object pool,怀疑在这里。
   ```
   WriteRequest* req = butil::get_object<WriteRequest>();
   ```
   
   能否在brpc上增加一个接口,将object pool信息导出到metrics接口,或者bvar上
   
   **To Reproduce (复现方法)**
   
   服务开启了snappy压缩;
   基本消息都很大,大多超过100K
   
   **Expected behavior (期望行为)**
   流量恢复后,concurrency,mem恢复正常。
   
   **Versions (各种版本)**
   OS:Ubuntu 20.04.3
   Compiler:clang : "8.0"
   brpc:1.4.0|HEAD|cfa1800
   protobuf:3.15.8
   
   **Additional context/screenshots (更多上下文/截图)**
   ![image](https://github.com/apache/brpc/assets/12888642/8539d257-b031-4057-a35f-941871019cca)
   ![image](https://github.com/apache/brpc/assets/12888642/97057863-d507-4b57-ac22-d6d8e22e00b9)
   ![image](https://github.com/apache/brpc/assets/12888642/90bcc584-82b3-41d3-9920-fbbb699452d6)
   ![image](https://github.com/apache/brpc/assets/12888642/dc331350-6181-4cbb-80a9-192d45c327b7)
   
   
   [growth_Insert Watermark.pdf](https://github.com/apache/brpc/files/13449005/growth_Insert.Watermark.pdf)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] 在大流量下,消息大,下游hang住情况下,内存暴涨 (brpc)

Posted by "sinomiko (via GitHub)" <gi...@apache.org>.
sinomiko closed issue #2457: 在大流量下,消息大,下游hang住情况下,内存暴涨
URL: https://github.com/apache/brpc/issues/2457


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] 在大流量下,消息大,下游hang住情况下,内存暴涨 (brpc)

Posted by "sinomiko (via GitHub)" <gi...@apache.org>.
sinomiko commented on issue #2457:
URL: https://github.com/apache/brpc/issues/2457#issuecomment-1842425833

   最初怀疑是`SharedObject` `IOBuf`导致:
   
   加了全局兼容`SharedObject`个数监控,申请和释放都是正常的。
   `IOBuf` 这块从已有监控看也是释放了的。
   
   
   从heap profile看是arena 有大量内存没有释放,排查protobuf有个类似问题
   [memory leak in PackFrom() on arena-allocated Any objects](https://github.com/protocolbuffers/protobuf/issues/8264)
   
   找到一个可以[优化的地方](https://github.com/apache/brpc/issues/2466)
   先关掉这个卡


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] 在大流量下,消息大,下游hang住情况下,内存暴涨 (brpc)

Posted by "imdouyu (via GitHub)" <gi...@apache.org>.
imdouyu commented on issue #2457:
URL: https://github.com/apache/brpc/issues/2457#issuecomment-3516563966

   这个问题有结论吗


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org