You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by GitBox <gi...@apache.org> on 2021/06/23 11:47:17 UTC

[GitHub] [incubator-brpc] rpbear88 opened a new issue #1446: 使用RDMA之后内存一直上涨？

rpbear88 opened a new issue #1446:
URL: https://github.com/apache/incubator-brpc/issues/1446


   和大家分享一个我们观察到的内存上涨的问题(目前已经解决)。我们在开启RDMA功能后，进程的内存会一直上涨，过一段时间后就会出现如下的日志信息：
   
   ![image](https://user-images.githubusercontent.com/61613299/123090930-51895800-d45b-11eb-8a9d-7e38cf88b87d.png)
   
   看了下代码加测试，发现是因为RDMA实现中使用的block pool的region个数达到上限(4个)了，这说明这个block pool可能存在内存泄漏。反复走读了代码没有发现问题，从socket中收取数据时会从block pool获取内存空间，在controller释放时会回收空间放回block pool，看起来没有问题。
   
   所以内存是怎么泄露的呢？
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org

[GitHub] [incubator-brpc] rpbear88 commented on issue #1446: 使用RDMA之后内存一直上涨？

Posted by GitBox <gi...@apache.org>.
rpbear88 commented on issue #1446:
URL: https://github.com/apache/incubator-brpc/issues/1446#issuecomment-866769000


   后来我们发现，因为开启RDMA之后，IOBuf进行内存分配的全局函数被修改为block pool的相关函数了，这导致只要是使用IOBuf的逻辑都会使用该内存池。
   
   而我们业务使用的braft的log storage是选择的内存类型，所以这些log实际上都在内存里占用了IOBuf一直未释放，后来我们加上了定期snapshot就解决了这个问题。
   
   ps：期间还出现了阶段性地重新选举的现象，经分析应该是我们在RDMA线程(开启了polling+inplace)分配过大的内存(10GB)时导致整个线程不能响应其他raft心跳等操作，导致leader失效了。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org

[GitHub] [incubator-brpc] rpbear88 closed issue #1446: 使用RDMA之后内存一直上涨？

Posted by GitBox <gi...@apache.org>.
rpbear88 closed issue #1446:
URL: https://github.com/apache/incubator-brpc/issues/1446


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org