You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by GitBox <gi...@apache.org> on 2021/12/20 13:07:04 UTC
[GitHub] [incubator-brpc] lintanghui opened a new issue #1648: brpc::Join(bthread_id_t)死锁
lintanghui opened a new issue #1648:
URL: https://github.com/apache/incubator-brpc/issues/1648
**Describe the bug (描述bug)**
下游节点故障时，调用下游无返回，call_method占用完所有bthread导致timeout无法获取bthread死锁
**To Reproduce (复现方法)**
并发请求下游，并发数超过worker count，下游节点网络故障没有返回resp时，会出现死锁
**Expected behavior (期望行为)**
timeout能够被唤醒执行结束call_method 。下游节点没响应的时候每个call_method占用了一个bthread。 此时即使设置了超时，但是由于超时timeer拿不到bthread导致死锁无法结束请求
**Versions (各种版本)**
OS: debian
Compiler:
brpc:
protobuf:
**Additional context/screenshots (更多上下文/截图)**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] zyearn commented on issue #1648: brpc::Join(bthread_id_t)死锁
Posted by GitBox <gi...@apache.org>.
zyearn commented on issue #1648:
URL: https://github.com/apache/incubator-brpc/issues/1648#issuecomment-1002745519
这个问题和 @lintanghui 线下讨论了下，场景是A(proxy)发请求到B (data node)，这个栈是在A上的。
问题发生在A->B网络发生故障（交换机）的时候，B一直没有rpc response（请求都不一定到了B），而A一直在接收请求并向B发送请求，此时A内存一直在变大，导致A上没有足够bthread栈分配，后面bthread就跑在了pthread栈上，一路就调用到了bthread::butex_wait_from_pthread上，导致所有worker都在等待。此时timer也因rq full卡在sleep上，只能重启解决。
给A设置限流不能解决问题，A->C/D/E都是没问题的，问题B节点久而久之会把所有并发都占满导致到C/D/E的请求受影响。这种情况可以试试熔断，或者不要在B/C/D/E里随机选下游而是根据inflight请求数量选择。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] wwbmmm commented on issue #1648: brpc::Join(bthread_id_t)死锁
Posted by GitBox <gi...@apache.org>.
wwbmmm commented on issue #1648:
URL: https://github.com/apache/incubator-brpc/issues/1648#issuecomment-1002414616
是不是开启了-usercode_in_pthread?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] lintanghui commented on issue #1648: brpc::Join(bthread_id_t)死锁
Posted by GitBox <gi...@apache.org>.
lintanghui commented on issue #1648:
URL: https://github.com/apache/incubator-brpc/issues/1648#issuecomment-1002652262
> 是不是开启了-usercode_in_pthread?
没有开启，应该是请求堆积以后导致内存不足从而触发了pthread的创建
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org