You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@brpc.apache.org by "legionxiong (via GitHub)" <gi...@apache.org> on 2025/12/12 02:51:58 UTC

[I] brpc socket broken 后一直没有被回收，client 与 server 端无法重建连接 (brpc)

legionxiong opened a new issue, #3172:
URL: https://github.com/apache/brpc/issues/3172

   **Describe the bug**
   brpc socket broken 后一直没有被回收，client 与 server 端无法重建连接。
   
   # This is a broken Socket
   version=1
   shared_part={
     ref_count=1
     socket_pool=null
     creator_socket=170
     in_size=1288977874
     in_num_messages=4850513
     out_size=20318875586
     out_num_messages=4850841
     recent_error_count=58527
   }
   nref=3
   nevent=1
   fd=94
   tos=0
   reset_fd_to_now=61999138754us
   remote_side=172.20.10.3:8000
   local_side=172.20.10.177:55204
   on_et_events=0x7f1019133de0
   user=(brpc::InputMessenger*)0x556efc440850
   this_id=170
   preferred_index=1 (baidu_std)
   hc_count=0
   avg_input_msg_size=85
   read_buf=0
   last_read_to_now=59650542887us
   last_write_to_now=59646841539us
   overcrowded=0
   id_wait_list={}
   parsing_context=0
   pipeline_q=0
   hc_interval_s=3
   is_hc_related_ref_held=1
   ninprocess=1
   auth_flag_error=0
   auth_id=2216203124737
   auth_context=0
   logoff_flag=0
   last_error=104
   _additional_ref_status=2
   total_streams_buffer_size=0
   ninflight_app_health_check=0
   agent_socket_id=(none)
   cid=0
   write_head=0
   ssl_state=SSL_OFF
   is_write_shutdown=0
   keepalive=0
   tcp_keepalive_time=7200
   tcp_keepalive_intvl=75
   tcp_keepalive_probes=9
   tcp_user_timeout=0
   tcpi={
     state=7
     ca_state=0
     retransmits=0
     probes=0
     backoff=0
     options=7
     snd_wscale=7
     rcv_wscale=7
     rto=202000
     ato=40000
     snd_mss=1448
     rcv_mss=1448
     unacked=0
     sacked=0
     lost=0
     retrans=0
     fackets=0
     last_data_sent=59646841
     last_ack_sent=0
     last_data_recv=59650543
     last_ack_recv=59646841
     pmtu=1500
     rcv_ssthresh=3145728
     rtt=1157
     rttvar=2223
     snd_ssthresh=640
     snd_cwnd=17
     advmss=1448
     reordering=3
   }
   bthread_tag=0
   #
   
   **To Reproduce**
   这个问题在我们的使用场景中只有一个 client 出现了， 原因是 server 挂了十几分钟后又恢复了， 而这个 client 在 10 小时后都无法与 server 通信。 客户端都默认开启了 HealthCheck，但是没有开 keepalive。 
   
   **Expected behavior**
   broken socket 被回收掉，client 恢复与 server 通信。
   
   **Versions**
   OS: Rocky 8.4
   Compiler:
   brpc: 1.13.0
   protobuf:
   
   **Additional context/screenshots**
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org

Re: [I] brpc broken socket 无法恢复 ，client 与 server 端无法重建连接 (brpc)

Posted by "legionxiong (via GitHub)" <gi...@apache.org>.
legionxiong commented on issue #3172:
URL: https://github.com/apache/brpc/issues/3172#issuecomment-3663088705

   初步分析， 和 #1773 描述的问题一样， NRefOfVRef(vref)是3，大于expected_nref=2， 导致 Socket::WaitAndReset 陷入死循环而无法重建连接。感觉特定异常场景会出现两个 HC 同时走到 WaitAndReset ， 然后都陷入死循环 ， 导致 HC 机制失效（hc_count=0）。
   
   <!-- Failed to upload "image.png" -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org

Re: [I] brpc socket broken 后一直没有被回收，client 与 server 端无法重建连接 (brpc)

Posted by "legionxiong (via GitHub)" <gi...@apache.org>.
legionxiong commented on issue #3172:
URL: https://github.com/apache/brpc/issues/3172#issuecomment-3644727543

   与 #1168 问题类似。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org

Re: [I] brpc broken socket 无法恢复 ，client 与 server 端无法重建连接 (brpc)

Posted by "legionxiong (via GitHub)" <gi...@apache.org>.
legionxiong commented on issue #3172:
URL: https://github.com/apache/brpc/issues/3172#issuecomment-3685599071

   > > 是不是可以加一个原子变量，记录一下同时存在的HC个数？
   > 
   > 所以，得记录一些状态或者日志，复现的时候才能进一步分析吧。
   
   这个问题最后定位出来是应用层自身的问题， OnNewMessages 返回时析构 last_msg 在 ProcessInputMessage 中会原地执行用户 done->Run()。 我们应用层代码走到了重试发 rpc 的逻辑，导致  OnNewMessages 没有释放掉 broken socket 的引用计数。 重试的 rpc address socket 失败又再次重试， 重试的次数有点大，与 WaitAndSock 循环依赖，导致整个业务流程卡住了。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org

Re: [I] brpc broken socket 无法恢复 ，client 与 server 端无法重建连接 (brpc)

Posted by "chenBright (via GitHub)" <gi...@apache.org>.
chenBright commented on issue #3172:
URL: https://github.com/apache/brpc/issues/3172#issuecomment-3668570585

   > 是不是可以加一个原子变量，记录一下同时存在的HC个数？
   
   所以，得记录一些状态或者日志，复现的时候才能进一步分析吧。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org

Re: [I] brpc broken socket 无法恢复 ，client 与 server 端无法重建连接 (brpc)

Posted by "legionxiong (via GitHub)" <gi...@apache.org>.
legionxiong commented on issue #3172:
URL: https://github.com/apache/brpc/issues/3172#issuecomment-3664715022

   > 是不是可以加一个原子变量，记录一下同时存在的HC个数？
   
   这个问题只出现过一次， 我们在测试环境不断随机杀服务没有再复现过， 触发条件目前还未知。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org

Re: [I] brpc socket broken 后一直没有被回收，client 与 server 端无法重建连接 (brpc)

Posted by "legionxiong (via GitHub)" <gi...@apache.org>.
legionxiong commented on issue #3172:
URL: https://github.com/apache/brpc/issues/3172#issuecomment-3644792001

   我们线上的版本已经包含了 https://github.com/apache/brpc/issues/1773 和 https://github.com/apache/brpc/issues/3058 这两个问题的补丁。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org

Re: [I] brpc broken socket 无法恢复 ，client 与 server 端无法重建连接 (brpc)

Posted by "legionxiong (via GitHub)" <gi...@apache.org>.
legionxiong closed issue #3172: brpc broken socket 无法恢复 ，client 与 server 端无法重建连接
URL: https://github.com/apache/brpc/issues/3172


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org

Re: [I] brpc broken socket 无法恢复 ，client 与 server 端无法重建连接 (brpc)

Posted by "chenBright (via GitHub)" <gi...@apache.org>.
chenBright commented on issue #3172:
URL: https://github.com/apache/brpc/issues/3172#issuecomment-3664452620

   是不是可以加一个原子变量，记录一下同时存在的HC个数？


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org

Re: [I] brpc broken socket 无法恢复 ，client 与 server 端无法重建连接 (brpc)

Posted by "dandyhuang (via GitHub)" <gi...@apache.org>.
dandyhuang commented on issue #3172:
URL: https://github.com/apache/brpc/issues/3172#issuecomment-4331796778

   M


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org