You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2020/01/02 14:29:27 UTC

[GitHub] [rocketmq-client-cpp] whk-isat opened a new issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

whk-isat opened a new issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211
 
 
   I used producer send sync message , sometimes it will send failed, some error info  like :GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
   log is show blow:
   ![image](https://user-images.githubusercontent.com/31062226/71671630-7c3aa900-2dae-11ea-8eaa-feb0153a8260.png)
   I find it in code , 
   ![image](https://user-images.githubusercontent.com/31062226/71671695-aab88400-2dae-11ea-9a42-cc2fbc0f0ea3.png)
   it seems that lock tcpTable failure, it return null tcp connection,then send failed , How can I do with it. 
   
   I use master code, its commit id : 817a81710dfeafff479aae42f9acfa92f5eb4619
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [rocketmq-client-cpp] ustcr7 edited a comment on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

Posted by GitBox <gi...@apache.org>.
ustcr7 edited a comment on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211#issuecomment-570995010
 
 
   ![image](https://user-images.githubusercontent.com/5212409/71794264-a4aaf780-307b-11ea-84e8-ebf00bfd9804.png)
   ![image](https://user-images.githubusercontent.com/5212409/71794273-b391aa00-307b-11ea-8dd5-ddb7d0cef587.png)
   this is the stacktrace of the process, m_tcpTableLock is hold by the dead lock thread, and deadlock is caused by a libevent bug :
   https://archives.seul.org/libevent/users/Aug-2012/msg00010.html
   https://github.com/libevent/libevent/blob/master/whatsnew-2.1.txt
   you should update the libevent to 2.1.xx version.
   @ShannonDing 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [rocketmq-client-cpp] ifplusor commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

Posted by GitBox <gi...@apache.org>.
ifplusor commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211#issuecomment-573551118
 
 
   这里的问题比较奇怪,在 CloseTransport 过程会对 TcpRemotingClient::m_tcpTableLock 加锁,然后调用 TcpTransport::disconnect -> TcpTransport::freeBufferEvent 通过操作 bufferevent 进入 libevent,最终是锁在了 libevent 里面。所以才有一系列的 timed_mutex timeout。要解决这个问题,需要先定位为什么会锁在 libevent 里。

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [rocketmq-client-cpp] ShannonDing commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

Posted by GitBox <gi...@apache.org>.
ShannonDing commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211#issuecomment-570479655
 
 
   link #204 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [rocketmq-client-cpp] ustcr7 commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

Posted by GitBox <gi...@apache.org>.
ustcr7 commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211#issuecomment-572363606
 
 
   > @ustcr7 I have read these page. Does it mean cleanup bufferevent in eventloop thread will avoid deadlock?
   
   No,it's a little complex. just upgrade libevent to release-2.1.11-stable can solve the deadlock question.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [rocketmq-client-cpp] whk-isat commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

Posted by GitBox <gi...@apache.org>.
whk-isat commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211#issuecomment-570225103
 
 
   @ShannonDing plz

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [rocketmq-client-cpp] ustcr7 commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

Posted by GitBox <gi...@apache.org>.
ustcr7 commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211#issuecomment-573542576
 
 
   > @ustcr7 Did you reproducer this issue again after update the libevent tp 2.1.x?
   no.  in our service, after update to libevent 1.1.11-stable, the deadlock question is solved.
   
   By the way, the deadlock problem is fairly easy to reproduce in test environment (concurrently send message in multi thread, kill broker and restart, then dealock is happend) ...
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [rocketmq-client-cpp] ShannonDing commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

Posted by GitBox <gi...@apache.org>.
ShannonDing commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211#issuecomment-570479385
 
 
   hi, it seems a bug on the transport layer and it is under checking.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [rocketmq-client-cpp] ustcr7 commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

Posted by GitBox <gi...@apache.org>.
ustcr7 commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211#issuecomment-573565121
 
 
   > 这里的问题比较奇怪,在 CloseTransport 过程会对 TcpRemotingClient::m_tcpTableLock 加锁,然后调用 TcpTransport::disconnect -> TcpTransport::freeBufferEvent 通过操作 bufferevent 进入 libevent,最终是锁在了 libevent 里面。所以才有一系列的 timed_mutex timeout。要解决这个问题,需要先定位为什么会锁在 libevent 里。
   
   其实就是libevent在高并发情况下,关闭连接时的bug。下面几个链接有说明。
   https://archives.seul.org/libevent/users/Aug-2012/msg00010.html  
   
   https://github.com/libevent/libevent/issues/225  
   
   https://archives.seul.org/libevent/users/Apr-2013/msg00002.html  
   
   我根据代码和链接里面的解释大概画了个图:
   ![image](https://user-images.githubusercontent.com/5212409/72243123-07753380-3626-11ea-8159-54a83dfa08f5.png)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [rocketmq-client-cpp] ShannonDing closed issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

Posted by GitBox <gi...@apache.org>.
ShannonDing closed issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [rocketmq-client-cpp] ShannonDing commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

Posted by GitBox <gi...@apache.org>.
ShannonDing commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211#issuecomment-573486474
 
 
   @ustcr7 Did you reproducer this issue again after update the libevent tp 2.1.x?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [rocketmq-client-cpp] ShannonDing commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

Posted by GitBox <gi...@apache.org>.
ShannonDing commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211#issuecomment-573584154
 
 
   link PR #30 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [rocketmq-client-cpp] ustcr7 edited a comment on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

Posted by GitBox <gi...@apache.org>.
ustcr7 edited a comment on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211#issuecomment-570995010
 
 
   ![image](https://user-images.githubusercontent.com/5212409/71794264-a4aaf780-307b-11ea-84e8-ebf00bfd9804.png)
   ![image](https://user-images.githubusercontent.com/5212409/71794273-b391aa00-307b-11ea-8dd5-ddb7d0cef587.png)
   this is the stacktrace of the process,m_tcpTableLock is hold by the dead lock thread, and deadlock is caused by a libevent bug :
   https://archives.seul.org/libevent/users/Aug-2012/msg00010.html
   https://github.com/libevent/libevent/blob/master/whatsnew-2.1.txt
   you should update the libevent to 2.1.xx version.
   @ShannonDing 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [rocketmq-client-cpp] ifplusor commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

Posted by GitBox <gi...@apache.org>.
ifplusor commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211#issuecomment-572292235
 
 
   @ustcr7 I have read these page. Does it mean cleanup bufferevent in eventloop thread will avoid deadlock?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [rocketmq-client-cpp] ustcr7 commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

Posted by GitBox <gi...@apache.org>.
ustcr7 commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211#issuecomment-570995010
 
 
   ![image](https://user-images.githubusercontent.com/5212409/71794264-a4aaf780-307b-11ea-84e8-ebf00bfd9804.png)
   ![image](https://user-images.githubusercontent.com/5212409/71794273-b391aa00-307b-11ea-8dd5-ddb7d0cef587.png)
   this is the stacktrace of the process,m_tcpTableLock is hold by the dead lock thread, and deadlock is caused by a libevent bug :
   https://archives.seul.org/libevent/users/Aug-2012/msg00010.html
   https://github.com/libevent/libevent/blob/master/whatsnew-2.1.txt
   you should update the libevent to 2.1.xx version.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [rocketmq-client-cpp] ifplusor commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout

Posted by GitBox <gi...@apache.org>.
ifplusor commented on issue #211: GetTransport of:x.x.x.x:yyyy get timed_mutex timeout
URL: https://github.com/apache/rocketmq-client-cpp/issues/211#issuecomment-573582848
 
 
   @ustcr7 thank you for your information. I think upgrade libevent to 2.1.11-stable is necessary, 2.0.x not contain all the bug fixes. @ShannonDing 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services