You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Kanishth Karthik (Jira)" <ji...@apache.org> on 2020/03/26 13:19:00 UTC
[jira] [Comment Edited] (THRIFT-4963) TNonblockingServer blocked
int addTask(IOThread) and notify(workerThread)
[ https://issues.apache.org/jira/browse/THRIFT-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067673#comment-17067673 ]
Kanishth Karthik edited comment on THRIFT-4963 at 3/26/20, 1:18 PM:
--------------------------------------------------------------------
I encountered the same issue in thrift 0.10.0 and it still persists in thrift 0.14.0. I have created a pull request to resolve this.
The problem is that the thread is stuck trying to obtain the lock on the first line in addTask and this lock is held by the worker (if the task has expired) trying to notify the IO thread. Both will not let go before the other hence deadlock.
This can be resolved if we release the lock while closing the connection and notifying just like the case of processing when the task is not expired and acquire it back once that is done.
Refer: ThreadManager::Task::run() for the same.
was (Author: kanishthkarthik):
I encountered the same issue in thrift 0.10.0 () and it still persists in thrift 0.14.0. I have created a pull request to resolve this.
The problem is that the thread is stuck trying to obtain the lock on the first line in addTask and this lock is held by the worker (if the task has expired) trying to notify the IO thread. Both will not let go before the other hence deadlock.
This can be resolved if we release the lock while closing the connection and notifying just like the case of processing when the task is not expired and acquire it back once that is done.
Refer: ThreadManager::Task::run() for the same.
> TNonblockingServer blocked int addTask(IOThread) and notify(workerThread)
> -------------------------------------------------------------------------
>
> Key: THRIFT-4963
> URL: https://issues.apache.org/jira/browse/THRIFT-4963
> Project: Thrift
> Issue Type: Bug
> Components: C++ - Library
> Affects Versions: 0.12.0
> Reporter: chenguang9239
> Priority: Major
> Attachments: CppClient.cpp, CppServer.cpp, deadlock_thread.png, wait_timeout.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> hello!
> when using c++ TNonblockingServer(with thread pool)，I found it blocked in high QPS status.
> I used pstack to print thread stack and found worker thread and IO thread blocked at:
> The worker thread will call notifyIOThread when it handle expired task, then call TNonblockingIOThread::notify and wait for POLLOUT in poll without timeout.
> The IO thread will call addTask when it gets requests. And IO threads will lock threadManager->mutex_ in addTask without a timeout
> Is it a bug of thrift 0.12.0?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)