[jira] [Resolved] (IMPALA-6346) Potential deadlock in KrpcDataStreamMgr

["Sailesh Mukil (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sailesh Mukil resolved IMPALA-6346.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.12.0

Commit in:
https://github.com/apache/impala/commit/ff86feaa67ff8bf703896e33d9a358e42739ae30

IMPALA-6346: Potential deadlock in KrpcDataStreamMgr

In KrpcDataStreamMgr::CreateRecvr() we take the lock_ and
then call recvr->TakeOverEarlySender() for all contexts.
recvr->TakeOverEarlySender() then calls
recvr_->mgr_->EnqueueDeserializeTask((), which can block if the
deserialize pool queue is full. The next thread to become available
in that queue will also have to acquire lock_, thus leading to a
deadlock.

We fix this by moving the EarlySendersList out of the
EarlySendersMap and dropping the lock before taking any actions on
the RPC contexts in the EarlySendersList. All functions called after
dropping 'lock_' do not require the lock to protect them as they are
thread safe.

Additionally modified the BE test data-stream-test to work with KRPC
as well.

Testing: Added a new test to data-stream-test to verify that the
deadlock does not happen. Also, I verified that this test hangs
without the fix.

Change-Id: Ib7d1a8f12a4821092ca61ccc8a6f20c0404d56c7
Reviewed-on: http://gerrit.cloudera.org:8080/8950
Reviewed-by: Sailesh Mukil <sa...@cloudera.com>
Tested-by: Impala Public Jenkins

> Potential deadlock in KrpcDataStreamMgr
> ---------------------------------------
>
>                 Key: IMPALA-6346
>                 URL: https://issues.apache.org/jira/browse/IMPALA-6346
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Distributed Exec
>    Affects Versions: Impala 2.11.0
>            Reporter: Lars Volker
>            Assignee: Sailesh Mukil
>            Priority: Major
>              Labels: deadlock
>             Fix For: Impala 2.12.0
>
>
> In {{KrpcDataStreamMgr::CreateRecvr()}} [we take the lock_|https://github.com/apache/impala/blob/061287d912d260ffe569ed9fce3e9ce12af2ab68/be/src/runtime/krpc-data-stream-mgr.cc#L102] and then call {{recvr->TakeOverEarlySender()}} for all contexts. {{recvr->TakeOverEarlySender()}} [then calls|https://github.com/apache/impala/blob/061287d912d260ffe569ed9fce3e9ce12af2ab68/be/src/runtime/krpc-data-stream-recvr.cc#L428] {{recvr_\->mgr_->EnqueueDeserializeTask(()}}, which can block if the deserialize pool queue is full. The next thread to become available in that queue will also have to acquire {{lock_}}, thus leading to a deadlock.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)