You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@arrow.apache.org by "Andrew Lamb (Jira)" <ji...@apache.org> on 2021/02/26 21:40:00 UTC

[jira] [Created] (ARROW-11802) [Rust][DataFusion] Mixing of crossbeam channel and async tasks can lead to deadlock

Andrew Lamb created ARROW-11802:
-----------------------------------

             Summary: [Rust][DataFusion] Mixing of crossbeam channel and async tasks can lead to deadlock
                 Key: ARROW-11802
                 URL: https://issues.apache.org/jira/browse/ARROW-11802
             Project: Apache Arrow
          Issue Type: Bug
          Components: Rust - DataFusion
            Reporter: Andrew Lamb


[~edrevo] noticed, on https://github.com/apache/arrow/pull/9523#issuecomment-786237494, that the use of crossbeam channels can potentially deadlock datafusion

The use of crossbeam channel is left over from earlier, non `async` implementations and get been fingered in some hangs that [~MikeSeddonAU] has observed in DataFusion ). Specifically the crossbeam channel can block a thread when the channel is full or empty, which can result in blocking all the tokio executor threads and deadlocking the system

The proposal is is to use tokio's mpsc channels instead of crossbeam which can properly yield back to tokio to run another task when the channel is either full or empty. .




--
This message was sent by Atlassian Jira
(v8.3.4#803005)