[jira] [Commented] (ARROW-6691) [Rust] [DataFusion] Use tokio and Futures instead of spawning threads

["Ben Sully (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024131#comment-17024131 ] 

Ben Sully commented on ARROW-6691:
----------------------------------

My hunch is also that `rayon` would be better suited to this part ('working' on lots of things in a threadpool, rather than 'waiting' for lots of things) than tokio + async/await, although it does sound like doing IO using gRPC might be well suited to tokio. I'm not familiar with the Datafusion codebase at all unfortunately, how approachable would this issue be for a first time Arrow contributor? :D

> [Rust] [DataFusion] Use tokio and Futures instead of spawning threads
> ---------------------------------------------------------------------
>
>                 Key: ARROW-6691
>                 URL: https://issues.apache.org/jira/browse/ARROW-6691
>             Project: Apache Arrow
>          Issue Type: Sub-task
>          Components: Rust, Rust - DataFusion
>            Reporter: Andy Grove
>            Priority: Major
>             Fix For: 1.0.0
>
>         Attachments: image-2019-12-07-17-54-57-862.png
>
>
> The current implementation of the physical query plan uses "thread::spawn" which is expensive. We should switch to using Futures, async!/await!, and tokio so that we are launching tasks in a thread pool instead and writing idiomatic Rust code with futures combinators to chain actions together.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)