[jira] [Comment Edited] (ARROW-8774) [Rust] [DataFusion] Improve threading model

["Wes McKinney (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105527#comment-17105527 ] 

Wes McKinney edited comment on ARROW-8774 at 5/12/20, 3:29 PM:
---------------------------------------------------------------

FYI we're also working on our thread scheduling and concurrent CPU/IO management APIs for C++ for nearly the exact same reasons you are. We might try to see if there are shared learnings, we can at least link issues on JIRA to point our design docs or work happening

cc [~fsaintjacques] [~lidavidm] [~apitrou]


was (Author: wesmckinn):
FYI we're also working on our thread scheduling and concurrent CPU/IO management APIs for C++. We might try to see if there are shared learnings, we can at least link issues on JIRA to point our design docs or work happening

cc [~fsaintjacques] [~lidavidm] [~apitrou]

> [Rust] [DataFusion] Improve threading model
> -------------------------------------------
>
>                 Key: ARROW-8774
>                 URL: https://issues.apache.org/jira/browse/ARROW-8774
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Rust, Rust - DataFusion
>            Reporter: Andy Grove
>            Assignee: Andy Grove
>            Priority: Major
>             Fix For: 1.0.0
>
>
> DataFusion currently spawns one thread per partition and this results in poor performance if there are more partitions than available cores/threads. It would be better to have a thread-pool that defaults to number of available cores.
> Here is a Google doc where we can collaborate on a design discussion.
> https://docs.google.com/document/d/1_wc6diy3YrRgEIhVIGzrO5AK8yhwfjWlmKtGnvbsrrY/edit?usp=sharing



--
This message was sent by Atlassian Jira
(v8.3.4#803005)