You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@arrow.apache.org by "Weston Pace (Jira)" <ji...@apache.org> on 2021/05/17 22:42:00 UTC

[jira] [Commented] (ARROW-10117) [C++] Implement work-stealing scheduler / multiple queues in ThreadPool

    [ https://issues.apache.org/jira/browse/ARROW-10117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17346470#comment-17346470 ] 

Weston Pace commented on ARROW-10117:
-------------------------------------

I'm a little confused by the word "workload" in the second paragraph.  Traditional work stealing attempts to keep tasks together based on thread/core to preserve cache coherency.  This is what appears to be described in the first paragraph.

 

In the second paragraph are you asking for the capability to also group tasks based on workload?  If so, I'm not sure what the benefit would be.  If not, I don't think we'll end up needing to modify the API.  A task can keep a thread_local reference to its queue.

> [C++] Implement work-stealing scheduler / multiple queues in ThreadPool
> -----------------------------------------------------------------------
>
>                 Key: ARROW-10117
>                 URL: https://issues.apache.org/jira/browse/ARROW-10117
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++
>            Reporter: Wes McKinney
>            Priority: Major
>
> This involves a change from a single task queue shared amongst all threads to a per-thread task queue and the ability for idle threads to take tasks from other threads' queues (work stealing). 
> As part of this, the task submission API would need to be evolved in some fashion to allow for tasks related to a particular workload to end up in the same task queue



--
This message was sent by Atlassian Jira
(v8.3.4#803005)