You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@arrow.apache.org by "Anton Malakhov (Jira)" <ji...@apache.org> on 2021/02/25 11:50:00 UTC

[jira] [Commented] (ARROW-7001) [C++] Develop threading APIs to accommodate nested parallelism

    [ https://issues.apache.org/jira/browse/ARROW-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17290873#comment-17290873 ] 

Anton Malakhov commented on ARROW-7001:
---------------------------------------

Please consider ideas from this blog: https://habr.com/en/company/intel/blog/542908/

> [C++] Develop threading APIs to accommodate nested parallelism 
> ---------------------------------------------------------------
>
>                 Key: ARROW-7001
>                 URL: https://issues.apache.org/jira/browse/ARROW-7001
>             Project: Apache Arrow
>          Issue Type: New Feature
>          Components: C++
>            Reporter: Wes McKinney
>            Assignee: Weston Pace
>            Priority: Major
>             Fix For: 4.0.0
>
>
> Tasks invoked in parallel may be able to submit their own subtasks, which in OpenMP and TBB documentation is often called "nested parallelism". 
> If a task blocks on the completion of subtasks, then outright deadlocks are possible -- running tasks are all blocking on their subtasks, but the thread pool will not schedule any further tasks.
> I suggest that such code have a way to indicate to the thread pool (if one is passed in) that it is blocking on the completion of other tasks so that further tasks can be run while the task waits for its child tasks to complete. One possible way to do this is to have a floating "soft limit" for concurrent tasks that can be incremented when tasks are waiting. 
> So if we normally allow 8 concurrent tasks, then this can be temporarily increased for each "suspended" task. Preferably we would provide some way for the dependent task group to "awaken" the suspended task so that it does not have to do any work while waiting for the task group to finish
> Note this feature can also be used in tasks that are waiting for IO calls



--
This message was sent by Atlassian Jira
(v8.3.4#803005)