You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Antoine Pitrou (Jira)" <ji...@apache.org> on 2020/09/18 14:28:00 UTC
[jira] [Commented] (ARROW-10038) SetCpuThreadPoolCapacity(1) spins
up nCPUs threads
[ https://issues.apache.org/jira/browse/ARROW-10038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198372#comment-17198372 ]
Antoine Pitrou commented on ARROW-10038:
----------------------------------------
While it may "go against the spirit", is there any concrete problem with it?
> API users would be expected to set the thread-pool capacity before creating the thread pool.
That's undesirable. The global thread pool can be created implicitly, and users must be able to change the thread count afterwards.
> ThreadPool could create threads on-demand instead of in the ctor. An unused ThreadPool would launch zero threads – resolving ARROW-10033 as a side-effect
That's a possibiliy indeed.
> SetCpuThreadPoolCapacity(1) spins up nCPUs threads
> --------------------------------------------------
>
> Key: ARROW-10038
> URL: https://issues.apache.org/jira/browse/ARROW-10038
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Affects Versions: 1.0.1
> Reporter: Adam Hooper
> Priority: Major
>
> When I call {{arrow::SetCpuThreadPoolCapacity(1);}}, Arrow does this:
> 1. Spins up a singleton {{ThreadPool}} with _the default thread count_;
> 2. Sets the number of threads on that {{ThreadPool}} to 1 -- killing the extra threads.
> On my Intel system, I'm forced to spin up four threads to set the CPU thread-pool capacity to 1. This goes against the spirit of the API method -- or at least, my understanding of it (and my experience with other thread pools).
> My workaround, for calling code: instead of calling {{arrow::SetCpuThreadPoolCapacity(1)}}, call {{setenv("OMP_NUM_THREADS", "1", 1)}}.
> Brainstorming, here are some ideas for Arrow's global thread pool that would stop launching {{>limit}} threads to set the limit:
> * {{cpu_thread_pool_capacity}} could be a global variable, not an attribute on the global {{ThreadPool}}. API users would be expected to set the thread-pool capacity _before_ creating the thread pool. (They're probably doing this anyway.)
> * {{SetCpuThreadPoolCapacity()}} could call {{setenv("OMP_NUM_THREADS", ...)}}
> * {{ThreadPool}} could create threads on-demand instead of in the ctor. An unused {{ThreadPool}} would launch zero threads -- resolving ARROW-10033 as a side-effect
--
This message was sent by Atlassian Jira
(v8.3.4#803005)