[jira] [Created] (ARROW-10038) SetCpuThreadPoolCapacity(1) spins up nCPUs threads

["Adam Hooper (Jira)" <ji...@apache.org>]

Adam Hooper created ARROW-10038:
-----------------------------------

             Summary: SetCpuThreadPoolCapacity(1) spins up nCPUs threads
                 Key: ARROW-10038
                 URL: https://issues.apache.org/jira/browse/ARROW-10038
             Project: Apache Arrow
          Issue Type: Bug
            Reporter: Adam Hooper


When I call {{arrow::SetCpuThreadPoolCapacity(1);}}, Arrow does this:

1. Spins up a singleton {{ThreadPool}} with _the default thread count_;
2. Sets the number of threads on that {{ThreadPool}} to 1 -- killing the extra threads.

On my Intel system, I'm forced to spin up four threads to set the CPU thread-pool capacity to 1. This goes against the spirit of the API method -- or at least, my understanding of it (and my experience with other thread pools).

My workaround, for calling code: instead of calling {{arrow::SetCpuThreadPoolCapacity(1)}}, call {{setenv("OMP_NUM_THREADS", "1", 1)}}.

Brainstorming, here are some ideas for Arrow's global thread pool that would stop launching {{>limit}} threads to set the limit:

* {{cpu_thread_pool_capacity}} could be a global variable, not an attribute on the global {{ThreadPool}}. API users would be expected to set the thread-pool capacity _before_ creating the thread pool. (They're probably doing this anyway.)
* {{SetCpuThreadPoolCapacity()}} could call {{setenv("OMP_NUM_THREADS", ...)}}
* {{ThreadPool}} could create threads on-demand instead of in the ctor. An unused {{ThreadPool}} would launch zero threads -- resolving ARROW-10033 as a side-effect



--
This message was sent by Atlassian Jira
(v8.3.4#803005)