[jira] [Created] (ARROW-15035) [C++] Create I/O thread pools per filesystem

["Weston Pace (Jira)" <ji...@apache.org>]

Weston Pace created ARROW-15035:
-----------------------------------

             Summary: [C++] Create I/O thread pools per filesystem
                 Key: ARROW-15035
                 URL: https://issues.apache.org/jira/browse/ARROW-15035
             Project: Apache Arrow
          Issue Type: Improvement
          Components: C++
            Reporter: Weston Pace


The IOContext gets us most of the way here but we still don't do this yet today.  One concrete advantage to this is it allows us to more intelligently set the number of I/O threads.

For example, 8 threads is often too small for an S3 filesystem (ARROW-14965)
On the other hand, in some cases, 8 threads can be too many for an HDD (ARROW-14354)

I doubt we will be able to figure out the ideal size of the I/O thread pool for any filesystem (e.g. on an S3 filesystem it depends on how many cores you have and how much bandwidth the system has) but we can possibly have more sensible defaults.

Furthermore, it will hopefully clarify to the user the connection between filesystem and I/O thread pool size.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)