[jira] [Assigned] (ARROW-15035) [C++] Create I/O thread pools per filesystem

["Todd Farmer (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-15035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Farmer reassigned ARROW-15035:
-----------------------------------

    Assignee:     (was: Vibhatha Lakmal Abeykoon)

This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

> [C++] Create I/O thread pools per filesystem
> --------------------------------------------
>
>                 Key: ARROW-15035
>                 URL: https://issues.apache.org/jira/browse/ARROW-15035
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++
>            Reporter: Weston Pace
>            Priority: Major
>
> The IOContext gets us most of the way here but we still don't do this yet today.  One concrete advantage to this is it allows us to more intelligently set the number of I/O threads.
> For example, 8 threads is often too small for an S3 filesystem (ARROW-14965)
> On the other hand, in some cases, 8 threads can be too many for an HDD (ARROW-14354)
> I doubt we will be able to figure out the ideal size of the I/O thread pool for any filesystem (e.g. on an S3 filesystem it depends on how many cores you have and how much bandwidth the system has) but we can possibly have more sensible defaults.
> Furthermore, it will hopefully clarify to the user the connection between filesystem and I/O thread pool size.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)