[jira] [Comment Edited] (ARROW-15036) [C++] Expose S3 SDK configuration parameter "maxConnections"

["Weston Pace (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17456047#comment-17456047 ] 

Weston Pace edited comment on ARROW-15036 at 12/8/21, 10:54 PM:
----------------------------------------------------------------

Actually, I think we could take two different routes here.

Option 1: Yet another S3Options config variable
  - Pro: More flexible
  - Con: Requires the user to set it correctly and line it up with the I/O thread pool size

Option 2: Set maxConnections to io_context_.executor()->GetCapacity()
  - Pro: Automatic configuration, don't need to burden the user with details
  - Con: May do the wrong thing if the user resizes the pool after creating the filesystem
  - Con: If the user is using S3Options::background_writes they may want more connections than I/O pool threads

[~apitrou] Any opinion or suggestion for a different approach?


was (Author: westonpace):
Actually, I think we could take two different routes here.

Option 1: Yet another S3Options config variable
  - Pros: More flexible
  - Cons: Requires the user to set it correctly and line it up with the I/O thread pool size

Option 2: Set maxConnections to io_context_.executor()->GetCapacity()
  - Pros: Automatic configuration, don't need to burden the user with details
  - Cons: May do the wrong thing if the user resizes the pool after creating the filesystem
  -  If the user is using S3Options::background_writes they may want more connections than I/O pool threads

[~apitrou] Any opinion or suggestion for a different approach?

> [C++] Expose S3 SDK configuration parameter "maxConnections"
> ------------------------------------------------------------
>
>                 Key: ARROW-15036
>                 URL: https://issues.apache.org/jira/browse/ARROW-15036
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++
>            Reporter: Weston Pace
>            Priority: Major
>
> This is primarily inspired by ARROW-14965 where it seems that the default (25) is limiting S3 filesystem performance.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)