You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Ahmar Suhail (Jira)" <ji...@apache.org> on 2022/01/14 11:04:00 UTC
[jira] [Commented] (HADOOP-16618) increase the default number of threads and http connections in S3A
[ https://issues.apache.org/jira/browse/HADOOP-16618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17476069#comment-17476069 ]
Ahmar Suhail commented on HADOOP-16618:
---------------------------------------
I started looking at this ticket and the default values are currently:
DEFAULT_MAXIMUM_CONNECTIONS = 96
DEFAULT_MAX_THREADS = 10
DEFAULT_SOCKET_SEND_BUFFER = 8 * 1024
I was wondering if you had any ideas for what the new numbers should be or what we can do to find these numbers? I tried tweaking the values and ran a couple of performance tests, see [concurrent renames|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3AConcurrentOps.java#L164] and [read file|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3AInputStreamPerformance.java#L244], but I don't know if that gives us enough data.
Also wanted to understand what advantage having a connection pool of size 96 gives? From what I understand, if we have max_threads to 10, we'll only ever use a max of 10 connections?
> increase the default number of threads and http connections in S3A
> ------------------------------------------------------------------
>
> Key: HADOOP-16618
> URL: https://issues.apache.org/jira/browse/HADOOP-16618
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2.1
> Reporter: Steve Loughran
> Priority: Major
>
> Enable bigger thread and http pools in the S3A connector, especially now that the transfer manager is doing parallel block transfer, as is rename()
> We can make a lot more with parallelism in a single thread, and for applications with multiple worker threads, we need bigger defaults
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org