You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by sl...@apache.org on 2024/01/25 14:33:29 UTC
(hadoop) branch branch-3.4.0 updated: HADOOP-19015. Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting for connection from pool. (#6372)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.4.0 by this push:
new cc1099785d90 HADOOP-19015. Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting for connection from pool. (#6372)
cc1099785d90 is described below
commit cc1099785d90f62f6ff2002fdfd89f442ba6a469
Author: Mukund Thakur <mt...@cloudera.com>
AuthorDate: Tue Jan 16 17:06:28 2024 -0600
HADOOP-19015. Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting for connection from pool. (#6372)
HADOOP-19015. Increase fs.s3a.connection.maximum to 500 to minimize the risk of Timeout waiting for connection from the pool
Contributed By: Mukund Thakur
---
.../hadoop-common/src/main/resources/core-default.xml | 3 ++-
.../hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java | 2 +-
.../hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md | 2 +-
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
index 5a5171056d04..29ec06db6598 100644
--- a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
+++ b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
@@ -1530,7 +1530,7 @@
<property>
<name>fs.s3a.connection.maximum</name>
- <value>200</value>
+ <value>500</value>
<description>Controls the maximum number of simultaneous connections to S3.
This must be bigger than the value of fs.s3a.threads.max so as to stop
threads being blocked waiting for new HTTPS connections.
@@ -1538,6 +1538,7 @@
</description>
</property>
+
<property>
<name>fs.s3a.connection.ssl.enabled</name>
<value>true</value>
diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
index e33f762cdfcf..636438afef25 100644
--- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
+++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
@@ -173,7 +173,7 @@ public final class Constants {
* Future releases are likely to increase this value.
* Keep in sync with the value in {@code core-default.xml}
*/
- public static final int DEFAULT_MAXIMUM_CONNECTIONS = 200;
+ public static final int DEFAULT_MAXIMUM_CONNECTIONS = 500;
/**
* Configuration option to configure expiration time of
diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md
index 4d506b6bfc49..4c03cca17161 100644
--- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md
+++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md
@@ -226,7 +226,7 @@ The choice is yours. Generally recovery is better, but sometimes fail-fast is mo
| Property | Default | V2 | Meaning |
|-----------------------------------------|---------|:----|-------------------------------------------------------|
-| `fs.s3a.connection.maximum` | `200` | | Connection pool size |
+| `fs.s3a.connection.maximum` | `500` | | Connection pool size |
| `fs.s3a.connection.keepalive` | `false` | `*` | Use TCP keepalive on open channels |
| `fs.s3a.connection.acquisition.timeout` | `60s` | `*` | Timeout for waiting for a connection from the pool. |
| `fs.s3a.connection.establish.timeout` | `30s` | | Time to establish the TCP/TLS connection |
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org