You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/08/26 08:02:54 UTC
[GitHub] [hadoop] dongjoon-hyun opened a new pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
dongjoon-hyun opened a new pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337
### Description of PR
This PR aims to increase `fs.s3a.connection.maximum` to be bigger than `fs.s3a.threads.max`.
### How was this patch tested?
HADOOP-15183 defined this variable like the following.
> Controls the maximum number of simultaneous connections to S3. This must be bigger than the value of fs.s3a.threads.max so as to stop threads being blocked waiting for new HTTPS connections.
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337#issuecomment-906351699
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 11m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 15s | | trunk passed |
| +1 :green_heart: | compile | 29m 25s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 24m 21s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 54s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 34s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 32s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 58s | | the patch passed |
| +1 :green_heart: | compile | 28m 22s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 28m 22s | | the patch passed |
| +1 :green_heart: | compile | 24m 41s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 24m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 49s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 1m 47s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 33s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 46s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 34s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 56s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not generate ASF License warnings. |
| | | 244m 24s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3337/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3337 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle |
| uname | Linux 7a722fe96015 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3e332aadd9b1a3ba7ceb8cedcaeb1cfcdaebc3cb |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3337/1/testReport/ |
| Max. process+thread count | 2849 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3337/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337#issuecomment-908542912
ok, in trunk & 3.3. Thanks for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dongjoon-hyun commented on pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337#issuecomment-906512783
Thank you, @steveloughran .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dongjoon-hyun commented on pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337#issuecomment-908629251
Thank you so much, @steveloughran and @sunchao !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dongjoon-hyun edited a comment on pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337#issuecomment-906188084
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dongjoon-hyun commented on pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337#issuecomment-908629251
Thank you so much, @steveloughran and @sunchao !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dongjoon-hyun commented on pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337#issuecomment-907821627
Thank you, @steveloughran . It sounds like a good plan.
cc @sunchao , too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337#issuecomment-907745372
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 11m 47s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 38s | | trunk passed |
| +1 :green_heart: | compile | 29m 11s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 24m 22s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 51s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 31s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 28s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 56s | | the patch passed |
| +1 :green_heart: | compile | 28m 19s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 28m 19s | | the patch passed |
| +1 :green_heart: | compile | 24m 28s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 24m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 50s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 1m 48s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 34s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 7s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 41s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 58s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not generate ASF License warnings. |
| | | 244m 51s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3337/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3337 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle |
| uname | Linux 6522380e4d89 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 057a45706927a20f44c6f316c49f67cd4cd1769e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3337/2/testReport/ |
| Max. process+thread count | 2504 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3337/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337#issuecomment-907628348
what do you think should be good values for modern systems? Say:? 64 threads and 96 connections? having lots of threads isn't so bad if they are all uploading to/downloading from S3 as they'll be waiting for network IO, not fighting for CPU time with worker threads
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337#issuecomment-907775866
LGTM
I've actually been thinking about having better thread pooling here with a RejectedExecutionHandler which
* increments a statistic
* hands on to ThreadPoolExecutor.CallerRunsPolicy
So rather than do blocking when the pool is full, fall back to executing in the current thread. Goal: when the pool is full we still get useful work done because the current thread is doing it.
I worry about this though when it comes to things like RenameOperation; is the use of CompletableFuture<> creating expectations in the code that the work will actually be done by a separate thread.
How about we merge this patch in as this, and then look at that problem in more detail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337#issuecomment-908542912
ok, in trunk & 3.3. Thanks for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dongjoon-hyun commented on a change in pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337#discussion_r696402072
##########
File path: hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
##########
@@ -1381,7 +1381,7 @@
<property>
<name>fs.s3a.connection.maximum</name>
- <value>48</value>
+ <value>65</value>
Review comment:
This is the smallest value which is bigger than `fs.s3a.threads.max`.
https://github.com/apache/hadoop/blob/3e332aadd9b1a3ba7ceb8cedcaeb1cfcdaebc3cb/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml#L1487
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dongjoon-hyun commented on pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337#issuecomment-906188084
Hi, @steveloughran .
The config description looks correct to me. Is there a reason why Apache Hadoop has 48 which is smaller than the default value of `fs.s3a.threads.max`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] dongjoon-hyun commented on pull request #3337: HADOOP-17869. `fs.s3a.connection.maximum` should be bigger than `fs.s3a.threads.max`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #3337:
URL: https://github.com/apache/hadoop/pull/3337#issuecomment-907722797
Changing `fs.s3a.connection.maximum` from 48 to 96 sounds good to me. I'll update this PR according to your opinion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org