You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/01/07 01:42:21 UTC
[GitHub] [hadoop] mustafaiman opened a new pull request #1795: HADOOP-16792:
Make S3 client request timeout configurable
mustafaiman opened a new pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mustafaiman commented on a change in pull request #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
mustafaiman commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#discussion_r368058613
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
##########
@@ -1284,7 +1284,10 @@ public static void initConnectionSettings(Configuration conf,
DEFAULT_SOCKET_SEND_BUFFER, 2048);
int sockRecvBuffer = intOption(conf, SOCKET_RECV_BUFFER,
DEFAULT_SOCKET_RECV_BUFFER, 2048);
+ int requestTimeoutMillis = intOption(conf, REQUEST_TIMEOUT,
Review comment:
Configuration.getTimeDuration() returns a long and awsConf.setRequestTimeout takes an int. Is there another common utility method to fail-fast a possible overflow?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mustafaiman commented on issue #1795: HADOOP-16792: Make
S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
mustafaiman commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-577374189
The code failed due to a compile error in trunk before. This now seems to be resolve. I am pushing it again to trigger automated tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1795: HADOOP-16792: Make
S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-578134555
test failure unrelated;
+1
committed to trunk with the warning text in the commit message. I don't want to be the one fielding support calls about writes not working,..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mustafaiman commented on issue #1795: HADOOP-16792: Make
S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
mustafaiman commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-577373810
@steveloughran I ran ITestS3AHugeFilesDiskBlocks#test_010_CreateHugeFile with some combinations.
The first experiments used default file size and partition for huge files. I set request timeout to 1 ms for the first experiment. Test file system failed to initialize. This is because verifyBuckets call in the beginning times out repeteadly. This is retried within AWS sdk code up to `com.amazonaws.ClientConfiguration#maxErrorRetry` times. This value is configurable from Hadoop side via property `fs.s3a.attempts.maximum`. All of this retries are opaque to Hadoop. At the end of this retry cycle, aws sdk returns failure to Hadoop's Invoker. Then, Invoker evaluates whether to retry this operation or not according to its configured retry policies. I saw that verifyBuckets call were not retried on Invoker level.
In a followup experiment, I set request timeout to 200ms, which is enough for verifyBuckets call to succeed but short enough that multi part uploads fail. In these cases, again AWS sdk retries these http requests up to `maxErrorRetry` times. After this http request fails `maxErrorRetry` times, Invoker's retry mechanism kicks in. I observed Invoker to retry these operations up to `fs.s3a.retry.limit` times conforming to configured exponential back-off limited retry policy. After all these `fs.s3a.retry.limit`*`maxErrorRetry` retries, Invoker bubbles up AWSClientIOException to the user as shown below:
```
org.apache.hadoop.fs.s3a.AWSClientIOException: upload part on tests3ascale/disk/hugefile: com.amazonaws.SdkClientException: Unable to execute HTTP request: Request did not complete before the request timeout configuration.: Unable to execute HTTP request: Request did not complete before the request timeout configuration.
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:205)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:112)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:315)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:407)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:311)
```
Later, I ran the test with 256M file size and 32M partitionsize. I set the request timeout to 5s. My goal was to introduce a few retries due to short request timeout, but complete the upload operation with the use of retries. I managed to do that. I saw some retries due to short request timeout, but they were retried and the upload operation completed successfully. The test failed anyway because it also expected that `TRANSFER_PART_FAILED_EVENT` be 0. This is obviously not the case because some transfers failed but they were retried. I checked S3 and verified that the file was there. I also verified that temporary partition files were cleared in my local drive.
When I run the same experiment with 8GB file and 128M partitions but with small request timeout, the test fails due to uploads not being completed.
I also ran a soak test with 8GB files with a large request timeout. This passed fine as expected because timeout value was high enough to let uploads complete.
@bgaborg
I did not add a functional test here. I am repeating what we talked offline to leave a record of the reasoning. The retry mechanism is entirely within AWS SDK as I explained earlier in this comment. To introduce a functional test, we need a mechanism to selectively delay/fail some requests because we want file system initialization to succeed but a subsequent dummy operation(like getFileStatus) to be delayed. Introducing such test support is very hard if not impossible since hadoop-aws does not have any visibility into this mechanism. So, we depend on AWS SDK to maintain this mechanism for which they expose a configuration option here. I think this is very reasonable assumption. If AWS SDK cuts support for this feature in the feature, the configuration test I added here will fail anyway. Because I expect AWS SDK to at least throw an exception trying to set a configuration on ClientConfiguration object that is not supported anymore, or they would remove the method completely which would cause a compile error.
@steveloughran @bgaborg
I addressed the other code comments in the code. This final version of code has passed hadoop-aws test suite against us-west-1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1795: HADOOP-16792: Make
S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-571414727
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 29m 23s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 30s | trunk passed |
| +1 :green_heart: | compile | 0m 30s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 49s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 59s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 26s | the patch passed |
| +1 :green_heart: | javac | 0m 26s | the patch passed |
| -0 :warning: | checkstyle | 0m 18s | hadoop-tools/hadoop-aws: The patch generated 1 new + 21 unchanged - 0 fixed = 22 total (was 21) |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 58s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | findbugs | 1m 2s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 16s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 89m 51s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9a4c42ae4d48 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 59aac00 |
| Default Java | 1.8.0_232 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/1/testReport/ |
| Max. process+thread count | 420 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1795: HADOOP-16792: Make
S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-576965145
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 29m 31s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 6m 49s | root in trunk failed. |
| +1 :green_heart: | compile | 0m 37s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | trunk passed |
| -1 :x: | shadedclient | 4m 23s | branch has errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | trunk passed |
| +0 :ok: | spotbugs | 0m 57s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 27s | the patch passed |
| +1 :green_heart: | javac | 0m 27s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| -1 :x: | shadedclient | 4m 22s | patch has errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed |
| +1 :green_heart: | findbugs | 1m 0s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 18s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 54m 18s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux e4364265aa8d 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 5e2ce37 |
| Default Java | 1.8.0_232 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/3/artifact/out/branch-mvninstall-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/3/testReport/ |
| Max. process+thread count | 422 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
bgaborg commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#discussion_r367913573
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
##########
@@ -983,6 +983,12 @@ options are covered in [Testing](./testing.md).
<description>Select which version of the S3 SDK's List Objects API to use.
Currently support 2 (default) and 1 (older API).</description>
</property>
+
+<property>
+ <name>fs.s3a.connection.request.timeout</name>
+ <value>0</value>
+ <description>Request timeout in milliseconds. Any non-positive value disables the timeout.</description>
Review comment:
Please describe what happens if the timeout is disabled. Is 0 a non-positive value? What will happen if the setting is 0?
Also, please include what will happen if the request will time out in cases like copying several large files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1795: HADOOP-16792: Make
S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-576972274
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 5s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 6m 24s | root in trunk failed. |
| +1 :green_heart: | compile | 0m 29s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | trunk passed |
| -1 :x: | shadedclient | 4m 19s | branch has errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed |
| +0 :ok: | spotbugs | 0m 56s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 53s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 19s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| -1 :x: | shadedclient | 4m 23s | patch has errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | findbugs | 1m 3s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 13s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| | | 25m 5s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 482157aa661b 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 5e2ce37 |
| Default Java | 1.8.0_232 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/4/artifact/out/branch-mvninstall-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/4/testReport/ |
| Max. process+thread count | 420 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#discussion_r370319123
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
##########
@@ -983,6 +983,14 @@ options are covered in [Testing](./testing.md).
<description>Select which version of the S3 SDK's List Objects API to use.
Currently support 2 (default) and 1 (older API).</description>
</property>
+
+<property>
+ <name>fs.s3a.connection.request.timeout</name>
+ <value>0</value>
+ <description>Controls timeout for S3 requests.
+ Any non-positive(0 or negative value) value disables the timeout.
Review comment:
needs to indicate that its now a time unit, and that playing with this number is dangerous.
Proposed (here and core-default)
```
Time out on HTTP requests to the AWS service; 0 means no timeout. Measured in seconds; the usual time suffixes are all supported
Important: this is the maximum duration of any AWS service call, including upload and copy operations. If non-zero, it must be larger than the time to upload multi-megabyte blocks to S3 from the client, and to rename many-GB files. Use with care.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-573271018
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 29m 25s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 20m 55s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 44s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed |
| +0 :ok: | spotbugs | 1m 0s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 59s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 19s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 4s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| +1 :green_heart: | findbugs | 1m 2s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 21s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 89m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 9b2903102ae5 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 2301b25 |
| Default Java | 1.8.0_232 |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/2/testReport/ |
| Max. process+thread count | 421 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1795: HADOOP-16792: Make
S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-577943862
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 6s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 11s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 0s | trunk passed |
| +1 :green_heart: | compile | 17m 43s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 54s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 51s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 2s | trunk passed |
| +0 :ok: | spotbugs | 1m 7s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 23s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 28s | the patch passed |
| +1 :green_heart: | compile | 17m 40s | the patch passed |
| +1 :green_heart: | javac | 17m 40s | the patch passed |
| +1 :green_heart: | checkstyle | 3m 31s | the patch passed |
| +1 :green_heart: | mvnsite | 2m 7s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | whitespace | 0m 0s | The patch 5 line(s) with tabs. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 34s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 2s | the patch passed |
| +1 :green_heart: | findbugs | 3m 36s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 9m 40s | hadoop-common in the patch failed. |
| +1 :green_heart: | unit | 1m 34s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | The patch does not generate ASF License warnings. |
| | | 128m 19s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.TestHarFileSystem |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux d19f3d348bf5 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 6c1fa24 |
| Default Java | 1.8.0_232 |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/6/artifact/out/whitespace-eol.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/6/artifact/out/whitespace-tabs.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/6/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/6/testReport/ |
| Max. process+thread count | 1344 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/6/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mustafaiman commented on a change in pull request #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
mustafaiman commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#discussion_r368058613
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
##########
@@ -1284,7 +1284,10 @@ public static void initConnectionSettings(Configuration conf,
DEFAULT_SOCKET_SEND_BUFFER, 2048);
int sockRecvBuffer = intOption(conf, SOCKET_RECV_BUFFER,
DEFAULT_SOCKET_RECV_BUFFER, 2048);
+ int requestTimeoutMillis = intOption(conf, REQUEST_TIMEOUT,
Review comment:
Configuration.getTimeDuration() returns a long and awsConf.setRequestTimeout takes an int. Is there another common utility method to fail-fast a lossy conversion?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mustafaiman commented on issue #1795: HADOOP-16792: Make
S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
mustafaiman commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-576819167
@steveloughran thanks for the detailed review, i'll adress the comments shortly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1795: HADOOP-16792: Make
S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-575676523
sorry, I had commented but I hadn't hit the "complete review" comment and so github didn't submit it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#discussion_r368497927
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
##########
@@ -1284,7 +1284,10 @@ public static void initConnectionSettings(Configuration conf,
DEFAULT_SOCKET_SEND_BUFFER, 2048);
int sockRecvBuffer = intOption(conf, SOCKET_RECV_BUFFER,
DEFAULT_SOCKET_RECV_BUFFER, 2048);
+ int requestTimeoutMillis = intOption(conf, REQUEST_TIMEOUT,
Review comment:
no, you will have to do some checks/cast, e.g timeout= max (duration, MAX_INT)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1795: HADOOP-16792: Make
S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-577406679
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 6s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 20m 28s | trunk passed |
| +1 :green_heart: | compile | 0m 32s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 42s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | the patch passed |
| +1 :green_heart: | compile | 0m 26s | the patch passed |
| +1 :green_heart: | javac | 0m 26s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 5s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed |
| +1 :green_heart: | findbugs | 1m 3s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 27s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| | | 60m 33s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux a14cf564963d 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 9520b2a |
| Default Java | 1.8.0_232 |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/5/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/5/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-576972274
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 5s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 6m 24s | root in trunk failed. |
| +1 :green_heart: | compile | 0m 29s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | trunk passed |
| -1 :x: | shadedclient | 4m 19s | branch has errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed |
| +0 :ok: | spotbugs | 0m 56s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 53s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 19s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| -1 :x: | shadedclient | 4m 23s | patch has errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | findbugs | 1m 3s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 13s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| | | 25m 5s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 482157aa661b 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 5e2ce37 |
| Default Java | 1.8.0_232 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/4/artifact/out/branch-mvninstall-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/4/testReport/ |
| Max. process+thread count | 420 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#discussion_r368079266
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java
##########
@@ -390,6 +390,19 @@ public void testCustomUserAgent() throws Exception {
awsConf.getUserAgentPrefix());
}
+ @Test
+ public void testRequestTimeout() throws Exception {
+ conf = new Configuration();
+ conf.setInt(REQUEST_TIMEOUT, 11895);
+ fs = S3ATestUtils.createTestFileSystem(conf);
+ assertNotNull(fs);
Review comment:
you don't need these; not given an NPE is raised immediately after
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mustafaiman commented on issue #1795: HADOOP-16792: Make
S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
mustafaiman commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-577902804
@steveloughran
```
we have the InconsistentS3AClient so simulate failures to S3; used for S3Guard. I've been wondering what it would take to actually simulate throttling there as well; some random probability of the client considering itself overloaded, and then having a window where it blocks. Or even better -could we actually let you configure a throttle load and have it trigger when the request rate exceeded it.
Simulating request timeouts would be simpler -but as or fault injecting client goes above the AWS SDK, it won't be testing their internals.
```
Simulating throttling looks achievable. AWS SDK passes throttle errors up to hadoop-aws and that is where we catch throttling error and implement retry mechanism. So you can simulate throttling in InconsistentAmazonS3Client. However, `fs.s3a.connection.request.timeout` affects the retry mechanism below AWS SDK. So as you already pointed out, it is not useful in testing this particular configuration.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#discussion_r370315573
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
##########
@@ -1284,6 +1285,16 @@ public static void initConnectionSettings(Configuration conf,
DEFAULT_SOCKET_SEND_BUFFER, 2048);
int sockRecvBuffer = intOption(conf, SOCKET_RECV_BUFFER,
DEFAULT_SOCKET_RECV_BUFFER, 2048);
+ long requestTimeoutMillis = conf.getTimeDuration(REQUEST_TIMEOUT,
+ DEFAULT_REQUEST_TIMEOUT, TimeUnit.MILLISECONDS);
+
+ if (requestTimeoutMillis > Integer.MAX_VALUE) {
+ LOG.debug(String.format("Request timeout is too high(%d ms)."
Review comment:
log uses {} for entries
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-577943862
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 6s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 11s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 0s | trunk passed |
| +1 :green_heart: | compile | 17m 43s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 54s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 51s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 2s | trunk passed |
| +0 :ok: | spotbugs | 1m 7s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 23s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 28s | the patch passed |
| +1 :green_heart: | compile | 17m 40s | the patch passed |
| +1 :green_heart: | javac | 17m 40s | the patch passed |
| +1 :green_heart: | checkstyle | 3m 31s | the patch passed |
| +1 :green_heart: | mvnsite | 2m 7s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | whitespace | 0m 0s | The patch 5 line(s) with tabs. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 34s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 2s | the patch passed |
| +1 :green_heart: | findbugs | 3m 36s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 9m 40s | hadoop-common in the patch failed. |
| +1 :green_heart: | unit | 1m 34s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | The patch does not generate ASF License warnings. |
| | | 128m 19s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.TestHarFileSystem |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux d19f3d348bf5 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 6c1fa24 |
| Default Java | 1.8.0_232 |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/6/artifact/out/whitespace-eol.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/6/artifact/out/whitespace-tabs.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/6/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/6/testReport/ |
| Max. process+thread count | 1344 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/6/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mustafaiman commented on issue #1795: HADOOP-16792: Make
S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
mustafaiman commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-578231504
@steveloughran @bgaborg thank you for reviews.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#discussion_r363769837
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
##########
@@ -1284,7 +1284,10 @@ public static void initConnectionSettings(Configuration conf,
DEFAULT_SOCKET_SEND_BUFFER, 2048);
int sockRecvBuffer = intOption(conf, SOCKET_RECV_BUFFER,
DEFAULT_SOCKET_RECV_BUFFER, 2048);
+ int requestTimeoutMillis = intOption(conf, REQUEST_TIMEOUT,
Review comment:
Use `Configuration.getTimeDuration()`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#discussion_r370319123
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
##########
@@ -983,6 +983,14 @@ options are covered in [Testing](./testing.md).
<description>Select which version of the S3 SDK's List Objects API to use.
Currently support 2 (default) and 1 (older API).</description>
</property>
+
+<property>
+ <name>fs.s3a.connection.request.timeout</name>
+ <value>0</value>
+ <description>Controls timeout for S3 requests.
+ Any non-positive(0 or negative value) value disables the timeout.
Review comment:
needs to indicate that its now a time unit, and that playing with this number is dangerous.
Proposed (here and core-default)
```
Time out on HTTP requests to the AWS service; 0 means no timeout.
Measured in seconds; the usual time suffixes are all supported
Important: this is the maximum duration of any AWS service call,
including upload and copy operations. If non-zero, it must be larger
than the time to upload multi-megabyte blocks to S3 from the client,
and to rename many-GB files. Use with care.
```
Also, in the troubleshooting doc can you add a section at the bottom with the error message and meaning "increase the timeout or set to 0"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mustafaiman commented on a change in pull request #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
mustafaiman commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#discussion_r368046503
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##########
@@ -187,6 +187,10 @@ private Constants() {
public static final String SOCKET_TIMEOUT = "fs.s3a.connection.timeout";
public static final int DEFAULT_SOCKET_TIMEOUT = 200000;
+ // milliseconds until a request is timed-out
+ public static final String REQUEST_TIMEOUT = "fs.s3a.connection.request.timeout";
+ public static final int DEFAULT_REQUEST_TIMEOUT = 0;
Review comment:
It is 0 by default already. It means, http client will not timeout. This preserves the existing behavior and is a good assumption for guys who does not have any reason to change it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #1795: HADOOP-16792:
Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#discussion_r370317165
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
##########
@@ -1284,6 +1285,16 @@ public static void initConnectionSettings(Configuration conf,
DEFAULT_SOCKET_SEND_BUFFER, 2048);
int sockRecvBuffer = intOption(conf, SOCKET_RECV_BUFFER,
DEFAULT_SOCKET_RECV_BUFFER, 2048);
+ long requestTimeoutMillis = conf.getTimeDuration(REQUEST_TIMEOUT,
+ DEFAULT_REQUEST_TIMEOUT, TimeUnit.MILLISECONDS);
Review comment:
can you make the default unit TimeUnit.Seconds, even if you take the range in millis. People should be using seconds for this timeout
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-571414727
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 29m 23s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 30s | trunk passed |
| +1 :green_heart: | compile | 0m 30s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 49s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 59s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 26s | the patch passed |
| +1 :green_heart: | javac | 0m 26s | the patch passed |
| -0 :warning: | checkstyle | 0m 18s | hadoop-tools/hadoop-aws: The patch generated 1 new + 21 unchanged - 0 fixed = 22 total (was 21) |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 58s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | findbugs | 1m 2s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 16s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 89m 51s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9a4c42ae4d48 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 59aac00 |
| Default Java | 1.8.0_232 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/1/testReport/ |
| Max. process+thread count | 420 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1795: HADOOP-16792: Make
S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-578131108
test failure looks like a regression of mine -should have been picked up earlier..interesting
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-576965145
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 29m 31s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 6m 49s | root in trunk failed. |
| +1 :green_heart: | compile | 0m 37s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | trunk passed |
| -1 :x: | shadedclient | 4m 23s | branch has errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | trunk passed |
| +0 :ok: | spotbugs | 0m 57s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 55s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 27s | the patch passed |
| +1 :green_heart: | javac | 0m 27s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| -1 :x: | shadedclient | 4m 22s | patch has errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed |
| +1 :green_heart: | findbugs | 1m 0s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 18s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 54m 18s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux e4364265aa8d 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 5e2ce37 |
| Default Java | 1.8.0_232 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/3/artifact/out/branch-mvninstall-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/3/testReport/ |
| Max. process+thread count | 422 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#discussion_r363771314
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##########
@@ -187,6 +187,10 @@ private Constants() {
public static final String SOCKET_TIMEOUT = "fs.s3a.connection.timeout";
public static final int DEFAULT_SOCKET_TIMEOUT = 200000;
+ // milliseconds until a request is timed-out
+ public static final String REQUEST_TIMEOUT = "fs.s3a.connection.request.timeout";
+ public static final int DEFAULT_REQUEST_TIMEOUT = 0;
Review comment:
what would a good default be here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1795: HADOOP-16792: Make
S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-573271018
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 29m 25s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 20m 55s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 44s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed |
| +0 :ok: | spotbugs | 1m 0s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 59s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 19s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 4s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| +1 :green_heart: | findbugs | 1m 2s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 21s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 89m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 9b2903102ae5 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 2301b25 |
| Default Java | 1.8.0_232 |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/2/testReport/ |
| Max. process+thread count | 421 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1795: HADOOP-16792: Make
S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#issuecomment-578066389
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 8s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 37s | trunk passed |
| +1 :green_heart: | compile | 17m 26s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 47s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 4s | trunk passed |
| +1 :green_heart: | shadedclient | 19m 53s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 56s | trunk passed |
| +0 :ok: | spotbugs | 1m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 6s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 20s | the patch passed |
| +1 :green_heart: | compile | 16m 46s | the patch passed |
| +1 :green_heart: | javac | 16m 46s | the patch passed |
| +1 :green_heart: | checkstyle | 2m 48s | the patch passed |
| +1 :green_heart: | mvnsite | 2m 4s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 58s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 54s | the patch passed |
| +1 :green_heart: | findbugs | 3m 26s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 9m 14s | hadoop-common in the patch failed. |
| +1 :green_heart: | unit | 1m 34s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | The patch does not generate ASF License warnings. |
| | | 123m 26s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.TestHarFileSystem |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux e04c8d8cf424 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 978c487 |
| Default Java | 1.8.0_232 |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/7/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/7/testReport/ |
| Max. process+thread count | 1346 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1795/7/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#discussion_r363770709
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java
##########
@@ -390,6 +390,19 @@ public void testCustomUserAgent() throws Exception {
awsConf.getUserAgentPrefix());
}
+ @Test
+ public void testRequestTimeout() throws Exception {
+ conf = new Configuration();
+ conf.setInt(REQUEST_TIMEOUT, 10);
+ fs = S3ATestUtils.createTestFileSystem(conf);
+ assertNotNull(fs);
+ AmazonS3 s3 = fs.getAmazonS3ClientForTesting("Request timeout");
+ assertNotNull(s3);
+ ClientConfiguration awsConf = getField(s3, ClientConfiguration.class,
+ "clientConfiguration");
+ assertEquals(10, awsConf.getRequestTimeout());
Review comment:
needs a message indicating what doesn't match
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1795:
HADOOP-16792: Make S3 client request timeout configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1795: HADOOP-16792: Make S3 client request timeout configurable
URL: https://github.com/apache/hadoop/pull/1795#discussion_r367999713
##########
File path: hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
##########
@@ -983,6 +983,12 @@ options are covered in [Testing](./testing.md).
<description>Select which version of the S3 SDK's List Objects API to use.
Currently support 2 (default) and 1 (older API).</description>
</property>
+
+<property>
+ <name>fs.s3a.connection.request.timeout</name>
+ <value>0</value>
+ <description>Request timeout in milliseconds. Any non-positive value disables the timeout.</description>
Review comment:
if you pass in a negative number intOption will reject. With a move to getTimeDuration you get to specify millis and hours, but you can't use negative time
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org