You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/01/31 11:48:02 UTC
[GitHub] [hadoop] steveloughran opened a new pull request #1826:
HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
steveloughran opened a new pull request #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826
Currently AWS S3 throttling is initially handled in the AWS SDK, only reaching the S3 client code after it has given up.
This means we don't always directly observe when throttling is taking place.
* disables throttling retries in the AWS client Library *for S3 only*
* add a quantile for the S3 throttle events, as DDB has
* isolate counters of s3 and DDB throttle events to classify issues better
* improvements to DDB throttling handling and testing
1. Because we are taking over the AWS retries, we need to expand the initial delay en retries and the number of retries we should support before giving up.
1. I can split the DDB and S3 side of this patch...they came in together as once I turned off throttling across all AWS client configs, scale tests against a 10 TPS DDB table showed we weren't retrying adequately in some of the tests, and retrying inefficiently in listChildren.
(reinstatment of #1814 which was accidentally closed)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1826:
HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#discussion_r377178960
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##########
@@ -174,10 +174,34 @@ private Constants() {
public static final String PROXY_DOMAIN = "fs.s3a.proxy.domain";
public static final String PROXY_WORKSTATION = "fs.s3a.proxy.workstation";
- // number of times we should retry errors
+ /**
+ * Number of times the AWS client library should retry errors before
+ * escalating to the S3A code: {@value}.
+ */
public static final String MAX_ERROR_RETRIES = "fs.s3a.attempts.maximum";
+
+ /**
+ * Default number of times the AWS client library should retry errors before
+ * escalating to the S3A code: {@value}.
+ */
public static final int DEFAULT_MAX_ERROR_RETRIES = 10;
+ /**
+ * Experimental/Unstable feature: should the AWS client library retry
+ * throttle responses before escalating to the S3A code: {@value}.
+ *
+ * When set to false, the S3A connector sees all S3 throttle events,
+ * And so can update it counters and the metrics, and use its own retry
+ * policy.
+ * However, this may have adverse effects on some operations where the S3A
+ * code cannot retry as efficiently as the AWS client library.
+ *
+ * This only applies to S3 operations, not to DynamoDB or other services.
+ */
+ @InterfaceStability.Unstable
+ public static final String EXPERIMENTAL_AWS_INTERNAL_THROTTLING =
+ "fs.s3a.experimental.aws.internal.throttling";
Review comment:
its true, but yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1826: HADOOP-16823.
Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-581039073
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 29m 8s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 31s | trunk passed |
| +1 :green_heart: | compile | 17m 20s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 47s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 3s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 1s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 55s | trunk passed |
| +0 :ok: | spotbugs | 1m 7s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 7s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 19s | the patch passed |
| +1 :green_heart: | compile | 16m 44s | the patch passed |
| +1 :green_heart: | javac | 16m 44s | the patch passed |
| -0 :warning: | checkstyle | 2m 48s | root: The patch generated 7 new + 69 unchanged - 2 fixed = 76 total (was 71) |
| +1 :green_heart: | mvnsite | 2m 2s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 8s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 55s | the patch passed |
| +1 :green_heart: | findbugs | 3m 23s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 36s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 33s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | The patch does not generate ASF License warnings. |
| | | 152m 36s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux b571c30a355c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 1e3a0b0 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/3/artifact/out/diff-checkstyle-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/3/testReport/ |
| Max. process+thread count | 1346 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1826:
HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#discussion_r377178960
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##########
@@ -174,10 +174,34 @@ private Constants() {
public static final String PROXY_DOMAIN = "fs.s3a.proxy.domain";
public static final String PROXY_WORKSTATION = "fs.s3a.proxy.workstation";
- // number of times we should retry errors
+ /**
+ * Number of times the AWS client library should retry errors before
+ * escalating to the S3A code: {@value}.
+ */
public static final String MAX_ERROR_RETRIES = "fs.s3a.attempts.maximum";
+
+ /**
+ * Default number of times the AWS client library should retry errors before
+ * escalating to the S3A code: {@value}.
+ */
public static final int DEFAULT_MAX_ERROR_RETRIES = 10;
+ /**
+ * Experimental/Unstable feature: should the AWS client library retry
+ * throttle responses before escalating to the S3A code: {@value}.
+ *
+ * When set to false, the S3A connector sees all S3 throttle events,
+ * And so can update it counters and the metrics, and use its own retry
+ * policy.
+ * However, this may have adverse effects on some operations where the S3A
+ * code cannot retry as efficiently as the AWS client library.
+ *
+ * This only applies to S3 operations, not to DynamoDB or other services.
+ */
+ @InterfaceStability.Unstable
+ public static final String EXPERIMENTAL_AWS_INTERNAL_THROTTLING =
+ "fs.s3a.experimental.aws.internal.throttling";
Review comment:
its false, but yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1826: HADOOP-16823.
Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-584219900
Gabor, thanks for the review.
yeah, you are right. Improvement.
Before I merge, do you want to look at `BulkDeleteRetryHandler` and see if you agree what I'm doing there?
XML parser errors are being treated as retry failures, as that is what I'm seeing during the load tests (i.e. not 503/slow down). https://issues.apache.org/jira/browse/HADOOP-13811 shows the history there (and yes, my test bucket is versioned for 24h).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1826:
HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1826: HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-583367137
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 46s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 9s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 25s | trunk passed |
| +1 :green_heart: | compile | 17m 41s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 35s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 14s | trunk passed |
| +1 :green_heart: | shadedclient | 19m 52s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 32s | trunk passed |
| +0 :ok: | spotbugs | 1m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 57s | trunk passed |
| -0 :warning: | patch | 2m 22s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 31s | the patch passed |
| +1 :green_heart: | compile | 17m 2s | the patch passed |
| +1 :green_heart: | javac | 17m 2s | the patch passed |
| -0 :warning: | checkstyle | 2m 42s | root: The patch generated 12 new + 75 unchanged - 2 fixed = 87 total (was 77) |
| +1 :green_heart: | mvnsite | 2m 46s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 9s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 38s | the patch passed |
| +1 :green_heart: | findbugs | 3m 27s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 15s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 36s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 52s | The patch does not generate ASF License warnings. |
| | | 127m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux fb3047deee9a 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 7dac7e1 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/7/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/7/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/7/testReport/ |
| Max. process+thread count | 1375 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/7/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1826: HADOOP-16823.
Large DeleteObject requests are their own Thundering Herd
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1826: HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-584690703
Did checkstyle changes and a diff with trunk to (a) reduce the diff and (b) see what I needed to improve with javadocs; mainly the RetryingCollection.
I got a failure on a -Dscale auth run
```
[ERROR] ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRecursiveRootListing:257->Assert.assertTrue:41->Assert.fail:88 files mismatch: between
"s3a://hwdev-steve-ireland-new/fork-0003/test/testBulkRenameAndDelete/src/file-1"
"s3a://hwdev-steve-ireland-new/fork-0003/test/testBulkRenameAndDelete/src/file-25"
"s3a://hwdev-steve-ireland-new/fork-0003/test/testBulkRenameAndDelete/src/file-16"
"s3a://hwdev-steve-ireland-new/fork-0003/test/testBulkRenameAndDelete/src/file-11"
"s3a://hwdev-steve-ireland-new/fork-0003/test/testBulkRenameAndDelete/src/file-7"
"s3a://hwdev-steve-ireland-new/fork-0003/test/testBulkRenameAndDelete/src/file-54"
"s3a://hwdev-steve-ireland-new/fork-0003/test/testBulkRenameAndDelete/src/file-14"
"s3a://hwdev-steve-ireland-new/fork-0003/test/testBulkRenameAndDelete/src/file-35"
"s3a://hwdev-steve-ireland-new/fork-0003/test/testBulkRenameAndDelete/src/file-48"
"s3a://hwdev-steve-ireland-new/fork-0003/test/testBulkRenameAndDelete/src/file-56"
"s3a://hwdev-steve-ireland-new/fork-0003/test/testBulkRenameAndDelete/src/file-29"
"s3a://hwdev-steve-ireland-new/fork-0003/test/testBulkRenameAndDelete/src/file-52"
"s3a://hwdev-steve-ireland-new/fork-0003/test/testBulkRenameAndDelete/src/file-40"
"s3a://hwdev-steve-ireland-new/fork-0003/test/testBulkRenameAndDelete/src/file-2"
"s3a://hwdev-steve-ireland-new/fork-0003/test/testBulkRenameAndDelete/src/file-24"
"s3a:
```
Now, I've been playing with older branch-2 versions recently, and could blame that -but "bulk" and "delete" describe exactly what I was working on in this patch.
It wasn't, but while working on this tests, with better renames, I managed to create a deadlock in the new code
1. S3ABlockOutputStream was waiting for space in the bounded thread pool so it can do an async put.
1. But that thread pool was blocked by threads waiting for their async directory operations to complete.
1. Outcome: total deadlock.
Surfaced in ITestS3ADeleteManyFiles during parallel file creation.
Actions
* remove the async stuff from the end of rename()
* keep dir marker delete operations in finishedWrite() async, but use the unbounded thread pool.
* Cleanup + enhancement of ITestS3ADeleteManyFiles so that it tests src and dest paths more rigorously, and sets a page size of 50 for better coverage of the paged rename sequence.
Makes me think we should do more parallel IO tests within the same process.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1826:
HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
bgaborg commented on a change in pull request #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#discussion_r377018136
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##########
@@ -174,10 +174,34 @@ private Constants() {
public static final String PROXY_DOMAIN = "fs.s3a.proxy.domain";
public static final String PROXY_WORKSTATION = "fs.s3a.proxy.workstation";
- // number of times we should retry errors
+ /**
+ * Number of times the AWS client library should retry errors before
+ * escalating to the S3A code: {@value}.
+ */
public static final String MAX_ERROR_RETRIES = "fs.s3a.attempts.maximum";
+
+ /**
+ * Default number of times the AWS client library should retry errors before
+ * escalating to the S3A code: {@value}.
+ */
public static final int DEFAULT_MAX_ERROR_RETRIES = 10;
+ /**
+ * Experimental/Unstable feature: should the AWS client library retry
+ * throttle responses before escalating to the S3A code: {@value}.
+ *
+ * When set to false, the S3A connector sees all S3 throttle events,
+ * And so can update it counters and the metrics, and use its own retry
+ * policy.
+ * However, this may have adverse effects on some operations where the S3A
+ * code cannot retry as efficiently as the AWS client library.
+ *
+ * This only applies to S3 operations, not to DynamoDB or other services.
+ */
+ @InterfaceStability.Unstable
+ public static final String EXPERIMENTAL_AWS_INTERNAL_THROTTLING =
+ "fs.s3a.experimental.aws.internal.throttling";
Review comment:
Where is the default value for this defined?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1826:
HADOOP-16823. Improve S3A Throttling in S3Guard and S3 bulk delete operations
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1826: HADOOP-16823. Improve S3A Throttling in S3Guard and S3 bulk delete operations
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-582521724
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 11s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 4s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 18m 6s | trunk passed |
| +1 :green_heart: | compile | 16m 37s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 42s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 16s | trunk passed |
| +1 :green_heart: | shadedclient | 19m 3s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 13s | trunk passed |
| +0 :ok: | spotbugs | 1m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 15s | trunk passed |
| -0 :warning: | patch | 1m 35s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 18s | the patch passed |
| +1 :green_heart: | compile | 15m 51s | the patch passed |
| +1 :green_heart: | javac | 15m 51s | the patch passed |
| -0 :warning: | checkstyle | 2m 37s | root: The patch generated 16 new + 63 unchanged - 2 fixed = 79 total (was 65) |
| +1 :green_heart: | mvnsite | 2m 15s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 12m 40s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 15s | the patch passed |
| +1 :green_heart: | findbugs | 3m 31s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 15s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 43s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 53s | The patch does not generate ASF License warnings. |
| | | 119m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 4aa192819c23 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ce7b8b5 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/6/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/6/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/6/testReport/ |
| Max. process+thread count | 1366 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/6/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1826: HADOOP-16823. Large
DeleteObject requests are their own Thundering Herd
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1826: HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-584758279
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 34s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 6s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 10s | trunk passed |
| +1 :green_heart: | compile | 16m 54s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 41s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 12s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 40s | trunk passed |
| +0 :ok: | spotbugs | 1m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 17s | trunk passed |
| -0 :warning: | patch | 1m 34s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 25s | the patch passed |
| +1 :green_heart: | compile | 16m 27s | the patch passed |
| +1 :green_heart: | javac | 16m 27s | the patch passed |
| -0 :warning: | checkstyle | 2m 40s | root: The patch generated 6 new + 75 unchanged - 2 fixed = 81 total (was 77) |
| +1 :green_heart: | mvnsite | 2m 21s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 15s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 44s | the patch passed |
| +1 :green_heart: | findbugs | 3m 48s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 22s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 30s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 52s | The patch does not generate ASF License warnings. |
| | | 124m 8s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 14c16a167728 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / cc8ae59 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/10/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/10/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/10/testReport/ |
| Max. process+thread count | 1448 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/10/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1826:
HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#discussion_r377181855
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
##########
@@ -56,7 +57,13 @@ public AmazonS3 createS3Client(URI name,
final String userAgentSuffix) throws IOException {
Configuration conf = getConf();
final ClientConfiguration awsConf = S3AUtils
- .createAwsConf(getConf(), bucket, Constants.AWS_SERVICE_IDENTIFIER_S3);
+ .createAwsConf(conf, bucket, Constants.AWS_SERVICE_IDENTIFIER_S3);
+
+ // throttling is explicitly disabled on the S3 client so that
+ // all failures are collected
+ awsConf.setUseThrottleRetries(
+ conf.getBoolean(EXPERIMENTAL_AWS_INTERNAL_THROTTLING, true));
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1826: HADOOP-16823.
Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-583367137
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 46s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 9s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 25s | trunk passed |
| +1 :green_heart: | compile | 17m 41s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 35s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 14s | trunk passed |
| +1 :green_heart: | shadedclient | 19m 52s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 32s | trunk passed |
| +0 :ok: | spotbugs | 1m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 57s | trunk passed |
| -0 :warning: | patch | 2m 22s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 31s | the patch passed |
| +1 :green_heart: | compile | 17m 2s | the patch passed |
| +1 :green_heart: | javac | 17m 2s | the patch passed |
| -0 :warning: | checkstyle | 2m 42s | root: The patch generated 12 new + 75 unchanged - 2 fixed = 87 total (was 77) |
| +1 :green_heart: | mvnsite | 2m 46s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 9s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 38s | the patch passed |
| +1 :green_heart: | findbugs | 3m 27s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 15s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 36s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 52s | The patch does not generate ASF License warnings. |
| | | 127m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux fb3047deee9a 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 7dac7e1 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/7/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/7/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/7/testReport/ |
| Max. process+thread count | 1375 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/7/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on a change in pull request #1826:
HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
bgaborg commented on a change in pull request #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#discussion_r377019280
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
##########
@@ -56,7 +57,13 @@ public AmazonS3 createS3Client(URI name,
final String userAgentSuffix) throws IOException {
Configuration conf = getConf();
final ClientConfiguration awsConf = S3AUtils
- .createAwsConf(getConf(), bucket, Constants.AWS_SERVICE_IDENTIFIER_S3);
+ .createAwsConf(conf, bucket, Constants.AWS_SERVICE_IDENTIFIER_S3);
+
+ // throttling is explicitly disabled on the S3 client so that
+ // all failures are collected
+ awsConf.setUseThrottleRetries(
+ conf.getBoolean(EXPERIMENTAL_AWS_INTERNAL_THROTTLING, true));
Review comment:
I'd rather add a setting for it than to set it here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1826: HADOOP-16823.
Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-583482153
experimental directory marker optimization feature removed. It was really broken and its very presence would only encourage people to turn it on, or at least, start demanding it was production ready within a short period of time..and be very disappointed when that couldn't happen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1826: HADOOP-16823.
Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-582574788
BTW, latest patch adds an experimental "optimize directory markers" switch which tells the S3a client to be less strict about creating and deleting directory markers. I'm explicitly vague about what that means, but currently in file creation it only ever looks one level up to delete markers. There's trouble there if applications don't call mkdir() on a path before creating the file, but otherwise it avoids the tombstones and scale problems on deep trees
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1826: HADOOP-16823. Large
DeleteObject requests are their own Thundering Herd
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1826: HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-585970809
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 25s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 1s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 10s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 4s | trunk passed |
| +1 :green_heart: | compile | 18m 4s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 48s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 8s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 51s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 29s | trunk passed |
| +0 :ok: | spotbugs | 1m 14s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 40s | trunk passed |
| -0 :warning: | patch | 1m 36s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 28s | the patch passed |
| +1 :green_heart: | compile | 19m 8s | the patch passed |
| +1 :green_heart: | javac | 19m 8s | the patch passed |
| -0 :warning: | checkstyle | 3m 4s | root: The patch generated 4 new + 75 unchanged - 2 fixed = 79 total (was 77) |
| +1 :green_heart: | mvnsite | 2m 24s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 16m 15s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 37s | the patch passed |
| +1 :green_heart: | findbugs | 4m 3s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 17s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 33s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | The patch does not generate ASF License warnings. |
| | | 136m 52s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux dc8447e3a59b 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / da99ac7 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/12/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/12/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/12/testReport/ |
| Max. process+thread count | 1393 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/12/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1826: HADOOP-16823. Large
DeleteObject requests are their own Thundering Herd
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1826: HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-584956951
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 12s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 31s | trunk passed |
| +1 :green_heart: | compile | 17m 55s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 49s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 8s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 48s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 27s | trunk passed |
| +0 :ok: | spotbugs | 1m 6s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 9s | trunk passed |
| -0 :warning: | patch | 1m 28s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 23s | the patch passed |
| +1 :green_heart: | compile | 18m 39s | the patch passed |
| +1 :green_heart: | javac | 18m 39s | the patch passed |
| -0 :warning: | checkstyle | 3m 35s | root: The patch generated 4 new + 75 unchanged - 2 fixed = 79 total (was 77) |
| +1 :green_heart: | mvnsite | 2m 12s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 19s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 27s | the patch passed |
| +1 :green_heart: | findbugs | 3m 27s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 25s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 31s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | The patch does not generate ASF License warnings. |
| | | 130m 19s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux accd65b42e91 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 9b8a78d |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/11/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/11/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/11/testReport/ |
| Max. process+thread count | 1348 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/11/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1826:
HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#discussion_r377181603
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
##########
@@ -174,10 +174,34 @@ private Constants() {
public static final String PROXY_DOMAIN = "fs.s3a.proxy.domain";
public static final String PROXY_WORKSTATION = "fs.s3a.proxy.workstation";
- // number of times we should retry errors
+ /**
+ * Number of times the AWS client library should retry errors before
+ * escalating to the S3A code: {@value}.
+ */
public static final String MAX_ERROR_RETRIES = "fs.s3a.attempts.maximum";
+
+ /**
+ * Default number of times the AWS client library should retry errors before
+ * escalating to the S3A code: {@value}.
+ */
public static final int DEFAULT_MAX_ERROR_RETRIES = 10;
+ /**
+ * Experimental/Unstable feature: should the AWS client library retry
+ * throttle responses before escalating to the S3A code: {@value}.
+ *
+ * When set to false, the S3A connector sees all S3 throttle events,
+ * And so can update it counters and the metrics, and use its own retry
+ * policy.
+ * However, this may have adverse effects on some operations where the S3A
+ * code cannot retry as efficiently as the AWS client library.
+ *
+ * This only applies to S3 operations, not to DynamoDB or other services.
+ */
+ @InterfaceStability.Unstable
+ public static final String EXPERIMENTAL_AWS_INTERNAL_THROTTLING =
+ "fs.s3a.experimental.aws.internal.throttling";
Review comment:
Updating the value, also changing the name to ""fs.s3a.experimental.aws.s3.throttling" to make clear its s3 only
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1826:
HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-580742619
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 9s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 37s | trunk passed |
| +1 :green_heart: | compile | 17m 46s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 47s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 6s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 6s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 57s | trunk passed |
| +0 :ok: | spotbugs | 1m 6s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 9s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 20s | the patch passed |
| +1 :green_heart: | compile | 16m 52s | the patch passed |
| +1 :green_heart: | javac | 16m 52s | the patch passed |
| -0 :warning: | checkstyle | 2m 55s | root: The patch generated 4 new + 69 unchanged - 2 fixed = 73 total (was 71) |
| +1 :green_heart: | mvnsite | 2m 11s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 6s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 59s | the patch passed |
| +1 :green_heart: | findbugs | 3m 26s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 24s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 20s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | The patch does not generate ASF License warnings. |
| | | 126m 29s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 911e02ed8c48 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / bf8686f |
| Default Java | 1.8.0_232 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/1/artifact/out/diff-checkstyle-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/1/testReport/ |
| Max. process+thread count | 1481 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1826: HADOOP-16823.
Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-582124713
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 30m 41s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 10s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 42s | trunk passed |
| +1 :green_heart: | compile | 17m 37s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 51s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 6s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 0s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 58s | trunk passed |
| +0 :ok: | spotbugs | 1m 6s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 8s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 19s | the patch passed |
| +1 :green_heart: | compile | 16m 49s | the patch passed |
| +1 :green_heart: | javac | 16m 49s | the patch passed |
| -0 :warning: | checkstyle | 2m 47s | root: The patch generated 16 new + 71 unchanged - 2 fixed = 87 total (was 73) |
| +1 :green_heart: | mvnsite | 2m 5s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 12s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 0s | the patch passed |
| +1 :green_heart: | findbugs | 3m 26s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 7s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 21s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | The patch does not generate ASF License warnings. |
| | | 155m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 2c2cceb2513c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 10a60fbe |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/5/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/5/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/5/testReport/ |
| Max. process+thread count | 1379 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/5/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1826:
HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-581039073
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 29m 8s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 31s | trunk passed |
| +1 :green_heart: | compile | 17m 20s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 47s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 3s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 1s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 55s | trunk passed |
| +0 :ok: | spotbugs | 1m 7s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 7s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 19s | the patch passed |
| +1 :green_heart: | compile | 16m 44s | the patch passed |
| +1 :green_heart: | javac | 16m 44s | the patch passed |
| -0 :warning: | checkstyle | 2m 48s | root: The patch generated 7 new + 69 unchanged - 2 fixed = 76 total (was 71) |
| +1 :green_heart: | mvnsite | 2m 2s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 8s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 55s | the patch passed |
| +1 :green_heart: | findbugs | 3m 23s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 36s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 33s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | The patch does not generate ASF License warnings. |
| | | 152m 36s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux b571c30a355c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 1e3a0b0 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/3/artifact/out/diff-checkstyle-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/3/testReport/ |
| Max. process+thread count | 1346 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on issue #1826: HADOOP-16823. Manage S3
Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
bgaborg commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-584100455
(also this is not a bug imho, more like an improvement)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1826: HADOOP-16823.
Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-582573793
```
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:216: @InterfaceStability.Unstable: 'member def modifier' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:218: "fs.s3a.experimental.optimized.directory.operations";: '"fs.s3a.experimental.optimized.directory.operations"' has incorrect indentation level 6, expected level should be 7. [Indentation]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:285: public static final String BULK_DELETE_PAGE_SIZE =: 'member def modifier' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:286: "fs.s3a.bulk.delete.page.size";: '"fs.s3a.bulk.delete.page.size"' has incorrect indentation level 6, expected level should be 7. [Indentation]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:322: public void initialize(URI name, Configuration originalConf):3: Method length is 154 lines (max allowed is 150). [MethodLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/DeleteOperation.java:50:import static org.apache.hadoop.fs.s3a.impl.CallableSupplier.waitForCompletion;:15: Unused import - org.apache.hadoop.fs.s3a.impl.CallableSupplier.waitForCompletion. [UnusedImports]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/StandardInvokeRetryHandler.java:180: THROTTLE_LOG.debug("Request throttled on {}", metastore ? "S3" : "DynamoDB");: Line is longer than 80 characters (found 81). [LineLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java:58:import org.apache.hadoop.fs.s3a.S3ATestUtils;:8: Unused import - org.apache.hadoop.fs.s3a.S3ATestUtils. [UnusedImports]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java:207: * @return true if the DDB table has prepaid IO and is small enough to throttle.: Line is longer than 80 characters (found 82). [LineLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java:519: public void test_999_delete_all_entries() throws Throwable {:15: Name 'test_999_delete_all_entries' must match pattern '^[a-z][a-zA-Z0-9]*$'. [MethodName]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ThrottleTracker.java:113: LOG.warn("No throttling detected in {} against {}", this, ddbms.toString());: Line is longer than 80 characters (found 82). [LineLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:117: @Parameterized.Parameters(name = "bulk-delete-client-retry={0}-requests={2}-size={1}"): Line is longer than 80 characters (found 88). [LineLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:184: public void test_010_Reset() throws Throwable {:15: Name 'test_010_Reset' must match pattern '^[a-z][a-zA-Z0-9]*$'. [MethodName]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:189: public void test_020_DeleteThrottling() throws Throwable {:15: Name 'test_020_DeleteThrottling' must match pattern '^[a-z][a-zA-Z0-9]*$'. [MethodName]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:203: public void test_030_Sleep() throws Throwable {:15: Name 'test_030_Sleep' must match pattern '^[a-z][a-zA-Z0-9]*$'. [MethodName]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:222: private File deleteFiles(final int requests, final int entries):38: 'requests' hides a field. [HiddenField]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1826:
HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1826: HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-584323280
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 24s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 17s | trunk passed |
| +1 :green_heart: | compile | 18m 51s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 48s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 13s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 55s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 30s | trunk passed |
| +0 :ok: | spotbugs | 1m 8s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 28s | trunk passed |
| -0 :warning: | patch | 1m 29s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 26s | the patch passed |
| +1 :green_heart: | compile | 18m 12s | the patch passed |
| +1 :green_heart: | javac | 18m 12s | the patch passed |
| -0 :warning: | checkstyle | 2m 52s | root: The patch generated 12 new + 75 unchanged - 2 fixed = 87 total (was 77) |
| +1 :green_heart: | mvnsite | 2m 8s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 53s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 27s | the patch passed |
| +1 :green_heart: | findbugs | 3m 29s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 12s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 28s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | The patch does not generate ASF License warnings. |
| | | 133m 57s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux c016059eec4a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / d5467d2 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/9/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/9/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/9/testReport/ |
| Max. process+thread count | 1383 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/9/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1826: HADOOP-16823.
Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-582521724
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 11s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 4s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 18m 6s | trunk passed |
| +1 :green_heart: | compile | 16m 37s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 42s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 16s | trunk passed |
| +1 :green_heart: | shadedclient | 19m 3s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 13s | trunk passed |
| +0 :ok: | spotbugs | 1m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 15s | trunk passed |
| -0 :warning: | patch | 1m 35s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 18s | the patch passed |
| +1 :green_heart: | compile | 15m 51s | the patch passed |
| +1 :green_heart: | javac | 15m 51s | the patch passed |
| -0 :warning: | checkstyle | 2m 37s | root: The patch generated 16 new + 63 unchanged - 2 fixed = 79 total (was 65) |
| +1 :green_heart: | mvnsite | 2m 15s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 12m 40s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 15s | the patch passed |
| +1 :green_heart: | findbugs | 3m 31s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 15s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 43s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 53s | The patch does not generate ASF License warnings. |
| | | 119m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 4aa192819c23 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ce7b8b5 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/6/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/6/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/6/testReport/ |
| Max. process+thread count | 1366 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/6/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1826: HADOOP-16823.
Large DeleteObject requests are their own Thundering Herd
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1826: HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-585948054
merged
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1826: HADOOP-16823. Large
DeleteObject requests are their own Thundering Herd
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1826: HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-584323280
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 24s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 13s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 17s | trunk passed |
| +1 :green_heart: | compile | 18m 51s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 48s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 13s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 55s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 30s | trunk passed |
| +0 :ok: | spotbugs | 1m 8s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 28s | trunk passed |
| -0 :warning: | patch | 1m 29s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 26s | the patch passed |
| +1 :green_heart: | compile | 18m 12s | the patch passed |
| +1 :green_heart: | javac | 18m 12s | the patch passed |
| -0 :warning: | checkstyle | 2m 52s | root: The patch generated 12 new + 75 unchanged - 2 fixed = 87 total (was 77) |
| +1 :green_heart: | mvnsite | 2m 8s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 53s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 27s | the patch passed |
| +1 :green_heart: | findbugs | 3m 29s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 12s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 28s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | The patch does not generate ASF License warnings. |
| | | 133m 57s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux c016059eec4a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / d5467d2 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/9/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/9/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/9/testReport/ |
| Max. process+thread count | 1383 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/9/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1826:
HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1826: HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-584758279
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 34s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 6s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 10s | trunk passed |
| +1 :green_heart: | compile | 16m 54s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 41s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 12s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 40s | trunk passed |
| +0 :ok: | spotbugs | 1m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 17s | trunk passed |
| -0 :warning: | patch | 1m 34s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 25s | the patch passed |
| +1 :green_heart: | compile | 16m 27s | the patch passed |
| +1 :green_heart: | javac | 16m 27s | the patch passed |
| -0 :warning: | checkstyle | 2m 40s | root: The patch generated 6 new + 75 unchanged - 2 fixed = 81 total (was 77) |
| +1 :green_heart: | mvnsite | 2m 21s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 15s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 44s | the patch passed |
| +1 :green_heart: | findbugs | 3m 48s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 22s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 30s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 52s | The patch does not generate ASF License warnings. |
| | | 124m 8s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 14c16a167728 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / cc8ae59 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/10/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/10/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/10/testReport/ |
| Max. process+thread count | 1448 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/10/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1826: HADOOP-16823.
Large DeleteObject requests are their own Thundering Herd
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1826: HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-584595196
style
```
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:278: public static final String BULK_DELETE_PAGE_SIZE =: 'member def modifier' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:279: "fs.s3a.bulk.delete.page.size";: '"fs.s3a.bulk.delete.page.size"' has incorrect indentation level 6, expected level should be 7. [Indentation]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:1965: * with the counter set to the number of keys, rather than the number of invocations: Line is longer than 80 characters (found 86). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:1967: * This is because S3 considers each key as one mutating operation on the store: Line is longer than 80 characters (found 81). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/DeleteOperation.java:50:import static org.apache.hadoop.fs.s3a.impl.CallableSupplier.waitForCompletion;:15: Unused import - org.apache.hadoop.fs.s3a.impl.CallableSupplier.waitForCompletion. [UnusedImports]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java:206: * @return true if the DDB table has prepaid IO and is small enough to throttle.: Line is longer than 80 characters (found 82). [LineLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java:518: public void test_999_delete_all_entries() throws Throwable {:15: Name 'test_999_delete_all_entries' must match pattern '^[a-z][a-zA-Z0-9]*$'. [MethodName]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ThrottleTracker.java:113: LOG.warn("No throttling detected in {} against {}", this, ddbms.toString());: Line is longer than 80 characters (found 82). [LineLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:117: @Parameterized.Parameters(name = "bulk-delete-client-retry={0}-requests={2}-size={1}"): Line is longer than 80 characters (found 88). [LineLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:184: public void test_010_Reset() throws Throwable {:15: Name 'test_010_Reset' must match pattern '^[a-z][a-zA-Z0-9]*$'. [MethodName]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:189: public void test_020_DeleteThrottling() throws Throwable {:15: Name 'test_020_DeleteThrottling' must match pattern '^[a-z][a-zA-Z0-9]*$'. [MethodName]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ILoadTestS3ABulkDeleteThrottling.java:202: public void test_030_Sleep() throws Throwable {:15: Name 'test_030_Sleep' must match pattern '^[a-z][a-zA-Z0-9]*$'. [MethodName]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1826: HADOOP-16823. Large
DeleteObject requests are their own Thundering Herd
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1826: HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-584280267
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 24m 57s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 14s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 31s | trunk passed |
| +1 :green_heart: | compile | 16m 55s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 42s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 20s | trunk passed |
| +1 :green_heart: | shadedclient | 22m 38s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 43s | trunk passed |
| +0 :ok: | spotbugs | 1m 17s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 39s | trunk passed |
| -0 :warning: | patch | 1m 44s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 39s | the patch passed |
| +1 :green_heart: | compile | 19m 23s | the patch passed |
| +1 :green_heart: | javac | 19m 23s | the patch passed |
| -0 :warning: | checkstyle | 2m 43s | root: The patch generated 12 new + 75 unchanged - 2 fixed = 87 total (was 77) |
| +1 :green_heart: | mvnsite | 2m 18s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 14s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 44s | the patch passed |
| +1 :green_heart: | findbugs | 3m 28s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 23s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 35s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 52s | The patch does not generate ASF License warnings. |
| | | 153m 56s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux dc89e8d171fe 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / d5467d2 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/8/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/8/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/8/testReport/ |
| Max. process+thread count | 1597 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/8/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1826: HADOOP-16823.
Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-581431289
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 6s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 18m 0s | trunk passed |
| +1 :green_heart: | compile | 16m 35s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 38s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 17s | trunk passed |
| +1 :green_heart: | shadedclient | 18m 44s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 13s | trunk passed |
| +0 :ok: | spotbugs | 1m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 17s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 22s | the patch passed |
| +1 :green_heart: | compile | 15m 54s | the patch passed |
| +1 :green_heart: | javac | 15m 54s | the patch passed |
| -0 :warning: | checkstyle | 2m 37s | root: The patch generated 7 new + 69 unchanged - 2 fixed = 76 total (was 71) |
| +1 :green_heart: | mvnsite | 2m 19s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 12m 28s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 17s | the patch passed |
| +1 :green_heart: | findbugs | 3m 27s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 5s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 33s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 54s | The patch does not generate ASF License warnings. |
| | | 118m 8s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 95e07a7275de 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 1e3a0b0 |
| Default Java | 1.8.0_232 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/4/artifact/out/diff-checkstyle-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/4/testReport/ |
| Max. process+thread count | 1594 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #1826: HADOOP-16823.
Large DeleteObject requests are their own Thundering Herd
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #1826: HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
URL: https://github.com/apache/hadoop/pull/1826
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1826: HADOOP-16823.
Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-584220746
(retested against s3 ireland, got the failure in testListingDelete[auth=true](org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations) which is from my auth mode patch against versioned buckets -will do a quick followup for that patch)
-Dparallel-tests -DtestsThreadCount=8 -Ds3guard -Ddynamo -Dauth
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1826: HADOOP-16823.
Large DeleteObject requests are their own Thundering Herd
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1826: HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-585902913
ooh, thanks for this!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1826:
HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1826: HADOOP-16823. Large DeleteObject requests are their own Thundering Herd
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-584280267
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 24m 57s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 14s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 31s | trunk passed |
| +1 :green_heart: | compile | 16m 55s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 42s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 20s | trunk passed |
| +1 :green_heart: | shadedclient | 22m 38s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 43s | trunk passed |
| +0 :ok: | spotbugs | 1m 17s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 39s | trunk passed |
| -0 :warning: | patch | 1m 44s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 39s | the patch passed |
| +1 :green_heart: | compile | 19m 23s | the patch passed |
| +1 :green_heart: | javac | 19m 23s | the patch passed |
| -0 :warning: | checkstyle | 2m 43s | root: The patch generated 12 new + 75 unchanged - 2 fixed = 87 total (was 77) |
| +1 :green_heart: | mvnsite | 2m 18s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 14s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 44s | the patch passed |
| +1 :green_heart: | findbugs | 3m 28s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 23s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 35s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 52s | The patch does not generate ASF License warnings. |
| | | 153m 56s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux dc89e8d171fe 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / d5467d2 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/8/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/8/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/8/testReport/ |
| Max. process+thread count | 1597 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/8/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1826:
HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-580925538
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 46m 26s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 9s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 58s | trunk passed |
| +1 :green_heart: | compile | 20m 49s | trunk passed |
| +1 :green_heart: | checkstyle | 3m 6s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 24s | trunk passed |
| +1 :green_heart: | shadedclient | 19m 28s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 19s | trunk passed |
| +0 :ok: | spotbugs | 1m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 18s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 27s | hadoop-aws in the patch failed. |
| -1 :x: | compile | 15m 40s | root in the patch failed. |
| -1 :x: | javac | 15m 40s | root in the patch failed. |
| -0 :warning: | checkstyle | 2m 32s | root: The patch generated 7 new + 69 unchanged - 2 fixed = 76 total (was 71) |
| -1 :x: | mvnsite | 0m 44s | hadoop-aws in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 12m 19s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 3s | the patch passed |
| -1 :x: | findbugs | 0m 41s | hadoop-aws in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 18s | hadoop-common in the patch passed. |
| -1 :x: | unit | 0m 40s | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 48s | The patch does not generate ASF License warnings. |
| | | 171m 37s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 04387364d8db 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / bf8686f |
| Default Java | 1.8.0_242 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/patch-compile-root.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/patch-compile-root.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/diff-checkstyle-root.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/testReport/ |
| Max. process+thread count | 1372 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1826:
HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-582124713
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 30m 41s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 10s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 42s | trunk passed |
| +1 :green_heart: | compile | 17m 37s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 51s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 6s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 0s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 58s | trunk passed |
| +0 :ok: | spotbugs | 1m 6s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 8s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 19s | the patch passed |
| +1 :green_heart: | compile | 16m 49s | the patch passed |
| +1 :green_heart: | javac | 16m 49s | the patch passed |
| -0 :warning: | checkstyle | 2m 47s | root: The patch generated 16 new + 71 unchanged - 2 fixed = 87 total (was 73) |
| +1 :green_heart: | mvnsite | 2m 5s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 12s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 0s | the patch passed |
| +1 :green_heart: | findbugs | 3m 26s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 7s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 21s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | The patch does not generate ASF License warnings. |
| | | 155m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint |
| uname | Linux 2c2cceb2513c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 10a60fbe |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/5/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/5/artifact/out/whitespace-eol.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/5/testReport/ |
| Max. process+thread count | 1379 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/5/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1826:
HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-581431289
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 6s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 18m 0s | trunk passed |
| +1 :green_heart: | compile | 16m 35s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 38s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 17s | trunk passed |
| +1 :green_heart: | shadedclient | 18m 44s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 13s | trunk passed |
| +0 :ok: | spotbugs | 1m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 17s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 22s | the patch passed |
| +1 :green_heart: | compile | 15m 54s | the patch passed |
| +1 :green_heart: | javac | 15m 54s | the patch passed |
| -0 :warning: | checkstyle | 2m 37s | root: The patch generated 7 new + 69 unchanged - 2 fixed = 76 total (was 71) |
| +1 :green_heart: | mvnsite | 2m 19s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 12m 28s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 17s | the patch passed |
| +1 :green_heart: | findbugs | 3m 27s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 5s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 33s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 54s | The patch does not generate ASF License warnings. |
| | | 118m 8s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 95e07a7275de 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 1e3a0b0 |
| Default Java | 1.8.0_232 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/4/artifact/out/diff-checkstyle-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/4/testReport/ |
| Max. process+thread count | 1594 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1826: HADOOP-16823.
Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-580742619
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 9s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 37s | trunk passed |
| +1 :green_heart: | compile | 17m 46s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 47s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 6s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 6s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 57s | trunk passed |
| +0 :ok: | spotbugs | 1m 6s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 9s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 20s | the patch passed |
| +1 :green_heart: | compile | 16m 52s | the patch passed |
| +1 :green_heart: | javac | 16m 52s | the patch passed |
| -0 :warning: | checkstyle | 2m 55s | root: The patch generated 4 new + 69 unchanged - 2 fixed = 73 total (was 71) |
| +1 :green_heart: | mvnsite | 2m 11s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 6s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 59s | the patch passed |
| +1 :green_heart: | findbugs | 3m 26s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 24s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 1m 20s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | The patch does not generate ASF License warnings. |
| | | 126m 29s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 911e02ed8c48 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / bf8686f |
| Default Java | 1.8.0_232 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/1/artifact/out/diff-checkstyle-root.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/1/testReport/ |
| Max. process+thread count | 1481 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1826: HADOOP-16823.
Manage S3 Throttling exclusively in S3A client.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
URL: https://github.com/apache/hadoop/pull/1826#issuecomment-580925538
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 46m 26s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 9s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 58s | trunk passed |
| +1 :green_heart: | compile | 20m 49s | trunk passed |
| +1 :green_heart: | checkstyle | 3m 6s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 24s | trunk passed |
| +1 :green_heart: | shadedclient | 19m 28s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 19s | trunk passed |
| +0 :ok: | spotbugs | 1m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 18s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 27s | hadoop-aws in the patch failed. |
| -1 :x: | compile | 15m 40s | root in the patch failed. |
| -1 :x: | javac | 15m 40s | root in the patch failed. |
| -0 :warning: | checkstyle | 2m 32s | root: The patch generated 7 new + 69 unchanged - 2 fixed = 76 total (was 71) |
| -1 :x: | mvnsite | 0m 44s | hadoop-aws in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 12m 19s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 3s | the patch passed |
| -1 :x: | findbugs | 0m 41s | hadoop-aws in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 18s | hadoop-common in the patch passed. |
| -1 :x: | unit | 0m 40s | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 48s | The patch does not generate ASF License warnings. |
| | | 171m 37s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 04387364d8db 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / bf8686f |
| Default Java | 1.8.0_242 |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/patch-compile-root.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/patch-compile-root.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/diff-checkstyle-root.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/testReport/ |
| Max. process+thread count | 1372 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1826/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org