You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/12/15 18:31:38 UTC
[GitHub] [hadoop] steveloughran opened a new pull request #2553: HADOOP-17271. S3A to support IOStatistics
steveloughran opened a new pull request #2553:
URL: https://github.com/apache/hadoop/pull/2553
This is the rebase of #2324 #2323 with the following changes
We're into the danger zone here as I'm adding more features; to get this
in it needs to be merged and then I can add as new patches on top.
Specifically I'm looking at: aws SDK (major) and better tracking of upload
progress (minor).
## Hadoop common
HADOOP-16830 API/Implementation changes for statistics
* tuning standard statistic names
* review javadoc
* can generate a o.a.h.fs.Statistics instance from an IOStatistics instance;
this can be used to unify statistics collection/reporting.
* Duration tracker to take long as argument
* another duration tracker lambda expression for any InvocationRaisingIOE void -> void.
## Hadoop-aws
HADOOP-17271. S3A Statistics Enhancement/tuning
* move S3A statistics from o.a.h.fs.s3a.impl.statistics into s3a.statistics
(interfaces) and s3a.statistics.impl for the implementations
* org.apache.hadoop.fs.s3a.Statistic enum adds a type for each entry.
this allows test/instrumentation setup to immediately determine which
are counters vs other types, so set things up properly. This will make
a different as more statistics are added
* S3AFileSystem.getStatistics() now serves up a dynamic
binding to S3AInstrumention.instanceIOStatistics
* ...so no need to have separate statistics updating
* Committer to track duration of: file upload, file commit
* Some review of block upload counting/gauges, but key changes
left for a follow-upon JIRA.
* javadocs everywhere!
## Testing:
S3 london with
```
-Dparallel-tests -DtestsThreadCount=4 -Dmarkers=delete -Ds3guard -Ddynamo -Dfs.s3a.directory.marker.audit=true
-Dparallel-tests -DtestsThreadCount=4 -Dmarkers=keep -Dfs.s3a.directory.marker.audit=true
```
I'll do a scale run next.
1. Unifying Statistics by making the Statistic class include a type, and then automatically registering things in the right place turns out to be a nice design pattern we should use in the other stores.
1. Serving up the IOStatistics counters as FileSystem StorageStatistics eliminates duplicate work/inconsistency. The core support for this is in hadoop-common.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2553: HADOOP-17271. S3A to support IOStatistics
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2553:
URL: https://github.com/apache/hadoop/pull/2553#issuecomment-747600304
Narrator: it was the mock. It just to a long time to debug the fact that durationTracking of deleteObject was triggering an NPE, which, in commit cleanup, was being swallowed. Joy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2553: HADOOP-17271. S3A to support IOStatistics
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2553:
URL: https://github.com/apache/hadoop/pull/2553#issuecomment-747521734
Mock test failure. As usual, as likely as a mock-related change as a functional code change. But as the committer has been instrumented with stats collection, I'm going to suspect the production code first
```
java.lang.AssertionError: [Committed objects compared to deleted paths org.apache.hadoop.fs.s3a.commit.staging.StagingTestBase$ClientResults@49d4e478{ requests=12, uploads=12, parts=12, tagsByUpload=12, commits=5, aborts=7, deletes=0}]
Expecting:
<["s3a://bucket-name/output/path/r_1_1_a8d9a7c6-060d-4e05-8225-96b2c412bb6f-ed6e0eb6-81de-4ffd-abfb-337f78e5c312",
"s3a://bucket-name/output/path/r_1_1_b54314cb-8266-4beb-8ecc-61e0663b3022-ed6e0eb6-81de-4ffd-abfb-337f78e5c312",
"s3a://bucket-name/output/path/r_0_0_65638697-4b34-4320-ad61-817a1ef996ca-ed6e0eb6-81de-4ffd-abfb-337f78e5c312",
"s3a://bucket-name/output/path/r_0_0_d34de224-e5dd-4cd7-93ba-45072d14b534-ed6e0eb6-81de-4ffd-abfb-337f78e5c312",
"s3a://bucket-name/output/path/r_0_0_1fcd9943-5a6e-448b-a887-452393227473-ed6e0eb6-81de-4ffd-abfb-337f78e5c312"]>
to contain exactly in any order:
<[]>
but the following elements were unexpected:
<["s3a://bucket-name/output/path/r_1_1_a8d9a7c6-060d-4e05-8225-96b2c412bb6f-ed6e0eb6-81de-4ffd-abfb-337f78e5c312",
"s3a://bucket-name/output/path/r_1_1_b54314cb-8266-4beb-8ecc-61e0663b3022-ed6e0eb6-81de-4ffd-abfb-337f78e5c312",
"s3a://bucket-name/output/path/r_0_0_65638697-4b34-4320-ad61-817a1ef996ca-ed6e0eb6-81de-4ffd-abfb-337f78e5c312",
"s3a://bucket-name/output/path/r_0_0_d34de224-e5dd-4cd7-93ba-45072d14b534-ed6e0eb6-81de-4ffd-abfb-337f78e5c312",
"s3a://bucket-name/output/path/r_0_0_1fcd9943-5a6e-448b-a887-452393227473-ed6e0eb6-81de-4ffd-abfb-337f78e5c312"]>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2553: HADOOP-17271. S3A to support IOStatistics
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2553:
URL: https://github.com/apache/hadoop/pull/2553#issuecomment-745606706
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 48 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 47s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 36s | | trunk passed |
| +1 :green_heart: | compile | 23m 2s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 22m 18s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 28m 22s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 22s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 3m 10s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 25s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 51s | | trunk passed |
| -0 :warning: | patch | 1m 48s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 39s | | the patch passed |
| +1 :green_heart: | compile | 26m 31s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 26m 31s | | root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 0 new + 2035 unchanged - 1 fixed = 2035 total (was 2036) |
| +1 :green_heart: | compile | 23m 9s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 23m 9s | | root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 0 new + 1930 unchanged - 1 fixed = 1930 total (was 1931) |
| -0 :warning: | checkstyle | 3m 18s | [/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/1/artifact/out/diff-checkstyle-root.txt) | root: The patch generated 17 new + 271 unchanged - 26 fixed = 288 total (was 297) |
| +1 :green_heart: | mvnsite | 3m 38s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 4s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 15s | | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 59s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/1/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javadoc | 0m 41s | [/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/1/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 21 new + 88 unchanged - 0 fixed = 109 total (was 88) |
| +1 :green_heart: | findbugs | 5m 21s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 56s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 7m 47s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 1m 41s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not generate ASF License warnings. |
| | | 233m 5s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2553 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint xml |
| uname | Linux 11bb9ad92fc8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / df35c7f5199 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/1/testReport/ |
| Max. process+thread count | 3337 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2553: HADOOP-17271. S3A to support IOStatistics
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2553:
URL: https://github.com/apache/hadoop/pull/2553#issuecomment-747727830
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 8s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 56 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 12s | | trunk passed |
| +1 :green_heart: | compile | 24m 12s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 18m 42s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 11s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 19s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 2m 57s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 18s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 57s | | trunk passed |
| -0 :warning: | patch | 1m 40s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 0s | | the patch passed |
| +1 :green_heart: | compile | 19m 15s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 19m 15s | | root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 0 new + 2044 unchanged - 1 fixed = 2044 total (was 2045) |
| +1 :green_heart: | compile | 17m 21s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 17m 21s | | root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 0 new + 1940 unchanged - 1 fixed = 1940 total (was 1941) |
| -0 :warning: | checkstyle | 2m 45s | [/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/3/artifact/out/diff-checkstyle-root.txt) | root: The patch generated 19 new + 272 unchanged - 26 fixed = 291 total (was 298) |
| +1 :green_heart: | mvnsite | 3m 18s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 12s | | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 1m 3s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/3/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javadoc | 0m 44s | [/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/3/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 1 new + 88 unchanged - 0 fixed = 89 total (was 88) |
| -1 :x: | findbugs | 1m 25s | [/new-findbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/3/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html) | hadoop-tools/hadoop-aws generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 43s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 7m 8s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 1m 36s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not generate ASF License warnings. |
| | | 213m 40s | | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-aws |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.getChangeTrackerStatistics() At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.getChangeTrackerStatistics() At S3AInstrumentation.java:[line 1056] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.getInputPolicy() At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.getInputPolicy() At S3AInstrumentation.java:[line 1266] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.inputPolicySet(int) At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.inputPolicySet(int) At S3AInstrumentation.java:[line 1045] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.merge(boolean) At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.merge(boolean) At S3AInstrumentation.java:[line 1111] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.promoteInputStreamCountersToMetrics() At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.promoteInputStreamCountersToMetrics() At S3AInstrumentation.java:[line 1151] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.toString() At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.toString() At S3AInstrumentation.java:[line 1072] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$OutputStreamStatistics.blockUploadStarted(Duration, int) At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$OutputStreamStatistics.blockUploadStarted(Duration, int) At S3AInstrumentation.java:[line 1411] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$OutputStreamStatistics.toString() At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$OutputStreamStatistics.toString() At S3AInstrumentation.java:[line 1557] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2553 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint xml |
| uname | Linux 2dd6dc4ac3d5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c2672bb2342 |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/3/testReport/ |
| Max. process+thread count | 1602 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/3/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2553: HADOOP-17271. S3A to support IOStatistics
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2553:
URL: https://github.com/apache/hadoop/pull/2553#issuecomment-746781643
Latest PR fixes test failures, does more
* delete duration now tracked
* and split into single/bulk delete calls
* delete/rename cost tests updated to match
* ...and to work correctly if bulk deletes are disabled.
* +moved most of the Statistic names into the hadoop-common constant files,
for reuse.
* New method IOStatisticsLogging.ioStatisticsToSortedString(); sorts the
metrics before printing. Adds cost of creation of and insertion into
treemaps, so should be used with care. Used for test reporting.
For an example of the sorted statistics log, see https://gist.github.com/steveloughran/50ad1cf3d22d6683d4f2f038453be803
That's across all the tests in ITestS3AOperationCost, remote access to S3 london from bristol; There's a 3 millis RTT of pinging my own base station from the laptop so subtract that from all the timings.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2553: HADOOP-17271. S3A to support IOStatistics
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2553:
URL: https://github.com/apache/hadoop/pull/2553#issuecomment-750880737
Hey @bgaborg! thanks for this. Will get it in. I need to understand those failures, one may be independent. ITestS3ADeleteCost is asserting over metrics, so I'm assuming its related. And I bet its a check only performance on unguarded runs. The other one: odd.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2553: HADOOP-17271. S3A to support IOStatistics
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2553:
URL: https://github.com/apache/hadoop/pull/2553#issuecomment-750933610
Rebased and pushed up
Gabor, I've run this repeatedly and not been able to recreate either test failure!
I'll merge this in as two patches (one for hadoop-common, one for hadoop-aws) and then we will see about the followups.
the MagicCommitProtocol one is going to be unrelated and implies that your test run didn't get unique job IDs. I'm wondering if that's due to some state from a previous test run, which would explain why I haven't seen it. Will need to investigate more.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2553: HADOOP-17271. S3A to support IOStatistics
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2553:
URL: https://github.com/apache/hadoop/pull/2553#issuecomment-745533557
Test failures in tests which are seeing 1 too many LIST calls. I think the wiring up of list duration and metrics is causing this; had something similar with object metadata and HEAD requests, hence the different key names for the duration and simple counter. If the duration has the same name then it updates the counter, see.
[ERROR] testCostOfListStatusOnEmptyDir[raw-delete-markers](org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost) Time elapsed: 6.77 s <<< FAILURE!
java.lang.AssertionError: operation returning [Lorg.apache.hadoop.fs.s3a.S3AFileStatus;@10e4be9d: object_list_request expected:<2> but was:<3>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.apache.hadoop.fs.s3a.S3ATestUtils$MetricDiff.assertDiffEquals(S3ATestUtils.java:1022)
at org.apache.hadoop.fs.s3a.performance.OperationCostValidator$ExpectSingleStatistic.verify(OperationCostValidator.java:392)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #2553: HADOOP-17271. S3A to support IOStatistics
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #2553:
URL: https://github.com/apache/hadoop/pull/2553
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2553: HADOOP-17271. S3A to support IOStatistics
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2553:
URL: https://github.com/apache/hadoop/pull/2553#issuecomment-753006191
Merging in as #2577 #2579 #2580; closing this one to avoid confusion
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2553: HADOOP-17271. S3A to support IOStatistics
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2553:
URL: https://github.com/apache/hadoop/pull/2553#issuecomment-751015541
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 3s | | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 56 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 42s | | trunk passed |
| +1 :green_heart: | compile | 21m 13s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 18m 3s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 19s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 3s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 2m 51s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 16s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 59s | | trunk passed |
| -0 :warning: | patch | 1m 37s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 7s | | the patch passed |
| +1 :green_heart: | compile | 21m 0s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 21m 0s | | root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 0 new + 2032 unchanged - 1 fixed = 2032 total (was 2033) |
| +1 :green_heart: | compile | 17m 26s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 17m 26s | | root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 0 new + 1927 unchanged - 1 fixed = 1927 total (was 1928) |
| -0 :warning: | checkstyle | 2m 47s | [/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/4/artifact/out/diff-checkstyle-root.txt) | root: The patch generated 19 new + 271 unchanged - 26 fixed = 290 total (was 297) |
| +1 :green_heart: | mvnsite | 3m 20s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 3s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 28s | | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 1m 2s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/4/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javadoc | 0m 43s | [/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/4/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 1 new + 88 unchanged - 0 fixed = 89 total (was 88) |
| -1 :x: | findbugs | 1m 26s | [/new-findbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/4/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html) | hadoop-tools/hadoop-aws generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 46s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 7m 7s | | hadoop-mapreduce-client-core in the patch passed. |
| +1 :green_heart: | unit | 1m 37s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not generate ASF License warnings. |
| | | 206m 47s | | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-aws |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.getChangeTrackerStatistics() At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.getChangeTrackerStatistics() At S3AInstrumentation.java:[line 1056] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.getInputPolicy() At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.getInputPolicy() At S3AInstrumentation.java:[line 1265] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.inputPolicySet(int) At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.inputPolicySet(int) At S3AInstrumentation.java:[line 1045] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.merge(boolean) At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.merge(boolean) At S3AInstrumentation.java:[line 1111] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.promoteInputStreamCountersToMetrics() At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.promoteInputStreamCountersToMetrics() At S3AInstrumentation.java:[line 1150] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.toString() At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.toString() At S3AInstrumentation.java:[line 1072] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$OutputStreamStatistics.blockUploadStarted(Duration, int) At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$OutputStreamStatistics.blockUploadStarted(Duration, int) At S3AInstrumentation.java:[line 1410] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$OutputStreamStatistics.toString() At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$OutputStreamStatistics.toString() At S3AInstrumentation.java:[line 1556] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2553 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint xml |
| uname | Linux 3ca05c9c5334 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 005b854f6ba |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/4/testReport/ |
| Max. process+thread count | 1576 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/4/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2553: HADOOP-17271. S3A to support IOStatistics
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2553:
URL: https://github.com/apache/hadoop/pull/2553#issuecomment-747060383
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 4s | | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 55 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 4s | | trunk passed |
| +1 :green_heart: | compile | 19m 58s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | compile | 17m 13s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 22s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 18s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javadoc | 2m 57s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +0 :ok: | spotbugs | 1m 17s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 57s | | trunk passed |
| -0 :warning: | patch | 1m 39s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 2s | | the patch passed |
| +1 :green_heart: | compile | 19m 26s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 |
| +1 :green_heart: | javac | 19m 26s | | root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 0 new + 2035 unchanged - 1 fixed = 2035 total (was 2036) |
| +1 :green_heart: | compile | 17m 12s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| +1 :green_heart: | javac | 17m 12s | | root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 0 new + 1930 unchanged - 1 fixed = 1930 total (was 1931) |
| -0 :warning: | checkstyle | 2m 45s | [/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/2/artifact/out/diff-checkstyle-root.txt) | root: The patch generated 10 new + 271 unchanged - 26 fixed = 281 total (was 297) |
| +1 :green_heart: | mvnsite | 3m 19s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 3s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 31s | | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 1m 3s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/2/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. |
| -1 :x: | javadoc | 0m 44s | [/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/2/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 generated 2 new + 88 unchanged - 0 fixed = 90 total (was 88) |
| -1 :x: | findbugs | 1m 26s | [/new-findbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/2/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html) | hadoop-tools/hadoop-aws generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 45s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 7m 8s | | hadoop-mapreduce-client-core in the patch passed. |
| -1 :x: | unit | 1m 36s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/2/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not generate ASF License warnings. |
| | | 202m 40s | | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-aws |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.getChangeTrackerStatistics() At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.getChangeTrackerStatistics() At S3AInstrumentation.java:[line 1056] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.getInputPolicy() At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.getInputPolicy() At S3AInstrumentation.java:[line 1265] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.inputPolicySet(int) At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.inputPolicySet(int) At S3AInstrumentation.java:[line 1045] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.merge(boolean) At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.merge(boolean) At S3AInstrumentation.java:[line 1111] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.promoteInputStreamCountersToMetrics() At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.promoteInputStreamCountersToMetrics() At S3AInstrumentation.java:[line 1150] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.toString() At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$InputStreamStatistics.toString() At S3AInstrumentation.java:[line 1072] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$OutputStreamStatistics.blockUploadStarted(Duration, int) At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$OutputStreamStatistics.blockUploadStarted(Duration, int) At S3AInstrumentation.java:[line 1410] |
| | Potentially ambiguous invocation of either an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$OutputStreamStatistics.toString() At S3AInstrumentation.java:an outer or inherited method org.apache.hadoop.fs.s3a.statistics.impl.AbstractS3AStatisticsSource.getIOStatistics() in org.apache.hadoop.fs.s3a.S3AInstrumentation$OutputStreamStatistics.toString() At S3AInstrumentation.java:[line 1556] |
| Failed junit tests | hadoop.fs.s3a.commit.staging.TestStagingCommitter |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2553 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint xml |
| uname | Linux 455b362413a2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / df7f1e5199e |
| Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/2/testReport/ |
| Max. process+thread count | 1579 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2553/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on pull request #2553: HADOOP-17271. S3A to support IOStatistics
Posted by GitBox <gi...@apache.org>.
bgaborg commented on pull request #2553:
URL: https://github.com/apache/hadoop/pull/2553#issuecomment-748642253
(tested with `mvn clean verify -Dparallel-tests -DtestsThreadCount=8`)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org