You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/02/10 15:21:28 UTC
[GitHub] [hadoop] ahmarsuhail opened a new pull request #3978: HADOOP-13704. Optimised getContentSummary()
ahmarsuhail opened a new pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978
### Description of PR
JIRA: https://issues.apache.org/jira/browse/HADOOP-13704
This PR implements an optimised version of getContentSummary which uses the result from the listFiles iterator.
Explanation of new `buildDirectorySet` method added:
Since the listFiles operation can return the directory `a/b/c` as a single object, we need to recurse over the path `a/b/c` to ensure we have counted all directories. We do this by keeping two sets, dirSet (Set of all directories under the base path) and pathTraversed (Set of paths we have recursed over so far).
Iterating over directory structure `basePath/a/b/c`, `basePath/a/b/d`, we will first find all the directories in `basePath/a/b/c`. Once this is completed, the pathTraversed set will have `{basePath/a/b}` and dirSet will have `{basePath/a, basePath/a/b, basePath/a/b/c}`.
Then for `basePath/a/b/d`, just add `basePath/a/b/d` to the dirSet and don't do any additional work as path `basePath/a/b` has already been traversed.
The Jira ticket mentions that we should add in some instrumentation to measure usage. T's already code that does this [here](https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3256)and usage is tested in an integration test [here](https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/ITestS3AMiscOperationCost.java#L144) .
### How was this patch tested?
Tested in eu-west-1 by running
`mvn -Dparallel-tests -DtestsThreadCount=16 clean verify`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ahmarsuhail commented on pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
ahmarsuhail commented on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1042971868
@steveloughran thanks for the review :) I've made the suggested changes and also updated `ITestS3ADirectoryPerformance.testListOperations()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1041671721
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 57s | | trunk passed |
| +1 :green_heart: | compile | 22m 32s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 19m 45s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 54s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 35s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 19s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 34s | | the patch passed |
| +1 :green_heart: | compile | 21m 53s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 53s | | the patch passed |
| +1 :green_heart: | compile | 19m 48s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 19m 48s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/2/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 33s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/2/artifact/out/results-checkstyle-root.txt) | root: The patch generated 53 new + 4 unchanged - 0 fixed = 57 total (was 4) |
| +1 :green_heart: | mvnsite | 2m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 49s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 36s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 25s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 50s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 41s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not generate ASF License warnings. |
| | | 218m 43s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3978 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux 89374a95dc5c 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ce88d31a45a4fd7455d7dffc6999475d90b51d28 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/2/testReport/ |
| Max. process+thread count | 1267 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1075201852
patch is in, though i realise i forgot to add the pr# to the header, which is automatic when done through the UI. never mind, at least the JIRA Is in. closing this PR as done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1044495276
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 32s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 29s | | trunk passed |
| +1 :green_heart: | compile | 24m 44s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 20m 44s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 28s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 7s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 36s | | the patch passed |
| +1 :green_heart: | compile | 24m 29s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 24m 29s | | the patch passed |
| +1 :green_heart: | compile | 21m 29s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 29s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/3/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 54s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/3/artifact/out/results-checkstyle-root.txt) | root: The patch generated 29 new + 4 unchanged - 0 fixed = 33 total (was 4) |
| +1 :green_heart: | mvnsite | 2m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 49s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javadoc | 0m 47s | [/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/3/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 2 new + 39 unchanged - 0 fixed = 41 total (was 39) |
| +1 :green_heart: | spotbugs | 4m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 52s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 25m 12s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 24s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not generate ASF License warnings. |
| | | 235m 5s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3978 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux 9d17fd41a2e1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 51bf192f3df13708414841ffd1c6b0b0b845bf90 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/3/testReport/ |
| Max. process+thread count | 3158 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1044978246
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 21s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 51s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 54s | | trunk passed |
| +1 :green_heart: | compile | 22m 16s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 19m 42s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 55s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 29s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 15s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 21m 43s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 43s | | the patch passed |
| +1 :green_heart: | compile | 19m 37s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 19m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 3m 32s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/4/artifact/out/results-checkstyle-root.txt) | root: The patch generated 29 new + 4 unchanged - 0 fixed = 33 total (was 4) |
| +1 :green_heart: | mvnsite | 2m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 54s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| -1 :x: | javadoc | 0m 48s | [/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/4/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 39 unchanged - 0 fixed = 40 total (was 39) |
| +1 :green_heart: | spotbugs | 4m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 13s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 31m 58s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 52s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not generate ASF License warnings. |
| | | 232m 39s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3978 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux f31a06407672 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 67f011a0ef63f319a74e03f0a75acc89a0d0825f |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/4/testReport/ |
| Max. process+thread count | 1281 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1044978246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1035421094
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 62m 6s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 50s | | trunk passed |
| +1 :green_heart: | compile | 22m 26s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 19m 41s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 55s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 35s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 6s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 35s | | the patch passed |
| +1 :green_heart: | compile | 21m 48s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 48s | | the patch passed |
| +1 :green_heart: | compile | 19m 43s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 19m 43s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/1/artifact/out/blanks-eol.txt) | The patch has 3 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 31s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/1/artifact/out/results-checkstyle-root.txt) | root: The patch generated 53 new + 4 unchanged - 0 fixed = 57 total (was 4) |
| +1 :green_heart: | mvnsite | 2m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 52s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 32s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 20s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 33s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 44s | | hadoop-aws in the patch passed. |
| -1 :x: | asflicense | 0m 59s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/1/artifact/out/results-asflicense.txt) | The patch generated 3 ASF License warnings. |
| | | 268m 34s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3978 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux 656603d35bca 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 061765df76561a47a98cae35ec7d0a43bbd09ab7 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/1/testReport/ |
| Max. process+thread count | 1267 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1063453062
.. not deliberately ignoring this, just falling behind on reviews while i try to get my manifest committer out the door. reviews there welcome, even though it targets abfs and gcs. #2971
i will pull some of this back into the s3a committer afterwards, including stat names and some IO enhancements to get parquet files writing faster (disabling existence/overwrite checks in __magic dirs)
#3289
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ahmarsuhail commented on a change in pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
ahmarsuhail commented on a change in pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#discussion_r809069339
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/GetContentSummaryOperation.java
##########
@@ -204,5 +233,6 @@ S3AFileStatus probePathStatus(Path path,
RemoteIterator<S3AFileStatus> listStatusIterator(Path path)
Review comment:
yup, have removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1046981805
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 20s | | trunk passed |
| +1 :green_heart: | compile | 27m 6s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 24m 53s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 27s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 11s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 49s | | the patch passed |
| +1 :green_heart: | compile | 26m 59s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 26m 59s | | the patch passed |
| +1 :green_heart: | compile | 24m 25s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 24m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 41s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 36s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 38s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 13s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 31m 3s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 47s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not generate ASF License warnings. |
| | | 261m 31s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3978 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux 1634af7db360 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3f3e1c4ba6c5f37181fa48ac077358846aa28990 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/6/testReport/ |
| Max. process+thread count | 1260 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1046907425
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 11s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 43s | | trunk passed |
| +1 :green_heart: | compile | 22m 28s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 19m 46s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 57s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 33s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 10s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 35s | | the patch passed |
| +1 :green_heart: | compile | 21m 53s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 53s | | the patch passed |
| +1 :green_heart: | compile | 19m 51s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 19m 51s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 32s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 51s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 34s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 22s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 32m 7s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 46s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not generate ASF License warnings. |
| | | 233m 23s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3978 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux a16af9ec9b1c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 945a9d9bfbe07a11e49cb325daaeb3bf05eb88ef |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/testReport/ |
| Max. process+thread count | 1664 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1041671721
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 57s | | trunk passed |
| +1 :green_heart: | compile | 22m 32s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 19m 45s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 54s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 35s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 19s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 34s | | the patch passed |
| +1 :green_heart: | compile | 21m 53s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 53s | | the patch passed |
| +1 :green_heart: | compile | 19m 48s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 19m 48s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/2/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 33s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/2/artifact/out/results-checkstyle-root.txt) | root: The patch generated 53 new + 4 unchanged - 0 fixed = 57 total (was 4) |
| +1 :green_heart: | mvnsite | 2m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 49s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 36s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 25s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 50s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 41s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not generate ASF License warnings. |
| | | 218m 43s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3978 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux 89374a95dc5c 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ce88d31a45a4fd7455d7dffc6999475d90b51d28 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/2/testReport/ |
| Max. process+thread count | 1267 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ahmarsuhail commented on a change in pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
ahmarsuhail commented on a change in pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#discussion_r809067257
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/GetContentSummaryOperation.java
##########
@@ -133,34 +133,63 @@ public ContentSummary execute() throws IOException {
* @throws IOException failure
*/
public ContentSummary getDirSummary(Path dir) throws IOException {
+
long totalLength = 0;
long fileCount = 0;
long dirCount = 1;
- final RemoteIterator<S3AFileStatus> it
- = callbacks.listStatusIterator(dir);
+
+ RemoteIterator<S3ALocatedFileStatus> it = callbacks.listFilesIterator(dir, true);
+
+ Set<Path> dirSet = new HashSet<>();
+ Set<Path> pathsTraversed = new HashSet<>();
while (it.hasNext()) {
- final S3AFileStatus s = it.next();
- if (s.isDirectory()) {
- try {
- ContentSummary c = getDirSummary(s.getPath());
- totalLength += c.getLength();
- fileCount += c.getFileCount();
- dirCount += c.getDirectoryCount();
- } catch (FileNotFoundException ignored) {
- // path was deleted during the scan; exclude from
- // summary.
- }
- } else {
- totalLength += s.getLen();
+ S3ALocatedFileStatus fileStatus = it.next();
+ Path filePath = fileStatus.getPath();
+
+ if (fileStatus.isDirectory() && !filePath.equals(dir)) {
+ dirSet.add(filePath);
+ buildDirectorySet(dirSet, pathsTraversed, dir, filePath.getParent());
+ } else if (!fileStatus.isDirectory()) {
fileCount += 1;
+ totalLength += fileStatus.getLen();
+ buildDirectorySet(dirSet, pathsTraversed, dir, filePath.getParent());
}
+
}
+
// Add the list's IOStatistics
iostatistics.aggregate(retrieveIOStatistics(it));
+
return new ContentSummary.Builder().length(totalLength).
- fileCount(fileCount).directoryCount(dirCount).
- spaceConsumed(totalLength).build();
+ fileCount(fileCount).directoryCount(dirCount + dirSet.size()).
+ spaceConsumed(totalLength).build();
+ }
+
+ /***
+ * This method builds the set of all directories found under the base path. We need to do this because if the
+ * directory structure /a/b/c was created with a single mkdirs() call, it is stored as 1 object in S3 and the list
+ * files iterator will only return a single entry /a/b/c.
+ *
+ * We keep track of paths traversed so far to prevent duplication of work. For eg, if we had a/b/c/file-1.txt and
+ * /a/b/c/file-2.txt, we will only recurse over the complete path once and won't have to do anything for file-2.txt.
+ *
+ * @param dirSet Set of all directories found in the path
+ * @param pathsTraversed Set of all paths traversed so far
+ * @param basePath Path of directory to scan
+ * @param parentPath Parent path of the current file/directory in the iterator
+ */
+ private void buildDirectorySet(Set<Path> dirSet, Set<Path> pathsTraversed, Path basePath, Path parentPath) {
+
+ if (parentPath == null || pathsTraversed.contains(parentPath) || parentPath.equals(basePath)) {
Review comment:
In most cases (eg: nested directories a/b/c.txt, a/b/d.txt) the contains condition will happen a lot more often than parentPath, so I think this order is probably the fastest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#discussion_r808346583
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractContentSummaryTest.java
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.contract;
+
+import org.apache.hadoop.fs.ContentSummary;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.assertj.core.api.Assertions;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.FileNotFoundException;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.touch;
+
+public abstract class AbstractContractContentSummaryTest extends AbstractFSContractTestBase {
+
+ @Test
+ public void testGetContentSummary() throws Throwable {
+ FileSystem fs = getFileSystem();
+
+ Path parent = path("parent");
+ Path nested = path(parent + "/a/b/c");
+ Path filePath = path(nested + "file.txt");
+
+ fs.mkdirs(parent);
+ fs.mkdirs(nested);
+ touch(getFileSystem(), filePath);
+
+ ContentSummary summary = fs.getContentSummary(parent);
+
+ Assertions.assertThat(summary.getDirectoryCount())
+ .as("Summary " + summary)
+ .isEqualTo(4);
+
+ Assertions.assertThat(summary.getFileCount())
+ .as("Summary " + summary)
+ .isEqualTo(1);
+ }
+
+ @Test
+ public void testGetContentSummaryIncorrectPath() throws Throwable {
+ FileSystem fs = getFileSystem();
+
+ Path parent = path("parent");
+ Path nested = path(parent + "/a");
+
+ fs.mkdirs(parent);
+
+ try {
Review comment:
use lambda test utils intercept() here
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractContentSummary.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.contract.s3a;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.ContentSummary;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.AbstractContractContentSummaryTest;
+import org.apache.hadoop.fs.contract.AbstractFSContract;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.assertj.core.api.Assertions;
+import org.junit.Test;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.touch;
+
+
+public class ITestS3AContractContentSummary extends AbstractContractContentSummaryTest {
+
+ @Test
+ public void testGetContentSummaryDir() throws Throwable {
+ describe("getContentSummary on test dir with children");
+ S3AFileSystem fs = getFileSystem();
+ Path baseDir = methodPath();
+
+ // Nested folders created separately will return as separate objects in listFiles()
+ fs.mkdirs(new Path(baseDir + "/a"));
Review comment:
better to use
```
new Path(basedir, "a");
```
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/GetContentSummaryOperation.java
##########
@@ -204,5 +233,6 @@ S3AFileStatus probePathStatus(Path path,
RemoteIterator<S3AFileStatus> listStatusIterator(Path path)
Review comment:
is this method obsolete now?
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractContentSummary.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.contract.s3a;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.ContentSummary;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.AbstractContractContentSummaryTest;
+import org.apache.hadoop.fs.contract.AbstractFSContract;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.assertj.core.api.Assertions;
Review comment:
nit: move to their own block above this one
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/GetContentSummaryOperation.java
##########
@@ -20,8 +20,10 @@
import java.io.FileNotFoundException;
import java.io.IOException;
+import java.util.HashSet;
import java.util.Set;
+import org.apache.hadoop.fs.s3a.S3ALocatedFileStatus;
Review comment:
nit: put all org.apache. imports in their own block under the others. note, some fixup of our move off guava means many of our current files break this rule ... and moving imports around makes cherrypicking harder. so we leave those alone
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/GetContentSummaryOperation.java
##########
@@ -133,34 +133,63 @@ public ContentSummary execute() throws IOException {
* @throws IOException failure
*/
public ContentSummary getDirSummary(Path dir) throws IOException {
+
long totalLength = 0;
long fileCount = 0;
long dirCount = 1;
- final RemoteIterator<S3AFileStatus> it
- = callbacks.listStatusIterator(dir);
+
+ RemoteIterator<S3ALocatedFileStatus> it = callbacks.listFilesIterator(dir, true);
+
+ Set<Path> dirSet = new HashSet<>();
+ Set<Path> pathsTraversed = new HashSet<>();
while (it.hasNext()) {
- final S3AFileStatus s = it.next();
- if (s.isDirectory()) {
- try {
- ContentSummary c = getDirSummary(s.getPath());
- totalLength += c.getLength();
- fileCount += c.getFileCount();
- dirCount += c.getDirectoryCount();
- } catch (FileNotFoundException ignored) {
- // path was deleted during the scan; exclude from
- // summary.
- }
- } else {
- totalLength += s.getLen();
+ S3ALocatedFileStatus fileStatus = it.next();
+ Path filePath = fileStatus.getPath();
+
+ if (fileStatus.isDirectory() && !filePath.equals(dir)) {
+ dirSet.add(filePath);
+ buildDirectorySet(dirSet, pathsTraversed, dir, filePath.getParent());
+ } else if (!fileStatus.isDirectory()) {
fileCount += 1;
+ totalLength += fileStatus.getLen();
+ buildDirectorySet(dirSet, pathsTraversed, dir, filePath.getParent());
}
+
}
+
// Add the list's IOStatistics
iostatistics.aggregate(retrieveIOStatistics(it));
+
return new ContentSummary.Builder().length(totalLength).
- fileCount(fileCount).directoryCount(dirCount).
- spaceConsumed(totalLength).build();
+ fileCount(fileCount).directoryCount(dirCount + dirSet.size()).
+ spaceConsumed(totalLength).build();
+ }
+
+ /***
+ * This method builds the set of all directories found under the base path. We need to do this because if the
+ * directory structure /a/b/c was created with a single mkdirs() call, it is stored as 1 object in S3 and the list
+ * files iterator will only return a single entry /a/b/c.
+ *
+ * We keep track of paths traversed so far to prevent duplication of work. For eg, if we had a/b/c/file-1.txt and
+ * /a/b/c/file-2.txt, we will only recurse over the complete path once and won't have to do anything for file-2.txt.
+ *
+ * @param dirSet Set of all directories found in the path
+ * @param pathsTraversed Set of all paths traversed so far
+ * @param basePath Path of directory to scan
+ * @param parentPath Parent path of the current file/directory in the iterator
+ */
+ private void buildDirectorySet(Set<Path> dirSet, Set<Path> pathsTraversed, Path basePath, Path parentPath) {
+
+ if (parentPath == null || pathsTraversed.contains(parentPath) || parentPath.equals(basePath)) {
Review comment:
would this be faster if the parentPath equals was done before the contained() probe? or is it rare enough that it wouldn't do much/be worse
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1035421094
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 4 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 62m 6s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 50s | | trunk passed |
| +1 :green_heart: | compile | 22m 26s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 19m 41s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 55s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 35s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 6s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 35s | | the patch passed |
| +1 :green_heart: | compile | 21m 48s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 48s | | the patch passed |
| +1 :green_heart: | compile | 19m 43s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 19m 43s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/1/artifact/out/blanks-eol.txt) | The patch has 3 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 31s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/1/artifact/out/results-checkstyle-root.txt) | root: The patch generated 53 new + 4 unchanged - 0 fixed = 57 total (was 4) |
| +1 :green_heart: | mvnsite | 2m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 52s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 32s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 20s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 33s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 44s | | hadoop-aws in the patch passed. |
| -1 :x: | asflicense | 0m 59s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/1/artifact/out/results-asflicense.txt) | The patch generated 3 ASF License warnings. |
| | | 268m 34s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3978 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux 656603d35bca 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 061765df76561a47a98cae35ec7d0a43bbd09ab7 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/1/testReport/ |
| Max. process+thread count | 1267 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978#issuecomment-1046907425
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 11s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 43s | | trunk passed |
| +1 :green_heart: | compile | 22m 28s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 19m 46s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 57s | | trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 33s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 10s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 35s | | the patch passed |
| +1 :green_heart: | compile | 21m 53s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 53s | | the patch passed |
| +1 :green_heart: | compile | 19m 51s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 19m 51s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 3m 32s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 51s | | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 2m 34s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 22s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 32m 7s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 46s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not generate ASF License warnings. |
| | | 233m 23s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3978 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle |
| uname | Linux a16af9ec9b1c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 945a9d9bfbe07a11e49cb325daaeb3bf05eb88ef |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/testReport/ |
| Max. process+thread count | 1664 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3978/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #3978: HADOOP-13704. Optimised getContentSummary()
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #3978:
URL: https://github.com/apache/hadoop/pull/3978
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org