You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/08/28 12:17:04 UTC
[GitHub] [hadoop] mukund-thakur opened a new pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
mukund-thakur opened a new pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257
S3AFileSystem.listStatus() to perform list operations
directly and then fallback to head checks for files
Tested using ap-south-1 bucket in following modes along with scale tests:
1) Raw s3.
2) Auth S3guard.
3) Non auth s3guard.
All good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-682588174
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 5s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 46s | trunk passed |
| +1 :green_heart: | compile | 1m 11s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 43s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 32s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 58s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 36s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 32s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 28s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 48s | the patch passed |
| +1 :green_heart: | compile | 0m 48s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 48s | the patch passed |
| +1 :green_heart: | compile | 0m 37s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 37s | the patch passed |
| -0 :warning: | checkstyle | 0m 23s | hadoop-tools/hadoop-aws: The patch generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13) |
| +1 :green_heart: | mvnsite | 0m 47s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 18m 48s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 18s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 35s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 5s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 12s | The patch does not generate ASF License warnings. |
| | | 96m 24s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2257 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 1af3e1f76ff6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bfd99f31fd6 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/testReport/ |
| Max. process+thread count | 344 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688314591
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 33m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 0m 22s | root in trunk failed. |
| -1 :x: | compile | 0m 11s | hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | compile | 0m 25s | hadoop-aws in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -0 :warning: | checkstyle | 0m 20s | The patch fails to run checkstyle in hadoop-aws |
| -1 :x: | mvnsite | 0m 10s | hadoop-aws in trunk failed. |
| -1 :x: | shadedclient | 0m 49s | branch has errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 16s | hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javadoc | 0m 22s | hadoop-aws in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| +0 :ok: | spotbugs | 1m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| -1 :x: | findbugs | 0m 22s | hadoop-aws in trunk failed. |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 41s | hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 23s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javac | 0m 23s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | compile | 0m 22s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | javac | 0m 22s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -0 :warning: | checkstyle | 0m 5s | The patch fails to run checkstyle in hadoop-aws |
| -1 :x: | mvnsite | 0m 12s | hadoop-aws in the patch failed. |
| -1 :x: | whitespace | 0m 0s | The patch has 38 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | whitespace | 0m 0s | The patch 600 line(s) with tabs. |
| +1 :green_heart: | shadedclient | 0m 21s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 22s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javadoc | 0m 23s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | findbugs | 0m 23s | hadoop-aws in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 22s | hadoop-aws in the patch failed. |
| +0 :ok: | asflicense | 0m 22s | ASF License check generated no output? |
| | | 43m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2257 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 667bf71456f4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 34fe74da0e9 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-mvnsite-hadoop-tools_hadoop-aws.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-findbugs-hadoop-tools_hadoop-aws.txt |
| mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/buildtool-patch-checkstyle-hadoop-tools_hadoop-aws.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt |
| whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/whitespace-eol.txt |
| whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/whitespace-tabs.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/testReport/ |
| Max. process+thread count | 51 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/console |
| versions | git=2.17.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#discussion_r479671878
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/performance/OperationCost.java
##########
@@ -110,6 +110,9 @@
public static final OperationCost LIST_FILES_LIST_OP =
new OperationCost(0, 1);
+ /** listStatus always does a LIST. */
+ public static final OperationCost LIST_STATUS_LIST_OP =
+ new OperationCost(0, 1);
Review comment:
is this just LIST_OPERATION? IF so, make the new constant a reference to that one to make clear that's what is happening. Probably good to do the same above too
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileOperationCost.java
##########
@@ -176,6 +176,77 @@ public void testCostOfListFilesOnNonExistingDir() throws Throwable {
.plus(GET_FILE_STATUS_FNFE)));
}
+ @Test
+ public void testCostOfListStatusOnFile() throws Throwable {
+ describe("Performing listStatus() on a file");
+ Path file = path(getMethodName() + ".txt");
+ S3AFileSystem fs = getFileSystem();
+ touch(fs, file);
+ verifyMetrics(() ->
+ fs.listStatus(file),
+ whenRaw(LIST_STATUS_LIST_OP
+ .plus(GET_FILE_STATUS_ON_FILE)),
+ whenAuthoritative(LIST_STATUS_LIST_OP),
+ whenNonauth(LIST_STATUS_LIST_OP));
+// resetMetricDiffs();
+// fs.listStatus(file);
+// if (!fs.hasMetadataStore()) {
+// metadataRequests.assertDiffEquals(1);
+// }
+// listRequests.assertDiffEquals(1);
+ }
+
+ @Test
+ public void testCostOfListStatusOnEmptyDir() throws Throwable {
+ describe("Performing listStatus() on an empty dir");
+ Path dir = path(getMethodName());
+ S3AFileSystem fs = getFileSystem();
+ fs.mkdirs(dir);
+ verifyMetrics(() ->
+ fs.listStatus(dir),
+ whenRaw(LIST_STATUS_LIST_OP
Review comment:
nice to see you picking up the new metrics validation I think it makes a big difference. If there are ways it can be improved, feel free to do so.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688822205
checkstyle complaining with fixable issues
```
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java:370: Listing.FileStatusListingIterator filesItr = createFileStatusListingIterator(: Line is longer than 80 characters (found 81). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:34:import java.util.Arrays;:8: Unused import - java.util.Arrays. [UnusedImports]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2672: private RemoteIterator<S3AFileStatus> innerListStatus(Path f) throws FileNotFoundException,: Line is longer than 80 characters (found 93). [LineLength]
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2686: }:5: '}' at column 5 should be on the same line as the next part of a multi-block statement (one that directly contains multiple blocks: if/else-if/else, do/while or try/catch/finally). [RightCurly]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java:45:import org.apache.hadoop.fs.s3a.impl.TestPartialDeleteFailures;:8: Unused import - org.apache.hadoop.fs.s3a.impl.TestPartialDeleteFailures. [UnusedImports]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java:138: S3AFileStatus[] result = S3AUtils.iteratorToStatuses(resultItr, new HashSet<>());: Line is longer than 80 characters (found 85). [LineLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java:205: S3AFileStatus[] result = S3AUtils.iteratorToStatuses(resultItr, new HashSet<>());: Line is longer than 80 characters (found 85). [LineLength]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-694763950
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 13s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 1s | trunk passed |
| +1 :green_heart: | compile | 0m 51s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 35s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 29s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | trunk passed |
| +1 :green_heart: | shadedclient | 18m 0s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 14s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 11s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | the patch passed |
| +1 :green_heart: | compile | 0m 38s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 38s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 21s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 39s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 35s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 16s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 11s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 39s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 81m 12s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2257 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9d99c637fb86 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / eacbe07b565 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/5/testReport/ |
| Max. process+thread count | 338 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/5/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688823862
> checkstyle complaining with fixable issues
>
> ```
> ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java:370: Listing.FileStatusListingIterator filesItr = createFileStatusListingIterator(: Line is longer than 80 characters (found 81). [LineLength]
> ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:34:import java.util.Arrays;:8: Unused import - java.util.Arrays. [UnusedImports]
> ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2672: private RemoteIterator<S3AFileStatus> innerListStatus(Path f) throws FileNotFoundException,: Line is longer than 80 characters (found 93). [LineLength]
> ./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2686: }:5: '}' at column 5 should be on the same line as the next part of a multi-block statement (one that directly contains multiple blocks: if/else-if/else, do/while or try/catch/finally). [RightCurly]
> ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java:45:import org.apache.hadoop.fs.s3a.impl.TestPartialDeleteFailures;:8: Unused import - org.apache.hadoop.fs.s3a.impl.TestPartialDeleteFailures. [UnusedImports]
> ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java:138: S3AFileStatus[] result = S3AUtils.iteratorToStatuses(resultItr, new HashSet<>());: Line is longer than 80 characters (found 85). [LineLength]
> ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java:205: S3AFileStatus[] result = S3AUtils.iteratorToStatuses(resultItr, new HashSet<>());: Line is longer than 80 characters (found 85). [LineLength]
> ```
Yes I am going to fix the checkstyle. How is the rest of code?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-694763950
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 13s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 1s | trunk passed |
| +1 :green_heart: | compile | 0m 51s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 35s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 29s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | trunk passed |
| +1 :green_heart: | shadedclient | 18m 0s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 14s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 11s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | the patch passed |
| +1 :green_heart: | compile | 0m 38s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 38s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 21s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 39s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 35s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 16s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 11s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 39s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 81m 12s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2257 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9d99c637fb86 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / eacbe07b565 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/5/testReport/ |
| Max. process+thread count | 338 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/5/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688762761
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 6s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 52s | trunk passed |
| +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 20s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 27s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 0s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 33s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| -0 :warning: | checkstyle | 0m 20s | hadoop-tools/hadoop-aws: The patch generated 7 new + 62 unchanged - 0 fixed = 69 total (was 62) |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 6s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 17s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 12s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 39s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 78m 48s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2257 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 27053c1e385f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ac7d4623aef |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/testReport/ |
| Max. process+thread count | 360 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-696773775
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-696211236
LGTM. +1
-thank you for a great piece of work here!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-694312667
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 28m 50s | trunk passed |
| +1 :green_heart: | compile | 0m 41s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 30s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 12s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 31s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 2s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 34s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 34s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| -0 :warning: | checkstyle | 0m 21s | hadoop-tools/hadoop-aws: The patch generated 4 new + 62 unchanged - 0 fixed = 66 total (was 62) |
| +1 :green_heart: | mvnsite | 0m 30s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 2s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | javadoc | 0m 25s | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) |
| +1 :green_heart: | findbugs | 1m 6s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 20s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 71m 12s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2257 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 177d1c552191 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 20a0e6278d6 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/4/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/4/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/4/testReport/ |
| Max. process+thread count | 456 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/4/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688762761
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 6s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 52s | trunk passed |
| +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 27s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 20s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 27s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 0s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 33s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| -0 :warning: | checkstyle | 0m 20s | hadoop-tools/hadoop-aws: The patch generated 7 new + 62 unchanged - 0 fixed = 69 total (was 62) |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 6s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 17s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 12s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 39s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 78m 48s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2257 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 27053c1e385f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ac7d4623aef |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/testReport/ |
| Max. process+thread count | 360 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/3/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-688314591
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 33m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 0m 22s | root in trunk failed. |
| -1 :x: | compile | 0m 11s | hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | compile | 0m 25s | hadoop-aws in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -0 :warning: | checkstyle | 0m 20s | The patch fails to run checkstyle in hadoop-aws |
| -1 :x: | mvnsite | 0m 10s | hadoop-aws in trunk failed. |
| -1 :x: | shadedclient | 0m 49s | branch has errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 16s | hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javadoc | 0m 22s | hadoop-aws in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| +0 :ok: | spotbugs | 1m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| -1 :x: | findbugs | 0m 22s | hadoop-aws in trunk failed. |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 41s | hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 23s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javac | 0m 23s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | compile | 0m 22s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | javac | 0m 22s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -0 :warning: | checkstyle | 0m 5s | The patch fails to run checkstyle in hadoop-aws |
| -1 :x: | mvnsite | 0m 12s | hadoop-aws in the patch failed. |
| -1 :x: | whitespace | 0m 0s | The patch has 38 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | whitespace | 0m 0s | The patch 600 line(s) with tabs. |
| +1 :green_heart: | shadedclient | 0m 21s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 22s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javadoc | 0m 23s | hadoop-aws in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | findbugs | 0m 23s | hadoop-aws in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 22s | hadoop-aws in the patch failed. |
| +0 :ok: | asflicense | 0m 22s | ASF License check generated no output? |
| | | 43m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2257 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 667bf71456f4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 34fe74da0e9 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-mvnsite-hadoop-tools_hadoop-aws.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/branch-findbugs-hadoop-tools_hadoop-aws.txt |
| mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/buildtool-patch-checkstyle-hadoop-tools_hadoop-aws.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt |
| whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/whitespace-eol.txt |
| whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/whitespace-tabs.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/testReport/ |
| Max. process+thread count | 51 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/2/console |
| versions | git=2.17.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-696793895
> Surprisingly large mount of merge problems surfacing here, not in the production code but in the tests. A sign of my patches all going near the same code you've worked on.
>
> I'm going to move the new inner classes in S3ATestUtils out of there and into their own classes in o.a.h.fs.s3a.test; reduces the diff to S3ATestUtils (which is often a source of spurious merge pain), so once isolated, less long term suffering.
>
> Doing that in #2310 which is a high priority change for me
Oh Yes I refactored some methods to S3ATestUtils thinking that would be the right place for them. Isolating them to o.a.h.fs.s3a.test seems a better idea.
Sorry for the inconvenience though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-696773775
Surprisingly large mount of merge problems surfacing here, not in the production code but in the tests. A sign of my patches all going near the same code you've worked on.
I'm going to move the new inner classes in S3ATestUtils out of there and into their own classes in o.a.h.fs.s3a.test; reduces the diff to S3ATestUtils (which is often a source of spurious merge pain), so once isolated, less long term suffering.
Doing that in #2310 which is a high priority change for me
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#discussion_r484868210
##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java
##########
@@ -362,14 +340,16 @@ public static BulkOperationState initiateBulkWrite(
* @return Final result of directory listing.
* @throws IOException if metadata store update failed
*/
- public static S3AFileStatus[] dirListingUnion(MetadataStore ms, Path path,
- List<S3AFileStatus> backingStatuses, DirListingMetadata dirMeta,
- boolean isAuthoritative, ITtlTimeProvider timeProvider)
- throws IOException {
+ public static RemoteIterator<S3AFileStatus> dirListingUnion(
+ MetadataStore ms, Path path,
+ RemoteIterator<S3AFileStatus> backingStatuses,
+ DirListingMetadata dirMeta, boolean isAuthoritative,
+ ITtlTimeProvider timeProvider, Listing listing)
Review comment:
we are into callback country here. Could you
1. create an interface (say in S3Guard) for createProvidedFileStatusIteratorForS3Guard (it could contain all params for `createProvidedFileStatusIterator` oterh than the dir meta list, 2. Have Listing implement it
3. Pass the interface in, not the reference to Listing.
I'm just trying to move away from having all the classes in here having intimate knowledge of every other class. An interface with a single method lets us control exactly what S3Guard does.
You could use a Lambda-expression if you want -for a single operation it would work, just take `Function<S3AFileStatus[], RemoteIterator<S3AFileStatus>` as a parameter and pass the operation in that way. But, given you use this operation in tests as well as production code, you should still implement this function as a single method in Listing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-694312667
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 28m 50s | trunk passed |
| +1 :green_heart: | compile | 0m 41s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 30s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 12s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 31s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 2s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 34s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 34s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| -0 :warning: | checkstyle | 0m 21s | hadoop-tools/hadoop-aws: The patch generated 4 new + 62 unchanged - 0 fixed = 66 total (was 62) |
| +1 :green_heart: | mvnsite | 0m 30s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 2s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | javadoc | 0m 25s | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) |
| +1 :green_heart: | findbugs | 1m 6s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 20s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 71m 12s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2257 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 177d1c552191 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 20a0e6278d6 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/4/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/4/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/4/testReport/ |
| Max. process+thread count | 456 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/4/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-694312667
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 28m 50s | trunk passed |
| +1 :green_heart: | compile | 0m 41s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 30s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 12s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 31s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 2s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 34s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 34s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| -0 :warning: | checkstyle | 0m 21s | hadoop-tools/hadoop-aws: The patch generated 4 new + 62 unchanged - 0 fixed = 66 total (was 62) |
| +1 :green_heart: | mvnsite | 0m 30s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 2s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | javadoc | 0m 25s | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) |
| +1 :green_heart: | findbugs | 1m 6s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 20s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 71m 12s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2257 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 177d1c552191 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 20a0e6278d6 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/4/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/4/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/4/testReport/ |
| Max. process+thread count | 456 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/4/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#discussion_r479223572
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileOperationCost.java
##########
@@ -176,6 +176,77 @@ public void testCostOfListFilesOnNonExistingDir() throws Throwable {
.plus(GET_FILE_STATUS_FNFE)));
}
+ @Test
+ public void testCostOfListStatusOnFile() throws Throwable {
+ describe("Performing listStatus() on a file");
+ Path file = path(getMethodName() + ".txt");
+ S3AFileSystem fs = getFileSystem();
+ touch(fs, file);
+ verifyMetrics(() ->
+ fs.listStatus(file),
+ whenRaw(LIST_STATUS_LIST_OP
+ .plus(GET_FILE_STATUS_ON_FILE)),
+ whenAuthoritative(LIST_STATUS_LIST_OP),
+ whenNonauth(LIST_STATUS_LIST_OP));
+// resetMetricDiffs();
+// fs.listStatus(file);
Review comment:
will remove the comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-694763950
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 13s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 1s | trunk passed |
| +1 :green_heart: | compile | 0m 51s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 35s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 29s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | trunk passed |
| +1 :green_heart: | shadedclient | 18m 0s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 14s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 11s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | the patch passed |
| +1 :green_heart: | compile | 0m 38s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 38s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 21s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 39s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 35s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 16s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 11s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 39s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 81m 12s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2257 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9d99c637fb86 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / eacbe07b565 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/5/testReport/ |
| Max. process+thread count | 338 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/5/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-682588174
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 5s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 46s | trunk passed |
| +1 :green_heart: | compile | 1m 11s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 43s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 32s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 58s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 36s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 32s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 28s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 48s | the patch passed |
| +1 :green_heart: | compile | 0m 48s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 48s | the patch passed |
| +1 :green_heart: | compile | 0m 37s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 37s | the patch passed |
| -0 :warning: | checkstyle | 0m 23s | hadoop-tools/hadoop-aws: The patch generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13) |
| +1 :green_heart: | mvnsite | 0m 47s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 18m 48s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 18s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 35s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 5s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 12s | The patch does not generate ASF License warnings. |
| | | 96m 24s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2257 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 1af3e1f76ff6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bfd99f31fd6 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/testReport/ |
| Max. process+thread count | 344 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2257/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-696793895
> Surprisingly large mount of merge problems surfacing here, not in the production code but in the tests. A sign of my patches all going near the same code you've worked on.
>
> I'm going to move the new inner classes in S3ATestUtils out of there and into their own classes in o.a.h.fs.s3a.test; reduces the diff to S3ATestUtils (which is often a source of spurious merge pain), so once isolated, less long term suffering.
>
> Doing that in #2310 which is a high priority change for me
Oh Yes I refactored some methods to S3ATestUtils thinking that would be the right place for them. Isolating them to o.a.h.fs.s3a.test seems a better idea.
Sorry for the inconvenience though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#discussion_r484259633
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileOperationCost.java
##########
@@ -176,6 +176,77 @@ public void testCostOfListFilesOnNonExistingDir() throws Throwable {
.plus(GET_FILE_STATUS_FNFE)));
}
+ @Test
+ public void testCostOfListStatusOnFile() throws Throwable {
+ describe("Performing listStatus() on a file");
+ Path file = path(getMethodName() + ".txt");
+ S3AFileSystem fs = getFileSystem();
+ touch(fs, file);
+ verifyMetrics(() ->
+ fs.listStatus(file),
+ whenRaw(LIST_STATUS_LIST_OP
+ .plus(GET_FILE_STATUS_ON_FILE)),
+ whenAuthoritative(LIST_STATUS_LIST_OP),
+ whenNonauth(LIST_STATUS_LIST_OP));
+// resetMetricDiffs();
+// fs.listStatus(file);
+// if (!fs.hasMetadataStore()) {
+// metadataRequests.assertDiffEquals(1);
+// }
+// listRequests.assertDiffEquals(1);
+ }
+
+ @Test
+ public void testCostOfListStatusOnEmptyDir() throws Throwable {
+ describe("Performing listStatus() on an empty dir");
+ Path dir = path(getMethodName());
+ S3AFileSystem fs = getFileSystem();
+ fs.mkdirs(dir);
+ verifyMetrics(() ->
+ fs.listStatus(dir),
+ whenRaw(LIST_STATUS_LIST_OP
Review comment:
Sure Thanks :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-696828885
S3AUtils and S3ATestUtils were where we stuck everything; creates lots of spurious merge pain. Isolation will ultimately improve life
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2257: HADOOP-17023 Tune S3AFileSystem.listStatus() api.
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2257:
URL: https://github.com/apache/hadoop/pull/2257#issuecomment-696211236
LGTM. +1
-thank you for a great piece of work here!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org