You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/06/30 21:15:08 UTC
[GitHub] [hadoop] jimmy-zuber-amzn opened a new pull request #2113: HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus
jimmy-zuber-amzn opened a new pull request #2113:
URL: https://github.com/apache/hadoop/pull/2113
S3AFS does not support symlinks, so attempting to resolve
symlinks in globStatus causes wasted S3 calls and worse
performance. Removing it will speed up some calls to
globStatus.
JIRA link: https://issues.apache.org/jira/browse/HADOOP-17105
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jimmy-zuber-amzn commented on pull request #2113: HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus
Posted by GitBox <gi...@apache.org>.
jimmy-zuber-amzn commented on pull request #2113:
URL: https://github.com/apache/hadoop/pull/2113#issuecomment-652562946
Looks like the javadoc failures would be fixed by -> https://github.com/apache/hadoop/pull/2094
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jimmy-zuber-amzn commented on pull request #2113: HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus
Posted by GitBox <gi...@apache.org>.
jimmy-zuber-amzn commented on pull request #2113:
URL: https://github.com/apache/hadoop/pull/2113#issuecomment-652572973
Revision 2: Fixed checkstyle warnings on line length
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #2113: HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #2113:
URL: https://github.com/apache/hadoop/pull/2113#discussion_r450705856
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileOperationCost.java
##########
@@ -574,4 +574,48 @@ public void testCreateCost() throws Throwable {
}
}
+
+ @Test
+ public void testCostOfGlobStatus() throws Throwable {
+ describe("Test globStatus has expected cost");
+ S3AFileSystem fs = getFileSystem();
+ assume("Unguarded FS only", !fs.hasMetadataStore());
+
+ Path basePath = path("testCostOfGlobStatus/nextFolder/");
+
+ // create a bunch of files
+ int filesToCreate = 10;
+ for (int i = 0; i < filesToCreate; i++) {
+ try (FSDataOutputStream out = fs.create(basePath.suffix("/" + i))) {
+ verifyOperationCount(1, 1);
+ }
+ }
+
+ fs.globStatus(basePath.suffix("/*"));
+ // 2 head + 1 list from getFileStatus on path,
+ // plus 1 list to match the glob pattern
+ verifyOperationCount(2, 2);
+ }
+
+ @Test
+ public void testCostOfGlobStatusNoSymlinkResolution() throws Throwable {
Review comment:
I got it it. https://github.com/apache/hadoop/blob/f77bbc2123e3b39117f42e2c9471eb83da98380e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java#L292
Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jimmy-zuber-amzn commented on pull request #2113: HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus
Posted by GitBox <gi...@apache.org>.
jimmy-zuber-amzn commented on pull request #2113:
URL: https://github.com/apache/hadoop/pull/2113#issuecomment-654534599
> patch LGTM. Which endpoint (e.g us-west-2) and what build CLI options did you use?
>
> we don't need that much detail, though if tests are failing that's good to call out so you can get some assistance debugging. e.g
>
> [#2076 (comment)](https://github.com/apache/hadoop/pull/2076#issuecomment-649564035)
I ran with the following test settings, no tests failed by the way:
* Region: us-west-2
* Commands
* w/ S3Guard: `mvn -T 1C verify -Dparallel-tests -DtestsThreadCount=6 -Ds3guard -Ddynamo -Dauth`
* without: `mvn clean verify -Dparallel-tests`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jimmy-zuber-amzn commented on a change in pull request #2113: HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus
Posted by GitBox <gi...@apache.org>.
jimmy-zuber-amzn commented on a change in pull request #2113:
URL: https://github.com/apache/hadoop/pull/2113#discussion_r450360857
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileOperationCost.java
##########
@@ -574,4 +574,48 @@ public void testCreateCost() throws Throwable {
}
}
+
+ @Test
+ public void testCostOfGlobStatus() throws Throwable {
+ describe("Test globStatus has expected cost");
+ S3AFileSystem fs = getFileSystem();
+ assume("Unguarded FS only", !fs.hasMetadataStore());
+
+ Path basePath = path("testCostOfGlobStatus/nextFolder/");
+
+ // create a bunch of files
+ int filesToCreate = 10;
+ for (int i = 0; i < filesToCreate; i++) {
+ try (FSDataOutputStream out = fs.create(basePath.suffix("/" + i))) {
+ verifyOperationCount(1, 1);
+ }
+ }
+
+ fs.globStatus(basePath.suffix("/*"));
+ // 2 head + 1 list from getFileStatus on path,
+ // plus 1 list to match the glob pattern
+ verifyOperationCount(2, 2);
+ }
+
+ @Test
+ public void testCostOfGlobStatusNoSymlinkResolution() throws Throwable {
Review comment:
So these tests two different things, a directory with multiple objects and a directory with one object. The directory with a single object is the special case that triggers attempted symlink resolution, so I wanted to carve out that special case in its own test.
The multiple-objects-in-a-directory test is a general test that it felt like globStatus should have, whereas the second one was specifically made to catch the regression.
If you don't think that the multiple objects test is justified, I can remove it. Thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #2113: HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #2113:
URL: https://github.com/apache/hadoop/pull/2113
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #2113: HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on a change in pull request #2113:
URL: https://github.com/apache/hadoop/pull/2113#discussion_r450016787
##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileOperationCost.java
##########
@@ -574,4 +574,48 @@ public void testCreateCost() throws Throwable {
}
}
+
+ @Test
+ public void testCostOfGlobStatus() throws Throwable {
+ describe("Test globStatus has expected cost");
+ S3AFileSystem fs = getFileSystem();
+ assume("Unguarded FS only", !fs.hasMetadataStore());
+
+ Path basePath = path("testCostOfGlobStatus/nextFolder/");
+
+ // create a bunch of files
+ int filesToCreate = 10;
+ for (int i = 0; i < filesToCreate; i++) {
+ try (FSDataOutputStream out = fs.create(basePath.suffix("/" + i))) {
+ verifyOperationCount(1, 1);
+ }
+ }
+
+ fs.globStatus(basePath.suffix("/*"));
+ // 2 head + 1 list from getFileStatus on path,
+ // plus 1 list to match the glob pattern
+ verifyOperationCount(2, 2);
+ }
+
+ @Test
+ public void testCostOfGlobStatusNoSymlinkResolution() throws Throwable {
Review comment:
Both these tests look almost the same in terms operation counts and also the symlink resolution is always disabled.
Can you please tell me why do we need two tests here? Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2113: HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2113:
URL: https://github.com/apache/hadoop/pull/2113#issuecomment-653479534
oh, and +1 pending that s3 endpoint declaration.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2113: HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2113:
URL: https://github.com/apache/hadoop/pull/2113#issuecomment-652080912
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 47s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 24m 34s | trunk passed |
| +1 :green_heart: | compile | 0m 41s | trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 0m 26s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 23s | branch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 34s | hadoop-aws in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 0m 29s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 1m 10s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 7s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | the patch passed |
| +1 :green_heart: | compile | 0m 38s | the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | javac | 0m 38s | the patch passed |
| +1 :green_heart: | compile | 0m 30s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 0m 30s | the patch passed |
| -0 :warning: | checkstyle | 0m 21s | hadoop-tools/hadoop-aws: The patch generated 4 new + 11 unchanged - 0 fixed = 15 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 35s | the patch passed |
| +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 21s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 26s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 1m 9s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 72m 8s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2113 |
| JIRA Issue | HADOOP-17105 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux bc2062ad8935 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / e8dc862d385 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/1/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/1/testReport/ |
| Max. process+thread count | 459 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2113: HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2113:
URL: https://github.com/apache/hadoop/pull/2113#issuecomment-652080912
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 47s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 24m 34s | trunk passed |
| +1 :green_heart: | compile | 0m 41s | trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 0m 26s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 23s | branch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 34s | hadoop-aws in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 0m 29s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 1m 10s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 7s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | the patch passed |
| +1 :green_heart: | compile | 0m 38s | the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | javac | 0m 38s | the patch passed |
| +1 :green_heart: | compile | 0m 30s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 0m 30s | the patch passed |
| -0 :warning: | checkstyle | 0m 21s | hadoop-tools/hadoop-aws: The patch generated 4 new + 11 unchanged - 0 fixed = 15 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 35s | the patch passed |
| +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 21s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 26s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 1m 9s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 72m 8s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2113 |
| JIRA Issue | HADOOP-17105 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux bc2062ad8935 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / e8dc862d385 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/1/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/1/testReport/ |
| Max. process+thread count | 459 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2113: HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2113:
URL: https://github.com/apache/hadoop/pull/2113#issuecomment-653479139
patch LGTM. Which endpoint (e.g us-west-2) and what build CLI options did you use?
we don't need that much detail, though if tests are failing that's good to call out so you can get some assistance debugging. e.g
https://github.com/apache/hadoop/pull/2076#issuecomment-649564035
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2113: HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2113:
URL: https://github.com/apache/hadoop/pull/2113#issuecomment-652600222
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 20m 16s | trunk passed |
| +1 :green_heart: | compile | 0m 40s | trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 0m 27s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 50s | branch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 34s | hadoop-aws in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 0m 28s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 57s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | the patch passed |
| +1 :green_heart: | compile | 0m 34s | the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 |
| +1 :green_heart: | javac | 0m 34s | the patch passed |
| +1 :green_heart: | compile | 0m 29s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 0m 29s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 20s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 33s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 48s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 28s | hadoop-aws in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04. |
| +1 :green_heart: | javadoc | 0m 27s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 1m 2s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 21s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 61m 59s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2113 |
| JIRA Issue | HADOOP-17105 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d9dfb535b6b2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 04abd0eb17b |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/2/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/2/testReport/ |
| Max. process+thread count | 457 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-2113/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2113: HADOOP-17105: S3AFS - Do not attempt to resolve symlinks in globStatus
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2113:
URL: https://github.com/apache/hadoop/pull/2113#issuecomment-657709962
+1 -merged to trunk. nice catch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org