You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/04/07 10:00:25 UTC
[GitHub] [hadoop] mukund-thakur opened a new pull request #1943: Hadoop
16465 listlocatedstatus optimisation
mukund-thakur opened a new pull request #1943: Hadoop 16465 listlocatedstatus optimisation
URL: https://github.com/apache/hadoop/pull/1943
Optimize S3AFileSystem.listLocatedStatus() to perform list
operations directly and then fallback to head checks for file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1943: HADOOP-16465
listLocatedStatus() optimisation
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1943: HADOOP-16465 listLocatedStatus() optimisation
URL: https://github.com/apache/hadoop/pull/1943#issuecomment-613521272
checkstyle is unfixable and only a line length problem
```
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:4319: private RemoteIterator<S3ALocatedFileStatus> getLocatedFileStatusIteratorForDir(: Line is longer than 80 characters (found 82). [LineLength]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on issue #1943: Hadoop 16465
listlocatedstatus optimisation
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on issue #1943: Hadoop 16465 listlocatedstatus optimisation
URL: https://github.com/apache/hadoop/pull/1943#issuecomment-612370398
Failures happening for guarded bucket.
This is a parameterised test which runs for both raw and guarded FS. If the guard settings are not enabled properly then tests actually skip rather than failing. So, I am not sure what am I missing here :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1943: Hadoop 16465
listlocatedstatus optimisation
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1943: Hadoop 16465 listlocatedstatus optimisation
URL: https://github.com/apache/hadoop/pull/1943#issuecomment-610992222
1. those failures happening when guarded or unguarded?
2. check what options you are passing down to the test runner in the IDE; it may be guarded or unguarded differently from the command line.
in HADOOP-13208 #1861 I'm parameterising these tests so that they will always test guarded +unguarded and dir marker keep vs delete. Makes for more complex assertions so I'm also improving how we assert metric diffs and report their failures. It'll make the suite very different, but it's the only way to have consistent estimates of the different codepath costs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1943: HADOOP-16465
listLocatedStatus() optimisation
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1943: HADOOP-16465 listLocatedStatus() optimisation
URL: https://github.com/apache/hadoop/pull/1943#issuecomment-613522528
LGTM. +1 from me, merging after a local test run
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on issue #1943: Hadoop 16465
listlocatedstatus optimisation
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on issue #1943: Hadoop 16465 listlocatedstatus optimisation
URL: https://github.com/apache/hadoop/pull/1943#issuecomment-612869023
Found the issue. Described here.
https://issues.apache.org/jira/browse/HADOOP-16979
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #1943: HADOOP-16465
listLocatedStatus() optimisation
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #1943: HADOOP-16465 listLocatedStatus() optimisation
URL: https://github.com/apache/hadoop/pull/1943
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1943: Hadoop 16465
listlocatedstatus optimisation
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1943: Hadoop 16465 listlocatedstatus optimisation
URL: https://github.com/apache/hadoop/pull/1943#issuecomment-610336818
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 6s | trunk passed |
| +1 :green_heart: | compile | 0m 45s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 29s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 48s | trunk passed |
| +1 :green_heart: | shadedclient | 19m 33s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 34s | trunk passed |
| +0 :ok: | spotbugs | 1m 11s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 9s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | the patch passed |
| +1 :green_heart: | compile | 0m 33s | the patch passed |
| +1 :green_heart: | javac | 0m 33s | the patch passed |
| -0 :warning: | checkstyle | 0m 21s | hadoop-tools/hadoop-aws: The patch generated 4 new + 10 unchanged - 0 fixed = 14 total (was 10) |
| +1 :green_heart: | mvnsite | 0m 39s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 18m 48s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 31s | the patch passed |
| +1 :green_heart: | findbugs | 1m 18s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 24s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 73m 9s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1943/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1943 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8a4843288838 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / c12ddbd |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1943/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1943/1/testReport/ |
| Max. process+thread count | 428 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1943/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on issue #1943: Hadoop 16465
listlocatedstatus optimisation
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on issue #1943: Hadoop 16465 listlocatedstatus optimisation
URL: https://github.com/apache/hadoop/pull/1943#issuecomment-610433133
2 of my new tests are failing on command line but succeeding in IDE.
`[ERROR] ITestS3AFileOperationCost.testCostOfListLocatedStatusOnEmptyDir:141->verifyOperationCount:190->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_list_requests starting=0 current=1 diff=1: object_list_requests expected:<0> but was:<1>
[ERROR] ITestS3AFileOperationCost.testCostOfListLocatedStatusOnNonEmptyDir:159->verifyOperationCount:190->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_list_requests starting=2 current=3 diff=1: object_list_requests expected:<0> but was:<1>`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on issue #1943: Hadoop 16465
listlocatedstatus optimisation
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on issue #1943: Hadoop 16465 listlocatedstatus optimisation
URL: https://github.com/apache/hadoop/pull/1943#issuecomment-610556018
@steveloughran @bgaborg @mehakmeet . Not sure why github is not letting me add anybody as reviewer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1943: Hadoop 16465
listlocatedstatus optimisation
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1943: Hadoop 16465 listlocatedstatus optimisation
URL: https://github.com/apache/hadoop/pull/1943#issuecomment-612890588
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 43s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 24s | trunk passed |
| +1 :green_heart: | compile | 0m 36s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 26s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | trunk passed |
| +1 :green_heart: | shadedclient | 18m 15s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | trunk passed |
| +0 :ok: | spotbugs | 1m 8s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 4s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 26s | the patch passed |
| +1 :green_heart: | javac | 0m 26s | the patch passed |
| -0 :warning: | checkstyle | 0m 18s | hadoop-tools/hadoop-aws: The patch generated 1 new + 10 unchanged - 0 fixed = 11 total (was 10) |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 36s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| +1 :green_heart: | findbugs | 1m 1s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 10s | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 67m 24s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1943/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1943 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 86cc067075db 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 8d49229 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1943/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1943/2/testReport/ |
| Max. process+thread count | 426 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1943/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on issue #1943: HADOOP-16465
listLocatedStatus() optimisation
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on issue #1943: HADOOP-16465 listLocatedStatus() optimisation
URL: https://github.com/apache/hadoop/pull/1943#issuecomment-613542503
Thanks :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org