You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/04/02 17:04:10 UTC
[GitHub] [hadoop] steveloughran opened a new pull request #1936:
HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail
against S3
steveloughran opened a new pull request #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936
Contributed by Steve Loughran.
Strips out all the attributes which have already been processed,
before deciding whether or not to query the far end for the status of the file.
This will avoid 404 caching-related issues in S3, and save time and a bit of IO
everywhere.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1936: HADOOP-16932.
distcp copy calls getFileStatus() needlessly and can fail against S3
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-608020742
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 34s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 25s | trunk passed |
| +1 :green_heart: | compile | 0m 29s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 28s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 21s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 0m 47s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 45s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | the patch passed |
| +1 :green_heart: | compile | 0m 21s | the patch passed |
| +1 :green_heart: | javac | 0m 21s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 18s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 23s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 50s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 18s | the patch passed |
| +1 :green_heart: | findbugs | 0m 46s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 11m 0s | hadoop-distcp in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 67m 13s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.tools.util.TestDistCpUtils |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1936 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b0dd8d2ac766 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 25361b0 |
| Default Java | 1.8.0_242 |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/1/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/1/testReport/ |
| Max. process+thread count | 482 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932.
distcp copy calls getFileStatus() needlessly and can fail against S3
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-608336023
yeah, looks like I didn't test for regressions. Sill, yetus gets to do that for github PRs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #1936: HADOOP-16932.
distcp copy calls getFileStatus() needlessly and can fail against S3
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1936: HADOOP-16932.
distcp copy calls getFileStatus() needlessly and can fail against S3
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-608690060
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 15s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 24m 0s | trunk passed |
| +1 :green_heart: | compile | 0m 25s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 30s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | trunk passed |
| +0 :ok: | spotbugs | 0m 42s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 40s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 20s | the patch passed |
| +1 :green_heart: | javac | 0m 20s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-distcp: The patch generated 2 new + 158 unchanged - 12 fixed = 160 total (was 170) |
| +1 :green_heart: | mvnsite | 0m 23s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 17m 11s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| +1 :green_heart: | findbugs | 0m 54s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 23m 21s | hadoop-distcp in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. |
| | | 89m 38s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.tools.TestDistCpSyncReverseFromTarget |
| | hadoop.tools.TestDistCpWithXAttrs |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1936 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9c7d9ef5c3f0 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 1695d8d |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/2/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/2/testReport/ |
| Max. process+thread count | 332 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932.
distcp copy calls getFileStatus() needlessly and can fail against S3
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-608343496
Various tests in .TestDistCpUtils failing, tests whose assertion's dont include error text. @mukund-thakur -this is a classic example of why I insist all new asserts are meaningful :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932.
distcp copy calls getFileStatus() needlessly and can fail against S3
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-609764508
local test run happy; the test failures are close but probably unrelated.
```
[INFO] Running org.apache.hadoop.tools.TestDistCpSystem
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.917 s - in org.apache.hadoop.tools.TestDistCpSystem
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 347, Failures: 0, Errors: 0, Skipped: 0
[INFO]
```
One thing to worry about: yetus doesn't normally run distCP tests because there are so few patches against it. Could be an intrinsic problem.
I'm going to rebase and resubmitted to see what happens.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1936:
HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail
against S3
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-608020742
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 34s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 25s | trunk passed |
| +1 :green_heart: | compile | 0m 29s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 28s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 21s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 0m 47s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 45s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | the patch passed |
| +1 :green_heart: | compile | 0m 21s | the patch passed |
| +1 :green_heart: | javac | 0m 21s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 18s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 23s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 50s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 18s | the patch passed |
| +1 :green_heart: | findbugs | 0m 46s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 11m 0s | hadoop-distcp in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 67m 13s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.tools.util.TestDistCpUtils |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1936 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b0dd8d2ac766 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 25361b0 |
| Default Java | 1.8.0_242 |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/1/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/1/testReport/ |
| Max. process+thread count | 482 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1936: HADOOP-16932.
distcp copy calls getFileStatus() needlessly and can fail against S3
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-609805287
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 51s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 31s | trunk passed |
| +1 :green_heart: | compile | 0m 25s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 26s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 25s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | trunk passed |
| +0 :ok: | spotbugs | 0m 38s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 37s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 24s | the patch passed |
| +1 :green_heart: | compile | 0m 19s | the patch passed |
| +1 :green_heart: | javac | 0m 19s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-distcp: The patch generated 2 new + 158 unchanged - 12 fixed = 160 total (was 170) |
| +1 :green_heart: | mvnsite | 0m 22s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 25s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | the patch passed |
| +1 :green_heart: | findbugs | 0m 43s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 57s | hadoop-distcp in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 73m 38s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1936 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux cdc0443a8acb 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ab7495d |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/3/testReport/ |
| Max. process+thread count | 350 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932.
distcp copy calls getFileStatus() needlessly and can fail against S3
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-607972418
adds a quick fix and tests; verified that without the changes in production code the test fails
```
java.io.FileNotFoundException: File does not exist: /tmp/testSkipsNeedlessAttributes/dest
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1729)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1722)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1737)
at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:210)
at org.apache.hadoop.tools.util.TestDistCpUtils.testSkipsNeedlessAttributes(TestDistCpUtils.java:234)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932.
distcp copy calls getFileStatus() needlessly and can fail against S3
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-610313950
thx
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on issue #1936: HADOOP-16932.
distcp copy calls getFileStatus() needlessly and can fail against S3
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-609000106
Looks good. I have one question btw.
I see replication factor is getting passed while creating the target file. But then why we are trying to preserve it again in DistCpUtils.preserve(). ??
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932.
distcp copy calls getFileStatus() needlessly and can fail against S3
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-609717556
you can set it in an API; I think it's propagating it so that even if you are doing an incremental write/append/update the replication factor will be updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932.
distcp copy calls getFileStatus() needlessly and can fail against S3
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-609848048
line length.
```
./hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java:231: String text = String.format("Source %s; dest %s: wrong ", srcStatus, destStatus);: Line is longer than 80 characters (found 85). [LineLength]
./hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java:232: // FileStatus.equals only compares path field, must explicitly compare all fields: Line is longer than 80 characters (found 85). [LineLeng
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1936:
HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail
against S3
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-608690060
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 15s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 24m 0s | trunk passed |
| +1 :green_heart: | compile | 0m 25s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 30s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | trunk passed |
| +0 :ok: | spotbugs | 0m 42s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 40s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 20s | the patch passed |
| +1 :green_heart: | javac | 0m 20s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-distcp: The patch generated 2 new + 158 unchanged - 12 fixed = 160 total (was 170) |
| +1 :green_heart: | mvnsite | 0m 23s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 17m 11s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| +1 :green_heart: | findbugs | 0m 54s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 23m 21s | hadoop-distcp in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. |
| | | 89m 38s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.tools.TestDistCpSyncReverseFromTarget |
| | hadoop.tools.TestDistCpWithXAttrs |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1936 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9c7d9ef5c3f0 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 1695d8d |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/2/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/2/testReport/ |
| Max. process+thread count | 332 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932.
distcp copy calls getFileStatus() needlessly and can fail against S3
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-609722501
error is minidfs cluster OOM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1936: HADOOP-16932.
distcp copy calls getFileStatus() needlessly and can fail against S3
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-610398485
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 21s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 27s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 50s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed |
| +0 :ok: | spotbugs | 0m 44s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 41s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | the patch passed |
| +1 :green_heart: | compile | 0m 21s | the patch passed |
| +1 :green_heart: | javac | 0m 21s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 19s | hadoop-tools/hadoop-distcp: The patch generated 0 new + 159 unchanged - 12 fixed = 159 total (was 171) |
| +1 :green_heart: | mvnsite | 0m 24s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 0s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed |
| +1 :green_heart: | findbugs | 0m 45s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 11m 20s | hadoop-distcp in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 67m 14s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1936 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 050e7d2a70ac 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / c12ddbd |
| Default Java | 1.8.0_242 |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/4/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on issue #1936: HADOOP-16932.
distcp copy calls getFileStatus() needlessly and can fail against S3
Posted by GitBox <gi...@apache.org>.
mukund-thakur commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
URL: https://github.com/apache/hadoop/pull/1936#issuecomment-608287003
Changes looks good. There are UT's failing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org