You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/01/06 11:10:45 UTC
[GitHub] [hadoop] taojieterry opened a new pull request #1794: HADOOP-15887:
Add an option to avoid writing data locally in Distcp
taojieterry opened a new pull request #1794: HADOOP-15887: Add an option to avoid writing data locally in Distcp
URL: https://github.com/apache/hadoop/pull/1794
In https://issues.apache.org/jira/browse/HDFS-3702 we add a flag in DFSClient to avoid replicating to the local datanode. We can make use of this flag in Distcp.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1794:
HADOOP-15887: Add an option to avoid writing data locally in Distcp
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1794: HADOOP-15887: Add an option to avoid writing data locally in Distcp
URL: https://github.com/apache/hadoop/pull/1794#discussion_r363379090
##########
File path: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java
##########
@@ -650,4 +650,22 @@ private Job runDistCpDirectWrite(final Path srcDir, final Path destDir)
Collections.singletonList(srcDir), destDir)
.withDirectWrite(true)));
}
+
+ @Test
+ public void testDistCpWithNoLocalWrite() throws Exception {
+ describe("test dictcp job compatibility with option: noLocalWrite");
+ Path target = distCpDeepDirectoryStructure(localFS, localDir, remoteFS,
+ remoteDir);
+ lsR("Local to update", localFS, localDir);
+ lsR("Remote before update", remoteFS, target);
+ Job job = runDistCp(buildWithStandardOptions(
+ new DistCpOptions.Builder(
+ Collections.singletonList(localDir), target)
+ .withDeleteMissing(true)
+ .withSyncFolder(true)
+ .withCRC(true)
+ .withOverwrite(false)
+ .withNoLocalWrite(true)));
+ assertTrue(job.isSuccessful());
Review comment:
Add a string to include in the exception text, e.g "job failed"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1794:
HADOOP-15887: Add an option to avoid writing data locally in Distcp
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1794: HADOOP-15887: Add an option to avoid writing data locally in Distcp
URL: https://github.com/apache/hadoop/pull/1794#discussion_r363376799
##########
File path: hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
##########
@@ -197,14 +200,25 @@ private long copyToFile(Path targetPath, FileSystem targetFS,
targetFS, targetPath);
final long blockSize = getBlockSize(fileAttributes, source,
targetFS, targetPath);
+ EnumSet<CreateFlag> createFlags =
+ EnumSet.of(CreateFlag.CREATE, CreateFlag.OVERWRITE);
+ if (noLocalWrite) {
+ createFlags.add(CreateFlag.NO_LOCAL_WRITE);
+ }
FSDataOutputStream out = targetFS.create(targetPath, permission,
- EnumSet.of(CreateFlag.CREATE, CreateFlag.OVERWRITE),
- copyBufferSize, repl, blockSize, context,
+ createFlags, copyBufferSize, repl, blockSize, context,
getChecksumOpt(fileAttributes, sourceChecksum));
outStream = new BufferedOutputStream(out);
} else {
- outStream = new BufferedOutputStream(targetFS.append(targetPath,
- copyBufferSize));
+ if (targetFS instanceof DistributedFileSystem && noLocalWrite) {
+ outStream = new BufferedOutputStream(
+ ((DistributedFileSystem) targetFS).append(targetPath,
Review comment:
you need to do an explicit cast here? Ugly. And it will mean we need hadoop-hdfs on the classpath, always, which object stores may not have. We'd be better off (carefully) pulling that HDFS method up
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1794:
HADOOP-15887: Add an option to avoid writing data locally in Distcp
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1794: HADOOP-15887: Add an option to avoid writing data locally in Distcp
URL: https://github.com/apache/hadoop/pull/1794#discussion_r363378165
##########
File path: hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm
##########
@@ -362,6 +362,7 @@ Command Line Options
| `-copybuffersize <copybuffersize>` | Size of the copy buffer to use. By default, `<copybuffersize>` is set to 8192B | |
| `-xtrack <path>` | Save information about missing source files to the specified path. | This option is only valid with `-update` option. This is an experimental property and it cannot be used with `-atomic` option. |
| `-direct` | Write directly to destination paths | Useful for avoiding potentially very expensive temporary file rename operations when the destination is an object store |
+| `-noLocalWrite` | Write data to target cluster with data locality disabled. | If this option is set, the distcp task will not write data replication to local datanode to avoid datanode being imbalanced. This option is suggested to be specified when the data to copy is very large and the DistCp job runs on the target cluster. |
Review comment:
suggest:
Write data to an HDFS cluster with data locality disabled. | If this option is set, the distcp tasks will not write data blocks to their local datanodes, so avoiding datanodes becoming imbalanced. Recommended when the amount of data to copy is very large, the target cluster is HDFS and the DistCp job runs on that target cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1794: HADOOP-15887: Add
an option to avoid writing data locally in Distcp
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1794: HADOOP-15887: Add an option to avoid writing data locally in Distcp
URL: https://github.com/apache/hadoop/pull/1794#issuecomment-571893263
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 10s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 24m 44s | trunk passed |
| +1 :green_heart: | compile | 0m 33s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 28s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 7s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed |
| +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | the patch passed |
| +1 :green_heart: | compile | 0m 22s | the patch passed |
| +1 :green_heart: | javac | 0m 22s | the patch passed |
| -0 :warning: | checkstyle | 0m 18s | hadoop-tools/hadoop-distcp: The patch generated 1 new + 113 unchanged - 2 fixed = 114 total (was 115) |
| +1 :green_heart: | mvnsite | 0m 24s | the patch passed |
| +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 28s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | findbugs | 0m 51s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 13m 34s | hadoop-distcp in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 81m 59s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1794 |
| JIRA Issue | HADOOP-15887 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4a750e828a67 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / a43c177 |
| Default Java | 1.8.0_232 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/2/testReport/ |
| Max. process+thread count | 449 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1794:
HADOOP-15887: Add an option to avoid writing data locally in Distcp
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1794: HADOOP-15887: Add an option to avoid writing data locally in Distcp
URL: https://github.com/apache/hadoop/pull/1794#discussion_r363378804
##########
File path: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
##########
@@ -804,4 +804,18 @@ public void testExclusionsOption() {
"hdfs://localhost:8020/target/"});
assertThat(options.getFiltersFile()).isEqualTo("/tmp/filters.txt");
}
+
+ @Test
+ public void testParseNoLocalWrite() {
+ DistCpOptions options = OptionsParser.parse(new String[] {
+ "hdfs://localhost:8020/source/first",
+ "hdfs://localhost:8020/target/"});
+ Assert.assertEquals(options.shouldNoLocalWrite(), false);
+
+ options = OptionsParser.parse(new String[] {
+ "-noLocalWrite",
+ "hdfs://localhost:8020/source/first",
+ "hdfs://localhost:8020/target/"});
+ Assert.assertEquals(options.shouldNoLocalWrite(), true);
Review comment:
use assertTrue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1794:
HADOOP-15887: Add an option to avoid writing data locally in Distcp
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1794: HADOOP-15887: Add an option to avoid writing data locally in Distcp
URL: https://github.com/apache/hadoop/pull/1794#discussion_r363378426
##########
File path: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java
##########
@@ -511,7 +518,27 @@ public void testCleanup() {
Assert.fail("testCleanup failed " + e.getMessage());
}
}
-
+
+ @Test(timeout=100000)
+ public void testNoLocalWrite() {
+ try {
+ addEntries(listFile, "singlefile1/file1");
+ createFiles("singlefile1/file1", "target");
+
+ Configuration conf = new Configuration(getConf());
+ conf.set("fs.file.impl", MockFileSystem.class.getName());
+ conf.setBoolean("fs.file.impl.disable.cache", true);
+ runTest(listFile, target, false, false, false, false, true, conf);
+
+ checkResult(target, 1);
+ } catch (IOException e) {
Review comment:
remove the catch, just have the test method throw IOException
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1794: HADOOP-15887: Add
an option to avoid writing data locally in Distcp
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1794: HADOOP-15887: Add an option to avoid writing data locally in Distcp
URL: https://github.com/apache/hadoop/pull/1794#issuecomment-571126994
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 30m 21s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 18m 9s | trunk passed |
| +1 :green_heart: | compile | 0m 30s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 25s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | trunk passed |
| +1 :green_heart: | shadedclient | 13m 33s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed |
| +0 :ok: | spotbugs | 0m 43s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 41s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | the patch passed |
| +1 :green_heart: | compile | 0m 21s | the patch passed |
| +1 :green_heart: | javac | 0m 21s | the patch passed |
| -0 :warning: | checkstyle | 0m 19s | hadoop-tools/hadoop-distcp: The patch generated 1 new + 113 unchanged - 2 fixed = 114 total (was 115) |
| +1 :green_heart: | mvnsite | 0m 24s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 8s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| +1 :green_heart: | findbugs | 0m 45s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 12m 51s | hadoop-distcp in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 95m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1794 |
| JIRA Issue | HADOOP-15887 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7b19cd995ff9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 4a76ab7 |
| Default Java | 1.8.0_232 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/1/testReport/ |
| Max. process+thread count | 411 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1794:
HADOOP-15887: Add an option to avoid writing data locally in Distcp
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1794: HADOOP-15887: Add an option to avoid writing data locally in Distcp
URL: https://github.com/apache/hadoop/pull/1794#issuecomment-571126994
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 30m 21s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 18m 9s | trunk passed |
| +1 :green_heart: | compile | 0m 30s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 25s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | trunk passed |
| +1 :green_heart: | shadedclient | 13m 33s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed |
| +0 :ok: | spotbugs | 0m 43s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 41s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | the patch passed |
| +1 :green_heart: | compile | 0m 21s | the patch passed |
| +1 :green_heart: | javac | 0m 21s | the patch passed |
| -0 :warning: | checkstyle | 0m 19s | hadoop-tools/hadoop-distcp: The patch generated 1 new + 113 unchanged - 2 fixed = 114 total (was 115) |
| +1 :green_heart: | mvnsite | 0m 24s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 8s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| +1 :green_heart: | findbugs | 0m 45s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 12m 51s | hadoop-distcp in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 95m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1794 |
| JIRA Issue | HADOOP-15887 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7b19cd995ff9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 4a76ab7 |
| Default Java | 1.8.0_232 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/1/testReport/ |
| Max. process+thread count | 411 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1794: HADOOP-15887: Add
an option to avoid writing data locally in Distcp
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1794: HADOOP-15887: Add an option to avoid writing data locally in Distcp
URL: https://github.com/apache/hadoop/pull/1794#issuecomment-574178727
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 25m 9s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 21s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 25s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 29s | trunk passed |
| +1 :green_heart: | shadedclient | 12m 59s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 0m 43s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 42s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 21s | the patch passed |
| +1 :green_heart: | javac | 0m 21s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-distcp: The patch generated 1 new + 112 unchanged - 2 fixed = 113 total (was 114) |
| +1 :green_heart: | mvnsite | 0m 23s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 52s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | findbugs | 0m 45s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 13m 13s | hadoop-distcp in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 91m 42s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1794 |
| JIRA Issue | HADOOP-15887 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9b3a5a022781 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 1c51f36 |
| Default Java | 1.8.0_232 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/3/testReport/ |
| Max. process+thread count | 421 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1794: HADOOP-15887: Add
an option to avoid writing data locally in Distcp
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1794: HADOOP-15887: Add an option to avoid writing data locally in Distcp
URL: https://github.com/apache/hadoop/pull/1794#issuecomment-572518897
thanks
* could those assertTrue calls have a string saying what failed? I hate having to debug a test failure without those (as I am doing right now...)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1794:
HADOOP-15887: Add an option to avoid writing data locally in Distcp
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1794: HADOOP-15887: Add an option to avoid writing data locally in Distcp
URL: https://github.com/apache/hadoop/pull/1794#issuecomment-571893263
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 2m 10s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 24m 44s | trunk passed |
| +1 :green_heart: | compile | 0m 33s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 28s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 7s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed |
| +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | the patch passed |
| +1 :green_heart: | compile | 0m 22s | the patch passed |
| +1 :green_heart: | javac | 0m 22s | the patch passed |
| -0 :warning: | checkstyle | 0m 18s | hadoop-tools/hadoop-distcp: The patch generated 1 new + 113 unchanged - 2 fixed = 114 total (was 115) |
| +1 :green_heart: | mvnsite | 0m 24s | the patch passed |
| +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 28s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| +1 :green_heart: | findbugs | 0m 51s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 13m 34s | hadoop-distcp in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 81m 59s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1794 |
| JIRA Issue | HADOOP-15887 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4a750e828a67 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / a43c177 |
| Default Java | 1.8.0_232 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/2/testReport/ |
| Max. process+thread count | 449 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1794/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1794:
HADOOP-15887: Add an option to avoid writing data locally in Distcp
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1794: HADOOP-15887: Add an option to avoid writing data locally in Distcp
URL: https://github.com/apache/hadoop/pull/1794#discussion_r363378663
##########
File path: hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
##########
@@ -804,4 +804,18 @@ public void testExclusionsOption() {
"hdfs://localhost:8020/target/"});
assertThat(options.getFiltersFile()).isEqualTo("/tmp/filters.txt");
}
+
+ @Test
+ public void testParseNoLocalWrite() {
+ DistCpOptions options = OptionsParser.parse(new String[] {
+ "hdfs://localhost:8020/source/first",
+ "hdfs://localhost:8020/target/"});
+ Assert.assertEquals(options.shouldNoLocalWrite(), false);
Review comment:
use assertFalse
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org