You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/03/23 21:20:18 UTC
[GitHub] [hadoop] ayushtkn opened a new pull request #2808: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).
ayushtkn opened a new pull request #2808:
URL: https://github.com/apache/hadoop/pull/2808
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2808: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2808:
URL: https://github.com/apache/hadoop/pull/2808#issuecomment-808184868
@ayushtkn have you got a PR for trunk for the change @aajisaka asked for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2808: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2808:
URL: https://github.com/apache/hadoop/pull/2808#issuecomment-808607806
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 24m 59s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 9 new or modified test files. |
|||| _ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 13m 43s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 8s | | branch-3.3 passed |
| +1 :green_heart: | compile | 18m 14s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 2m 49s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 2m 52s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 2m 39s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 4m 21s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 18m 12s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 53s | | the patch passed |
| +1 :green_heart: | compile | 17m 28s | | the patch passed |
| +1 :green_heart: | javac | 17m 28s | | root generated 0 new + 1948 unchanged - 1 fixed = 1948 total (was 1949) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 2m 45s | | root: The patch generated 0 new + 93 unchanged - 5 fixed = 93 total (was 98) |
| +1 :green_heart: | mvnsite | 2m 51s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 36s | | the patch passed |
| +1 :green_heart: | spotbugs | 5m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 35s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 16m 58s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 14m 11s | | hadoop-distcp in the patch passed. |
| +1 :green_heart: | unit | 1m 56s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not generate ASF License warnings. |
| | | 198m 41s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2808/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2808 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux f54de7037819 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 782ed0cbb11bb21aa37931411cd386117381a1dd |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~18.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2808/2/testReport/ |
| Max. process+thread count | 3143 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2808/2/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2808: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2808:
URL: https://github.com/apache/hadoop/pull/2808#issuecomment-805383059
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 27m 1s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 9 new or modified test files. |
|||| _ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 13m 31s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 24s | | branch-3.3 passed |
| +1 :green_heart: | compile | 18m 23s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 2m 47s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 2m 49s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 2m 38s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 4m 21s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 18m 13s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 52s | | the patch passed |
| +1 :green_heart: | compile | 17m 36s | | the patch passed |
| +1 :green_heart: | javac | 17m 36s | | root generated 0 new + 1871 unchanged - 1 fixed = 1871 total (was 1872) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 2m 45s | | root: The patch generated 0 new + 93 unchanged - 5 fixed = 93 total (was 98) |
| +1 :green_heart: | mvnsite | 2m 51s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | javadoc | 2m 40s | | the patch passed |
| +1 :green_heart: | spotbugs | 4m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 22s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 16m 53s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 14m 53s | | hadoop-distcp in the patch passed. |
| +1 :green_heart: | unit | 1m 59s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not generate ASF License warnings. |
| | | 201m 15s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2808/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2808 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux adbcf2c417ee 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / c4a28680aa75129a67c281e77fbd199dc55a6701 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~18.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2808/1/testReport/ |
| Max. process+thread count | 3143 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-distcp hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2808/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn merged pull request #2808: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).
Posted by GitBox <gi...@apache.org>.
ayushtkn merged pull request #2808:
URL: https://github.com/apache/hadoop/pull/2808
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #2808: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on pull request #2808:
URL: https://github.com/apache/hadoop/pull/2808#issuecomment-808642850
Merged both the main and the addendum commit as part of this PR, Thanx Everyone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #2808: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #2808:
URL: https://github.com/apache/hadoop/pull/2808#discussion_r600212994
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java
##########
@@ -896,5 +916,132 @@ public static int getTestsThreadCount() {
}
return threadCount;
}
+ /**
+ * Write the text to a file asynchronously. Logs the operation duration.
+ * @param fs filesystem
+ * @param path path
+ * @return future to the patch created.
+ */
+ private static CompletableFuture<Path> put(FileSystem fs,
+ Path path, String text) {
+ return submit(EXECUTOR, () -> {
+ try (DurationInfo ignore =
+ new DurationInfo(LOG, false, "Creating %s", path)) {
+ createFile(fs, path, true, text.getBytes(Charsets.UTF_8));
Review comment:
Hey @aajisaka
This is actually a backport of https://github.com/apache/hadoop/pull/2732
Will it be ok, if I merge this as is and raise an Addendum PR for trunk as well?
I pulled it up from `ITestPartialRenamesDeletes` as is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a change in pull request #2808: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #2808:
URL: https://github.com/apache/hadoop/pull/2808#discussion_r600212994
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java
##########
@@ -896,5 +916,132 @@ public static int getTestsThreadCount() {
}
return threadCount;
}
+ /**
+ * Write the text to a file asynchronously. Logs the operation duration.
+ * @param fs filesystem
+ * @param path path
+ * @return future to the patch created.
+ */
+ private static CompletableFuture<Path> put(FileSystem fs,
+ Path path, String text) {
+ return submit(EXECUTOR, () -> {
+ try (DurationInfo ignore =
+ new DurationInfo(LOG, false, "Creating %s", path)) {
+ createFile(fs, path, true, text.getBytes(Charsets.UTF_8));
Review comment:
Hey @aajisaka
This is actually a backport of https://github.com/apache/hadoop/pull/2732
Will it be ok, if I merge this as is and raise a Addendum PR for trunk as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #2808: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on pull request #2808:
URL: https://github.com/apache/hadoop/pull/2808#issuecomment-806913688
@aajisaka does that make sense? Let me know your thoughts, Plan to conclude this by tomorrow EOD.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #2808: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on pull request #2808:
URL: https://github.com/apache/hadoop/pull/2808#issuecomment-805537866
Ran the S3A and optional HDFS tests:
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.tools.contract.OptionalTestHDFSContractDistCp
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 234.777 s - in org.apache.hadoop.tools.contract.OptionalTestHDFSContractDistCp
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 767.429 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0
AWS Region: ap-south-1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2808: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2808:
URL: https://github.com/apache/hadoop/pull/2808#issuecomment-805821254
+1 on the patch from me, but if @aajisaka has some feedback then let's address that as a followup on trunk before the backport.
I think a followup-patch with the same JIRA ID would be enough for something that minor, and in -3.3 we'd just combine the pair
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2808: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2808:
URL: https://github.com/apache/hadoop/pull/2808#issuecomment-808751684
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #2808: HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).
Posted by GitBox <gi...@apache.org>.
aajisaka commented on a change in pull request #2808:
URL: https://github.com/apache/hadoop/pull/2808#discussion_r600209565
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java
##########
@@ -896,5 +916,132 @@ public static int getTestsThreadCount() {
}
return threadCount;
}
+ /**
+ * Write the text to a file asynchronously. Logs the operation duration.
+ * @param fs filesystem
+ * @param path path
+ * @return future to the patch created.
+ */
+ private static CompletableFuture<Path> put(FileSystem fs,
+ Path path, String text) {
+ return submit(EXECUTOR, () -> {
+ try (DurationInfo ignore =
+ new DurationInfo(LOG, false, "Creating %s", path)) {
+ createFile(fs, path, true, text.getBytes(Charsets.UTF_8));
Review comment:
java.nio.charset.StandardCharsets.UTF_8 can be used instead of shaded guava.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org