You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by st...@apache.org on 2020/01/02 15:38:15 UTC
[hadoop] branch branch-3.2 updated: HADOOP-16775. DistCp reuses the
same temp file within the task for different files.
This is an automated email from the ASF dual-hosted git repository.
stevel pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 5410732 HADOOP-16775. DistCp reuses the same temp file within the task for different files.
5410732 is described below
commit 5410732cffebf728405e2d494f238bcf033d7698
Author: Steve Loughran <st...@cloudera.com>
AuthorDate: Thu Jan 2 15:36:25 2020 +0000
HADOOP-16775. DistCp reuses the same temp file within the task for different files.
Contributed by Amir Shenavandeh.
This avoids overwrite consistency issues with S3 and other stores -though
given S3's copy operation is O(data), you are still best of using -direct
when distcp-ing to it.
Change-Id: I8dc9f048ad0cc57ff01543b849da1ce4eaadf8c3
---
.../java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
index fa91930..52834b9 100644
--- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
+++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
@@ -229,7 +229,8 @@ public class RetriableFileCopyCommand extends RetriableCommand {
Path root = target.equals(targetWorkPath) ? targetWorkPath.getParent()
: targetWorkPath;
Path tempFile = new Path(root, ".distcp.tmp." +
- context.getTaskAttemptID().toString());
+ context.getTaskAttemptID().toString() +
+ "." + String.valueOf(System.currentTimeMillis()));
LOG.info("Creating temp file: {}", tempFile);
return tempFile;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org