[jira] [Commented] (MAPREDUCE-6917) DistCp does not clean staging folder if class extends DistCp

["lawrence andrews (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095249#comment-16095249 ] 

lawrence andrews commented on MAPREDUCE-6917:
---------------------------------------------

Following is the fix

{noformat}
git diff
diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
index 06e08e428c1..f69de6285c6 100644
--- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
+++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
@@ -487,12 +487,17 @@ private static Configuration getDefaultConf() {
     return config;
   }
 
-  private synchronized void cleanup() {
+  /**
+   * Clean the staging folder created by distcp
+   */
+  public void cleanup() {
     try {
       if (metaFolder == null) return;
 
-      jobFS.delete(metaFolder, true);
-      metaFolder = null;
+      synchronized (this) {
+        jobFS.delete(metaFolder, true);
+        metaFolder = null;
+      }
     } catch (IOException e) {
       LOG.error("Unable to cleanup meta folder: " + metaFolder, e);
     }
{noformat}

> DistCp does not clean staging folder if class extends DistCp
> ------------------------------------------------------------
>
>                 Key: MAPREDUCE-6917
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6917
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: distcp
>            Reporter: lawrence andrews
>            Priority: Minor
>
> My code extends Distcp class and for some reason if distcp fails staging folder is not delete and this staging folder piles up occupying space on hdfs.
> After checking the code i found that cleanup() function is private. Making the cleanup() method as public, user should be able to invoke the cleanup if job fails.
> This works fine with command line argument. But fails only if we extend Distcp class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org