[jira] Commented: (MAPREDUCE-1213) TaskTrackers restart is very slow because ti deletes distributed cache directory synchronously

["dhruba borthakur (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777470#action_12777470 ] 

dhruba borthakur commented on MAPREDUCE-1213:
---------------------------------------------

One option is to rename the currentworkdir to a temporary location and then delete these file asynchronously via the technique used in HDFS-611

> TaskTrackers restart is very slow because ti deletes distributed cache directory synchronously
> ----------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1213
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1213
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: dhruba borthakur
>
> We are seeing that when we restart a tasktracker, it tries to recursively delete all the file in the distributed cache. It invoked FileUtil.fullyDelete() which is very very slow. This means that the TaskTracker cannot join the cluster for an extended period of time (upto 2 hours for us). The problem is acute if the number of files in a distributed cache is a few-thousands.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.