[jira] [Created] (MAPREDUCE-5661) ShuffleHandler using yarn.nodemanager.local-dirs instead of mapreduce.cluster.local.dir

["Eric Sirianni (JIRA)" <ji...@apache.org>]

Eric Sirianni created MAPREDUCE-5661:
----------------------------------------

             Summary: ShuffleHandler using yarn.nodemanager.local-dirs instead of mapreduce.cluster.local.dir
                 Key: MAPREDUCE-5661
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5661
             Project: Hadoop Map/Reduce
          Issue Type: Bug
    Affects Versions: 2.2.0
            Reporter: Eric Sirianni
            Priority: Trivial


While debugging an issue where a MapReduce job is failing due to running out of disk space, I noticed that the {{ShuffleHandler}} uses {{yarn.nodemanager.local-dirs}} for its {{LocalDirAllocator}} whereas all of the other MapReduce classes use {{mapreduce.cluster.local.dir}}:

{noformat}
$ find hadoop-mapreduce-project/hadoop-mapreduce-client/*/src/main/java/ -name "*.java" | xargs grep "new LocalDirAllocator("
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java:    LocalDirAllocator lDirAlloc = new LocalDirAllocator(MRConfig.LOCAL_DIR);
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnOutputFiles.java:    new LocalDirAllocator(MRConfig.LOCAL_DIR);
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java:      new LocalDirAllocator(MRConfig.LOCAL_DIR);
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/BackupStore.java:      this.lDirAlloc = new LocalDirAllocator(MRConfig.LOCAL_DIR);
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MROutputFiles.java:    new LocalDirAllocator(MRConfig.LOCAL_DIR);
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java:    new LocalDirAllocator(MRConfig.LOCAL_DIR);
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java:    this.lDirAlloc = new LocalDirAllocator(MRConfig.LOCAL_DIR);

*****hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java:      new LocalDirAllocator(YarnConfiguration.NM_LOCAL_DIRS);
{noformat}

This inconsistency feels like something that is likely to confuse admins.  



--
This message was sent by Atlassian JIRA
(v6.1#6144)