You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by "Jason Lowe (JIRA)" <ji...@apache.org> on 2013/12/05 22:32:37 UTC
[jira] [Resolved] (MAPREDUCE-5661) ShuffleHandler using
yarn.nodemanager.local-dirs instead of mapreduce.cluster.local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe resolved MAPREDUCE-5661.
-----------------------------------
Resolution: Not A Problem
Closing as shufflehandler should continue to use the yarn property since it's part of the nodemanager process.
> ShuffleHandler using yarn.nodemanager.local-dirs instead of mapreduce.cluster.local.dir
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5661
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5661
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.2.0
> Reporter: Eric Sirianni
> Priority: Trivial
>
> While debugging an issue where a MapReduce job is failing due to running out of disk space, I noticed that the {{ShuffleHandler}} uses {{yarn.nodemanager.local-dirs}} for its {{LocalDirAllocator}} whereas all of the other MapReduce classes use {{mapreduce.cluster.local.dir}}:
> {noformat}
> $ find hadoop-mapreduce-project/hadoop-mapreduce-client/*/src/main/java/ -name "*.java" | xargs grep "new LocalDirAllocator("
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java: LocalDirAllocator lDirAlloc = new LocalDirAllocator(MRConfig.LOCAL_DIR);
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnOutputFiles.java: new LocalDirAllocator(MRConfig.LOCAL_DIR);
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java: new LocalDirAllocator(MRConfig.LOCAL_DIR);
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/BackupStore.java: this.lDirAlloc = new LocalDirAllocator(MRConfig.LOCAL_DIR);
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MROutputFiles.java: new LocalDirAllocator(MRConfig.LOCAL_DIR);
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java: new LocalDirAllocator(MRConfig.LOCAL_DIR);
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java: this.lDirAlloc = new LocalDirAllocator(MRConfig.LOCAL_DIR);
> *****hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java: new LocalDirAllocator(YarnConfiguration.NM_LOCAL_DIRS);
> {noformat}
> This inconsistency feels like something that is likely to confuse admins.
--
This message was sent by Atlassian JIRA
(v6.1#6144)