You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "NING DING (JIRA)" <ji...@apache.org> on 2015/12/27 02:30:49 UTC
[jira] [Updated] (MAPREDUCE-6590) Update
MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV value for loading correct navite
lib when using ubertask feature
[ https://issues.apache.org/jira/browse/MAPREDUCE-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
NING DING updated MAPREDUCE-6590:
---------------------------------
Attachment: MAPREDUCE-6590.patch
> Update MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV value for loading correct navite lib when using ubertask feature
> -----------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6590
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6590
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
> Affects Versions: 2.7.1
> Reporter: NING DING
> Assignee: NING DING
> Attachments: MAPREDUCE-6590.patch
>
>
> By default, the MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library, then you can find the error message as below.
> {panel}
> org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable.
> {panel}
> I found the patch in https://issues.apache.org/jira/browse/MAPREDUCE-5799 update YARNRunner code and use MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV as the default value. The code is as below.
> {code:title=YARNRunner.java|borderStyle=solid}
> MRApps.setEnvFromInputString(environment,
> conf.get(MRJobConfig.MR_AM_ADMIN_USER_ENV,
> MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV), conf);
> {code}
> In fact the parameters yarn.app.mapreduce.am.env and yarn.app.mapreduce.am.admin.user.env are resolved in YARNRunner running on hadoop client host.
> But their values, e.g. LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, will be used on hadoop cluster hosts.
> If the hadoop client host installs hadoop on a different path from that on hadoop cluster hosts,
> the $HADOOP_COMMON_HOME/lib/native will be resolved to a path which is not existed on hadoop cluster hosts.
> Then MRAppMaster running on hadoop cluster host cannot load native lib.
> If don't use MAPREDUCE-5799 patch, I set my hadoop client's mapred-site.xml with the following content.
> {code:xml}
> <property>
> <name>yarn.app.mapreduce.am.admin.user.env</name>
> <value>LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native</value>
> </property>
> {code}
> In this way, the YARNRunner will put LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native into environment.
> The method ContainerLaunch.expandEnvironment running in NodeManager can translate LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native
> to LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native.
> The host running NodeManager can find its $HADOOP_COMMON_HOME/lib/native path.
> I suggest MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV should be defined as below.
> {code:title=MRJobConfig.java|borderStyle=solid}
> public final String DEFAULT_MAPRED_ADMIN_USER_ENV =
> Shell.WINDOWS ?
> "PATH={{PATH}};{{HADOOP_COMMON_HOME}}\\bin":
> "LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native";
> {code}
> Please see my patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)