You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "NING DING (JIRA)" <ji...@apache.org> on 2015/09/11 10:19:46 UTC
[jira] [Commented] (MAPREDUCE-5799) add default value of
MR_AM_ADMIN_USER_ENV
[ https://issues.apache.org/jira/browse/MAPREDUCE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740384#comment-14740384 ]
NING DING commented on MAPREDUCE-5799:
--------------------------------------
The parameters yarn.app.mapreduce.am.env and yarn.app.mapreduce.am.admin.user.env are resolved in YARNRunner running on hadoop client host.
But their values, e.g. LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, will be used on hadoop cluster hosts.
If the hadoop client host installs hadoop on a different path from that on hadoop cluster hosts,
the $HADOOP_COMMON_HOME/lib/native will be resolved to a path which is not existed on hadoop cluster hosts.
Then MRAppMaster running on hadoop cluster host cannot load native lib.
To solve this issue, I set my hadoop client's mapred-site.xml with the following content.
{code}
<property>
<name>yarn.app.mapreduce.am.admin.user.env</name>
<value>LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native</value>
</property>
{code}
{noformat}
In this way, the YARNRunner will put LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native into environment.
The method ContainerLaunch.expandEnvironment running in NodeManager can translate LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native
to LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native.
The host running NodeManager can find its $HADOOP_COMMON_HOME/lib/native path.
{noformat}
I suggest MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV should be defined as below.
{code}
public final String DEFAULT_MAPRED_ADMIN_USER_ENV =
Shell.WINDOWS ?
"PATH={{PATH}};{{HADOOP_COMMON_HOME}}\\bin":
"LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native";
{code}
Any ideas?
> add default value of MR_AM_ADMIN_USER_ENV
> -----------------------------------------
>
> Key: MAPREDUCE-5799
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5799
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 2.3.0
> Reporter: Liyin Liang
> Assignee: Rajesh Kartha
> Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5799-1.diff, MAPREDUCE-5799.002.patch, MAPREDUCE-5799.diff
>
>
> Submit a 1 map + 1 reduce sleep job with the following config:
> {code}
> <property>
> <name>mapreduce.map.output.compress</name>
> <value>true</value>
> </property>
> <property>
> <name>mapreduce.map.output.compress.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> </property>
> <property>
> <name>mapreduce.job.ubertask.enable</name>
> <value>true</value>
> </property>
> {code}
> And the LinuxContainerExecutor is enable on NodeManager.
> This job will fail with the following error:
> {code}
> 2014-03-18 21:28:20,153 FATAL [uber-SubtaskRunner] org.apache.hadoop.mapred.LocalContainerLauncher: Error running local (uberized) 'child' : java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
> at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
> at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
> at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:115)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1583)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1462)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:700)
> at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1990)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:774)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.runSubtask(LocalContainerLauncher.java:317)
> at org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.run(LocalContainerLauncher.java:232)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> When create a ContainerLaunchContext for task in TaskAttemptImpl.createCommonContainerLaunchContext(), the DEFAULT_MAPRED_ADMIN_USER_ENV which is "LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native" is added to the environment. Where when create a ContainerLaunchContext for mrappmaster in YARNRunner.createApplicationSubmissionContext(), there is no default environment. So the ubermode job fails to find native lib.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)