You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Jason Lowe (JIRA)" <ji...@apache.org> on 2015/12/18 16:29:46 UTC
[jira] [Comment Edited] (MAPREDUCE-6577) MR AM unable to load
native library without MR_AM_ADMIN_USER_ENV set
[ https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064085#comment-15064085 ]
Jason Lowe edited comment on MAPREDUCE-6577 at 12/18/15 3:29 PM:
-----------------------------------------------------------------
The environment variable reference should be like this:
{code}
public static final String DEFAULT_MR_AM_ADMIN_USER_ENV =
Shell.WINDOWS ?
"PATH={{PATH}};{{HADOOP_COMMON_HOME}}\\" + RELATIVE_LIB_PATH :
"LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/" + RELATIVE_LIB_PATH;
{code}
Otherwise it will be expanded by the job client's version of those environment variables rather than the nodemanager. The client could have a totally bogus PATH or HADOOP_COMMON_HOME setting relative to what the cluster nodes need. It works for tasks today because it's being setup by the AM, and the AM is inheriting the nodemanager's variables instead of the job client's.
Otherwise patch looks good. I'm not a Windows expert, but those changes look OK to me.
was (Author: jlowe):
The environment variable reference should be like this:
{code}
public static final String DEFAULT_MR_AM_ADMIN_USER_ENV =
Shell.WINDOWS ?
"PATH={{PATH}};{{HADOOP_COMMON_HOME}}\\" + RELATIVE_LIB_PATH :
"LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/" + RELATIVE_LIB_PATH;
{code}
Otherwise it will be expanded by the job client's version of those environment variables rather than the nodemanager. The client could have a totally bogus PATH or HADOOP_COMMON_HOME setting relative to what the cluster nodes need. It works for tasks today because it's being setup by the AM, and the AM is inheriting the nodemanager's variables instead of the job client's.
Otherwise patch looks good. I'm not a Windows expert, but those changes look OK to me.
Otherwise patch looks good
> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
> Affects Versions: 2.6.0
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will fail. For example, an uber-AM with lz4 compression for the mapper task will fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner] org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local (uberized) 'child' : java.lang.RuntimeException: native lz4 library not available
> at org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
> at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114)
> at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
> at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
> at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
> at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)