You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Mostafa Elhemali (JIRA)" <ji...@apache.org> on 2013/06/24 23:34:21 UTC

[jira] [Commented] (YARN-729) MR uses LD_LIBRARY_PATH which doesn't mean anything in Windows

    [ https://issues.apache.org/jira/browse/YARN-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13692409#comment-13692409 ] 

Mostafa Elhemali commented on YARN-729:
---------------------------------------

Chuan: PATH and LD_LIBRARY_PATH can also be used for JNI libraries - it's not very well documented so I can't find a link off a casual search, but you can test this behavior yourself. Typically yes, we do use java.library.path, but in MR for some reason we rely on LD_LIBRARY_PATH not java.library.path, so it doesn't work (see MRJobConfig.java, DEFAULT_MAPRED_ADMIN_USER_ENV). I suspect you don't see this error because you just happen to have the native hadoop.dll in one of the directories in your PATH already, but if you don't have that then you'll run into this problem. Manually setting mapreduce.admin.user.env works around this problem in Windows, but we should fix that so by default it runs properly on Windows.
                
> MR uses LD_LIBRARY_PATH which doesn't mean anything in Windows
> --------------------------------------------------------------
>
>                 Key: YARN-729
>                 URL: https://issues.apache.org/jira/browse/YARN-729
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>         Environment: Windows
>            Reporter: Mostafa Elhemali
>
> In order to set the path for loading native libraries, MR relies on the default value of the mapreduce.admin.user.env configuration setting the LD_LIBRARY_PATH environment entry. There are two problems with this setting in Windows:
> a) LD_LIBRARY_PATH doesn't mean anything in Windows.
> b) It sets it using $HADOOP_COMMON_HOME, instead of %HADOOP_COMMON_HOME%.
> The default value here should be platform-dependent (use the PATH variable in Windows instead of LD_LIBRARY_PATH), or we should rely on another mechanism. The net effect is that in Windows unless this configuration is over-ridden MR jobs fail with this error:
> {code}
> 2013-05-29 13:51:41,049 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
> 	at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
> 	at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:393)
> 	at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:928)
> 	at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:177)
> 	at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:164)
> 	at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:98)
> 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:288)
> 	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:431)
> 	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
> 	at org.apache.hadoop.mapred.YarnChild.configureLocalDirs(YarnChild.java:235)
> 	at org.apache.hadoop.mapred.YarnChild.configureTask(YarnChild.java:294)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira