[jira] [Commented] (YARN-1025) NodeManager does not propagate java.library.path to launched child containers on Windows

["Chris Nauroth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13728957#comment-13728957 ] 

Chris Nauroth commented on YARN-1025:
-------------------------------------

As a workaround, you can set the PATH environment variable to point to the directory containing hadoop.dll before launching NodeManager.  The environment variable will propagate to child container processes, so this works.

I have a question on how this is expected to work on Linux.  I see code in the {{yarn}} shell script and {{LinuxContainerExecutor}} that propagates java.library.path to children, but I also see checks in {{YARNRunner#warnForJavaLibPath}} that warn if a caller is attempting to control java.library.path and state that the caller should set LD_LIBRARY_PATH instead.  What is the expected way for this to work on Linux?

If the expectation is for the caller to set LD_LIBRARY_PATH, then this issue is likely a Won't Fix.  Setting PATH on Windows is roughly equivalent to setting LD_LIBRARY_PATH on Linux.  I'm already using this technique successfully as a workaround.

If the expectation is that the shell scripts set up java.library.path, and NodeManager propagates that to its children, then we do have a bug on Windows and need to make some code changes.  The first part is changing {{yarn.cmd}} to set java.library.path.  That's a small easy change that gets NodeManager to load hadoop.dll.  To cover the child processes, we'll need additional code changes, probably in {{DefaultContainerExecutor}}.
                
> NodeManager does not propagate java.library.path to launched child containers on Windows
> ----------------------------------------------------------------------------------------
>
>                 Key: YARN-1025
>                 URL: https://issues.apache.org/jira/browse/YARN-1025
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 3.0.0, 2.1.1-beta
>            Reporter: Chris Nauroth
>
> Neither the NodeManager process itself nor the child container processes that it launches have the correct setting for java.library.path on Windows.  This prevents the processes from loading native code from hadoop.dll.  The native code is required for correct functioning on Windows (not optional), so this ultimately can cause failures in MapReduce jobs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira