You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org> on 2012/04/02 18:09:24 UTC
[jira] [Updated] (MAPREDUCE-4072) User set java.library.path seems
to overwrite default creating problems native lib loading
[ https://issues.apache.org/jira/browse/MAPREDUCE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4072:
-------------------------------------------
Status: Open (was: Patch Available)
I don't think that this is the correct place to make these changes. You are removing using java.library.path from the localizer command line. This is the part of the code that will download a file from HDFS to the local file system as a particular user. Is this the process that was causing the error? I don't see how because it has nothing to do with mapred.child.java.opts. mapred.child.java.opts is something that the MRAM will use when produing the container launch context that it sends to the NM.
So we either need to update the AM strip out java.library.path from mapred.child.java.opts, and add it to LD_LIBRARY_PATH, or just put in a documentation change. In the mapred-default.xml under mapred.child.java.opts say that the usage of -Djava.library.path can cause your programs to no longer function and that these values should be set as part of LD_LIBRARY_PATH.
In either case we probably also want to update hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm to have something in it about how setting -Djava.library.path on the command line while launching a container can cause native libraries used by hadoop to not be loaded correctly and can result in errors. It is better to just use LD_LIBRARY_PATH.
> User set java.library.path seems to overwrite default creating problems native lib loading
> ------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4072
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Attachments: MAPREDUCE-4072-branch-23.patch
>
>
> This was found by Peeyush Bishnoi.
> While running a distributed cache example with Hadoop-0.23,
> tasks are failing as follows:
> ------------------------------------------------------------------------------------------------------------
> Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException: at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at
> org.apache.hadoop.util.Shell.run(Shell.java:188) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at
> java.util.concurrent.FutureTask.run(FutureTask.java:138) at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user
> is <user>
> ------------------------------------------------------------------------------------------------------------
> Same Pig script and command work successfully on 0.20
> See this in the stderr:
> Exception in thread "main" java.lang.ExceptionInInitializerError
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264)
> at org.apache.hadoop.security.Groups.(Groups.java:54)
> at
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178)
> at
> org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223)
> at
> org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75)
> Caused by: java.lang.RuntimeException: Bailing out since native library
> couldn't be loaded
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48)
> ... 12 more
> Pig command:
> $ pig -Dmapred.job.queue.name=<queue> -Dmapred.cache.archives=<archives> -Dmapred.child.java.opts="-Djava.library.path=./ygeo/lib
> -Dip2geo.preLoadLibraries=<some other libs>" -Djava.io.tmpdir=/grid/0/tmp -Dmapred.create.symlink=yes -Dmapred.job.map.memory.mb=3072 piggeoscript.pig
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira