You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ramya Sunil (JIRA)" <ji...@apache.org> on 2011/08/10 20:48:27 UTC
[jira] [Created] (MAPREDUCE-2801) Include the native libs in
java.library.path
Include the native libs in java.library.path
---------------------------------------------
Key: MAPREDUCE-2801
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Fix For: 0.23.0
For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103085#comment-13103085 ]
Arun C Murthy commented on MAPREDUCE-2801:
------------------------------------------
Makes sense, I'll open a separate jira.
Also, we shud ensure "." is added to both java.library.path and LD_LIBRRAY_PATH as part of this patch.
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2801:
-------------------------------------
Priority: Blocker (was: Major)
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch, MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2801:
-------------------------------------------
Attachment: MR-2801.patch
This patch should also apply to branch-0.23.
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102874#comment-13102874 ]
Robert Joseph Evans commented on MAPREDUCE-2801:
------------------------------------------------
The find bugs warning does not come from this patch. It appears to be something related to cache archive localization that was missed before.
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2801:
-------------------------------------------
Attachment: MR-2801.patch
Upmerged to latest trunk, to resolve some merge conflicts.
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch, MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103077#comment-13103077 ]
Mahadev konar commented on MAPREDUCE-2801:
------------------------------------------
@Arun,
I think its going to require a little bit of more work to add it to dist cache, (with respect to different OS's, different version (64/32 bits)). For now we can just do what Bobby has and work on something like a MAPREDUCE_HOME for a job where all the native libaries/environment is defined on job submission and is setup during executing of MAPREDUCE containers.
Bobby,
We should add the ability for adding the native libaries if specified by the client to the native libraries classpath as done in 0.20.***.
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103102#comment-13103102 ]
Arun C Murthy commented on MAPREDUCE-2801:
------------------------------------------
bq. What do you mean by the native libraries classpath?
Robert - the clients can pass -Djava.library.path in mapred.child.java.opts. We need to append in that case - that is what Mahadev is referring to. Currently TaskRunner handles this in 0.20.
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2801:
-------------------------------------
Status: Open (was: Patch Available)
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch, MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103644#comment-13103644 ]
Robert Joseph Evans commented on MAPREDUCE-2801:
------------------------------------------------
To clarify a little Mahadev's question TaskRunner.java (in 0.23) line 392 calls parseChildJavaOpts.
parseChildJavaOpts is replaced by lines 348 and 349 in YARNRunner.java
{code}
String[] javaOptsSplit = conf.get(MRJobConfig.MR_AM_COMMAND_OPTS,
MRJobConfig.DEFAULT_MR_AM_COMMAND_OPTS).split(" ");
{code}
and also the method getChildJavaOpts and lines 223 and 225 in MapReduceChildJVM.java
{code}
String javaOpts = getChildJavaOpts(conf, task.isMapTask());
{code}
{code}
String [] javaOptsSplit = javaOpts.split(" ");
{code}
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102865#comment-13102865 ]
Hadoop QA commented on MAPREDUCE-2801:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12494078/MR-2801.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/707//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/707//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/707//console
This message is automatically generated.
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103063#comment-13103063 ]
Arun C Murthy commented on MAPREDUCE-2801:
------------------------------------------
Robert - sorry for coming in late.
It seems simpler to add the container's cwd to the java.library.path by default and get the MR AM to use the DistributedCache to distribute the libraries?
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105372#comment-13105372 ]
Robert Joseph Evans commented on MAPREDUCE-2801:
------------------------------------------------
I am fine with doing it through the distributed cache. You made a very good point that the java.library.path/LD_LIBRARY_PATH for the nodemanager is likely to be different from what we want. It may fix a couple of issues with native code in common, but not the native code in map/reduce. So the patch is wrong and really should not go in. If Mahadev agrees I will dupe this issue to MAPREDUCE-2880.
Arun, do you need help any with MAPREDUCE-2880?
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch, MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098388#comment-13098388 ]
Robert Joseph Evans commented on MAPREDUCE-2801:
------------------------------------------------
In the 0.20.20X code the tasktracker via the TaskRunner.java added in its own java.library.path to the options passed down to the child. Under YARN this will not work. It is the application master that determines the command line that is used to launch the container not the nodemanager. So if the application master is running on a system with a different java.library.path on it then the system the container is running on then it will not work (Think Hadoop is installed in a different location on different machines). Also the application master does not have the java.library.path set because the client did not set it when launching.
I think the best way to solve this is to add in code to the nodemanager that would do a replace a @hadoop-java.library.path@ with the nodemanager's java.library.path similar to how the application master will replace @taskid@ with the task id for the application. Any thoughts on this?
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2801:
-------------------------------------------
Status: Patch Available (was: Open)
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans reassigned MAPREDUCE-2801:
----------------------------------------------
Assignee: Robert Joseph Evans
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103092#comment-13103092 ]
Robert Joseph Evans commented on MAPREDUCE-2801:
------------------------------------------------
@Mahadev,
What do you mean by the native libraries classpath? I understand the java.library.path and LD_LIBRARY_PATH for finding the necessary dependencies for the .so files in java.library.path. I also understand the classpath for finding java .class files, but I am not sure what is meant by native libraries classpath.
@Arun,
I believe that "." is already added to "java.library.path" and "LD_LIBRARY_PATH"
If you look at TestMapReduceChildJVM the first test ensures that LD_LIBRARY_PATH is equal to nmLdLibraryPath+":"+pwd. And the Second test ensures that command line contains -Djava.library.path=WORK_DIR:<HADOOP_LIBRARY_PATH> where WORK_DIR is the working directory to launch the container in.
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104777#comment-13104777 ]
Hadoop QA commented on MAPREDUCE-2801:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12494478/MR-2801.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/737//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/737//console
This message is automatically generated.
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch, MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105023#comment-13105023 ]
Arun C Murthy commented on MAPREDUCE-2801:
------------------------------------------
Bobby, the more I think about this approach the less I like it.
Essentially relying on NM to supply critical env for the task is a broken concept - this ensures that you can't run the same task with different env. Here 'env' can be jars, shared-objects etc.
I'm thinking we shud just bite the bullet, go the whole 9 yards and fix the classpath/ld_library_path etc. in the *correct* way i.e. throw out our reliance on bin/mapred and do everything via the DistributedCache i.e. MAPREDUCE-2880.
Thoughts?
Apologies for dragging this. IAC, this won't fix the original issue - the NM doesn't have the right java.library.path for setting up compression libs correctly.
Arguably we can fix this, but I'd rather do it the *right* way i.e. MAPREDUCE-2880.
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch, MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli resolved MAPREDUCE-2801.
------------------------------------------------
Resolution: Duplicate
Arun already started working on this at MAPREDUCE-2880.
Bobby, I am closing this as duplicate as you are fine too.
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch, MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2801:
-------------------------------------------
Status: Patch Available (was: Open)
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch, MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2801:
-------------------------------------
Status: Open (was: Patch Available)
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103111#comment-13103111 ]
Robert Joseph Evans commented on MAPREDUCE-2801:
------------------------------------------------
Yes, that code was already in before this patch was added. I did add that code into YARNRunner.java, but it was already present in MapReduceChildJVM.java. The code in MapReduceChildJVM.java was almost a copy.paste from the code in TaskRunner.java, except the code to add in the java.library.path from the current TaskTracker JVM was deleted.
If you want me to add in some more tests to verify this I am happy to, but I think the patch is valid as is.
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2801) Include the native libs in
java.library.path
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103107#comment-13103107 ]
Mahadev konar commented on MAPREDUCE-2801:
------------------------------------------
@Bobby, Sorry to have caused confusion.
To add to what Arun said take a look at:
TaskRunner.java (in 0.23): line 392, wherein it parses mapred.child.java.opts and adds it to the java.library.path.
> Include the native libs in java.library.path
> ---------------------------------------------
>
> Key: MAPREDUCE-2801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
> Attachments: MR-2801.patch
>
>
> For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to <path to native libs>:$PWD
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira