You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2008/02/15 19:02:28 UTC
[jira] Created: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
------------------------------------------------------------------------------------------
Key: HADOOP-2838
URL: https://issues.apache.org/jira/browse/HADOOP-2838
Project: Hadoop Core
Issue Type: Improvement
Components: scripts
Affects Versions: 0.16.0
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Fix For: 0.17.0
Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-2838) Add HADOOP_LIBRARY_PATH
config setting so Hadoop will include external directories for jni
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597525#action_12597525 ]
acmurthy edited comment on HADOOP-2838 at 5/16/08 9:18 AM:
----------------------------------------------------------------
I propose we generalise this feature and let users set _any_ environment variable for their tasks... maybe we should add a mapred.child.env configuration variable which takes in a comma-separated list of environment variables:
{noformat}
<property>
<name>mapred.child.env</name>
<value>LD_LIBRARY_PATH=/foo,PATH=/blah</value>
</property>
{noformat}
Of course we should ensure that each of the env-vars in that list is appended to the env inherited from the parent TT.
Thoughts?
was (Author: acmurthy):
I propose we generalise this feature and let users set _any_ environment variable for their tasks... maybe we should add a mapred.child.env configuration variable which takes in a comma-separated list of environment variables:
{noformat}
<property>
<name>mapred.child.env</name>
<value>LD_LIBRARY_PATH=/foo,PATH=/blah</value>
</property>
Of course we should ensure that each of the env-vars in that list is appended to the env inherited from the parent TT.
Thoughts?
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: scripts
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716574#action_12716574 ]
Amar Kamat commented on HADOOP-2838:
------------------------------------
I tried that with the testcase and the job fails. Wondering if this requires to be added in the testcase.
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2-branch-20-example.patch, HADOOP-2838-v2.2.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-2838:
-------------------------------
Release Note:
Now the users can set environment variables using mapred.child.env. They can do the following
X=Y : set X to Y
X=$X:Y : Append Y to X (which should be taken from the tasktracker)
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2-branch-20-example.patch, HADOOP-2838-v2.2.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-2838:
-------------------------------
Attachment: HADOOP-2838-v2.2-branch-20-example.patch
Not to be committed to branch 20. Just an example patch.
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2-branch-20-example.patch, HADOOP-2838-v2.2.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-2838:
-------------------------------
Component/s: (was: scripts)
mapred
Assignee: Amar Kamat (was: Owen O'Malley)
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Attachments: HADOOP-2838-v1.0.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708428#action_12708428 ]
Amar Kamat commented on HADOOP-2838:
------------------------------------
+1 for mapred.child.env which will be a comma separated.
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: scripts
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-2838:
------------------------------------
Fix Version/s: (was: 0.17.0)
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: scripts
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716275#action_12716275 ]
Sharad Agarwal commented on HADOOP-2838:
----------------------------------------
+1 patch looks fine to me.
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-2838:
-------------------------------
Attachment: HADOOP-2838-v1.1.patch
Attaching a new patch with a bug fix. Result of test-patch
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
Ant tests passed on my box. Currently trying to optimize the patch.
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-2838:
-------------------------------
Attachment: HADOOP-2838-v1.0.patch
Attaching a patch that provides the facility to add/set/append env variables for child processes. Result of test-patch
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
Todo:
# ant test
# optimize and re-iterate over the patch.
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: scripts
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Attachments: HADOOP-2838-v1.0.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641195#action_12641195 ]
Christian Kunz commented on HADOOP-2838:
----------------------------------------
I see a use case for this proposal.
HADOOP-2867 allows to load shared libraries via symbolic links pointing to localized cache files.
This would apply to libhdfs library
On the other hand, libhdfs is part of the Hadoop release and is likely locally available on all the nodes. By being able to set LD_LIBRARY_PATH to some static location, the need to use DistributedCache for dynamically loading libhdfs could be avoided.
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: scripts
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715785#action_12715785 ]
Amar Kamat commented on HADOOP-2838:
------------------------------------
Todd, I think for now we can keep this simple and get the basic functionality (i.e for paths) working. We can change/add if needed. Thoughts?
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal resolved HADOOP-2838.
------------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
I just committed this. Thanks Amar!
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716262#action_12716262 ]
Amar Kamat commented on HADOOP-2838:
------------------------------------
Ant tests passed on my box.
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715563#action_12715563 ]
Todd Lipcon commented on HADOOP-2838:
-------------------------------------
Using a comma separated conf variable here seems slightly dangerous. I imagine there must be some cases where people want to pass commas as part of their environment. I don't have a concrete example, but it seems it's only a matter of time until some streaming application comes along and really needs a comma in its environment, and then we're stuck dealing with escaping, etc.
What about using the entire mapred.child.env namespace, and iterating over it? So mapred.child.env.LD_LIBRARY_PATH would hold the value for that environment variable.
Unfortunately this would either involve iterating over and filtering all of the conf vars, or switching the "Properties" class used for storage inside Configuration to be a SortedMap so we can do a prefix match. I don't particularly like the former, and the latter is a bigger change, but the interface does seem simpler and also more robust. For example, a sysadmin could set a final property of LD_LIBRARY_PATH="path/to/some.so" to provide some kind of security sandboxing while still allowing the user to modify other variables in the environment.
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717140#action_12717140 ]
Hemanth Yamijala commented on HADOOP-2838:
------------------------------------------
Will it fail mysteriously or will it be easy to know that the failure is because of the invalid syntax ? If not, I think it would be useful to make it easy to know this is the cause in some way. Maybe through the diagnostic string.
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2-branch-20-example.patch, HADOOP-2838-v2.2.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-2838:
-------------------------------
Attachment: HADOOP-2838-v2.2.patch
Attaching a patch incorporating Sharad's comments. Result of test-patch
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 6 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
Changes :
# Improved the test case
# Merged TestMiniMRTaskTempDir with env checks.
Running ant tests now.
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715477#action_12715477 ]
Sharad Agarwal commented on HADOOP-2838:
----------------------------------------
Adding a new MiniDFSCluster based test for this looks to be an overkill. Instead can we write a test by separating this functionality in a method and just testing that? Alternatively we can add to one of the existing MiniDFSCluster test.
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716571#action_12716571 ]
Amar Kamat commented on HADOOP-2838:
------------------------------------
bq. Stumbled on this patch.. what happens if the value of mapred.child.env does not have a well formed string - like doesn't have "key=value" but only key ? Has this been taken care of ?
TaskRunner will fail in child launch leading to task failure ultimately leading to job failure. I see it as something similar to bad mapred code.
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2-branch-20-example.patch, HADOOP-2838-v2.2.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597525#action_12597525 ]
Arun C Murthy commented on HADOOP-2838:
---------------------------------------
I propose we generalise this feature and let users set _any_ environment variable for their tasks... maybe we should add a mapred.child.env configuration variable which takes in a comma-separated list of environment variables:
{noformat}
<property>
<name>mapred.child.env</name>
<value>LD_LIBRARY_PATH=/foo,PATH=/blah</value>
</property>
Of course we should ensure that each of the env-vars in that list is appended to the env inherited from the parent TT.
Thoughts?
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: scripts
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715787#action_12715787 ]
Todd Lipcon commented on HADOOP-2838:
-------------------------------------
Yep, that seems reasonable to me, Amar.
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2838) Add HADOOP_LIBRARY_PATH config
setting so Hadoop will include external directories for jni
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716558#action_12716558 ]
Hemanth Yamijala commented on HADOOP-2838:
------------------------------------------
Stumbled on this patch.. what happens if the value of mapred.child.env does not have a well formed string - like doesn't have "key=value" but only key ? Has this been taken care of ?
> Add HADOOP_LIBRARY_PATH config setting so Hadoop will include external directories for jni
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-2838
> URL: https://issues.apache.org/jira/browse/HADOOP-2838
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Owen O'Malley
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-2838-v1.0.patch, HADOOP-2838-v1.1.patch, HADOOP-2838-v2.2-branch-20-example.patch, HADOOP-2838-v2.2.patch
>
>
> Currently there is no way to configure Hadoop to use external JNI directories. I propose we add a new variable like HADOOP_CLASS_PATH that is added to the JAVA_LIBRARY_PATH before the process is run.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.