You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by "Roman Shaposhnik (Created) (JIRA)" <ji...@apache.org> on 2012/01/19 03:32:39 UTC
[jira] [Created] (MAPREDUCE-3693) mapreduce's MapReduceChildJVM no
longer proxies value of java.library.path
mapreduce's MapReduceChildJVM no longer proxies value of java.library.path
---------------------------------------------------------------------------
Key: MAPREDUCE-3693
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3693
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Roman Shaposhnik
Fix For: 0.23.1
I have noticed that org.apache.hadoop.mapred.MapReduceChildJVM doesn't forward the value of -Djava.library.path= from the parent JVM to the child JVM. Thus if one wants to use native libraries for compression the only option seems to be to manually include relevant java.library.path settings into the mapred-site.xml (as mapred.[map|reduce].child.java.opts).
This seems to be a change in behavior compared to MR1 where TaskRunner.java used to do that:
{noformat}
String libraryPath = System.getProperty("java.library.path");
if (libraryPath == null) {
libraryPath = workDir.getAbsolutePath();
} else {
libraryPath += SYSTEM_PATH_SEPARATOR + workDir;
}
boolean hasUserLDPath = false;
for(int i=0; i<javaOptsSplit.length ;i++) {
if(javaOptsSplit[i].startsWith("-Djava.library.path=")) {
javaOptsSplit[i] += SYSTEM_PATH_SEPARATOR + libraryPath;
hasUserLDPath = true;
break;
}
}
if(!hasUserLDPath) {
vargs.add("-Djava.library.path=" + libraryPath);
}
for (int i = 0; i < javaOptsSplit.length; i++) {
vargs.add(javaOptsSplit[i]);
}
{noformat}
Is this a regression or a deliberate choice?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-3693) Job's shouldn't have to
configure the location of compression libraries on the server in
mapred-site.xml
Posted by "Roman Shaposhnik (Reopened) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik reopened MAPREDUCE-3693:
-----------------------------------------
Assignee: Roman Shaposhnik
> Job's shouldn't have to configure the location of compression libraries on the server in mapred-site.xml
> --------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3693
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3693
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Roman Shaposhnik
> Assignee: Roman Shaposhnik
> Attachments: MAPREDUCE-3693.patch.txt
>
>
> I have noticed that org.apache.hadoop.mapred.MapReduceChildJVM doesn't forward the value of -Djava.library.path= from the parent JVM to the child JVM. Thus if one wants to use native libraries for compression the only option seems to be to manually include relevant java.library.path settings into the mapred-site.xml (as mapred.[map|reduce].child.java.opts).
> This seems to be a change in behavior compared to MR1 where TaskRunner.java used to do that:
> {noformat}
> String libraryPath = System.getProperty("java.library.path");
> if (libraryPath == null) {
> libraryPath = workDir.getAbsolutePath();
> } else {
> libraryPath += SYSTEM_PATH_SEPARATOR + workDir;
> }
> boolean hasUserLDPath = false;
> for(int i=0; i<javaOptsSplit.length ;i++) {
> if(javaOptsSplit[i].startsWith("-Djava.library.path=")) {
> javaOptsSplit[i] += SYSTEM_PATH_SEPARATOR + libraryPath;
> hasUserLDPath = true;
> break;
> }
> }
> if(!hasUserLDPath) {
> vargs.add("-Djava.library.path=" + libraryPath);
> }
> for (int i = 0; i < javaOptsSplit.length; i++) {
> vargs.add(javaOptsSplit[i]);
> }
> {noformat}
> Is this a regression or a deliberate choice?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-3693) mapreduce's MapReduceChildJVM no
longer proxies value of java.library.path
Posted by "Eli Collins (Reopened) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins reopened MAPREDUCE-3693:
------------------------------------
> mapreduce's MapReduceChildJVM no longer proxies value of java.library.path
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-3693
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3693
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Roman Shaposhnik
> Fix For: 0.23.1
>
>
> I have noticed that org.apache.hadoop.mapred.MapReduceChildJVM doesn't forward the value of -Djava.library.path= from the parent JVM to the child JVM. Thus if one wants to use native libraries for compression the only option seems to be to manually include relevant java.library.path settings into the mapred-site.xml (as mapred.[map|reduce].child.java.opts).
> This seems to be a change in behavior compared to MR1 where TaskRunner.java used to do that:
> {noformat}
> String libraryPath = System.getProperty("java.library.path");
> if (libraryPath == null) {
> libraryPath = workDir.getAbsolutePath();
> } else {
> libraryPath += SYSTEM_PATH_SEPARATOR + workDir;
> }
> boolean hasUserLDPath = false;
> for(int i=0; i<javaOptsSplit.length ;i++) {
> if(javaOptsSplit[i].startsWith("-Djava.library.path=")) {
> javaOptsSplit[i] += SYSTEM_PATH_SEPARATOR + libraryPath;
> hasUserLDPath = true;
> break;
> }
> }
> if(!hasUserLDPath) {
> vargs.add("-Djava.library.path=" + libraryPath);
> }
> for (int i = 0; i < javaOptsSplit.length; i++) {
> vargs.add(javaOptsSplit[i]);
> }
> {noformat}
> Is this a regression or a deliberate choice?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3693) mapreduce's MapReduceChildJVM no
longer proxies value of java.library.path
Posted by "Arun C Murthy (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy resolved MAPREDUCE-3693.
--------------------------------------
Resolution: Not A Problem
In MR2 the parent JVM i.e. MR AM is unrelated to child JVM (map/reduce) and hence it's java.library.path isn't passed along.
Very different from MR1 which, arguably, was a bug to pass TT's java.library.path to child (map/reduce).
This is a good time to not do this anymore.
Similarly, we shouldn't pass NM's library path either. It's the AM's responsibility to set it up.
> mapreduce's MapReduceChildJVM no longer proxies value of java.library.path
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-3693
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3693
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Roman Shaposhnik
> Fix For: 0.23.1
>
>
> I have noticed that org.apache.hadoop.mapred.MapReduceChildJVM doesn't forward the value of -Djava.library.path= from the parent JVM to the child JVM. Thus if one wants to use native libraries for compression the only option seems to be to manually include relevant java.library.path settings into the mapred-site.xml (as mapred.[map|reduce].child.java.opts).
> This seems to be a change in behavior compared to MR1 where TaskRunner.java used to do that:
> {noformat}
> String libraryPath = System.getProperty("java.library.path");
> if (libraryPath == null) {
> libraryPath = workDir.getAbsolutePath();
> } else {
> libraryPath += SYSTEM_PATH_SEPARATOR + workDir;
> }
> boolean hasUserLDPath = false;
> for(int i=0; i<javaOptsSplit.length ;i++) {
> if(javaOptsSplit[i].startsWith("-Djava.library.path=")) {
> javaOptsSplit[i] += SYSTEM_PATH_SEPARATOR + libraryPath;
> hasUserLDPath = true;
> break;
> }
> }
> if(!hasUserLDPath) {
> vargs.add("-Djava.library.path=" + libraryPath);
> }
> for (int i = 0; i < javaOptsSplit.length; i++) {
> vargs.add(javaOptsSplit[i]);
> }
> {noformat}
> Is this a regression or a deliberate choice?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3693) Job's shouldn't have to
configure the location of compression libraries on the server in
mapred-site.xml
Posted by "Arun C Murthy (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy resolved MAPREDUCE-3693.
--------------------------------------
Resolution: Not A Problem
> Job's shouldn't have to configure the location of compression libraries on the server in mapred-site.xml
> --------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3693
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3693
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Roman Shaposhnik
>
> I have noticed that org.apache.hadoop.mapred.MapReduceChildJVM doesn't forward the value of -Djava.library.path= from the parent JVM to the child JVM. Thus if one wants to use native libraries for compression the only option seems to be to manually include relevant java.library.path settings into the mapred-site.xml (as mapred.[map|reduce].child.java.opts).
> This seems to be a change in behavior compared to MR1 where TaskRunner.java used to do that:
> {noformat}
> String libraryPath = System.getProperty("java.library.path");
> if (libraryPath == null) {
> libraryPath = workDir.getAbsolutePath();
> } else {
> libraryPath += SYSTEM_PATH_SEPARATOR + workDir;
> }
> boolean hasUserLDPath = false;
> for(int i=0; i<javaOptsSplit.length ;i++) {
> if(javaOptsSplit[i].startsWith("-Djava.library.path=")) {
> javaOptsSplit[i] += SYSTEM_PATH_SEPARATOR + libraryPath;
> hasUserLDPath = true;
> break;
> }
> }
> if(!hasUserLDPath) {
> vargs.add("-Djava.library.path=" + libraryPath);
> }
> for (int i = 0; i < javaOptsSplit.length; i++) {
> vargs.add(javaOptsSplit[i]);
> }
> {noformat}
> Is this a regression or a deliberate choice?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira