You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Devaraj Das (Created) (JIRA)" <ji...@apache.org> on 2012/03/20 00:45:37 UTC
[jira] [Created] (HADOOP-8187) Improve the discovery of the jvm
library during the build process
Improve the discovery of the jvm library during the build process
-----------------------------------------------------------------
Key: HADOOP-8187
URL: https://issues.apache.org/jira/browse/HADOOP-8187
Project: Hadoop Common
Issue Type: Improvement
Reporter: Devaraj Das
Improve the discovery of the jvm library during the build of native libraries/libhdfs/fuse-dfs, etc. A couple of different ways are currently used (discussed in HADOOP-6924). We should clean this part up and also consider builds of native stuff on OSX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8187) Improve the discovery of the jvm
library during the build process
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462313#comment-13462313 ]
Colin Patrick McCabe commented on HADOOP-8187:
----------------------------------------------
Now that you've put forward a concrete proposal, we can discuss it.
I was not aware of {{sun.boot.library.path}} prior to this, so I checked out sun.com to see what it was. At http://bugs.sun.com/view_bug.do?bug_id=6819213 I found this information:
{code}
If a class is loaded using the bootclasspath, then any native libraries needed
by that the class will be loaded from the sun.boot.library.path. This
property (sun.boot.library.path) could be changed during the vm startup phase,
in much earlier releases, but a feature implementation rendered this property immutable.
It is the desire to make this property mutable by the java, javafx and the java plugin
launchers.
[...]
Though this will continue to be sun specific implementation and will
be used as such, external developers should not rely on this mechanism,
and it will be deprecated or changed in the future.
Posted Date : 2009-03-18 22:27:00.0
{code}
Some more Googling revealed that {{sun.boot.library.path}} was often set to {{C:\sdk\jre\bin}} on Windows. Correct me if I am wrong, but basically it seems like a non-portable, Sun-specific way of inferring the location of the JRE (not the JDK). Please note that {{sun.boot.library.path}} is not a single path, but a series of paths, one of which might be the JRE path.
There are still a lot of unanswered questions here. Is using {{sun.boot.library.path}} better than using {{java.library.path}}? Why or why not? What specific problem would this solve? You mentioned MacOS. Do you think using {{sun.boot.library.path}} would allow users to build on MacOS without setting {{JAVA_HOME}}? Is it safe to rely on {{sun.boot.library.path}} when using OpenJDK or other runtime environments?
bq. - As usual, OS X is a bit of a troublemaker since (by default) libjvm.dylib is only 32-bit on at least Snow Leopard in some JVM versions. (I'll leave it as an exercise for the reader to figure out why.) But by passing by the appropriate flags at compile time, it will get built the fat binary version of libjvm and the problem goes away.
I find this confusing. Are you suggesting that MacOS users recompile the JDK itself? Or something else?
> Improve the discovery of the jvm library during the build process
> -----------------------------------------------------------------
>
> Key: HADOOP-8187
> URL: https://issues.apache.org/jira/browse/HADOOP-8187
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Devaraj Das
>
> Improve the discovery of the jvm library during the build of native libraries/libhdfs/fuse-dfs, etc. A couple of different ways are currently used (discussed in HADOOP-6924). We should clean this part up and also consider builds of native stuff on OSX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8187) Improve the discovery of the jvm
library during the build process
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462255#comment-13462255 ]
Colin Patrick McCabe commented on HADOOP-8187:
----------------------------------------------
As far as I can see, this was addressed by HADOOP-8737. If there's anything I'm missing, then please reopen this with a description of how we could improve.
> Improve the discovery of the jvm library during the build process
> -----------------------------------------------------------------
>
> Key: HADOOP-8187
> URL: https://issues.apache.org/jira/browse/HADOOP-8187
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Devaraj Das
>
> Improve the discovery of the jvm library during the build of native libraries/libhdfs/fuse-dfs, etc. A couple of different ways are currently used (discussed in HADOOP-6924). We should clean this part up and also consider builds of native stuff on OSX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8187) Improve the discovery of the jvm
library during the build process
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462291#comment-13462291 ]
Colin Patrick McCabe commented on HADOOP-8187:
----------------------------------------------
bq. Description of how to improve this is in the linked jiras.
Allen, the linked JIRAs are closed with resolution "fixed." There's nothing more to improve there.
If you have ideas for improving this further, could you add them to the description of this JIRA?
> Improve the discovery of the jvm library during the build process
> -----------------------------------------------------------------
>
> Key: HADOOP-8187
> URL: https://issues.apache.org/jira/browse/HADOOP-8187
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Devaraj Das
>
> Improve the discovery of the jvm library during the build of native libraries/libhdfs/fuse-dfs, etc. A couple of different ways are currently used (discussed in HADOOP-6924). We should clean this part up and also consider builds of native stuff on OSX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8187) Improve the discovery of the jvm
library during the build process
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe resolved HADOOP-8187.
------------------------------------------
Resolution: Duplicate
> Improve the discovery of the jvm library during the build process
> -----------------------------------------------------------------
>
> Key: HADOOP-8187
> URL: https://issues.apache.org/jira/browse/HADOOP-8187
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Devaraj Das
>
> Improve the discovery of the jvm library during the build of native libraries/libhdfs/fuse-dfs, etc. A couple of different ways are currently used (discussed in HADOOP-6924). We should clean this part up and also consider builds of native stuff on OSX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8187) Improve the discovery of the jvm
library during the build process
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463127#comment-13463127 ]
Colin Patrick McCabe commented on HADOOP-8187:
----------------------------------------------
I think we should just require developers on MacOS, Windows, etc, to set {{JAVA_HOME}} to the location of a valid JDK, and error out if it's not.
Playing guessing games never works well in practice, and it leads to mysterious build problems or even misbuilds when the "clever" code grabs the wrong JDK.
At least that's my opinion.
> Improve the discovery of the jvm library during the build process
> -----------------------------------------------------------------
>
> Key: HADOOP-8187
> URL: https://issues.apache.org/jira/browse/HADOOP-8187
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Devaraj Das
>
> Improve the discovery of the jvm library during the build of native libraries/libhdfs/fuse-dfs, etc. A couple of different ways are currently used (discussed in HADOOP-6924). We should clean this part up and also consider builds of native stuff on OSX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8187) Improve the discovery of the jvm
library during the build process
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462298#comment-13462298 ]
Allen Wittenauer commented on HADOOP-8187:
------------------------------------------
I guess you didn't bother actually reading that JIRA. Here, I'll summarize for you:
Deveraj opened this jira to track future work. The primarily thought was actually asking the JVM where the libraries are via system properties. HADOOP-8737 is a pretty terrible hack that only works on some platforms. By asking the JVM, one should only need to port to a JVM rather than an entire OS*. Sample code I wrote a while back for Sun's JVM exists in HADOOP-6924.
* - As usual, OS X is a bit of a troublemaker since (by default) libjvm.dylib is only 32-bit on at least Snow Leopard in some JVM versions. (I'll leave it as an exercise for the reader to figure out why.) But by passing by the appropriate flags at compile time, it will get built the fat binary version of libjvm and the problem goes away.
> Improve the discovery of the jvm library during the build process
> -----------------------------------------------------------------
>
> Key: HADOOP-8187
> URL: https://issues.apache.org/jira/browse/HADOOP-8187
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Devaraj Das
>
> Improve the discovery of the jvm library during the build of native libraries/libhdfs/fuse-dfs, etc. A couple of different ways are currently used (discussed in HADOOP-6924). We should clean this part up and also consider builds of native stuff on OSX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-8187) Improve the discovery of the jvm
library during the build process
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer reopened HADOOP-8187:
--------------------------------------
Description of how to improve this is in the linked jiras.
> Improve the discovery of the jvm library during the build process
> -----------------------------------------------------------------
>
> Key: HADOOP-8187
> URL: https://issues.apache.org/jira/browse/HADOOP-8187
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Devaraj Das
>
> Improve the discovery of the jvm library during the build of native libraries/libhdfs/fuse-dfs, etc. A couple of different ways are currently used (discussed in HADOOP-6924). We should clean this part up and also consider builds of native stuff on OSX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8187) Improve the discovery of the jvm
library during the build process
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462335#comment-13462335 ]
Allen Wittenauer commented on HADOOP-8187:
------------------------------------------
bq. it seems like a non-portable, Sun-specific way of inferring the location of the JRE
Right. Which is why I said at that point you'd be porting to different JVMs. IBM has a similar property (whose name escape me at the moment). We *should* have a fallback method in case we can't find it, the property doesn't exist, whatever.
But my experience at least on a few platforms is that this is significantly more reliable than $JAVA_HOME or doing finds or any of the other weird things we've tried in the past, especially on multi-arch JVMs where multiple libraries are generally present. (The -d param isn't just for decoration, folks...)
Windows appears to be another (not surprising) outlier.
bq. Are you suggesting that MacOS users recompile the JDK itself?
Apple had a special agreement with Sun that resulted in a lot of chaos with regards to file locations and even how to build JNI code in order to fit it into the NeXTSTEP/OpenStep/Darwin mold. The normal rules do not apply and treating it as such usually ends in tears. I haven't had a chance to look at Mountain Lion to see if things are any better/more standard. I'm suspecting not.
> Improve the discovery of the jvm library during the build process
> -----------------------------------------------------------------
>
> Key: HADOOP-8187
> URL: https://issues.apache.org/jira/browse/HADOOP-8187
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Devaraj Das
>
> Improve the discovery of the jvm library during the build of native libraries/libhdfs/fuse-dfs, etc. A couple of different ways are currently used (discussed in HADOOP-6924). We should clean this part up and also consider builds of native stuff on OSX.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira