You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Colin Patrick McCabe (JIRA)" <ji...@apache.org> on 2012/09/14 00:28:07 UTC
[jira] [Created] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Colin Patrick McCabe created HADOOP-8806:
--------------------------------------------
Summary: libhadoop.so: search java.library.path when calling dlopen
Key: HADOOP-8806
URL: https://issues.apache.org/jira/browse/HADOOP-8806
Project: Hadoop Common
Issue Type: Improvement
Reporter: Colin Patrick McCabe
Priority: Minor
libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8806:
-----------------------------------------
Attachment: rpathtest2.tar.gz
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz, rpathtest.tar.gz
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8806:
-----------------------------------------
Attachment: HADOOP-8806.003.patch
* insert '$ORIGIN/' into the RPATH of {{libhadoop.so}} on Linux.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457787#comment-13457787 ]
Hudson commented on HADOOP-8806:
--------------------------------
Integrated in Hadoop-Hdfs-trunk #1169 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1169/])
HADOOP-8806. libhadoop.so: dlopen should be better at locating libsnappy.so, etc. Contributed by Colin Patrick McCabe (Revision 1386784)
HADOOP-8806. libhadoop.so: dlopen should be better at locating libsnappy.so, etc. Contributed by Colin Patrick McCabe (Revision 1386780)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1386784
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1386780
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: 2.0.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz, rpathtest.tar.gz
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455531#comment-13455531 ]
Andy Isaacson commented on HADOOP-8806:
---------------------------------------
Another potential issue -- there is plenty of fun debugging waiting for the first developer who tries to have a dynamic libsnappy.so and a static snappy.a-in-libhadoop.so in the same executable. Supposedly that scenario can be made to work, but I've had no end of trouble with similar scenarios previously.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457251#comment-13457251 ]
Hudson commented on HADOOP-8806:
--------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2800 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2800/])
HADOOP-8806. libhadoop.so: dlopen should be better at locating libsnappy.so, etc. Contributed by Colin Patrick McCabe (Revision 1386784)
HADOOP-8806. libhadoop.so: dlopen should be better at locating libsnappy.so, etc. Contributed by Colin Patrick McCabe (Revision 1386780)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1386784
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1386780
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: 2.0.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz, rpathtest.tar.gz
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456030#comment-13456030 ]
Colin Patrick McCabe commented on HADOOP-8806:
----------------------------------------------
What do you think about the proposed RPATH hack? Basically allowing us to find libraries relative to libhadoop.so. I haven't tried it yet, but it seems like a good way to go if it works?
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456019#comment-13456019 ]
Todd Lipcon commented on HADOOP-8806:
-------------------------------------
It's not only about MR jobs. Apps like HBase and Flume can also depend on Snappy, and it would be nice to allow them to get everything they need by just setting the java.library.path to include the hadoop lib/native dir without also futzing with LD_LIBRARY_PATH
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455485#comment-13455485 ]
Roman Shaposhnik commented on HADOOP-8806:
------------------------------------------
bq. [epel discussion]
I think this is a bit of a red herring here. I'm confident that libsnappy will get into the distros with time and when this happens we have to be able to offer a choice that is not -- recompile your libhadoop.so. The current situation where libsnappy.so gets bundled with hadoop as a separate object that can ignored (if needed) is ideal from that standpoint. Statically linking all of it into the libhadoop.so is a hammer that I'd rather not use right away.
At this point the problem is that we've got 2 code path. One in org.apache.hadoop.io.compress.snappy that does System.loadLibrary("snappy"); and is fine and the other one, apparently, in libhadoop.so that uses dlopen(). Would it be completely out of the question to focus on unifying the two?
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Kihwal Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455813#comment-13455813 ]
Kihwal Lee commented on HADOOP-8806:
------------------------------------
bq. These libraries can be bundled in the $HADOOP_ROOT/lib/native directory. For example, the -Dbundle.snappy build option copies libsnappy.so to this directory. However, snappy can't be loaded from this directory unless LD_LIBRARY_PATH is set to include this directory.
If this is only about MR jobs, isn't setting {{LD_LIBRARY_PATH}} in {{mapreduce.admin.user.env}} enough?
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457250#comment-13457250 ]
Hudson commented on HADOOP-8806:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #2737 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2737/])
HADOOP-8806. libhadoop.so: dlopen should be better at locating libsnappy.so, etc. Contributed by Colin Patrick McCabe (Revision 1386784)
HADOOP-8806. libhadoop.so: dlopen should be better at locating libsnappy.so, etc. Contributed by Colin Patrick McCabe (Revision 1386780)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1386784
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1386780
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: 2.0.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz, rpathtest.tar.gz
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457236#comment-13457236 ]
Andy Isaacson commented on HADOOP-8806:
---------------------------------------
I didn't intend my exploratory tarball to derail my Friday +1. +1.
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz, rpathtest.tar.gz
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455375#comment-13455375 ]
Todd Lipcon commented on HADOOP-8806:
-------------------------------------
IMO we don't want to link against the system's libsnappy for the forseeable future, since it's not widely packaged by distributions enough. libz being an ancient library would be more reasonable to link against.
So, I think we should do one of:
1) Continue to use dlopen, but explicitly search the java.library.path. It's probably easy enough with JNI to grab this system property.
or 2) Statically link libsnappy.a into libhadoop at build time.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455495#comment-13455495 ]
Allen Wittenauer commented on HADOOP-8806:
------------------------------------------
bq. However, snappy can't be loaded from this directory unless LD_LIBRARY_PATH is set to include this directory
Or, IIRC, dlopen will look in the shared libraries run path (-rpath for those using GNU LD, -R for just about everyone else). This is the preferred way to deal with this outside of Java. See also the $ORIGIN 'macro' to make the path dynamic based upon the executable location. There is no reason to really hard-code any paths or set LD_LIBRARY_PATH in modern linkers due to these features unless you are absolutely doing something crazy.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455527#comment-13455527 ]
Allen Wittenauer commented on HADOOP-8806:
------------------------------------------
The problem with LD_LIBRARY_PATH is if you are running something not Java, you may accidentally introduce a different/conflicting library than the one the compiled program is expecting. That's going to lead to some very strange errors to the user. The other possibility is that the end user will override LD_LIBRARY_PATH themselves, which puts us back to the original problem.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455372#comment-13455372 ]
Colin Patrick McCabe commented on HADOOP-8806:
----------------------------------------------
One issue is that if we ever move from using {{dlopen}} to linking directory against libsnappy or libz, this trick won't work. so maybe {{LD_LIBRARY_PATH}} is the better way after all? Hmm.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456291#comment-13456291 ]
Andy Isaacson commented on HADOOP-8806:
---------------------------------------
I ran a standalone test of {{DT_RPATH $ORIGIN}}. There's good news and bad news which is probably ok.
On the plus side, setting DT_RPATH=$ORIGIN on libhadoop.so does allow it to find libsnappy.so in the same directory. This is good.
On the downside, setting DT_RPATH=$ORIGIN on libhadoop.so pollutes the search path for later dlopens in the main executable. In my test,
{noformat}
main.c
-> dlopens /path/to/libbar.so with DT_RPATH=$ORIGIN
-> dlopens libfoo.so
-> dlopens libfoo.so
{noformat}
Since main.c is supposed to be unaffected by the behavior of libbar, the final dlopen should fail since main's search path does not include libfoo.
Unfortunately the final dlopen succeeds. This means that the libfoo opened by libbar, while loaded, is available for open by main.
Modifying the test so that libbar.so dlclose()s libfoo before returning fixes the problem; the final open of libfoo fails again. So it's not a problem of libbar's DT_RPATH polluting the main executable's search path, but rather, a single table of currently open objects.
What does this mean? Suppose we have a libsnappy.1.0.14 in $ORIGIN, and the system has a libsnappy.1.0.20 in /usr/lib. A program which uses libhadoop will get 1.0.14 if libhadoop is opened before libsnappy, and 1.0.20 if libsnappy is opened before libhadoop. This kinda sucks. But, it's definitely a corner case, so maybe it's OK.
If we used RUNPATH rather than RPATH, the user could work around the above problem by setting LD_LIBRARY_PATH. Since it's probably safer from a build perspective to use RPATH -- I couldn't even figure out how to get ld-2.22 to set RUNPATH -- then LD_LIBRARY_PATH will not work.
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HADOOP-8806:
--------------------------------
Component/s: build
Target Version/s: 2.0.3-alpha
Affects Version/s: 2.0.0-alpha
Hadoop Flags: Reviewed
+1 lgtm as well. Thanks for the deleted explanation guys.
Also, thanks for testing both the java.library.patch and LD_LIBRARY_PATH scenarios Colin. The test failure and findbugs from the earlier test-patch run are unrelated to this change. I've tested a native build on my host for sanity.
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: 2.0.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz, rpathtest.tar.gz
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HADOOP-8806:
--------------------------------
Resolution: Fixed
Fix Version/s: 2.0.3-alpha
Target Version/s: (was: 2.0.3-alpha)
Status: Resolved (was: Patch Available)
I've committed this and merged to branch-2. Thanks Colin.
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: 2.0.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz, rpathtest.tar.gz
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455613#comment-13455613 ]
Colin Patrick McCabe commented on HADOOP-8806:
----------------------------------------------
Alan, if you have a patch that actually fixes this, maybe you could share it with us?
After doing a little more research, it seems that {{rpath}} is not just for binaries. Dynamic libraries can have it too. Although the man page for {{ld.so}} only mentions it in the context of executables, it seems like it can embedded into shared libraries as well. Combine that with ${ORIGIN}, and at least in theory we could find {{libsnappy.so}} by using the path of {{libhadoop.so}}.
There's some discussion here: http://stackoverflow.com/questions/6323603/ld-using-rpath-origin-inside-a-shared-library-recursive
It's all a little undocumented and weird, and ${ORIGIN} is definitely Linux- (and maybe Solaris?) specific, but it might be better than {{LD_LIBRARY_PATH}}. Maybe.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456120#comment-13456120 ]
Colin Patrick McCabe commented on HADOOP-8806:
----------------------------------------------
I have tested this patch with Yarn running TestDFSIO, and it works. I can drop {{libsnappy.so}} into {{$HADOOP_ROOT/lib/native}} (the same directory that contains {{libhadoop.so}}) and everything "just works."
No {{LD_LIBRARY_PATH}} required, and no code changes required.
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455440#comment-13455440 ]
Roman Shaposhnik commented on HADOOP-8806:
------------------------------------------
FYI: snappy is now pretty widely available. Even on CentOS 5: http://pkgs.org/search/?keyword=snappy
With that in mind, I'd rather link against it dynamically. Especially if we are not getting rid of dynamic aspect alltogether (libz will remain to be dynamically linked).
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455525#comment-13455525 ]
Andy Isaacson commented on HADOOP-8806:
---------------------------------------
{quote}
bq. you cannot link a .a into a .so
Sure you can. You can always use ar to pull out the objects and then include them into your own library.
{quote}
Only if the objects were compiled {{-fPIC}} and any other requirements are met. My understanding is that PIC is still an issue in the amd64 ABI but I'd have to go check to make sure...
I'd strongly recommend that we continue to dynamically link against libsnappy.so, using LD_LIBRARY_PATH if at all possible, but even parsing {{java.library.path}} and iterating it to dlopen would be OK.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455470#comment-13455470 ]
Colin Patrick McCabe commented on HADOOP-8806:
----------------------------------------------
bq. With that in mind, I'd rather link against it dynamically. Especially if we are not getting rid of dynamic aspect alltogether (libz will remain to be dynamically linked).
I think a lot of people still don't have libsnappy installed, and they would perceive being required to install it to use libhadoop as a regression. In contrast, nearly every system in existence has libz installed.
bq. re static linking, we should make sure we don't accidentally export the libsnappy symbols, though – we'd like someone to be able to pull in libhadoop.so but separately link their own snappy from somewhere else if they so choose.
I agree that we should not export the libsnappy symbols. I mean really we should not be exporting any symbols except the ones that the JVM invokes. But that's a bit of a separate issue.
bq. [epel discussion]
EPEL isn't officially supported by Red Hat, and a lot of systems don't install packages from there. There are other third-party repos for Red Hat, and some of them conflict with one another, as I found out (but that's another story)
For now, I think we have to assume that most users will not have libsnappy provided by their base OS. When that changes in a few years we can revisit this.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Isaacson updated HADOOP-8806:
----------------------------------
Attachment: rpathtest.tar.gz
Attaching rpathtest.tar.gz which contains a test program for various rpath scenarios.
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch, rpathtest.tar.gz
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455459#comment-13455459 ]
Todd Lipcon commented on HADOOP-8806:
-------------------------------------
bq. FYI: snappy is now pretty widely available. Even on CentOS 5: http://pkgs.org/search/?keyword=snappy
Yea, but only in EPEL/elforge, which is a little annoying (not always enabled in production systems)
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456310#comment-13456310 ]
Hadoop QA commented on HADOOP-8806:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12545262/rpathtest.tar.gz
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1468//console
This message is automatically generated.
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch, rpathtest.tar.gz
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456148#comment-13456148 ]
Hadoop QA commented on HADOOP-8806:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12545207/HADOOP-8806.003.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverController
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1462//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1462//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1462//console
This message is automatically generated.
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8806:
-----------------------------------------
Assignee: Colin Patrick McCabe
Status: Patch Available (was: Open)
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456531#comment-13456531 ]
Hadoop QA commented on HADOOP-8806:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12545302/rpathtest2.tar.gz
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1472//console
This message is automatically generated.
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz, rpathtest.tar.gz
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455549#comment-13455549 ]
Colin Patrick McCabe commented on HADOOP-8806:
----------------------------------------------
On x86_64, you cannot link a .a into a .so unless the .a was compiled with -fPIC. Give it a try if you are curious.
The issue here as I see it as the a lot of people seem to want to put {{libsnappy.so}} in the same folder as {{libhadoop.so}}. They believe that by doing this, we will use that library. However, currently we do not. So we need to eliminate that difference between people's expectations and reality somehow.
A lot of things have been proposed:
* we could manually search {{java.library.path}}, but that is more complex. Also, it doesn't work for shared libraries that we link against normally. Since every discussion we've ever had about {{dlopen}} has ended with "... and eventually, we won't have to do this," that seems like a major downside.
* we could add {{java.library.path}} to {{LD_LIBRARY_PATH}}. That solves the problem for both dlopen'ed and normally linked shared libraries, but it requires some changes to initialization scripts. Alan has argued that this may lead to unintended code being loaded. However, if you can drop evil jars into the {{java.library.path}}, you can already compromise the system, so this seems specious. (You could also drop an evil {{libhadoop.so}} into {{java.library.path}}, if you have write access to that path.) Basically if you can write to {{java.library.path}}, you have own the system-- simple as that.
* we could use {{System.loadLibrary}} to load the shared library, and then use {{dlopen(RTLD_NOLOAD | RTLD_GLOBAL)}} to make the library's symbols accessible to {{libhadoop.so}}. This solves the problem with minimal code change, but it's Linux specific, and suffers from a lot of the same problems as the first solution.
* static linking was proposed-- but it seems to be infeasible, so forget that.
I think I'm leaning towards solution #2, which would basically mean closing this JIRA as WONTFIX.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455519#comment-13455519 ]
Allen Wittenauer commented on HADOOP-8806:
------------------------------------------
(p.s., this is pretty much what the compiler does when you statically link...)
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456035#comment-13456035 ]
Todd Lipcon commented on HADOOP-8806:
-------------------------------------
Seems reasonable to me.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456228#comment-13456228 ]
Eli Collins commented on HADOOP-8806:
-------------------------------------
Overall seems reasonable.
- Why use RPATH instead of RUNPATH?
- Have you tested with libsnappy.so in $HADOOP_ROOT/lib/native as well as installed in the system (ie in LD_LIBRARY_PATH)?
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457280#comment-13457280 ]
Hudson commented on HADOOP-8806:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2761 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2761/])
HADOOP-8806. libhadoop.so: dlopen should be better at locating libsnappy.so, etc. Contributed by Colin Patrick McCabe (Revision 1386784)
HADOOP-8806. libhadoop.so: dlopen should be better at locating libsnappy.so, etc. Contributed by Colin Patrick McCabe (Revision 1386780)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1386784
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1386780
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: 2.0.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz, rpathtest.tar.gz
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455552#comment-13455552 ]
Allen Wittenauer commented on HADOOP-8806:
------------------------------------------
It's pretty clear that I'm not making my point given the summary, so I'm just going to let it drop and prepare yet another local patch to back this total mess out after it inevitably gets committed.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455433#comment-13455433 ]
Todd Lipcon commented on HADOOP-8806:
-------------------------------------
Makes sense to me.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8806:
-----------------------------------------
Description:
libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
was:
libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
Should we also search {{java.library.path}} when loading these libraries?
Summary: libhadoop.so: dlopen should be better at locating libsnappy.so, etc. (was: libhadoop.so: search java.library.path when calling dlopen)
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455434#comment-13455434 ]
Todd Lipcon commented on HADOOP-8806:
-------------------------------------
re static linking, we should make sure we don't accidentally export the libsnappy symbols, though -- we'd like someone to be able to pull in libhadoop.so but separately link their own snappy from somewhere else if they so choose.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456004#comment-13456004 ]
Colin Patrick McCabe commented on HADOOP-8806:
----------------------------------------------
bq. If this is only about MR jobs, isn't setting LD_LIBRARY_PATH in mapreduce.admin.user.env enough?
Yes, it is enough. The question is whether we could do anything to make this easier.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456529#comment-13456529 ]
Colin Patrick McCabe commented on HADOOP-8806:
----------------------------------------------
bq. On the downside, setting DT_RPATH=$ORIGIN on libhadoop.so pollutes the search path for later dlopens in the main executable. In my test:
{code}
main.c
-> dlopens /path/to/libbar.so with DT_RPATH=$ORIGIN
-> dlopens libfoo.so
-> dlopens libfoo.so
{code}
I'm afraid you have been misled.
At the point when you are dlopening libfoo.so a second time, it is already linked to the executable. Essentially, the dynamic linker realizes that you already have libfoo loaded, and doesn't try to reload it a second time.
This behavior is sort of hinted at in the man page for dlopen, although it's not spelled out explicitly. See the documentation for {{RTLD_NOLOAD}}, for example. According to the man page, "This can be used to test if the library is already resident." To me, this implies that:
* libraries can be either resident or not
* libraries can't be loaded twice in the same executable
Although those stipulations aren't spelled out explicitly. (Time to submit a patch?)
Here's a modification of your test program which illustrates what I mean. Basically it does:
{code}
main.c
-> dlopens /path/to/libbar.so with DT_RPATH=$ORIGIN
-> dlopens libfoo.so
-> dlopens libbaz.so
{code}
The last dlopen fails because {{libbaz.so}} is not in the library search path (although it would succeed for {{libfoo.so}} opening it, due to {{RPATH}}.)
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz, rpathtest.tar.gz
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456244#comment-13456244 ]
Colin Patrick McCabe commented on HADOOP-8806:
----------------------------------------------
bq. Why use RPATH instead of RUNPATH?
RUNPATH support is a little newer than RPATH, so it seemed less likely to work with older versions of CMake or older distributions. RUNPATH is also a Linux-ism, apparently.
I guess we could use RUNPATH, but it seemed simpler to go with the default (RPATH). The only difference is that RPATH supercedes {{LD_LIBRARY_PATH}}, and RUNPATH does not.
bq. Have you tested with libsnappy.so in $HADOOP_ROOT/lib/native as well as installed in the system (ie in LD_LIBRARY_PATH)?
Yes.
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455512#comment-13455512 ]
Colin Patrick McCabe commented on HADOOP-8806:
----------------------------------------------
bq. [rpath discussion]
The problem is, we don't know at compile-time where libsnappy.so will be. Normally there's a "make install" step where rpaths get injected, but there is nothing like that for Hadoop.
Sadly, I have encountered an issue that I think puts the kibosh on the "static libsnappy" idea-- you cannot link a .a into a .so. I don't know why I didn't think of that earlier.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455426#comment-13455426 ]
Colin Patrick McCabe commented on HADOOP-8806:
----------------------------------------------
What if we
* linked against the system {{libz.so}}
* statically linked in {{libsnappy.a}}
I think that would simplify things considerably and eliminate a lot of hair-pulling over obscure configuration issues.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: search
java.library.path when calling dlopen
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455515#comment-13455515 ]
Allen Wittenauer commented on HADOOP-8806:
------------------------------------------
That's why $ORIGIN is a way out of this. At install time, build a symlink to a known path to the out-of-the-way location.
bq. you cannot link a .a into a .so
Sure you can. You can always use ar to pull out the objects and then include them into your own library.
> libhadoop.so: search java.library.path when calling dlopen
> ----------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Priority: Minor
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Should we also search {{java.library.path}} when loading these libraries?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457370#comment-13457370 ]
Roman Shaposhnik commented on HADOOP-8806:
------------------------------------------
Since Linux (GNU really) documentation leaves quite a bit open for "interpretation" as far as $ORIGIN and dependencies of unbundled products go, here's a much more complete treatment of the from Solaris' Linkers and Libraries doc: http://docs.oracle.com/cd/E19253-01/817-1984/appendixc-4/index.html
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: 2.0.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz, rpathtest.tar.gz
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456261#comment-13456261 ]
Andy Isaacson commented on HADOOP-8806:
---------------------------------------
This seems like a reasonable stopgap to me. +1.
Note that {{$ORIGIN}} is documented in {{ld-linux(8)}}:
{noformat}
RPATH TOKEN EXPANSION
The runtime linker provides a number of tokens that can be used in an
rpath specification (DT_RPATH or DT_RUNPATH).
$ORIGIN
ld.so understands the string $ORIGIN (or equivalently ${ORIGIN})
in an rpath specification to mean the directory containing the
application executable.
{noformat}
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8806.003.patch
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8806) libhadoop.so: dlopen should be
better at locating libsnappy.so, etc.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457829#comment-13457829 ]
Hudson commented on HADOOP-8806:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk #1200 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1200/])
HADOOP-8806. libhadoop.so: dlopen should be better at locating libsnappy.so, etc. Contributed by Colin Patrick McCabe (Revision 1386784)
HADOOP-8806. libhadoop.so: dlopen should be better at locating libsnappy.so, etc. Contributed by Colin Patrick McCabe (Revision 1386780)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1386784
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1386780
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
> libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
> --------------------------------------------------------------------
>
> Key: HADOOP-8806
> URL: https://issues.apache.org/jira/browse/HADOOP-8806
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: 2.0.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8806.003.patch, rpathtest2.tar.gz, rpathtest.tar.gz
>
>
> libhadoop calls {{dlopen}} to load {{libsnappy.so}} and {{libz.so}}. These libraries can be bundled in the {{$HADOOP_ROOT/lib/native}} directory. For example, the {{-Dbundle.snappy}} build option copies {{libsnappy.so}} to this directory. However, snappy can't be loaded from this directory unless {{LD_LIBRARY_PATH}} is set to include this directory.
> Can we make this configuration "just work" without needing to rely on {{LD_LIBRARY_PATH}}?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira