You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Colin Patrick McCabe (JIRA)" <ji...@apache.org> on 2012/09/01 03:02:07 UTC
[jira] [Created] (HADOOP-8756) libsnappy loader issues
Colin Patrick McCabe created HADOOP-8756:
--------------------------------------------
Summary: libsnappy loader issues
Key: HADOOP-8756
URL: https://issues.apache.org/jira/browse/HADOOP-8756
Project: Hadoop Common
Issue Type: Bug
Components: native
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
* We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. It seems like the System.loadLibrary nonsense is completely redundant-- we would only need this if we planned on accessing libsnappy methods directly from Java (via the 'native' keyword)- - which we don't, and can't, since snappy is not JNI. At the end of the day, snappy is only accessible through libhadoop.so, so I think snappy-related issues should be handled there.
* We should log the search path(s) we use for {{libsnappy.so}}, so that it's easier to diagnose configuration issues.
* It seems like the static initialization code in the LoadSnappy.java class references the static initialization code in NativeCodeLoader. Basically, if the init method in NativeCodeLoader doesn't run before the init method in LoadSnappy, there could be problems. We should avoid having this kind of issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8756:
-----------------------------------------
Attachment: HADOOP-8756.004.patch
* fix whitespace
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8756) libsnappy loader issues
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8756:
-----------------------------------------
Attachment: HADOOP-8756.002.patch
* remove HADOOP_RUNAS_HOME (runAs has been removed)
* remove SnappyCodec.java. Instead, load the native code from SnappyCompressor.java and SnappyDecompressor.java. This is similar to the way the zlib stuff works now.
* If we try to instantiate SnappyCodec, but snappy is not loaded, throw an exception. Formerly, we might get undefined behavior like segfaults in this case.
* Be more helpful about why snappy could not be loaded: was it because the build was compiled without snappy? Or some other reason found in the exception thrown from {{SnappyDecompressor#initIDs}} or {{SnappyCompressor#initIDs}}?
* We don't need to call {{System.loadLibrary("snappy")}}. It's irrelevant because libsnappy is not a JNI library. Again, zlib shows how this should be done: just use {{dlopen}}, not {{System.loadLibrary}}. Note that calliing {{System.loadLibrary}} does *not* make the symbols visible to {{libhadoop.so}} because the JVM does not use {{RTLD_GLOBAL}}.
> libsnappy loader issues
> -----------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.2.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) libsnappy loader issues
Posted by "Ahmed Radwan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451077#comment-13451077 ]
Ahmed Radwan commented on HADOOP-8756:
--------------------------------------
The TestZKFailoverController failures seems unrelated (see HADOOP-8591).
> libsnappy loader issues
> -----------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) libsnappy loader issues
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450269#comment-13450269 ]
Hadoop QA commented on HADOOP-8756:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12544147/HADOOP-8756.003.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverController
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1420//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1420//console
This message is automatically generated.
> libsnappy loader issues
> -----------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.2.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454377#comment-13454377 ]
Hadoop QA commented on HADOOP-8756:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12544869/HADOOP-8756.004.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverController
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1444//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1444//console
This message is automatically generated.
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8756) libsnappy loader issues
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8756:
-----------------------------------------
Description:
We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
was:
* We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. It seems like the System.loadLibrary nonsense is completely redundant-- we would only need this if we planned on accessing libsnappy methods directly from Java (via the 'native' keyword)- - which we don't, and can't, since snappy is not JNI. At the end of the day, snappy is only accessible through libhadoop.so, so I think snappy-related issues should be handled there.
* We should log the search path(s) we use for {{libsnappy.so}}, so that it's easier to diagnose configuration issues.
> libsnappy loader issues
> -----------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.2.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468548#comment-13468548 ]
Hudson commented on HADOOP-8756:
--------------------------------
Integrated in Hadoop-Hdfs-trunk #1184 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1184/])
HADOOP-8756. Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH. Contributed by Colin Patrick McCabe (Revision 1393243)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1393243
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/config.h.cmake
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/SnappyCodec.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/LoadSnappy.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCodeLoader.c
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468209#comment-13468209 ]
Hudson commented on HADOOP-8756:
--------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2865 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2865/])
HADOOP-8756. Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH. Contributed by Colin Patrick McCabe (Revision 1393243)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1393243
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/config.h.cmake
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/SnappyCodec.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/LoadSnappy.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCodeLoader.c
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468205#comment-13468205 ]
Eli Collins commented on HADOOP-8756:
-------------------------------------
Forgot to mention, I verified hadoop fs catting a snappy file works with a build with snappy bundled in.
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8756) libsnappy loader issues
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8756:
-----------------------------------------
Status: Patch Available (was: Open)
> libsnappy loader issues
> -----------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.2.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8756) libsnappy loader issues
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8756:
-----------------------------------------
Attachment: HADOOP-8756.003.patch
We previously allowed Codecs to be constructed even if they couldn't be used; the tests relied on this. So let's keep the old behavior for now.
> libsnappy loader issues
> -----------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.2.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8756) libsnappy loader issues
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8756:
-----------------------------------------
Description:
* We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. It seems like the System.loadLibrary nonsense is completely redundant-- we would only need this if we planned on accessing libsnappy methods directly from Java (via the 'native' keyword)- - which we don't, and can't, since snappy is not JNI. At the end of the day, snappy is only accessible through libhadoop.so, so I think snappy-related issues should be handled there.
* We should log the search path(s) we use for {{libsnappy.so}}, so that it's easier to diagnose configuration issues.
was:
* We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. It seems like the System.loadLibrary nonsense is completely redundant-- we would only need this if we planned on accessing libsnappy methods directly from Java (via the 'native' keyword)- - which we don't, and can't, since snappy is not JNI. At the end of the day, snappy is only accessible through libhadoop.so, so I think snappy-related issues should be handled there.
* We should log the search path(s) we use for {{libsnappy.so}}, so that it's easier to diagnose configuration issues.
* It seems like the static initialization code in the LoadSnappy.java class references the static initialization code in NativeCodeLoader. Basically, if the init method in NativeCodeLoader doesn't run before the init method in LoadSnappy, there could be problems. We should avoid having this kind of issue.
> libsnappy loader issues
> -----------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.2.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
>
> * We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. It seems like the System.loadLibrary nonsense is completely redundant-- we would only need this if we planned on accessing libsnappy methods directly from Java (via the 'native' keyword)- - which we don't, and can't, since snappy is not JNI. At the end of the day, snappy is only accessible through libhadoop.so, so I think snappy-related issues should be handled there.
> * We should log the search path(s) we use for {{libsnappy.so}}, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454333#comment-13454333 ]
Roman Shaposhnik commented on HADOOP-8756:
------------------------------------------
This seems to be related to the HADOOP-8781. Just to make sure -- is the patches posted to this JIRA meant to provide an alternative fix for the issue that won't require us changing the value of LD_LIBRARY_PATH ? If so, shouldn't your patches include the changes that would restore the old behavior?
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HADOOP-8756:
--------------------------------
Resolution: Fixed
Fix Version/s: 2.0.3-alpha
Target Version/s: (was: 2.0.2-alpha)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed this and merged to branch-2. Thanks Colin!
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468574#comment-13468574 ]
Hudson commented on HADOOP-8756:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk #1215 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1215/])
HADOOP-8756. Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH. Contributed by Colin Patrick McCabe (Revision 1393243)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1393243
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/config.h.cmake
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/SnappyCodec.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/LoadSnappy.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCodeLoader.c
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8756) libsnappy loader issues
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8756:
-----------------------------------------
Target Version/s: 2.2.0-alpha
Affects Version/s: 2.2.0-alpha
> libsnappy loader issues
> -----------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.2.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
>
> * We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. It seems like the System.loadLibrary nonsense is completely redundant-- we would only need this if we planned on accessing libsnappy methods directly from Java (via the 'native' keyword)- - which we don't, and can't, since snappy is not JNI. At the end of the day, snappy is only accessible through libhadoop.so, so I think snappy-related issues should be handled there.
> * We should log the search path(s) we use for {{libsnappy.so}}, so that it's easier to diagnose configuration issues.
> * It seems like the static initialization code in the LoadSnappy.java class references the static initialization code in NativeCodeLoader. Basically, if the init method in NativeCodeLoader doesn't run before the init method in LoadSnappy, there could be problems. We should avoid having this kind of issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454365#comment-13454365 ]
Roman Shaposhnik commented on HADOOP-8756:
------------------------------------------
bq. The patch could be revised to manually search java.library.path, I guess. Would that be worthwhile?
Well, my concern is around all the clients that don't use hadoop launcher script but need to access snappy codec *on the client* side. Flume would be a good example here: since it launches the JVM directly it has to also makes sure it sets up LD_LIBRARY_PATH if we don't provide the manual search capability in core hadoop itself.
I guess what I'm trying to say is that we have to have a solution for things like Flume. It would be nice if it worked automagically.
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468212#comment-13468212 ]
Hudson commented on HADOOP-8756:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #2803 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2803/])
HADOOP-8756. Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH. Contributed by Colin Patrick McCabe (Revision 1393243)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1393243
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/config.h.cmake
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/SnappyCodec.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/LoadSnappy.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCodeLoader.c
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455371#comment-13455371 ]
Colin Patrick McCabe commented on HADOOP-8756:
----------------------------------------------
Filed HADOOP-8806 to discuss searching {{java.library.path}}
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8756:
-----------------------------------------
Summary: Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH (was: libsnappy loader issues)
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454454#comment-13454454 ]
Colin Patrick McCabe commented on HADOOP-8756:
----------------------------------------------
Roman:
bq. It would be nice if it [searched java.library.path]
Let's file a separate JIRA for that. It's not really related to this JIRA, which is just about fixing a segfault. Smaller patches are easier for people to review, as well.
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454214#comment-13454214 ]
Andy Isaacson commented on HADOOP-8756:
---------------------------------------
{code}
@@ -295,4 +298,5 @@ public class SnappyCompressor implements Compressor {
private native static void initIDs();
private native int compressBytesDirect();
+
}
{code}
Unneeded whitespace change.
Other than that, the patch looks right and is a nice conceptual cleanup as well as fixing the SEGV. +1.
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468232#comment-13468232 ]
Hudson commented on HADOOP-8756:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2826 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2826/])
HADOOP-8756. Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH. Contributed by Colin Patrick McCabe (Revision 1393243)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1393243
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/config.h.cmake
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/SnappyCodec.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/LoadSnappy.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCodeLoader.c
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 2.0.3-alpha
>
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) libsnappy loader issues
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450132#comment-13450132 ]
Hadoop QA commented on HADOOP-8756:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12544127/HADOOP-8756.002.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverController
org.apache.hadoop.io.compress.TestCodec
org.apache.hadoop.io.compress.TestCodecFactory
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1418//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1418//console
This message is automatically generated.
> libsnappy loader issues
> -----------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.2.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465980#comment-13465980 ]
Andy Isaacson commented on HADOOP-8756:
---------------------------------------
+1
I've reviewed HADOOP-8756.004.patch and I have no further issues. This is a good localized fix that is not dependent on nor obsoleted by the other patches under discussion or checked in. Without this patch a local "mvn -Pnative,dist clean package -Dmaven.javadoc.skip=true -DskipTests; mvn test -Dtest=TestCodec" fails.
{code}
Running org.apache.hadoop.io.compress.TestCodec
Tests run: 21, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 15.956 sec <<< FAILURE!
Results :
Failed tests: testSnappyCodec(org.apache.hadoop.io.compress.TestCodec): Snappy native available but Hadoop native not
{code}
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468192#comment-13468192 ]
Eli Collins commented on HADOOP-8756:
-------------------------------------
+1 looks good to me as well
For those who haven't been following along, we're segfaulting because the compressor/decompressor ignore failures in initIDS, we now detect this failure and check for it in SnappyCodec so we don't run with snappy when we failed to load it. We still require that libsnappy is present in LD_LIBRARY_PATH (HADOOP-8806 improves how it is located).
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8756) Fix SEGV when libsnappy is in
java.library.path but not LD_LIBRARY_PATH
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454344#comment-13454344 ]
Colin Patrick McCabe commented on HADOOP-8756:
----------------------------------------------
bq. Just to make sure – is the patches posted to this JIRA meant to provide an alternative fix for the issue that won't require us changing the value of LD_LIBRARY_PATH
No, you will still need to set {{LD_LIBRARY_PATH}}. The patch could be revised to manually search {{java.library.path}}, I guess. Would that be worthwhile?
bq. If so, shouldn't your patches include the changes that would restore the old behavior?
The behavior hasn't changed. You always needed to have {{libsnappy.so}} in your {{LD_LIBRARY_PATH}} or system library path in order to load it with {{dlopen}}.
> Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
> -----------------------------------------------------------------------
>
> Key: HADOOP-8756
> URL: https://issues.apache.org/jira/browse/HADOOP-8756
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.2-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8756.002.patch, HADOOP-8756.003.patch, HADOOP-8756.004.patch
>
>
> We use {{System.loadLibrary("snappy")}} from the Java side. However in libhadoop, we use {{dlopen}} to open libsnappy.so dynamically. System.loadLibrary uses {{java.library.path}} to resolve libraries, and {{dlopen}} uses {{LD_LIBRARY_PATH}} and the system paths to resolve libraries. Because of this, the two library loading functions can be at odds.
> We should fix this so we only load the library once, preferably using the standard Java {{java.library.path}}.
> We should also log the search path(s) we use for {{libsnappy.so}} when loading fails, so that it's easier to diagnose configuration issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira