You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Colin Patrick McCabe (JIRA)" <ji...@apache.org> on 2012/10/09 02:14:02 UTC
[jira] [Created] (HADOOP-8901) GZip and Snappy support may not work
without unversioned libraries
Colin Patrick McCabe created HADOOP-8901:
--------------------------------------------
Summary: GZip and Snappy support may not work without unversioned libraries
Key: HADOOP-8901
URL: https://issues.apache.org/jira/browse/HADOOP-8901
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.3-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-8901.001.patch
Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
For libz, I believe we should simply link normally against it, rather than loading it using {{dlopen}}. It is part of the base Linux install for every distribution I'm aware of, so let's not overcomplicate things.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8901) GZip and Snappy support may not work
without unversioned libraries
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8901:
-----------------------------------------
Attachment: HADOOP-8901.001.patch
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> For libz, I believe we should simply link normally against it, rather than loading it using {{dlopen}}. It is part of the base Linux install for every distribution I'm aware of, so let's not overcomplicate things.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476336#comment-13476336 ]
Hudson commented on HADOOP-8901:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2884 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2884/])
HADOOP-8901. GZip and Snappy support may not work without unversioned libraries. Contributed by Colin Patrick McCabe. (Revision 1398416)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398416
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472011#comment-13472011 ]
Andy Isaacson commented on HADOOP-8901:
---------------------------------------
I agree with Todd, let's just fix the regression rather than adding zlib as a link-time dependency.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> For libz, I believe we should simply link normally against it, rather than loading it using {{dlopen}}. It is part of the base Linux install for every distribution I'm aware of, so let's not overcomplicate things.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8901) GZip and Snappy support may not work
without unversioned libraries
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Radim Kolar updated HADOOP-8901:
--------------------------------
Attachment: freebsd-so.txt
fix for finding libraries on freebsd. Tested and it works with both zlib and snappy.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: freebsd-so.txt, HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481866#comment-13481866 ]
Radim Kolar commented on HADOOP-8901:
-------------------------------------
on freebsd there is no /lib/libz.so and still cmake handles it nicely with find_package 1. This patch seems unnecessary, maybe bug reporter has outdated cmake.
ponto:(crawler)hadoop-common-project/hadoop-common>ls -l /lib/libz*
-r--r--r-- 1 root wheel 85328 Jun 24 12:41 /lib/libz.so.5
-r--r--r-- 1 root wheel 275040 Jun 24 12:41 /lib/libzfs.so.2
-r--r--r-- 1 root wheel 1347800 Jun 24 12:41 /lib/libzpool.so.2
ponto:(crawler)hadoop-common-project/hadoop-common>
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472191#comment-13472191 ]
Hadoop QA commented on HADOOP-8901:
-----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12548366/HADOOP-8901.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1587//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1587//console
This message is automatically generated.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Radim Kolar reopened HADOOP-8901:
---------------------------------
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472149#comment-13472149 ]
Colin Patrick McCabe commented on HADOOP-8901:
----------------------------------------------
Windows doesn't even have {{dlopen}} at all. Yet another reason to implement HADOOP-6069.
Anyway, I will add a platform check. I could take a stab at doing the right thing on Mac OS X, but someone else would need to confirm, because I don't have access to a Mac. It might be better to file a follow-on JIRA for that.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8901) GZip and Snappy support may not work
without unversioned libraries
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Isaacson updated HADOOP-8901:
----------------------------------
Description:
Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
Thanks to Brandon Vargo for reporting this bug.
was:
Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471989#comment-13471989 ]
Todd Lipcon commented on HADOOP-8901:
-------------------------------------
This seems to incorporate HADOOP-6069, which, the first time it was floated, had some objections. So rather than sneaking that change into this bug fix, we should have this just be a fix for the cmake regression, and use HADOOP-6069 if we want to make libz non-dynamic.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> For libz, I believe we should simply link normally against it, rather than loading it using {{dlopen}}. It is part of the base Linux install for every distribution I'm aware of, so let's not overcomplicate things.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471978#comment-13471978 ]
Colin Patrick McCabe commented on HADOOP-8901:
----------------------------------------------
I tested this patch on CentOS 5.8 and verified that I could run TestDFSIO without {{libz.so}} present (but with {{libz.so.1}} present), with {{mapred.map.output.compress.codec}} set to {{org.apache.hadoop.io.compress.GzipCodec}}.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> For libz, I believe we should simply link normally against it, rather than loading it using {{dlopen}}. It is part of the base Linux install for every distribution I'm aware of, so let's not overcomplicate things.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472019#comment-13472019 ]
Colin Patrick McCabe commented on HADOOP-8901:
----------------------------------------------
Tests are not provided because this is a build change.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8901) GZip and Snappy support may not work
without unversioned libraries
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8901:
-----------------------------------------
Attachment: HADOOP-8901.003.patch
Multi-platform-ish version.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481890#comment-13481890 ]
Andy Isaacson commented on HADOOP-8901:
---------------------------------------
bq. I'll test your suggestion on Linux to see if it has the needed behavior
I tested with
{noformat}
--- a/hadoop-common-project/hadoop-common/src/CMakeLists.txt
+++ b/hadoop-common-project/hadoop-common/src/CMakeLists.txt
@@ -55,7 +55,7 @@ if (NOT GENERATED_JAVAH)
MESSAGE(FATAL_ERROR "You must set the cmake variable GENERATED_JAVAH")
endif (NOT GENERATED_JAVAH)
find_package(JNI REQUIRED)
-find_package(ZLIB REQUIRED)
+find_package(ZLIB 1 REQUIRED)
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g -Wall -O2")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -D_REENTRANT -D_FILE_OFFSET_BITS=64")
{noformat}
and found that the resulting value for {{HADOOP_ZLIB_LIBRARY}} remained {{"libz.so"}}. This doesn't work on standard Linux installs because {{dlopen}} uses the string it is passed as an exact filename match, and without the {{-dev}} packages, Linux installations do not have {{libfoo.so}} symlinks installed.
Colin's fix which was committed on this Jira addresses the problem by ensuring that {{HADOOP_ZLIB_LIBRARY}} expands to a string that specifies the library ABI version, {{"libz.so.1"}}.
It seems like you've actually got the same problem on BSD, but the hardcoded "1" breaks because BSD considers zlib to have a larger ABI revision. Possibly this is related to an issue in the upstream Linux system where the zlib ABI number is not properly being incremented when the ABI changes -- certainly BSD has been much more careful / aware of such issues over the years.
I think the right fix is to figure out how to have CMake determine the correct ABI revision number at build time, no?
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Andy Isaacson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481868#comment-13481868 ]
Andy Isaacson commented on HADOOP-8901:
---------------------------------------
Radim,
Thanks for noting this issue! With {{find_package(ZLIB 1 REQUIRED)}} , what path does dlopen end up using on BSD to open the libz.so? If the pathname is {{"/usr/lib/libz.so"}} then there's a dependency on that symlink pointing to the correct library.
I'll test your suggestion on Linux to see if it has the needed behavior -- in order to support Linux systems without a {{"libz.so"}} symlink, we need to ensure that dlopen gets a pathname like {{"libz.so.1"}}.
Also, just out of curiosity, what BSD are you testing this on?
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473547#comment-13473547 ]
Colin Patrick McCabe commented on HADOOP-8901:
----------------------------------------------
The latest patch should address the portability concerns.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8901) GZip and Snappy support may not work
without unversioned libraries
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8901:
-----------------------------------------
Status: Patch Available (was: Open)
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> For libz, I believe we should simply link normally against it, rather than loading it using {{dlopen}}. It is part of the base Linux install for every distribution I'm aware of, so let's not overcomplicate things.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8901) GZip and Snappy support may not work
without unversioned libraries
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HADOOP-8901:
--------------------------------
Resolution: Fixed
Fix Version/s: 2.0.3-alpha
3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to branch-2 and trunk, thanks Colin.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482286#comment-13482286 ]
Hadoop QA commented on HADOOP-8901:
-----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12550445/freebsd-so.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1663//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1663//console
This message is automatically generated.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: freebsd-so.txt, HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476301#comment-13476301 ]
Hudson commented on HADOOP-8901:
--------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2923 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2923/])
HADOOP-8901. GZip and Snappy support may not work without unversioned libraries. Contributed by Colin Patrick McCabe. (Revision 1398416)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398416
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475345#comment-13475345 ]
Todd Lipcon commented on HADOOP-8901:
-------------------------------------
+1, will commit this this afternoon unless there are any comments.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472017#comment-13472017 ]
Hadoop QA commented on HADOOP-8901:
-----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12548334/HADOOP-8901.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1580//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1580//console
This message is automatically generated.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472045#comment-13472045 ]
Todd Lipcon commented on HADOOP-8901:
-------------------------------------
Can cmake currently build on Windows and Mac? If so, I think this would break that, since the Snappy and libz libraries wouldn't be .so.1 -- instead being .dll or .dylib or something.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482460#comment-13482460 ]
Radim Kolar commented on HADOOP-8901:
-------------------------------------
my bsd patch was committed as part of HADOOP-8811.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: freebsd-so.txt, HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8901) GZip and Snappy support may not work
without unversioned libraries
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Radim Kolar updated HADOOP-8901:
--------------------------------
Tags: native
Hadoop Flags: (was: Reviewed)
Status: Patch Available (was: Reopened)
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: freebsd-so.txt, HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481863#comment-13481863 ]
Radim Kolar commented on HADOOP-8901:
-------------------------------------
This patch broke it on BSD because there is /lib/libz.so.5, not so.1 as you expect. Why not simple use "find_package(ZLIB 1 REQUIRED)" ?
this works on bsd nicely: Found ZLIB: /usr/lib/libz.so (found suitable version "1.2.3", required is "1")
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482243#comment-13482243 ]
Radim Kolar commented on HADOOP-8901:
-------------------------------------
This fixes zlib on BSD but its still not enough to fix libsnappy. i could not determine why libsnappy is broken because version d2b46d18183ad70c0aad5ce278f49fbd52c88a08 works fine.
{noformat}
macro(set_find_shared_library_version LVERS)
IF(${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
# Mac OS uses .dylib
SET(CMAKE_FIND_LIBRARY_SUFFIXES ".${LVERS}.dylib")
ELSEIF(${CMAKE_SYSTEM_NAME} MATCHES "FreeBSD")
# FreeBSD has always .so installed.
ELSEIF(${CMAKE_SYSTEM_NAME} MATCHES "Windows")
# Windows doesn't support finding shared libraries by version.
ELSE()
# Most UNIX variants use .so
SET(CMAKE_FIND_LIBRARY_SUFFIXES ".so.${LVERS}")
ENDIF()
endmacro(set_find_shared_library_version LVERS)
{noformat}
on freebsd libsnappy is in /usr/local/lib and it has version 2
{noformat}
slush:(hduser)target/native>ls -l /usr/local/lib/libsnappy.*
-rw-r--r-- 1 root wheel 38336 Sep 18 00:17 /usr/local/lib/libsnappy.a
-rwxr-xr-x 1 root wheel 951 Sep 18 00:17 /usr/local/lib/libsnappy.la
lrwxr-xr-x 1 root wheel 14 Sep 18 00:17 /usr/local/lib/libsnappy.so -> libsnappy.so.2
-rwxr-xr-x 1 root wheel 33767 Sep 18 00:17 /usr/local/lib/libsnappy.so.2
{noformat}
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481884#comment-13481884 ]
Radim Kolar commented on HADOOP-8901:
-------------------------------------
just discovered that on bsd is a symlink: /usr/lib/libz.so -> /lib/libz.so.5
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481896#comment-13481896 ]
Radim Kolar commented on HADOOP-8901:
-------------------------------------
freebsd has always -dev packages installed because bsd do not splits include and other stuff out of base packages.
determining correct version number at runtime should be fine. another problem is with libsnappy, on bsd it have abi number 2 not 1
ls -l /usr/local/lib/libsnappy.*
-rw-r--r-- 1 root wheel 39776 Sep 14 13:48 /usr/local/lib/libsnappy.a
-rwxr-xr-x 1 root wheel 951 Sep 14 13:48 /usr/local/lib/libsnappy.la
lrwxr-xr-x 1 root wheel 14 Sep 14 13:48 /usr/local/lib/libsnappy.so -> libsnappy.so.2
-rwxr-xr-x 1 root wheel 37639 Sep 14 13:48 /usr/local/lib/libsnappy.so.2
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472038#comment-13472038 ]
Hadoop QA commented on HADOOP-8901:
-----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12548340/HADOOP-8901.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1582//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1582//console
This message is automatically generated.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476302#comment-13476302 ]
Hudson commented on HADOOP-8901:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #2861 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2861/])
HADOOP-8901. GZip and Snappy support may not work without unversioned libraries. Contributed by Colin Patrick McCabe. (Revision 1398416)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398416
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8901) GZip and Snappy support may not work
without unversioned libraries
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8901:
-----------------------------------------
Description:
Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
was:
Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
For libz, I believe we should simply link normally against it, rather than loading it using {{dlopen}}. It is part of the base Linux install for every distribution I'm aware of, so let's not overcomplicate things.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481881#comment-13481881 ]
Radim Kolar commented on HADOOP-8901:
-------------------------------------
I am testing on FreeBSD 8.3 64bit. There is no symlink as you can see zlibrary is sitting in /lib
in config.h on bsd is:
#define HADOOP_ZLIB_LIBRARY "libz.so"
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481920#comment-13481920 ]
Colin Patrick McCabe commented on HADOOP-8901:
----------------------------------------------
One way to do it would be to use something like {{get_filename_component(HADOOP_ZLIB_LIBRARY $\{HADOOP_ZLIB_LIBRARY\} REALPATH)}} to follow the symlinks all the way to the fully versioned library. Then you need to strip off the minor version, which is a small bit of text manipulation.
Another way would be to figure out what versions are in use on BSD and hardcode those. Arguably, depending on a specific version is a feature not a bug, since the major version number should only be bumped when there was an incompatible zlib ABI change that could (in theory) break our code.
If BSD users are convinced that the versionless variant will always be present, we could add an IF(BSD) or something to CMakeLists.txt and do it that way. Personally I do not have enough knowledge of BSD to say if the devel packages are always installed there-- seems strange, to me.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472000#comment-13472000 ]
Colin Patrick McCabe commented on HADOOP-8901:
----------------------------------------------
I also tested this with snappy.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> For libz, I believe we should simply link normally against it, rather than loading it using {{dlopen}}. It is part of the base Linux install for every distribution I'm aware of, so let's not overcomplicate things.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8901) GZip and Snappy support may not work
without unversioned libraries
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HADOOP-8901:
-----------------------------------------
Attachment: HADOOP-8901.002.patch
OK, here is a verison that doesn't remove the dlopen for libz.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> For libz, I believe we should simply link normally against it, rather than loading it using {{dlopen}}. It is part of the base Linux install for every distribution I'm aware of, so let's not overcomplicate things.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471981#comment-13471981 ]
Colin Patrick McCabe commented on HADOOP-8901:
----------------------------------------------
Note that this is a regression introduced by the conversion from automake to cmake.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Attachments: HADOOP-8901.001.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> For libz, I believe we should simply link normally against it, rather than loading it using {{dlopen}}. It is part of the base Linux install for every distribution I'm aware of, so let's not overcomplicate things.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477855#comment-13477855 ]
Hudson commented on HADOOP-8901:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk #1228 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1228/])
HADOOP-8901. GZip and Snappy support may not work without unversioned libraries. Contributed by Colin Patrick McCabe. (Revision 1398416)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1398416
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Colin Patrick McCabe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482483#comment-13482483 ]
Colin Patrick McCabe commented on HADOOP-8901:
----------------------------------------------
Thanks, Radim. And thanks for doing the work to get this stuff working on BSD.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: freebsd-so.txt, HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8901) GZip and Snappy support may not work
without unversioned libraries
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Radim Kolar updated HADOOP-8901:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: freebsd-so.txt, HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not
work without unversioned libraries
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482185#comment-13482185 ]
Radim Kolar commented on HADOOP-8901:
-------------------------------------
If (BSD) seems fine. its easiest to fix it that way.
> GZip and Snappy support may not work without unversioned libraries
> ------------------------------------------------------------------
>
> Key: HADOOP-8901
> URL: https://issues.apache.org/jira/browse/HADOOP-8901
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 2.0.3-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Priority: Minor
> Fix For: 3.0.0, 2.0.3-alpha
>
> Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, HADOOP-8901.003.patch
>
>
> Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get Gzip and Snappy support, respectively.
> However, this is not correct; we should be dlopening {{libsnappy.so.1}} instead. The versionless form of the shared library is not commonly installed except by development packages. Also, we may run into subtle compatibility problems if a new version of libsnappy comes out.
> Thanks to Brandon Vargo for reporting this bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira