You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Travis Thompson (JIRA)" <ji...@apache.org> on 2014/03/15 00:28:44 UTC
[jira] [Created] (HADOOP-10409) Bzip2 error message isn't clear
Travis Thompson created HADOOP-10409:
----------------------------------------
Summary: Bzip2 error message isn't clear
Key: HADOOP-10409
URL: https://issues.apache.org/jira/browse/HADOOP-10409
Project: Hadoop Common
Issue Type: Improvement
Components: io
Affects Versions: 2.3.0
Reporter: Travis Thompson
If you compile hadoop without {{bzip2-devel}} installed (on RHEL), bzip2 doesn't get compiled into libhadoop, as is expected. This is not documented however and the error message thrown from {{hadoop checknative -a}} is not helpful.
{noformat}
[tthompso@eat1-hcl4060 bin]$ hadoop checknative -a
14/03/13 00:51:02 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
14/03/13 00:51:02 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /export/apps/hadoop/hadoop-2.3.0.li7-1-bin/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: true /usr/lib64/libsnappy.so.1
lz4: true revision:99
bzip2: false
14/03/13 00:51:02 INFO util.ExitUtil: Exiting with status 1
{noformat}
You can see that it wasn't compiled in here:
{noformat}
[mislam@eat1-hcl4060 ~]$ strings /export/apps/hadoop/latest/lib/native/libhadoop.so | grep initIDs
Java_org_apache_hadoop_io_compress_lz4_Lz4Compressor_initIDs
Java_org_apache_hadoop_io_compress_lz4_Lz4Decompressor_initIDs
Java_org_apache_hadoop_io_compress_snappy_SnappyCompressor_initIDs
Java_org_apache_hadoop_io_compress_snappy_SnappyDecompressor_initIDs
Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_initIDs
Java_org_apache_hadoop_io_compress_zlib_ZlibDecompressor_initIDs
{noformat}
After installing bzip2-devel and recompiling:
{noformat}
[tthompso@eat1-hcl4060 ~]$ hadoop checknative -a
14/03/14 23:00:08 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
14/03/14 23:00:08 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /export/apps/hadoop/hadoop-2.3.0.11-2-bin/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: true /usr/lib64/libsnappy.so.1
lz4: true revision:99
bzip2: true /lib64/libbz2.so.1
{noformat}
{noformat}
tthompso@esv4-hcl261:~/hadoop-common(li-2.3.0⚡) » strings ./hadoop-common-project/hadoop-common/target/native/target/usr/local/lib/libhadoop.so |grep initIDs
Java_org_apache_hadoop_io_compress_lz4_Lz4Compressor_initIDs
Java_org_apache_hadoop_io_compress_lz4_Lz4Decompressor_initIDs
Java_org_apache_hadoop_io_compress_snappy_SnappyCompressor_initIDs
Java_org_apache_hadoop_io_compress_snappy_SnappyDecompressor_initIDs
Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_initIDs
Java_org_apache_hadoop_io_compress_zlib_ZlibDecompressor_initIDs
Java_org_apache_hadoop_io_compress_bzip2_Bzip2Compressor_initIDs
Java_org_apache_hadoop_io_compress_bzip2_Bzip2Decompressor_initIDs
{noformat}
The error message thrown should hint that perhaps libhadoop wasn't compiled with the bzip2 headers installed. It would also be nice if compile time dependencies were documented somewhere... :)
--
This message was sent by Atlassian JIRA
(v6.2#6252)