You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Matt Foley (JIRA)" <ji...@apache.org> on 2012/11/20 22:46:59 UTC
[jira] [Updated] (HADOOP-7825) Hadoop wrapper script not picking up
native libs correctly
[ https://issues.apache.org/jira/browse/HADOOP-7825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-7825:
-------------------------------
Target Version/s: (was: 1.2.0)
> Hadoop wrapper script not picking up native libs correctly
> ----------------------------------------------------------
>
> Key: HADOOP-7825
> URL: https://issues.apache.org/jira/browse/HADOOP-7825
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
> Affects Versions: 0.20.205.0
> Environment: Debian 6.0 x64_64
> java version "1.6.0_26"
> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
> Reporter: stephen mulcahy
>
> Originally discussed in https://mail-archives.apache.org/mod_mbox/hadoop-common-user/201111.mbox/%3C4EC3A3AE.7060402%40deri.org%3E
> I'm testing out native lib support on our test amd64 test cluster
> running 0.20.205 running the following
> ./bin/hadoop jar hadoop-test-0.20.205.0.jar testsequencefile -seed 0
> -count 1000 -compressType RECORD xxx -codec
> org.apache.hadoop.io.compress.GzipCodec -check 2
> it fails with
> WARN util.NativeCodeLoader: Unable to load native-hadoop library for
> your platform... using builtin-java classes where applicable
> Looking at
> bin/hadoop
> it seems to successfully detect that the native libs are available (they
> seem to come pre-compiled with 0.20.205 which is nice)
> if [ -d "${HADOOP_HOME}/lib/native" ]; then
> if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
>
> JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:${HADOOP_HOME}/lib/native/${JAVA_PLATFORM}
> else
> JAVA_LIBRARY_PATH=${HADOOP_HOME}/lib/native/${JAVA_PLATFORM}
> fi
> fi
> and sets JAVA_LIBRARY_PATH to contain them.
> Then in the following line, if ${HADOOP_HOME}/lib contains libhadoop.a
> (which is seems to in the stock tar) then it proceeds to ignore the
> native libs
> if [ -e "${HADOOP_PREFIX}/lib/libhadoop.a" ]; then
> JAVA_LIBRARY_PATH=${HADOOP_PREFIX}/lib
> fi
> The libhadoop.a in ${HADOOP_HOME}/lib seems to be a copy of the lib/native/Linux-i386-32 going from the filesizes (and also noted by https://mail-archives.apache.org/mod_mbox/hadoop-common-user/201111.mbox/%3CCAOcnVr2AzUDnN0LFhmTqUmAyujYtvhfkmMm_J0R-bmxw2wU+9A@mail.gmail.com%3E)
> hadoop@testhbase01:~$ ls -la hadoop/lib/libhadoop.*
> -rw-r--r-- 1 hadoop hadoop 237244 Oct 7 08:20 hadoop/lib/libhadoop.a
> -rw-r--r-- 1 hadoop hadoop 877 Oct 7 08:20 hadoop/lib/libhadoop.la
> -rw-r--r-- 1 hadoop hadoop 160438 Oct 7 08:20 hadoop/lib/libhadoop.so
> -rw-r--r-- 1 hadoop hadoop 160438 Oct 7 08:19 hadoop/lib/libhadoop.so.1.0.0
> hadoop@testhbase01:~$ ls -la hadoop/lib/native/Linux-i386-32/
> total 728
> drwxr-xr-x 3 hadoop hadoop 4096 Nov 15 14:05 .
> drwxr-xr-x 5 hadoop hadoop 4096 Oct 7 08:24 ..
> -rw-r--r-- 1 hadoop hadoop 237244 Oct 7 08:20 libhadoop.a
> -rw-r--r-- 1 hadoop hadoop 877 Oct 7 08:20 libhadoop.la
> -rw-r--r-- 1 hadoop hadoop 160438 Oct 7 08:20 libhadoop.so
> -rw-r--r-- 1 hadoop hadoop 160438 Oct 7 08:20 libhadoop.so.1
> -rw-r--r-- 1 hadoop hadoop 160438 Oct 7 08:20 libhadoop.so.1.0.0
> A possible solution includes removing libhadoop.a and friends from ${HADOOP_HOME}/lib and possibly also modifying the hadoop wrapper to remove
> if [ -e "${HADOOP_PREFIX}/lib/libhadoop.a" ]; then
> JAVA_LIBRARY_PATH=${HADOOP_PREFIX}/lib
> fi
> unless there is some other reason for this to exist.
> This was also noted in a comment to HADOOP-6453
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira