You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Derek Brown <de...@media6degrees.com> on 2010/02/22 21:51:37 UTC
Native libraries on OS X
I was (seemingly) able to build the native libraries, but still get the
"unable to load native-hadoop library" message at run-time (with a simple
app of my own to read/write a gzip-compressed sequence file).
Based on https://issues.apache.org/jira/browse/HADOOP-3659, "Patch to allow
hadoop native to compile on Mac OS X", and other research, I did the
following:
$ cd hadoop-0.20.1/
$ export
LDFLAGS=-L/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Libraries
$ patch -p0 -b < HADOOP-3659.patch
$ cd src/native/
$ # To regenerate the configure script:
$ autoreconf -ifv
$ cd ../..
$ ant compile-native
$ ls -l build/native/Mac_OS_X-x86_64-64/lib
total 248
-rw-r--r-- 1 derek staff 14007 Feb 22 13:26 Makefile
lrwxr-xr-x 1 derek staff 17 Feb 22 13:26 libhadoop.1.0.0.dylib@ ->
libhadoop.1.dylib
-rwxr-xr-x 1 derek staff 8404 Feb 22 13:26 libhadoop.1.dylib*
-rw-r--r-- 1 derek staff 84880 Feb 22 13:26 libhadoop.a
lrwxr-xr-x 1 derek staff 17 Feb 22 13:26 libhadoop.dylib@ ->
libhadoop.1.dylib
-rw-r--r-- 1 derek staff 900 Feb 22 13:26 libhadoop.la
But the this still comes up when running my code:
10/02/22 15:10:29 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.IllegalArgumentException: SequenceFile
doesn't work with GzipCodec without native-hadoop code!
I'm running my code as follows:
$ hadoop jar my.jar com.media6.MyClass [args to MyClass here]
I'm on OS X 10.5.8 Leopard, with an Intel Core 2 Duo MacBook.
I didn't install any other libraries, which there are some references to in
the Jira issue, since the build succeeded.
hadoop.native.lib is true by default, but to be explicit, I added it to
core-site.xml anyway.
Thanks for any pointers,
Derek
Re: Native libraries on OS X
Posted by Alex Kozlov <al...@cloudera.com>.
Have you set java.library.path? Add "-*Djava.library.path=$HOME**/**
hadoop-0.20.1/build/native/Mac_OS_X-x86_64-64/lib"* to
mapred.child.java.opts.
On Mon, Feb 22, 2010 at 12:51 PM, Derek Brown <de...@media6degrees.com>wrote:
> I was (seemingly) able to build the native libraries, but still get the
> "unable to load native-hadoop library" message at run-time (with a simple
> app of my own to read/write a gzip-compressed sequence file).
> Based on https://issues.apache.org/jira/browse/HADOOP-3659, "Patch to
> allow
> hadoop native to compile on Mac OS X", and other research, I did the
> following:
>
> $ cd hadoop-0.20.1/
> $ export
>
> LDFLAGS=-L/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Libraries
> $ patch -p0 -b < HADOOP-3659.patch
> $ cd src/native/
> $ # To regenerate the configure script:
> $ autoreconf -ifv
> $ cd ../..
> $ ant compile-native
> $ ls -l build/native/Mac_OS_X-x86_64-64/lib
> total 248
> -rw-r--r-- 1 derek staff 14007 Feb 22 13:26 Makefile
> lrwxr-xr-x 1 derek staff 17 Feb 22 13:26 libhadoop.1.0.0.dylib@ ->
> libhadoop.1.dylib
> -rwxr-xr-x 1 derek staff 8404 Feb 22 13:26 libhadoop.1.dylib*
> -rw-r--r-- 1 derek staff 84880 Feb 22 13:26 libhadoop.a
> lrwxr-xr-x 1 derek staff 17 Feb 22 13:26 libhadoop.dylib@ ->
> libhadoop.1.dylib
> -rw-r--r-- 1 derek staff 900 Feb 22 13:26 libhadoop.la
>
> But the this still comes up when running my code:
>
> 10/02/22 15:10:29 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Exception in thread "main" java.lang.IllegalArgumentException: SequenceFile
> doesn't work with GzipCodec without native-hadoop code!
>
> I'm running my code as follows:
>
> $ hadoop jar my.jar com.media6.MyClass [args to MyClass here]
>
> I'm on OS X 10.5.8 Leopard, with an Intel Core 2 Duo MacBook.
>
> I didn't install any other libraries, which there are some references to in
> the Jira issue, since the build succeeded.
>
> hadoop.native.lib is true by default, but to be explicit, I added it to
> core-site.xml anyway.
>
> Thanks for any pointers,
> Derek
>
Re: Native libraries on OS X
Posted by Allen Wittenauer <aw...@linkedin.com>.
What happens if you use the test sequence file code:
bin/hadoop jar hadoop-0.20.1-test.jar testsequencefile -seed 0 -count 1000
-compressType RECORD blah.seq -codec org.apache.hadoop.io.compress.GzipCodec
-check
On 2/22/10 12:51 PM, "Derek Brown" <de...@media6degrees.com> wrote:
> I was (seemingly) able to build the native libraries, but still get the
> "unable to load native-hadoop library" message at run-time (with a simple
> app of my own to read/write a gzip-compressed sequence file).
> Based on https://issues.apache.org/jira/browse/HADOOP-3659, "Patch to allow
> hadoop native to compile on Mac OS X", and other research, I did the
> following:
>
> $ cd hadoop-0.20.1/
> $ export
> LDFLAGS=-L/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Libraries
> $ patch -p0 -b < HADOOP-3659.patch
> $ cd src/native/
> $ # To regenerate the configure script:
> $ autoreconf -ifv
> $ cd ../..
> $ ant compile-native
> $ ls -l build/native/Mac_OS_X-x86_64-64/lib
> total 248
> -rw-r--r-- 1 derek staff 14007 Feb 22 13:26 Makefile
> lrwxr-xr-x 1 derek staff 17 Feb 22 13:26 libhadoop.1.0.0.dylib@ ->
> libhadoop.1.dylib
> -rwxr-xr-x 1 derek staff 8404 Feb 22 13:26 libhadoop.1.dylib*
> -rw-r--r-- 1 derek staff 84880 Feb 22 13:26 libhadoop.a
> lrwxr-xr-x 1 derek staff 17 Feb 22 13:26 libhadoop.dylib@ ->
> libhadoop.1.dylib
> -rw-r--r-- 1 derek staff 900 Feb 22 13:26 libhadoop.la
>
> But the this still comes up when running my code:
>
> 10/02/22 15:10:29 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Exception in thread "main" java.lang.IllegalArgumentException: SequenceFile
> doesn't work with GzipCodec without native-hadoop code!
>
> I'm running my code as follows:
>
> $ hadoop jar my.jar com.media6.MyClass [args to MyClass here]
>
> I'm on OS X 10.5.8 Leopard, with an Intel Core 2 Duo MacBook.
>
> I didn't install any other libraries, which there are some references to in
> the Jira issue, since the build succeeded.
>
> hadoop.native.lib is true by default, but to be explicit, I added it to
> core-site.xml anyway.
>
> Thanks for any pointers,
> Derek