You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mimi Sun <mi...@rapleaf.com> on 2009/02/10 20:06:40 UTC

Loading native libraries

Hi,

I'm new to Hadoop and I'm wondering what the recommended method is for  
using native libraries in mapred jobs.
I've tried the following separately:
1. set LD_LIBRARY_PATH in .bashrc
2. set LD_LIBRARY_PATH and  JAVA_LIBRARY_PATH in hadoop-env.sh
3. set -Djava.library.path=... for mapred.child.java.opts
4. change bin/hadoop to include  $LD_LIBRARY_PATH in addition to the  
path it generates:  HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path= 
$LD_LIBRARY_PATH:$JAVA_LIBRARY_PATH"
5. drop the .so files I need into hadoop/lib/native/...

1~3 didn't work, 4 and 5 did but seem to be hacks. I also read that I  
can do this using DistributedCache, but that seems to be extra work  
for loading libraries that are already present on each machine. (I'm  
using the JNI libs for berkeley db).
It seems that there should be a way to configure java.library.path for  
the mapred jobs.  Perhaps bin/hadoop should make use of LD_LIBRARY_PATH?

Thanks,
- Mimi

Re: Loading native libraries

Posted by Arun C Murthy <ac...@yahoo-inc.com>.
On Feb 10, 2009, at 12:24 PM, Mimi Sun wrote:

> I see UnsatisfiedLinkError.  Also I'm calling   
> System.getProperty("java.library.path") in the reducer and logging  
> it. The only thing that prints out is ...hadoop-0.18.2/bin/../lib/ 
> native/Mac_OS_X-i386-32
> I'm using Cascading, not sure if that affects anything.
>

Hmm... that's odd. The framework does try to pass the user provided  
java.library.path down to the launched jvm. I assume your  
mapred.child.java.opts looks something like
"-Xmx <heapsize> -Djava.library.path=<path>" ?

Arun


Re: Loading native libraries

Posted by Rasit OZDAS <ra...@gmail.com>.
I have also the same problem.
It would be wonderful if someone has some info about this..

Rasit

2009/2/10 Mimi Sun <mi...@rapleaf.com>:
> I see UnsatisfiedLinkError.  Also I'm calling
>  System.getProperty("java.library.path") in the reducer and logging it. The
> only thing that prints out is
> ...hadoop-0.18.2/bin/../lib/native/Mac_OS_X-i386-32
> I'm using Cascading, not sure if that affects anything.
>
> - Mimi
>
> On Feb 10, 2009, at 11:40 AM, Arun C Murthy wrote:
>
>>
>> On Feb 10, 2009, at 11:06 AM, Mimi Sun wrote:
>>
>>> Hi,
>>>
>>> I'm new to Hadoop and I'm wondering what the recommended method is for
>>> using native libraries in mapred jobs.
>>> I've tried the following separately:
>>> 1. set LD_LIBRARY_PATH in .bashrc
>>> 2. set LD_LIBRARY_PATH and  JAVA_LIBRARY_PATH in hadoop-env.sh
>>> 3. set -Djava.library.path=... for mapred.child.java.opts
>>
>> For what you are trying (i.e. given that the JNI libs are present on all
>> machines at a constant path) setting -Djava.library.path for the child task
>> via mapred.child.java.opts should work. What are you seeing?
>>
>> Arun
>>
>>>
>>> 4. change bin/hadoop to include  $LD_LIBRARY_PATH in addition to the path
>>> it generates:  HADOOP_OPTS="$HADOOP_OPTS
>>> -Djava.library.path=$LD_LIBRARY_PATH:$JAVA_LIBRARY_PATH"
>>> 5. drop the .so files I need into hadoop/lib/native/...
>>>
>>> 1~3 didn't work, 4 and 5 did but seem to be hacks. I also read that I can
>>> do this using DistributedCache, but that seems to be extra work for loading
>>> libraries that are already present on each machine. (I'm using the JNI libs
>>> for berkeley db).
>>> It seems that there should be a way to configure java.library.path for
>>> the mapred jobs.  Perhaps bin/hadoop should make use of LD_LIBRARY_PATH?
>>>
>>> Thanks,
>>> - Mimi
>>
>
>



-- 
M. Raşit ÖZDAŞ

Re: Loading native libraries

Posted by Mimi Sun <mi...@rapleaf.com>.
I see UnsatisfiedLinkError.  Also I'm calling   
System.getProperty("java.library.path") in the reducer and logging it.  
The only thing that prints out is ...hadoop-0.18.2/bin/../lib/native/ 
Mac_OS_X-i386-32
I'm using Cascading, not sure if that affects anything.

- Mimi

On Feb 10, 2009, at 11:40 AM, Arun C Murthy wrote:

>
> On Feb 10, 2009, at 11:06 AM, Mimi Sun wrote:
>
>> Hi,
>>
>> I'm new to Hadoop and I'm wondering what the recommended method is  
>> for using native libraries in mapred jobs.
>> I've tried the following separately:
>> 1. set LD_LIBRARY_PATH in .bashrc
>> 2. set LD_LIBRARY_PATH and  JAVA_LIBRARY_PATH in hadoop-env.sh
>> 3. set -Djava.library.path=... for mapred.child.java.opts
>
> For what you are trying (i.e. given that the JNI libs are present on  
> all machines at a constant path) setting -Djava.library.path for the  
> child task via mapred.child.java.opts should work. What are you  
> seeing?
>
> Arun
>
>>
>> 4. change bin/hadoop to include  $LD_LIBRARY_PATH in addition to  
>> the path it generates:  HADOOP_OPTS="$HADOOP_OPTS - 
>> Djava.library.path=$LD_LIBRARY_PATH:$JAVA_LIBRARY_PATH"
>> 5. drop the .so files I need into hadoop/lib/native/...
>>
>> 1~3 didn't work, 4 and 5 did but seem to be hacks. I also read that  
>> I can do this using DistributedCache, but that seems to be extra  
>> work for loading libraries that are already present on each  
>> machine. (I'm using the JNI libs for berkeley db).
>> It seems that there should be a way to configure java.library.path  
>> for the mapred jobs.  Perhaps bin/hadoop should make use of  
>> LD_LIBRARY_PATH?
>>
>> Thanks,
>> - Mimi
>


Re: Loading native libraries

Posted by Arun C Murthy <ac...@yahoo-inc.com>.
On Feb 10, 2009, at 11:06 AM, Mimi Sun wrote:

> Hi,
>
> I'm new to Hadoop and I'm wondering what the recommended method is  
> for using native libraries in mapred jobs.
> I've tried the following separately:
> 1. set LD_LIBRARY_PATH in .bashrc
> 2. set LD_LIBRARY_PATH and  JAVA_LIBRARY_PATH in hadoop-env.sh
> 3. set -Djava.library.path=... for mapred.child.java.opts

For what you are trying (i.e. given that the JNI libs are present on  
all machines at a constant path) setting -Djava.library.path for the  
child task via mapred.child.java.opts should work. What are you seeing?

Arun

>
> 4. change bin/hadoop to include  $LD_LIBRARY_PATH in addition to the  
> path it generates:  HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path= 
> $LD_LIBRARY_PATH:$JAVA_LIBRARY_PATH"
> 5. drop the .so files I need into hadoop/lib/native/...
>
> 1~3 didn't work, 4 and 5 did but seem to be hacks. I also read that  
> I can do this using DistributedCache, but that seems to be extra  
> work for loading libraries that are already present on each machine.  
> (I'm using the JNI libs for berkeley db).
> It seems that there should be a way to configure java.library.path  
> for the mapred jobs.  Perhaps bin/hadoop should make use of  
> LD_LIBRARY_PATH?
>
> Thanks,
> - Mimi