You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by He Chen <ai...@gmail.com> on 2010/12/09 23:00:28 UTC
Jcuda on Hadoop
Hello everyone, I 've got a problem when I write some Jcuda program based on
Hadoop MapReduce. I use the jcudaUtill. The KernelLauncherSample can be
successfully executed on my worker node. However, When I submit a program
containing jcuda to Hadoop MapReduce. I got following errors. Any reply will
be appreciated! 10/12/09 15:41:39 INFO mapred.JobClient: Running job:
job_201012091523_0002 10/12/09 15:41:40 INFO mapred.JobClient: map 0% reduce
0% 10/12/09 15:41:53 INFO mapred.JobClient: Task Id :
attempt_201012091523_0002_m_000000_0, Status : FAILED Error: Could not load
native library attempt_201012091523_0002_m_000000_0: [GC
7402K->1594K(12096K), 0.0045650 secs] attempt_201012091523_0002_m_000000_0:
[GC 108666K->104610K(116800K), 0.0106860 secs]
attempt_201012091523_0002_m_000000_0: [Full GC 104610K->104276K(129856K),
0.0482530 secs] attempt_201012091523_0002_m_000000_0: Error while loading
native library with base name "JCudaDriver"
attempt_201012091523_0002_m_000000_0: Operating system name: Linux
attempt_201012091523_0002_m_000000_0: Architecture : amd64
attempt_201012091523_0002_m_000000_0: Architecture bit size: 64 10/12/09
15:42:00 INFO mapred.JobClient: Task Id :
attempt_201012091523_0002_m_000000_1, Status : FAILED Error: Could not load
native library attempt_201012091523_0002_m_000000_1: [GC
7373K->1573K(18368K), 0.0045230 secs] attempt_201012091523_0002_m_000000_1:
Error while loading native library with base name "JCudaDriver"
attempt_201012091523_0002_m_000000_1: Operating system name: Linux
attempt_201012091523_0002_m_000000_1: Architecture : amd64
attempt_201012091523_0002_m_000000_1: Architecture bit size: 64 It looks
like The jcuda library file can not be sucessfully loaded. Actually, I tried
many combinations. 1) I include all the jcuda library files in my
TaskTracker's classpath, and also include jcuda library file into my
mapreduce program. 2) TaskTracker's classpath w/o jcuda library, but my
program contains them 3) TaskTracker's classpath w/ jcuda librara, but my
program w/o them All of them report the same error above.
Re: Jcuda on Hadoop
Posted by He Chen <ai...@gmail.com>.
Thank you Mathias, It works
On Thu, Dec 9, 2010 at 6:26 PM, Mathias Herberts <mathias.herberts@gmail.com
> wrote:
> Put the native libs (.so) with the other common libs that hadoop
> already has (libhadoop, libcompression, ...). Or put them in a
> specific 'lib/native' in your job jar and set 'java.library.path' to
> 'job.local.dir'/../jars/lib/native in your MR job.
>
> Mathias.
>
> On Thu, Dec 9, 2010 at 23:17, He Chen <ai...@gmail.com> wrote:
> > I am still in the test stage. I mean I only start one JobTracker and one
> > TaskTracker.
> > I copied all jcuda.*.jar into HADOOP_HOME/lib/
> >
> > from the ps aux|grep java
> >
> > I can confirm the JT and TT processes all contain the jcuda.*.jar files
> >
> > On Thu, Dec 9, 2010 at 4:05 PM, Mathias Herberts <
> mathias.herberts@gmail.com
> >> wrote:
> >
> >> You need to have the native libs on all tasktrackers and have
> >> java.library.path correctly set.
> >> On Dec 9, 2010 11:01 PM, "He Chen" <ai...@gmail.com> wrote:
> >> > Hello everyone, I 've got a problem when I write some Jcuda program
> based
> >> on
> >> > Hadoop MapReduce. I use the jcudaUtill. The KernelLauncherSample can
> be
> >> > successfully executed on my worker node. However, When I submit a
> program
> >> > containing jcuda to Hadoop MapReduce. I got following errors. Any
> reply
> >> will
> >> > be appreciated! 10/12/09 15:41:39 INFO mapred.JobClient: Running job:
> >> > job_201012091523_0002 10/12/09 15:41:40 INFO mapred.JobClient: map 0%
> >> reduce
> >> > 0% 10/12/09 15:41:53 INFO mapred.JobClient: Task Id :
> >> > attempt_201012091523_0002_m_000000_0, Status : FAILED Error: Could not
> >> load
> >> > native library attempt_201012091523_0002_m_000000_0: [GC
> >> > 7402K->1594K(12096K), 0.0045650 secs]
> >> attempt_201012091523_0002_m_000000_0:
> >> > [GC 108666K->104610K(116800K), 0.0106860 secs]
> >> > attempt_201012091523_0002_m_000000_0: [Full GC
> 104610K->104276K(129856K),
> >> > 0.0482530 secs] attempt_201012091523_0002_m_000000_0: Error while
> loading
> >> > native library with base name "JCudaDriver"
> >> > attempt_201012091523_0002_m_000000_0: Operating system name: Linux
> >> > attempt_201012091523_0002_m_000000_0: Architecture : amd64
> >> > attempt_201012091523_0002_m_000000_0: Architecture bit size: 64
> 10/12/09
> >> > 15:42:00 INFO mapred.JobClient: Task Id :
> >> > attempt_201012091523_0002_m_000000_1, Status : FAILED Error: Could not
> >> load
> >> > native library attempt_201012091523_0002_m_000000_1: [GC
> >> > 7373K->1573K(18368K), 0.0045230 secs]
> >> attempt_201012091523_0002_m_000000_1:
> >> > Error while loading native library with base name "JCudaDriver"
> >> > attempt_201012091523_0002_m_000000_1: Operating system name: Linux
> >> > attempt_201012091523_0002_m_000000_1: Architecture : amd64
> >> > attempt_201012091523_0002_m_000000_1: Architecture bit size: 64 It
> looks
> >> > like The jcuda library file can not be sucessfully loaded. Actually, I
> >> tried
> >> > many combinations. 1) I include all the jcuda library files in my
> >> > TaskTracker's classpath, and also include jcuda library file into my
> >> > mapreduce program. 2) TaskTracker's classpath w/o jcuda library, but
> my
> >> > program contains them 3) TaskTracker's classpath w/ jcuda librara, but
> my
> >> > program w/o them All of them report the same error above.
> >>
> >
>
Re: Jcuda on Hadoop
Posted by Mathias Herberts <ma...@gmail.com>.
Put the native libs (.so) with the other common libs that hadoop
already has (libhadoop, libcompression, ...). Or put them in a
specific 'lib/native' in your job jar and set 'java.library.path' to
'job.local.dir'/../jars/lib/native in your MR job.
Mathias.
On Thu, Dec 9, 2010 at 23:17, He Chen <ai...@gmail.com> wrote:
> I am still in the test stage. I mean I only start one JobTracker and one
> TaskTracker.
> I copied all jcuda.*.jar into HADOOP_HOME/lib/
>
> from the ps aux|grep java
>
> I can confirm the JT and TT processes all contain the jcuda.*.jar files
>
> On Thu, Dec 9, 2010 at 4:05 PM, Mathias Herberts <mathias.herberts@gmail.com
>> wrote:
>
>> You need to have the native libs on all tasktrackers and have
>> java.library.path correctly set.
>> On Dec 9, 2010 11:01 PM, "He Chen" <ai...@gmail.com> wrote:
>> > Hello everyone, I 've got a problem when I write some Jcuda program based
>> on
>> > Hadoop MapReduce. I use the jcudaUtill. The KernelLauncherSample can be
>> > successfully executed on my worker node. However, When I submit a program
>> > containing jcuda to Hadoop MapReduce. I got following errors. Any reply
>> will
>> > be appreciated! 10/12/09 15:41:39 INFO mapred.JobClient: Running job:
>> > job_201012091523_0002 10/12/09 15:41:40 INFO mapred.JobClient: map 0%
>> reduce
>> > 0% 10/12/09 15:41:53 INFO mapred.JobClient: Task Id :
>> > attempt_201012091523_0002_m_000000_0, Status : FAILED Error: Could not
>> load
>> > native library attempt_201012091523_0002_m_000000_0: [GC
>> > 7402K->1594K(12096K), 0.0045650 secs]
>> attempt_201012091523_0002_m_000000_0:
>> > [GC 108666K->104610K(116800K), 0.0106860 secs]
>> > attempt_201012091523_0002_m_000000_0: [Full GC 104610K->104276K(129856K),
>> > 0.0482530 secs] attempt_201012091523_0002_m_000000_0: Error while loading
>> > native library with base name "JCudaDriver"
>> > attempt_201012091523_0002_m_000000_0: Operating system name: Linux
>> > attempt_201012091523_0002_m_000000_0: Architecture : amd64
>> > attempt_201012091523_0002_m_000000_0: Architecture bit size: 64 10/12/09
>> > 15:42:00 INFO mapred.JobClient: Task Id :
>> > attempt_201012091523_0002_m_000000_1, Status : FAILED Error: Could not
>> load
>> > native library attempt_201012091523_0002_m_000000_1: [GC
>> > 7373K->1573K(18368K), 0.0045230 secs]
>> attempt_201012091523_0002_m_000000_1:
>> > Error while loading native library with base name "JCudaDriver"
>> > attempt_201012091523_0002_m_000000_1: Operating system name: Linux
>> > attempt_201012091523_0002_m_000000_1: Architecture : amd64
>> > attempt_201012091523_0002_m_000000_1: Architecture bit size: 64 It looks
>> > like The jcuda library file can not be sucessfully loaded. Actually, I
>> tried
>> > many combinations. 1) I include all the jcuda library files in my
>> > TaskTracker's classpath, and also include jcuda library file into my
>> > mapreduce program. 2) TaskTracker's classpath w/o jcuda library, but my
>> > program contains them 3) TaskTracker's classpath w/ jcuda librara, but my
>> > program w/o them All of them report the same error above.
>>
>
Re: Jcuda on Hadoop
Posted by He Chen <ai...@gmail.com>.
Thank you so much， Mathias Herberts
This really helps
On Thu, Dec 9, 2010 at 4:17 PM, He Chen <ai...@gmail.com> wrote:
> I am still in the test stage. I mean I only start one JobTracker and one
> TaskTracker.
> I copied all jcuda.*.jar into HADOOP_HOME/lib/
>
> from the ps aux|grep java
>
> I can confirm the JT and TT processes all contain the jcuda.*.jar files
>
>
> On Thu, Dec 9, 2010 at 4:05 PM, Mathias Herberts <
> mathias.herberts@gmail.com> wrote:
>
>> You need to have the native libs on all tasktrackers and have
>> java.library.path correctly set.
>> On Dec 9, 2010 11:01 PM, "He Chen" <ai...@gmail.com> wrote:
>> > Hello everyone, I 've got a problem when I write some Jcuda program
>> based
>> on
>> > Hadoop MapReduce. I use the jcudaUtill. The KernelLauncherSample can be
>> > successfully executed on my worker node. However, When I submit a
>> program
>> > containing jcuda to Hadoop MapReduce. I got following errors. Any reply
>> will
>> > be appreciated! 10/12/09 15:41:39 INFO mapred.JobClient: Running job:
>> > job_201012091523_0002 10/12/09 15:41:40 INFO mapred.JobClient: map 0%
>> reduce
>> > 0% 10/12/09 15:41:53 INFO mapred.JobClient: Task Id :
>> > attempt_201012091523_0002_m_000000_0, Status : FAILED Error: Could not
>> load
>> > native library attempt_201012091523_0002_m_000000_0: [GC
>> > 7402K->1594K(12096K), 0.0045650 secs]
>> attempt_201012091523_0002_m_000000_0:
>> > [GC 108666K->104610K(116800K), 0.0106860 secs]
>> > attempt_201012091523_0002_m_000000_0: [Full GC
>> 104610K->104276K(129856K),
>> > 0.0482530 secs] attempt_201012091523_0002_m_000000_0: Error while
>> loading
>> > native library with base name "JCudaDriver"
>> > attempt_201012091523_0002_m_000000_0: Operating system name: Linux
>> > attempt_201012091523_0002_m_000000_0: Architecture : amd64
>> > attempt_201012091523_0002_m_000000_0: Architecture bit size: 64 10/12/09
>> > 15:42:00 INFO mapred.JobClient: Task Id :
>> > attempt_201012091523_0002_m_000000_1, Status : FAILED Error: Could not
>> load
>> > native library attempt_201012091523_0002_m_000000_1: [GC
>> > 7373K->1573K(18368K), 0.0045230 secs]
>> attempt_201012091523_0002_m_000000_1:
>> > Error while loading native library with base name "JCudaDriver"
>> > attempt_201012091523_0002_m_000000_1: Operating system name: Linux
>> > attempt_201012091523_0002_m_000000_1: Architecture : amd64
>> > attempt_201012091523_0002_m_000000_1: Architecture bit size: 64 It looks
>> > like The jcuda library file can not be sucessfully loaded. Actually, I
>> tried
>> > many combinations. 1) I include all the jcuda library files in my
>> > TaskTracker's classpath, and also include jcuda library file into my
>> > mapreduce program. 2) TaskTracker's classpath w/o jcuda library, but my
>> > program contains them 3) TaskTracker's classpath w/ jcuda librara, but
>> my
>> > program w/o them All of them report the same error above.
>>
>
>
Re: Jcuda on Hadoop
Posted by He Chen <ai...@gmail.com>.
I am still in the test stage. I mean I only start one JobTracker and one
TaskTracker.
I copied all jcuda.*.jar into HADOOP_HOME/lib/
from the ps aux|grep java
I can confirm the JT and TT processes all contain the jcuda.*.jar files
On Thu, Dec 9, 2010 at 4:05 PM, Mathias Herberts <mathias.herberts@gmail.com
> wrote:
> You need to have the native libs on all tasktrackers and have
> java.library.path correctly set.
> On Dec 9, 2010 11:01 PM, "He Chen" <ai...@gmail.com> wrote:
> > Hello everyone, I 've got a problem when I write some Jcuda program based
> on
> > Hadoop MapReduce. I use the jcudaUtill. The KernelLauncherSample can be
> > successfully executed on my worker node. However, When I submit a program
> > containing jcuda to Hadoop MapReduce. I got following errors. Any reply
> will
> > be appreciated! 10/12/09 15:41:39 INFO mapred.JobClient: Running job:
> > job_201012091523_0002 10/12/09 15:41:40 INFO mapred.JobClient: map 0%
> reduce
> > 0% 10/12/09 15:41:53 INFO mapred.JobClient: Task Id :
> > attempt_201012091523_0002_m_000000_0, Status : FAILED Error: Could not
> load
> > native library attempt_201012091523_0002_m_000000_0: [GC
> > 7402K->1594K(12096K), 0.0045650 secs]
> attempt_201012091523_0002_m_000000_0:
> > [GC 108666K->104610K(116800K), 0.0106860 secs]
> > attempt_201012091523_0002_m_000000_0: [Full GC 104610K->104276K(129856K),
> > 0.0482530 secs] attempt_201012091523_0002_m_000000_0: Error while loading
> > native library with base name "JCudaDriver"
> > attempt_201012091523_0002_m_000000_0: Operating system name: Linux
> > attempt_201012091523_0002_m_000000_0: Architecture : amd64
> > attempt_201012091523_0002_m_000000_0: Architecture bit size: 64 10/12/09
> > 15:42:00 INFO mapred.JobClient: Task Id :
> > attempt_201012091523_0002_m_000000_1, Status : FAILED Error: Could not
> load
> > native library attempt_201012091523_0002_m_000000_1: [GC
> > 7373K->1573K(18368K), 0.0045230 secs]
> attempt_201012091523_0002_m_000000_1:
> > Error while loading native library with base name "JCudaDriver"
> > attempt_201012091523_0002_m_000000_1: Operating system name: Linux
> > attempt_201012091523_0002_m_000000_1: Architecture : amd64
> > attempt_201012091523_0002_m_000000_1: Architecture bit size: 64 It looks
> > like The jcuda library file can not be sucessfully loaded. Actually, I
> tried
> > many combinations. 1) I include all the jcuda library files in my
> > TaskTracker's classpath, and also include jcuda library file into my
> > mapreduce program. 2) TaskTracker's classpath w/o jcuda library, but my
> > program contains them 3) TaskTracker's classpath w/ jcuda librara, but my
> > program w/o them All of them report the same error above.
>
Re: Jcuda on Hadoop
Posted by Mathias Herberts <ma...@gmail.com>.
You need to have the native libs on all tasktrackers and have
java.library.path correctly set.
On Dec 9, 2010 11:01 PM, "He Chen" <ai...@gmail.com> wrote:
> Hello everyone, I 've got a problem when I write some Jcuda program based
on
> Hadoop MapReduce. I use the jcudaUtill. The KernelLauncherSample can be
> successfully executed on my worker node. However, When I submit a program
> containing jcuda to Hadoop MapReduce. I got following errors. Any reply
will
> be appreciated! 10/12/09 15:41:39 INFO mapred.JobClient: Running job:
> job_201012091523_0002 10/12/09 15:41:40 INFO mapred.JobClient: map 0%
reduce
> 0% 10/12/09 15:41:53 INFO mapred.JobClient: Task Id :
> attempt_201012091523_0002_m_000000_0, Status : FAILED Error: Could not
load
> native library attempt_201012091523_0002_m_000000_0: [GC
> 7402K->1594K(12096K), 0.0045650 secs]
attempt_201012091523_0002_m_000000_0:
> [GC 108666K->104610K(116800K), 0.0106860 secs]
> attempt_201012091523_0002_m_000000_0: [Full GC 104610K->104276K(129856K),
> 0.0482530 secs] attempt_201012091523_0002_m_000000_0: Error while loading
> native library with base name "JCudaDriver"
> attempt_201012091523_0002_m_000000_0: Operating system name: Linux
> attempt_201012091523_0002_m_000000_0: Architecture : amd64
> attempt_201012091523_0002_m_000000_0: Architecture bit size: 64 10/12/09
> 15:42:00 INFO mapred.JobClient: Task Id :
> attempt_201012091523_0002_m_000000_1, Status : FAILED Error: Could not
load
> native library attempt_201012091523_0002_m_000000_1: [GC
> 7373K->1573K(18368K), 0.0045230 secs]
attempt_201012091523_0002_m_000000_1:
> Error while loading native library with base name "JCudaDriver"
> attempt_201012091523_0002_m_000000_1: Operating system name: Linux
> attempt_201012091523_0002_m_000000_1: Architecture : amd64
> attempt_201012091523_0002_m_000000_1: Architecture bit size: 64 It looks
> like The jcuda library file can not be sucessfully loaded. Actually, I
tried
> many combinations. 1) I include all the jcuda library files in my
> TaskTracker's classpath, and also include jcuda library file into my
> mapreduce program. 2) TaskTracker's classpath w/o jcuda library, but my
> program contains them 3) TaskTracker's classpath w/ jcuda librara, but my
> program w/o them All of them report the same error above.