You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by edward choi <mp...@gmail.com> on 2010/06/07 04:57:16 UTC
Hadoop cannot find my mapper class
Hi, I am trying to run a JNI application on Hadoop and I keep getting an
error message that my Mapper class cannot be found.
I have a dynamic library(libetri_lmi.so).
A java file that calls the library(freeparser.java). Freeparser.java
declares native functions.
Another java file which is the hadoop application(LangAnal.java).
LangAnal.java imports freeparser.java.
LangAnal.java has one Mapper function and zero Reducer function.
I have compiled freeparser.java and LangAnal.java.
I got three classes (freeparser.class, LangAnal.class,
LangAnal$LangAnalMapper.class)
I have archived the three classes into one jar file(LangAnal.jar)
Then I run
Hadoop jar ./LangAnal.jar org.etri.mp2893.LangAnal ./input ./output
Then I get the following message.
I am definitely sure that my Mapper class is in the jar file.
I did not get this message before. So I tried formating the namenode, but
didn't work.
What can I be doing wrong??
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10/06/07 10:31:15 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
10/06/07 10:31:15 WARN mapred.JobClient: No job jar file set. User classes
may not be found. See JobConf(Class) or JobConf#setJar(String).
10/06/07 10:31:15 INFO input.FileInputFormat: Total input paths to process :
1
10/06/07 10:31:16 INFO mapred.JobClient: Running job: job_201006071023_0003
10/06/07 10:31:17 INFO mapred.JobClient: map 0% reduce 0%
10/06/07 10:31:27 INFO mapred.JobClient: Task Id :
attempt_201006071023_0003_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.etri.mp2893.LangAnal$LangAnalMapper
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at
org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException:
org.etri.mp2893.LangAnal$LangAnalMapper
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 4 more
10/06/07 10:31:33 INFO mapred.JobClient: Task Id :
attempt_201006071023_0003_m_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.etri.mp2893.LangAnal$LangAnalMapper
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at
org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException:
org.etri.mp2893.LangAnal$LangAnalMapper
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 4 more
10/06/07 10:31:39 INFO mapred.JobClient: Task Id :
attempt_201006071023_0003_m_000000_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.etri.mp2893.LangAnal$LangAnalMapper
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at
org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException:
org.etri.mp2893.LangAnal$LangAnalMapper
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
... 4 more
10/06/07 10:31:48 INFO mapred.JobClient: Job complete: job_201006071023_0003
10/06/07 10:31:48 INFO mapred.JobClient: Counters: 4
10/06/07 10:31:48 INFO mapred.JobClient: Job Counters
10/06/07 10:31:48 INFO mapred.JobClient: Rack-local map tasks=2
10/06/07 10:31:48 INFO mapred.JobClient: Launched map tasks=4
10/06/07 10:31:48 INFO mapred.JobClient: Data-local map tasks=2
10/06/07 10:31:48 INFO mapred.JobClient: Failed map tasks=1
RE: Hadoop cannot find my mapper class
Posted by "Buttler, David" <bu...@llnl.gov>.
According to your output, you did not specify the jar to load the class from, e.g.:
job.setJarByClass(BaseReconcileAnnotation.class);
Also, you need to be sure that the native library is on all nodes of your cluster as it won't be automatically copied by Hadoop to each node.
Dave
-----Original Message-----
From: edward choi [mailto:mp2893@gmail.com]
Sent: Sunday, June 06, 2010 10:15 PM
To: common-user@hadoop.apache.org
Subject: Re: Hadoop cannot find my mapper class
I have solved this problem.
I used to declare the package of LangAnal.java as "org.etri.mp2893"
But when I changed it to "org.myorg", the problem was gone.
This is really baffling.
Does anybody have any idea why this happens?
2010/6/7 edward choi <mp...@gmail.com>
> Hi, I am trying to run a JNI application on Hadoop and I keep getting an
> error message that my Mapper class cannot be found.
>
> I have a dynamic library(libetri_lmi.so).
> A java file that calls the library(freeparser.java). Freeparser.java
> declares native functions.
> Another java file which is the hadoop application(LangAnal.java).
>
> LangAnal.java imports freeparser.java.
> LangAnal.java has one Mapper function and zero Reducer function.
>
> I have compiled freeparser.java and LangAnal.java.
> I got three classes (freeparser.class, LangAnal.class,
> LangAnal$LangAnalMapper.class)
> I have archived the three classes into one jar file(LangAnal.jar)
>
> Then I run
> Hadoop jar ./LangAnal.jar org.etri.mp2893.LangAnal ./input ./output
>
> Then I get the following message.
>
> I am definitely sure that my Mapper class is in the jar file.
> I did not get this message before. So I tried formating the namenode, but
> didn't work.
> What can I be doing wrong??
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> 10/06/07 10:31:15 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 10/06/07 10:31:15 WARN mapred.JobClient: No job jar file set. User classes
> may not be found. See JobConf(Class) or JobConf#setJar(String).
> 10/06/07 10:31:15 INFO input.FileInputFormat: Total input paths to process
> : 1
> 10/06/07 10:31:16 INFO mapred.JobClient: Running job: job_201006071023_0003
> 10/06/07 10:31:17 INFO mapred.JobClient: map 0% reduce 0%
> 10/06/07 10:31:27 INFO mapred.JobClient: Task Id :
> attempt_201006071023_0003_m_000000_0, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.etri.mp2893.LangAnal$LangAnalMapper
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
> at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.ClassNotFoundException:
> org.etri.mp2893.LangAnal$LangAnalMapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
> ... 4 more
>
> 10/06/07 10:31:33 INFO mapred.JobClient: Task Id :
> attempt_201006071023_0003_m_000000_1, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.etri.mp2893.LangAnal$LangAnalMapper
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
> at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.ClassNotFoundException:
> org.etri.mp2893.LangAnal$LangAnalMapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
> ... 4 more
>
> 10/06/07 10:31:39 INFO mapred.JobClient: Task Id :
> attempt_201006071023_0003_m_000000_2, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.etri.mp2893.LangAnal$LangAnalMapper
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
> at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.ClassNotFoundException:
> org.etri.mp2893.LangAnal$LangAnalMapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
> ... 4 more
>
> 10/06/07 10:31:48 INFO mapred.JobClient: Job complete:
> job_201006071023_0003
> 10/06/07 10:31:48 INFO mapred.JobClient: Counters: 4
> 10/06/07 10:31:48 INFO mapred.JobClient: Job Counters
> 10/06/07 10:31:48 INFO mapred.JobClient: Rack-local map tasks=2
> 10/06/07 10:31:48 INFO mapred.JobClient: Launched map tasks=4
> 10/06/07 10:31:48 INFO mapred.JobClient: Data-local map tasks=2
> 10/06/07 10:31:48 INFO mapred.JobClient: Failed map tasks=1
>
Re: Hadoop cannot find my mapper class
Posted by edward choi <mp...@gmail.com>.
I have solved this problem.
I used to declare the package of LangAnal.java as "org.etri.mp2893"
But when I changed it to "org.myorg", the problem was gone.
This is really baffling.
Does anybody have any idea why this happens?
2010/6/7 edward choi <mp...@gmail.com>
> Hi, I am trying to run a JNI application on Hadoop and I keep getting an
> error message that my Mapper class cannot be found.
>
> I have a dynamic library(libetri_lmi.so).
> A java file that calls the library(freeparser.java). Freeparser.java
> declares native functions.
> Another java file which is the hadoop application(LangAnal.java).
>
> LangAnal.java imports freeparser.java.
> LangAnal.java has one Mapper function and zero Reducer function.
>
> I have compiled freeparser.java and LangAnal.java.
> I got three classes (freeparser.class, LangAnal.class,
> LangAnal$LangAnalMapper.class)
> I have archived the three classes into one jar file(LangAnal.jar)
>
> Then I run
> Hadoop jar ./LangAnal.jar org.etri.mp2893.LangAnal ./input ./output
>
> Then I get the following message.
>
> I am definitely sure that my Mapper class is in the jar file.
> I did not get this message before. So I tried formating the namenode, but
> didn't work.
> What can I be doing wrong??
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> 10/06/07 10:31:15 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 10/06/07 10:31:15 WARN mapred.JobClient: No job jar file set. User classes
> may not be found. See JobConf(Class) or JobConf#setJar(String).
> 10/06/07 10:31:15 INFO input.FileInputFormat: Total input paths to process
> : 1
> 10/06/07 10:31:16 INFO mapred.JobClient: Running job: job_201006071023_0003
> 10/06/07 10:31:17 INFO mapred.JobClient: map 0% reduce 0%
> 10/06/07 10:31:27 INFO mapred.JobClient: Task Id :
> attempt_201006071023_0003_m_000000_0, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.etri.mp2893.LangAnal$LangAnalMapper
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
> at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.ClassNotFoundException:
> org.etri.mp2893.LangAnal$LangAnalMapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
> ... 4 more
>
> 10/06/07 10:31:33 INFO mapred.JobClient: Task Id :
> attempt_201006071023_0003_m_000000_1, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.etri.mp2893.LangAnal$LangAnalMapper
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
> at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.ClassNotFoundException:
> org.etri.mp2893.LangAnal$LangAnalMapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
> ... 4 more
>
> 10/06/07 10:31:39 INFO mapred.JobClient: Task Id :
> attempt_201006071023_0003_m_000000_2, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.etri.mp2893.LangAnal$LangAnalMapper
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
> at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.ClassNotFoundException:
> org.etri.mp2893.LangAnal$LangAnalMapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:807)
> ... 4 more
>
> 10/06/07 10:31:48 INFO mapred.JobClient: Job complete:
> job_201006071023_0003
> 10/06/07 10:31:48 INFO mapred.JobClient: Counters: 4
> 10/06/07 10:31:48 INFO mapred.JobClient: Job Counters
> 10/06/07 10:31:48 INFO mapred.JobClient: Rack-local map tasks=2
> 10/06/07 10:31:48 INFO mapred.JobClient: Launched map tasks=4
> 10/06/07 10:31:48 INFO mapred.JobClient: Data-local map tasks=2
> 10/06/07 10:31:48 INFO mapred.JobClient: Failed map tasks=1
>