You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Subramanian, Hema " <he...@citi.com> on 2012/01/30 23:37:18 UTC
Re: ClassNotFound just started with custom mapper
I am facing issues while trying to run a job from windows (through eclipse) on my hadoop cluster on my RHEL VM's. When I run it as "run on hadoop" it works fine, but when I run it as a java application, it throws classnotfound exception
INFO: Task Id : attempt_201201101527_0037_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: TestHadoop$Map
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:195)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:718)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Below is the stub:
public class TestHadoop extends Configured {
public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
context.write(word, one);
}
}
}
public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val:values) {
sum += val.get();
}
context.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration(true);
conf.set("fs.default.name","hdfs://vm-acd2-4c51:54310/");
conf.set("mapred.job.tracker","hdfs://vm-acd2-4c51:54311/");
conf.set("mapreduce.jobtracker.staging.root.dir","/app/hadoop/mapred/staging");
Job jobconf = new Job(conf,"TestHadoop");
jobconf.setJarByClass(TestHadoop.class);
jobconf.setOutputKeyClass(Text.class);
jobconf.setOutputValueClass(IntWritable.class);
jobconf.setMapperClass(Map.class);
jobconf.setCombinerClass(Reduce.class);
jobconf.setReducerClass(Reduce.class);
jobconf.setInputFormatClass(TextInputFormat.class);
jobconf.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.setInputPaths(jobconf, new Path("/tmp/Hadoop_Temp_Data/Input/"));
FileOutputFormat.setOutputPath(jobconf, new Path("/tmp/Hadoop_Temp_Data/Output1/"));
jobconf.waitForCompletion(true);
}
}
Any help will be greatly appreciated!
Thanks
Hema Subramanian
Re: ClassNotFound just started with custom mapper
Posted by Anil Gupta <an...@gmail.com>.
Hi Hema,
I had set-up a Hadoop cluster in which the name has hyphen character and it works fine. So, it don't think this problem is related to hyphen character.
The problem is related to your Hadoop classpath settings. So, check your Hadoop classpath.
I don't have experience of running the job from windows but here is what you can do to run the jar itself.
Copy the jar on master node and Use the Hadoop jar command to run the jar.
Best Regards,
Anil
On Jan 30, 2012, at 9:52 PM, hadoop hive <ha...@gmail.com> wrote:
> hey Hema,
>
> I m not sure but the problem is with you hdfs name
> *hdfs://vm-acd2-4c51:54310/ ,
> *change you host name and it ll run fine. specially remove "-" from
> hostname.
> regards
>
> Vikas Srivastava
>
> On Tue, Jan 31, 2012 at 4:07 AM, Subramanian, Hema <
> hema.subramanian@citi.com> wrote:
>
>> I am facing issues while trying to run a job from windows (through
>> eclipse) on my hadoop cluster on my RHEL VM's. When I run it as "run on
>> hadoop" it works fine, but when I run it as a java application, it throws
>> classnotfound exception
>>
>> INFO: Task Id : attempt_201201101527_0037_m_000000_0, Status : FAILED
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> TestHadoop$Map
>> at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
>> at
>> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:195)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:718)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>>
>> Below is the stub:
>>
>> public class TestHadoop extends Configured {
>>
>> public static class Map extends Mapper<LongWritable, Text, Text,
>> IntWritable> {
>> private final static IntWritable one = new IntWritable(1);
>> private Text word = new Text();
>>
>> public void map(LongWritable key, Text value, Context
>> context) throws IOException, InterruptedException {
>> String line = value.toString();
>> StringTokenizer tokenizer = new
>> StringTokenizer(line);
>> while (tokenizer.hasMoreTokens()) {
>> word.set(tokenizer.nextToken());
>> context.write(word, one);
>>
>> }
>> }
>> }
>>
>> public static class Reduce extends Reducer<Text, IntWritable, Text,
>> IntWritable> {
>> public void reduce(Text key, Iterable<IntWritable> values,
>> Context context) throws IOException, InterruptedException {
>> int sum = 0;
>> for (IntWritable val:values) {
>> sum += val.get();
>> }
>> context.write(key, new IntWritable(sum));
>> }
>> }
>>
>> public static void main(String[] args) throws Exception {
>> Configuration conf = new Configuration(true);
>> conf.set("fs.default.name","hdfs://vm-acd2-4c51:54310/");
>> conf.set("mapred.job.tracker","hdfs://vm-acd2-4c51:54311/");
>>
>> conf.set("mapreduce.jobtracker.staging.root.dir","/app/hadoop/mapred/staging");
>> Job jobconf = new Job(conf,"TestHadoop");
>> jobconf.setJarByClass(TestHadoop.class);
>> jobconf.setOutputKeyClass(Text.class);
>> jobconf.setOutputValueClass(IntWritable.class);
>>
>> jobconf.setMapperClass(Map.class);
>> jobconf.setCombinerClass(Reduce.class);
>> jobconf.setReducerClass(Reduce.class);
>>
>> jobconf.setInputFormatClass(TextInputFormat.class);
>> jobconf.setOutputFormatClass(TextOutputFormat.class);
>>
>> FileInputFormat.setInputPaths(jobconf, new
>> Path("/tmp/Hadoop_Temp_Data/Input/"));
>> FileOutputFormat.setOutputPath(jobconf, new
>> Path("/tmp/Hadoop_Temp_Data/Output1/"));
>> jobconf.waitForCompletion(true);
>> }
>>
>> }
>>
>> Any help will be greatly appreciated!
>>
>> Thanks
>> Hema Subramanian
>>
Re: ClassNotFound just started with custom mapper
Posted by hadoop hive <ha...@gmail.com>.
hey Hema,
I m not sure but the problem is with you hdfs name
*hdfs://vm-acd2-4c51:54310/ ,
*change you host name and it ll run fine. specially remove "-" from
hostname.
regards
Vikas Srivastava
On Tue, Jan 31, 2012 at 4:07 AM, Subramanian, Hema <
hema.subramanian@citi.com> wrote:
> I am facing issues while trying to run a job from windows (through
> eclipse) on my hadoop cluster on my RHEL VM's. When I run it as "run on
> hadoop" it works fine, but when I run it as a java application, it throws
> classnotfound exception
>
> INFO: Task Id : attempt_201201101527_0037_m_000000_0, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> TestHadoop$Map
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
> at
> org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:195)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:718)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>
> Below is the stub:
>
> public class TestHadoop extends Configured {
>
> public static class Map extends Mapper<LongWritable, Text, Text,
> IntWritable> {
> private final static IntWritable one = new IntWritable(1);
> private Text word = new Text();
>
> public void map(LongWritable key, Text value, Context
> context) throws IOException, InterruptedException {
> String line = value.toString();
> StringTokenizer tokenizer = new
> StringTokenizer(line);
> while (tokenizer.hasMoreTokens()) {
> word.set(tokenizer.nextToken());
> context.write(word, one);
>
> }
> }
> }
>
> public static class Reduce extends Reducer<Text, IntWritable, Text,
> IntWritable> {
> public void reduce(Text key, Iterable<IntWritable> values,
> Context context) throws IOException, InterruptedException {
> int sum = 0;
> for (IntWritable val:values) {
> sum += val.get();
> }
> context.write(key, new IntWritable(sum));
> }
> }
>
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration(true);
> conf.set("fs.default.name","hdfs://vm-acd2-4c51:54310/");
> conf.set("mapred.job.tracker","hdfs://vm-acd2-4c51:54311/");
>
> conf.set("mapreduce.jobtracker.staging.root.dir","/app/hadoop/mapred/staging");
> Job jobconf = new Job(conf,"TestHadoop");
> jobconf.setJarByClass(TestHadoop.class);
> jobconf.setOutputKeyClass(Text.class);
> jobconf.setOutputValueClass(IntWritable.class);
>
> jobconf.setMapperClass(Map.class);
> jobconf.setCombinerClass(Reduce.class);
> jobconf.setReducerClass(Reduce.class);
>
> jobconf.setInputFormatClass(TextInputFormat.class);
> jobconf.setOutputFormatClass(TextOutputFormat.class);
>
> FileInputFormat.setInputPaths(jobconf, new
> Path("/tmp/Hadoop_Temp_Data/Input/"));
> FileOutputFormat.setOutputPath(jobconf, new
> Path("/tmp/Hadoop_Temp_Data/Output1/"));
> jobconf.waitForCompletion(true);
> }
>
> }
>
> Any help will be greatly appreciated!
>
> Thanks
> Hema Subramanian
>