You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Harit Himanshu <ha...@gmail.com> on 2012/08/02 21:59:31 UTC
Hadoop : java.lang.ClassCastException: org.apache.hadoop.io.LongWritable
cannot be cast to org.apache.hadoop.io.Text
StackOverflow link -
http://stackoverflow.com/questions/11784729/hadoop-java-lang-classcastexception-org-apache-hadoop-io-longwritable-cannot
----------
My program looks like
public class TopKRecord extends Configured implements Tool {
public static class MapClass extends Mapper<Text, Text, Text, Text> {
public void map(Text key, Text value, Context context) throws
IOException, InterruptedException {
// your map code goes here
String[] fields = value.toString().split(",");
String year = fields[1];
String claims = fields[8];
if (claims.length() > 0 && (!claims.startsWith("\""))) {
context.write(new Text(year.toString()), new
Text(claims.toString()));
}
}
}
public int run(String args[]) throws Exception {
Job job = new Job();
job.setJarByClass(TopKRecord.class);
job.setMapperClass(MapClass.class);
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setJobName("TopKRecord");
job.setMapOutputValueClass(Text.class);
job.setNumReduceTasks(0);
boolean success = job.waitForCompletion(true);
return success ? 0 : 1;
}
public static void main(String args[]) throws Exception {
int ret = ToolRunner.run(new TopKRecord(), args);
System.exit(ret);
}
}
The data looks like
"PATENT","GYEAR","GDATE","APPYEAR","COUNTRY","POSTATE","ASSIGNEE","ASSCODE","CLAIMS","NCLASS","CAT","SUBCAT","CMADE","CRECEIVE","RATIOCIT","GENERAL","ORIGINAL","FWDAPLAG","BCKGTLAG","SELFCTUB","SELFCTLB","SECDUPBD","SECDLWBD"
3070801,1963,1096,,"BE","",,1,,269,6,69,,1,,0,,,,,,,
3070802,1963,1096,,"US","TX",,1,,2,6,63,,0,,,,,,,,,
3070803,1963,1096,,"US","IL",,1,,2,6,63,,9,,0.3704,,,,,,,
3070804,1963,1096,,"US","OH",,1,,2,6,63,,3,,0.6667,,,,,,,
On running this program I see the following on console
12/08/02 12:43:34 INFO mapred.JobClient: Task Id :
attempt_201208021025_0007_m_000000_0, Status : FAILED
java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
be cast to org.apache.hadoop.io.Text
at com.hadoop.programs.TopKRecord$MapClass.map(TopKRecord.java:26)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
I believe that the Class Types are mapped correctly, Class
Mapper<http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Mapper.html>
,
Please let me know what is that I am doing wrong here?
Thank you
+ Harit Himanshu
Re: Hadoop : java.lang.ClassCastException: org.apache.hadoop.io.LongWritable
cannot be cast to org.apache.hadoop.io.Text
Posted by Harit Himanshu <ha...@gmail.com>.
That works perfectly well, thank you Robert!
On Thu, Aug 2, 2012 at 1:09 PM, Robert Evans <ev...@yahoo-inc.com> wrote:
> The default text input format has a key of a LongWritable that is the
> offset into the file. The value is the full line.
>
> On 8/2/12 2:59 PM, "Harit Himanshu" <ha...@gmail.com> wrote:
>
> >StackOverflow link -
> >
> http://stackoverflow.com/questions/11784729/hadoop-java-lang-classcastexce
> >ption-org-apache-hadoop-io-longwritable-cannot
> >
> >----------
> >
> >My program looks like
> >
> >public class TopKRecord extends Configured implements Tool {
> >
> > public static class MapClass extends Mapper<Text, Text, Text, Text> {
> >
> > public void map(Text key, Text value, Context context) throws
> >IOException, InterruptedException {
> > // your map code goes here
> > String[] fields = value.toString().split(",");
> > String year = fields[1];
> > String claims = fields[8];
> >
> > if (claims.length() > 0 && (!claims.startsWith("\""))) {
> > context.write(new Text(year.toString()), new
> >Text(claims.toString()));
> > }
> > }
> > }
> > public int run(String args[]) throws Exception {
> > Job job = new Job();
> > job.setJarByClass(TopKRecord.class);
> >
> > job.setMapperClass(MapClass.class);
> >
> > FileInputFormat.setInputPaths(job, new Path(args[0]));
> > FileOutputFormat.setOutputPath(job, new Path(args[1]));
> >
> > job.setJobName("TopKRecord");
> > job.setMapOutputValueClass(Text.class);
> > job.setNumReduceTasks(0);
> > boolean success = job.waitForCompletion(true);
> > return success ? 0 : 1;
> > }
> >
> > public static void main(String args[]) throws Exception {
> > int ret = ToolRunner.run(new TopKRecord(), args);
> > System.exit(ret);
> > }
> >}
> >
> >The data looks like
> >
> >"PATENT","GYEAR","GDATE","APPYEAR","COUNTRY","POSTATE","ASSIGNEE","ASSCODE
> >","CLAIMS","NCLASS","CAT","SUBCAT","CMADE","CRECEIVE","RATIOCIT","GENERAL"
> >,"ORIGINAL","FWDAPLAG","BCKGTLAG","SELFCTUB","SELFCTLB","SECDUPBD","SECDLW
> >BD"
> >3070801,1963,1096,,"BE","",,1,,269,6,69,,1,,0,,,,,,,
> >3070802,1963,1096,,"US","TX",,1,,2,6,63,,0,,,,,,,,,
> >3070803,1963,1096,,"US","IL",,1,,2,6,63,,9,,0.3704,,,,,,,
> >3070804,1963,1096,,"US","OH",,1,,2,6,63,,3,,0.6667,,,,,,,
> >
> >On running this program I see the following on console
> >
> >12/08/02 12:43:34 INFO mapred.JobClient: Task Id :
> >attempt_201208021025_0007_m_000000_0, Status : FAILED
> >java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
> >be cast to org.apache.hadoop.io.Text
> > at com.hadoop.programs.TopKRecord$MapClass.map(TopKRecord.java:26)
> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
> >java:1121)
> > at org.apache.hadoop.mapred.Child.main(Child.java:249)
> >
> >I believe that the Class Types are mapped correctly, Class
> >Mapper<
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/
> >mapreduce/Mapper.html>
> >,
> >
> >Please let me know what is that I am doing wrong here?
> >
> >
> >Thank you
> >
> >+ Harit Himanshu
>
>
Re: Hadoop : java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.io.Text
Posted by Devi Kumarappan <kp...@att.net>.
After specifying NLineInputFormat option, streaming job fails with
Error from attempt_201205171448_0092_m_000000_0: java.lang.RuntimeException:
PipeMapRed.waitOutputThreads(): subprocess failed with code 2
It spawns two mappers, but i am not sure whether the mapper runs with file names
specified in the input option. I was expecting one mapper to run with
/user/devi/s_input/a.txt and one mapper to run with /user/devi/s_input/b.txt. I
digged into the task files, but could not find anything.
Here is the simple mapper perl script .All does is it reads the file and prints
it. (It needs to do much more stuff, but I could not get the basic job itself to
run).
$i = 0;
$userinput = <STDIN>;
open(INFILE,"$userinput") || die "could not open the file $userinput \n";
while (<INFILE>) {
my $line = $_;
print "$i".$line ;
$i++;
}
close(INFILE);
exit;
My command is hadoop jar
/usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar -input
/user/devi/file.txt -output /user/devi/s_output -mapper "/usr/bin/perl
/home/devi/Perl/crash_parser.pl" -inputformat
org.apache.hadoop.mapred.lib.NLineInputFormat
Really appreciate your help.
Devi
________________________________
From: Robert Evans <ev...@yahoo-inc.com>
To: "common-user@hadoop.apache.org" <co...@hadoop.apache.org>
Sent: Thu, August 2, 2012 1:15:05 PM
Subject: Re: Hadoop : java.lang.ClassCastException:
org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.io.Text
The default text input format has a key of a LongWritable that is the
offset into the file. The value is the full line.
On 8/2/12 2:59 PM, "Harit Himanshu" <ha...@gmail.com> wrote:
>StackOverflow link -
>http://stackoverflow.com/questions/11784729/hadoop-java-lang-classcastexce
>ption-org-apache-hadoop-io-longwritable-cannot
>
>----------
>
>My program looks like
>
>public class TopKRecord extends Configured implements Tool {
>
> public static class MapClass extends Mapper<Text, Text, Text, Text> {
>
> public void map(Text key, Text value, Context context) throws
>IOException, InterruptedException {
> // your map code goes here
> String[] fields = value.toString().split(",");
> String year = fields[1];
> String claims = fields[8];
>
> if (claims.length() > 0 && (!claims.startsWith("\""))) {
> context.write(new Text(year.toString()), new
>Text(claims.toString()));
> }
> }
> }
> public int run(String args[]) throws Exception {
> Job job = new Job();
> job.setJarByClass(TopKRecord.class);
>
> job.setMapperClass(MapClass.class);
>
> FileInputFormat.setInputPaths(job, new Path(args[0]));
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
> job.setJobName("TopKRecord");
> job.setMapOutputValueClass(Text.class);
> job.setNumReduceTasks(0);
> boolean success = job.waitForCompletion(true);
> return success ? 0 : 1;
> }
>
> public static void main(String args[]) throws Exception {
> int ret = ToolRunner.run(new TopKRecord(), args);
> System.exit(ret);
> }
>}
>
>The data looks like
>
>"PATENT","GYEAR","GDATE","APPYEAR","COUNTRY","POSTATE","ASSIGNEE","ASSCODE
>","CLAIMS","NCLASS","CAT","SUBCAT","CMADE","CRECEIVE","RATIOCIT","GENERAL"
>,"ORIGINAL","FWDAPLAG","BCKGTLAG","SELFCTUB","SELFCTLB","SECDUPBD","SECDLW
>BD"
>3070801,1963,1096,,"BE","",,1,,269,6,69,,1,,0,,,,,,,
>3070802,1963,1096,,"US","TX",,1,,2,6,63,,0,,,,,,,,,
>3070803,1963,1096,,"US","IL",,1,,2,6,63,,9,,0.3704,,,,,,,
>3070804,1963,1096,,"US","OH",,1,,2,6,63,,3,,0.6667,,,,,,,
>
>On running this program I see the following on console
>
>12/08/02 12:43:34 INFO mapred.JobClient: Task Id :
>attempt_201208021025_0007_m_000000_0, Status : FAILED
>java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
>be cast to org.apache.hadoop.io.Text
> at com.hadoop.programs.TopKRecord$MapClass.map(TopKRecord.java:26)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>I believe that the Class Types are mapped correctly, Class
>Mapper<http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/
>mapreduce/Mapper.html>
>,
>
>Please let me know what is that I am doing wrong here?
>
>
>Thank you
>
>+ Harit Himanshu
Re: Hadoop : java.lang.ClassCastException:
org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.io.Text
Posted by Robert Evans <ev...@yahoo-inc.com>.
The default text input format has a key of a LongWritable that is the
offset into the file. The value is the full line.
On 8/2/12 2:59 PM, "Harit Himanshu" <ha...@gmail.com> wrote:
>StackOverflow link -
>http://stackoverflow.com/questions/11784729/hadoop-java-lang-classcastexce
>ption-org-apache-hadoop-io-longwritable-cannot
>
>----------
>
>My program looks like
>
>public class TopKRecord extends Configured implements Tool {
>
> public static class MapClass extends Mapper<Text, Text, Text, Text> {
>
> public void map(Text key, Text value, Context context) throws
>IOException, InterruptedException {
> // your map code goes here
> String[] fields = value.toString().split(",");
> String year = fields[1];
> String claims = fields[8];
>
> if (claims.length() > 0 && (!claims.startsWith("\""))) {
> context.write(new Text(year.toString()), new
>Text(claims.toString()));
> }
> }
> }
> public int run(String args[]) throws Exception {
> Job job = new Job();
> job.setJarByClass(TopKRecord.class);
>
> job.setMapperClass(MapClass.class);
>
> FileInputFormat.setInputPaths(job, new Path(args[0]));
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
> job.setJobName("TopKRecord");
> job.setMapOutputValueClass(Text.class);
> job.setNumReduceTasks(0);
> boolean success = job.waitForCompletion(true);
> return success ? 0 : 1;
> }
>
> public static void main(String args[]) throws Exception {
> int ret = ToolRunner.run(new TopKRecord(), args);
> System.exit(ret);
> }
>}
>
>The data looks like
>
>"PATENT","GYEAR","GDATE","APPYEAR","COUNTRY","POSTATE","ASSIGNEE","ASSCODE
>","CLAIMS","NCLASS","CAT","SUBCAT","CMADE","CRECEIVE","RATIOCIT","GENERAL"
>,"ORIGINAL","FWDAPLAG","BCKGTLAG","SELFCTUB","SELFCTLB","SECDUPBD","SECDLW
>BD"
>3070801,1963,1096,,"BE","",,1,,269,6,69,,1,,0,,,,,,,
>3070802,1963,1096,,"US","TX",,1,,2,6,63,,0,,,,,,,,,
>3070803,1963,1096,,"US","IL",,1,,2,6,63,,9,,0.3704,,,,,,,
>3070804,1963,1096,,"US","OH",,1,,2,6,63,,3,,0.6667,,,,,,,
>
>On running this program I see the following on console
>
>12/08/02 12:43:34 INFO mapred.JobClient: Task Id :
>attempt_201208021025_0007_m_000000_0, Status : FAILED
>java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
>be cast to org.apache.hadoop.io.Text
> at com.hadoop.programs.TopKRecord$MapClass.map(TopKRecord.java:26)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>I believe that the Class Types are mapped correctly, Class
>Mapper<http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/
>mapreduce/Mapper.html>
>,
>
>Please let me know what is that I am doing wrong here?
>
>
>Thank you
>
>+ Harit Himanshu