You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Jens Scheidtmann <je...@gmail.com> on 2013/05/30 22:09:28 UTC
Types and SequenceFiles
Dear list,
I have created a sequence file like this:
seqWriter = SequenceFile.createWriter(fs, getConf(), new
Path(hdfsPath), IntWritable.class, BytesWritable.class,
SequenceFile.CompressionType.NONE);
seqWriter.append(new IntWritable(index++), new BytesWritable(buf));
(with buf a byte array.)
Now, when reading the same sequence file in a map reduce job, I specify the
mapper like this:
public static class NoOfMovesMapper
extends Mapper<IntWritable, BytesWritable, IntWritable,
IntWritable> {
and configure the SequenceFile as:
SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
Path(args[i]));
This job fails with:
java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
be cast to org.apache.hadoop.io.IntWritable
at
org.gostats.hadoop.NoOfMoves$NoOfMovesMapper.map(NoOfMoves.java:1)
I have to specify the mapper as
extends Mapper<LongWritable, Text, IntWritable, IntWritable> {
to read the sequence file. But then the number of records and invocations
of the map is much larger than I would expect. I thought that I will have
as many invocations of map as records in the sequence file.
What am I doing wrong? Were am I wrong?
Thanks in advance,
Jens
Re: Types and SequenceFiles
Posted by Harsh J <ha...@cloudera.com>.
There is a solution discussed in the mail you've quoted. Did you do
that? Share your code snippets please?
On Fri, Jul 26, 2013 at 5:07 AM, qwerty <ni...@gmail.com> wrote:
> Harsh J <ha...@...> writes:
>
>>
>> Ah, sorry I didn't read the exact problem.
>>
>> Yes that static call you make to addInputPath goes all the way up to
>> (inheritance!) FileInputFormat.addInputPath, which just adds input
>> paths and doesn't automatically imprint itself as the input format
>> class at the same time.
>>
>> On Fri, May 31, 2013 at 9:35 PM, Jens Scheidtmann
>> <je...@...> wro0te:
>> > Dear Harsh,
>> >
>> >
>> > thanks for your answer. Your post talks about the intermediate and final
>> > result types.
>> > These are already configured in my job as:
>> > job.setOutputKeyClass(IntWritable.class);
>> > job.setOutputValueClass(IntWritable.class);
>> >
>> > My problem was input key and value types, though.
>> > Your post let me look in the right direction. I added
>> >
>> > job.setInputFormatClass(SequenceFileInputFormat.class);
>> >
>> > which did the trick.I thought this would be done by the
>> >
>> > SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
>> > Path(args[i]));
>> >
>> > Best regards,
>> >
>> > Jens
>>
>
> Hey, I have a similar problem. I am trying to read a sequence
> file(compressed with snappy)
>
> I did my Mapper class as:
> public static class Map extends Mapper<LongWritable, BytesWritable, Text,
> NullWritable>
>
> My map function as : public void map(LongWritable key, Text value, Context
> context)
>
> When I try to run it says 'Type mismatch in key from map: expected
> org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable'
>
> Can anyone please help?
>
>
>
>
--
Harsh J
Re: Types and SequenceFiles
Posted by Harsh J <ha...@cloudera.com>.
There is a solution discussed in the mail you've quoted. Did you do
that? Share your code snippets please?
On Fri, Jul 26, 2013 at 5:07 AM, qwerty <ni...@gmail.com> wrote:
> Harsh J <ha...@...> writes:
>
>>
>> Ah, sorry I didn't read the exact problem.
>>
>> Yes that static call you make to addInputPath goes all the way up to
>> (inheritance!) FileInputFormat.addInputPath, which just adds input
>> paths and doesn't automatically imprint itself as the input format
>> class at the same time.
>>
>> On Fri, May 31, 2013 at 9:35 PM, Jens Scheidtmann
>> <je...@...> wro0te:
>> > Dear Harsh,
>> >
>> >
>> > thanks for your answer. Your post talks about the intermediate and final
>> > result types.
>> > These are already configured in my job as:
>> > job.setOutputKeyClass(IntWritable.class);
>> > job.setOutputValueClass(IntWritable.class);
>> >
>> > My problem was input key and value types, though.
>> > Your post let me look in the right direction. I added
>> >
>> > job.setInputFormatClass(SequenceFileInputFormat.class);
>> >
>> > which did the trick.I thought this would be done by the
>> >
>> > SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
>> > Path(args[i]));
>> >
>> > Best regards,
>> >
>> > Jens
>>
>
> Hey, I have a similar problem. I am trying to read a sequence
> file(compressed with snappy)
>
> I did my Mapper class as:
> public static class Map extends Mapper<LongWritable, BytesWritable, Text,
> NullWritable>
>
> My map function as : public void map(LongWritable key, Text value, Context
> context)
>
> When I try to run it says 'Type mismatch in key from map: expected
> org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable'
>
> Can anyone please help?
>
>
>
>
--
Harsh J
Re: Types and SequenceFiles
Posted by Harsh J <ha...@cloudera.com>.
There is a solution discussed in the mail you've quoted. Did you do
that? Share your code snippets please?
On Fri, Jul 26, 2013 at 5:07 AM, qwerty <ni...@gmail.com> wrote:
> Harsh J <ha...@...> writes:
>
>>
>> Ah, sorry I didn't read the exact problem.
>>
>> Yes that static call you make to addInputPath goes all the way up to
>> (inheritance!) FileInputFormat.addInputPath, which just adds input
>> paths and doesn't automatically imprint itself as the input format
>> class at the same time.
>>
>> On Fri, May 31, 2013 at 9:35 PM, Jens Scheidtmann
>> <je...@...> wro0te:
>> > Dear Harsh,
>> >
>> >
>> > thanks for your answer. Your post talks about the intermediate and final
>> > result types.
>> > These are already configured in my job as:
>> > job.setOutputKeyClass(IntWritable.class);
>> > job.setOutputValueClass(IntWritable.class);
>> >
>> > My problem was input key and value types, though.
>> > Your post let me look in the right direction. I added
>> >
>> > job.setInputFormatClass(SequenceFileInputFormat.class);
>> >
>> > which did the trick.I thought this would be done by the
>> >
>> > SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
>> > Path(args[i]));
>> >
>> > Best regards,
>> >
>> > Jens
>>
>
> Hey, I have a similar problem. I am trying to read a sequence
> file(compressed with snappy)
>
> I did my Mapper class as:
> public static class Map extends Mapper<LongWritable, BytesWritable, Text,
> NullWritable>
>
> My map function as : public void map(LongWritable key, Text value, Context
> context)
>
> When I try to run it says 'Type mismatch in key from map: expected
> org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable'
>
> Can anyone please help?
>
>
>
>
--
Harsh J
Re: Types and SequenceFiles
Posted by Harsh J <ha...@cloudera.com>.
There is a solution discussed in the mail you've quoted. Did you do
that? Share your code snippets please?
On Fri, Jul 26, 2013 at 5:07 AM, qwerty <ni...@gmail.com> wrote:
> Harsh J <ha...@...> writes:
>
>>
>> Ah, sorry I didn't read the exact problem.
>>
>> Yes that static call you make to addInputPath goes all the way up to
>> (inheritance!) FileInputFormat.addInputPath, which just adds input
>> paths and doesn't automatically imprint itself as the input format
>> class at the same time.
>>
>> On Fri, May 31, 2013 at 9:35 PM, Jens Scheidtmann
>> <je...@...> wro0te:
>> > Dear Harsh,
>> >
>> >
>> > thanks for your answer. Your post talks about the intermediate and final
>> > result types.
>> > These are already configured in my job as:
>> > job.setOutputKeyClass(IntWritable.class);
>> > job.setOutputValueClass(IntWritable.class);
>> >
>> > My problem was input key and value types, though.
>> > Your post let me look in the right direction. I added
>> >
>> > job.setInputFormatClass(SequenceFileInputFormat.class);
>> >
>> > which did the trick.I thought this would be done by the
>> >
>> > SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
>> > Path(args[i]));
>> >
>> > Best regards,
>> >
>> > Jens
>>
>
> Hey, I have a similar problem. I am trying to read a sequence
> file(compressed with snappy)
>
> I did my Mapper class as:
> public static class Map extends Mapper<LongWritable, BytesWritable, Text,
> NullWritable>
>
> My map function as : public void map(LongWritable key, Text value, Context
> context)
>
> When I try to run it says 'Type mismatch in key from map: expected
> org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable'
>
> Can anyone please help?
>
>
>
>
--
Harsh J
Re: Types and SequenceFiles
Posted by qwerty <ni...@gmail.com>.
Harsh J <ha...@...> writes:
>
> Ah, sorry I didn't read the exact problem.
>
> Yes that static call you make to addInputPath goes all the way up to
> (inheritance!) FileInputFormat.addInputPath, which just adds input
> paths and doesn't automatically imprint itself as the input format
> class at the same time.
>
> On Fri, May 31, 2013 at 9:35 PM, Jens Scheidtmann
> <je...@...> wro0te:
> > Dear Harsh,
> >
> >
> > thanks for your answer. Your post talks about the intermediate and final
> > result types.
> > These are already configured in my job as:
> > job.setOutputKeyClass(IntWritable.class);
> > job.setOutputValueClass(IntWritable.class);
> >
> > My problem was input key and value types, though.
> > Your post let me look in the right direction. I added
> >
> > job.setInputFormatClass(SequenceFileInputFormat.class);
> >
> > which did the trick.I thought this would be done by the
> >
> > SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
> > Path(args[i]));
> >
> > Best regards,
> >
> > Jens
>
Hey, I have a similar problem. I am trying to read a sequence
file(compressed with snappy)
I did my Mapper class as:
public static class Map extends Mapper<LongWritable, BytesWritable, Text,
NullWritable>
My map function as : public void map(LongWritable key, Text value, Context
context)
When I try to run it says 'Type mismatch in key from map: expected
org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable'
Can anyone please help?
Re: Types and SequenceFiles
Posted by qwerty <ni...@gmail.com>.
Harsh J <ha...@...> writes:
>
> Ah, sorry I didn't read the exact problem.
>
> Yes that static call you make to addInputPath goes all the way up to
> (inheritance!) FileInputFormat.addInputPath, which just adds input
> paths and doesn't automatically imprint itself as the input format
> class at the same time.
>
> On Fri, May 31, 2013 at 9:35 PM, Jens Scheidtmann
> <je...@...> wro0te:
> > Dear Harsh,
> >
> >
> > thanks for your answer. Your post talks about the intermediate and final
> > result types.
> > These are already configured in my job as:
> > job.setOutputKeyClass(IntWritable.class);
> > job.setOutputValueClass(IntWritable.class);
> >
> > My problem was input key and value types, though.
> > Your post let me look in the right direction. I added
> >
> > job.setInputFormatClass(SequenceFileInputFormat.class);
> >
> > which did the trick.I thought this would be done by the
> >
> > SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
> > Path(args[i]));
> >
> > Best regards,
> >
> > Jens
>
Hey, I have a similar problem. I am trying to read a sequence
file(compressed with snappy)
I did my Mapper class as:
public static class Map extends Mapper<LongWritable, BytesWritable, Text,
NullWritable>
My map function as : public void map(LongWritable key, Text value, Context
context)
When I try to run it says 'Type mismatch in key from map: expected
org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable'
Can anyone please help?
Re: Types and SequenceFiles
Posted by qwerty <ni...@gmail.com>.
Harsh J <ha...@...> writes:
>
> Ah, sorry I didn't read the exact problem.
>
> Yes that static call you make to addInputPath goes all the way up to
> (inheritance!) FileInputFormat.addInputPath, which just adds input
> paths and doesn't automatically imprint itself as the input format
> class at the same time.
>
> On Fri, May 31, 2013 at 9:35 PM, Jens Scheidtmann
> <je...@...> wro0te:
> > Dear Harsh,
> >
> >
> > thanks for your answer. Your post talks about the intermediate and final
> > result types.
> > These are already configured in my job as:
> > job.setOutputKeyClass(IntWritable.class);
> > job.setOutputValueClass(IntWritable.class);
> >
> > My problem was input key and value types, though.
> > Your post let me look in the right direction. I added
> >
> > job.setInputFormatClass(SequenceFileInputFormat.class);
> >
> > which did the trick.I thought this would be done by the
> >
> > SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
> > Path(args[i]));
> >
> > Best regards,
> >
> > Jens
>
Hey, I have a similar problem. I am trying to read a sequence
file(compressed with snappy)
I did my Mapper class as:
public static class Map extends Mapper<LongWritable, BytesWritable, Text,
NullWritable>
My map function as : public void map(LongWritable key, Text value, Context
context)
When I try to run it says 'Type mismatch in key from map: expected
org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable'
Can anyone please help?
Re: Types and SequenceFiles
Posted by qwerty <ni...@gmail.com>.
Harsh J <ha...@...> writes:
>
> Ah, sorry I didn't read the exact problem.
>
> Yes that static call you make to addInputPath goes all the way up to
> (inheritance!) FileInputFormat.addInputPath, which just adds input
> paths and doesn't automatically imprint itself as the input format
> class at the same time.
>
> On Fri, May 31, 2013 at 9:35 PM, Jens Scheidtmann
> <je...@...> wro0te:
> > Dear Harsh,
> >
> >
> > thanks for your answer. Your post talks about the intermediate and final
> > result types.
> > These are already configured in my job as:
> > job.setOutputKeyClass(IntWritable.class);
> > job.setOutputValueClass(IntWritable.class);
> >
> > My problem was input key and value types, though.
> > Your post let me look in the right direction. I added
> >
> > job.setInputFormatClass(SequenceFileInputFormat.class);
> >
> > which did the trick.I thought this would be done by the
> >
> > SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
> > Path(args[i]));
> >
> > Best regards,
> >
> > Jens
>
Hey, I have a similar problem. I am trying to read a sequence
file(compressed with snappy)
I did my Mapper class as:
public static class Map extends Mapper<LongWritable, BytesWritable, Text,
NullWritable>
My map function as : public void map(LongWritable key, Text value, Context
context)
When I try to run it says 'Type mismatch in key from map: expected
org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable'
Can anyone please help?
Re: Types and SequenceFiles
Posted by Harsh J <ha...@cloudera.com>.
Ah, sorry I didn't read the exact problem.
Yes that static call you make to addInputPath goes all the way up to
(inheritance!) FileInputFormat.addInputPath, which just adds input
paths and doesn't automatically imprint itself as the input format
class at the same time.
On Fri, May 31, 2013 at 9:35 PM, Jens Scheidtmann
<je...@gmail.com> wrote:
> Dear Harsh,
>
>
> thanks for your answer. Your post talks about the intermediate and final
> result types.
> These are already configured in my job as:
> job.setOutputKeyClass(IntWritable.class);
> job.setOutputValueClass(IntWritable.class);
>
> My problem was input key and value types, though.
> Your post let me look in the right direction. I added
>
> job.setInputFormatClass(SequenceFileInputFormat.class);
>
> which did the trick.I thought this would be done by the
>
> SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
> Path(args[i]));
>
> Best regards,
>
> Jens
--
Harsh J
Re: Types and SequenceFiles
Posted by Harsh J <ha...@cloudera.com>.
Ah, sorry I didn't read the exact problem.
Yes that static call you make to addInputPath goes all the way up to
(inheritance!) FileInputFormat.addInputPath, which just adds input
paths and doesn't automatically imprint itself as the input format
class at the same time.
On Fri, May 31, 2013 at 9:35 PM, Jens Scheidtmann
<je...@gmail.com> wrote:
> Dear Harsh,
>
>
> thanks for your answer. Your post talks about the intermediate and final
> result types.
> These are already configured in my job as:
> job.setOutputKeyClass(IntWritable.class);
> job.setOutputValueClass(IntWritable.class);
>
> My problem was input key and value types, though.
> Your post let me look in the right direction. I added
>
> job.setInputFormatClass(SequenceFileInputFormat.class);
>
> which did the trick.I thought this would be done by the
>
> SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
> Path(args[i]));
>
> Best regards,
>
> Jens
--
Harsh J
Re: Types and SequenceFiles
Posted by Harsh J <ha...@cloudera.com>.
Ah, sorry I didn't read the exact problem.
Yes that static call you make to addInputPath goes all the way up to
(inheritance!) FileInputFormat.addInputPath, which just adds input
paths and doesn't automatically imprint itself as the input format
class at the same time.
On Fri, May 31, 2013 at 9:35 PM, Jens Scheidtmann
<je...@gmail.com> wrote:
> Dear Harsh,
>
>
> thanks for your answer. Your post talks about the intermediate and final
> result types.
> These are already configured in my job as:
> job.setOutputKeyClass(IntWritable.class);
> job.setOutputValueClass(IntWritable.class);
>
> My problem was input key and value types, though.
> Your post let me look in the right direction. I added
>
> job.setInputFormatClass(SequenceFileInputFormat.class);
>
> which did the trick.I thought this would be done by the
>
> SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
> Path(args[i]));
>
> Best regards,
>
> Jens
--
Harsh J
Re: Types and SequenceFiles
Posted by Harsh J <ha...@cloudera.com>.
Ah, sorry I didn't read the exact problem.
Yes that static call you make to addInputPath goes all the way up to
(inheritance!) FileInputFormat.addInputPath, which just adds input
paths and doesn't automatically imprint itself as the input format
class at the same time.
On Fri, May 31, 2013 at 9:35 PM, Jens Scheidtmann
<je...@gmail.com> wrote:
> Dear Harsh,
>
>
> thanks for your answer. Your post talks about the intermediate and final
> result types.
> These are already configured in my job as:
> job.setOutputKeyClass(IntWritable.class);
> job.setOutputValueClass(IntWritable.class);
>
> My problem was input key and value types, though.
> Your post let me look in the right direction. I added
>
> job.setInputFormatClass(SequenceFileInputFormat.class);
>
> which did the trick.I thought this would be done by the
>
> SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
> Path(args[i]));
>
> Best regards,
>
> Jens
--
Harsh J
Re: Types and SequenceFiles
Posted by Jens Scheidtmann <je...@gmail.com>.
Dear Harsh,
thanks for your answer. Your post talks about the intermediate and final
result types.
These are already configured in my job as:
job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(IntWritable.class);
My problem was input key and value types, though.
Your post let me look in the right direction. I added
job.setInputFormatClass(SequenceFileInputFormat.class);
which did the trick.I thought this would be done by the
SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
Path(args[i]));
Best regards,
Jens
Re: Types and SequenceFiles
Posted by Jens Scheidtmann <je...@gmail.com>.
Dear Harsh,
thanks for your answer. Your post talks about the intermediate and final
result types.
These are already configured in my job as:
job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(IntWritable.class);
My problem was input key and value types, though.
Your post let me look in the right direction. I added
job.setInputFormatClass(SequenceFileInputFormat.class);
which did the trick.I thought this would be done by the
SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
Path(args[i]));
Best regards,
Jens
Re: Types and SequenceFiles
Posted by Jens Scheidtmann <je...@gmail.com>.
Dear Harsh,
thanks for your answer. Your post talks about the intermediate and final
result types.
These are already configured in my job as:
job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(IntWritable.class);
My problem was input key and value types, though.
Your post let me look in the right direction. I added
job.setInputFormatClass(SequenceFileInputFormat.class);
which did the trick.I thought this would be done by the
SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
Path(args[i]));
Best regards,
Jens
Re: Types and SequenceFiles
Posted by Jens Scheidtmann <je...@gmail.com>.
Dear Harsh,
thanks for your answer. Your post talks about the intermediate and final
result types.
These are already configured in my job as:
job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(IntWritable.class);
My problem was input key and value types, though.
Your post let me look in the right direction. I added
job.setInputFormatClass(SequenceFileInputFormat.class);
which did the trick.I thought this would be done by the
SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
Path(args[i]));
Best regards,
Jens
Re: Types and SequenceFiles
Posted by Harsh J <ha...@cloudera.com>.
Hi Jens,
Please read this old thread at http://search-hadoop.com/m/WHvZDCfVsD
which covers the issue, the solution and more.
On Fri, May 31, 2013 at 1:39 AM, Jens Scheidtmann
<je...@gmail.com> wrote:
> Dear list,
>
> I have created a sequence file like this:
>
> seqWriter = SequenceFile.createWriter(fs, getConf(), new Path(hdfsPath),
> IntWritable.class, BytesWritable.class, SequenceFile.CompressionType.NONE);
> seqWriter.append(new IntWritable(index++), new BytesWritable(buf));
>
> (with buf a byte array.)
>
> Now, when reading the same sequence file in a map reduce job, I specify the
> mapper like this:
>
> public static class NoOfMovesMapper
> extends Mapper<IntWritable, BytesWritable, IntWritable, IntWritable>
> {
>
> and configure the SequenceFile as:
>
> SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
> Path(args[i]));
>
> This job fails with:
>
> java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
> be cast to org.apache.hadoop.io.IntWritable
> at
> org.gostats.hadoop.NoOfMoves$NoOfMovesMapper.map(NoOfMoves.java:1)
>
> I have to specify the mapper as
>
> extends Mapper<LongWritable, Text, IntWritable, IntWritable> {
>
> to read the sequence file. But then the number of records and invocations of
> the map is much larger than I would expect. I thought that I will have as
> many invocations of map as records in the sequence file.
>
> What am I doing wrong? Were am I wrong?
>
> Thanks in advance,
>
> Jens
--
Harsh J
Re: Types and SequenceFiles
Posted by Harsh J <ha...@cloudera.com>.
Hi Jens,
Please read this old thread at http://search-hadoop.com/m/WHvZDCfVsD
which covers the issue, the solution and more.
On Fri, May 31, 2013 at 1:39 AM, Jens Scheidtmann
<je...@gmail.com> wrote:
> Dear list,
>
> I have created a sequence file like this:
>
> seqWriter = SequenceFile.createWriter(fs, getConf(), new Path(hdfsPath),
> IntWritable.class, BytesWritable.class, SequenceFile.CompressionType.NONE);
> seqWriter.append(new IntWritable(index++), new BytesWritable(buf));
>
> (with buf a byte array.)
>
> Now, when reading the same sequence file in a map reduce job, I specify the
> mapper like this:
>
> public static class NoOfMovesMapper
> extends Mapper<IntWritable, BytesWritable, IntWritable, IntWritable>
> {
>
> and configure the SequenceFile as:
>
> SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
> Path(args[i]));
>
> This job fails with:
>
> java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
> be cast to org.apache.hadoop.io.IntWritable
> at
> org.gostats.hadoop.NoOfMoves$NoOfMovesMapper.map(NoOfMoves.java:1)
>
> I have to specify the mapper as
>
> extends Mapper<LongWritable, Text, IntWritable, IntWritable> {
>
> to read the sequence file. But then the number of records and invocations of
> the map is much larger than I would expect. I thought that I will have as
> many invocations of map as records in the sequence file.
>
> What am I doing wrong? Were am I wrong?
>
> Thanks in advance,
>
> Jens
--
Harsh J
Re: Types and SequenceFiles
Posted by Harsh J <ha...@cloudera.com>.
Hi Jens,
Please read this old thread at http://search-hadoop.com/m/WHvZDCfVsD
which covers the issue, the solution and more.
On Fri, May 31, 2013 at 1:39 AM, Jens Scheidtmann
<je...@gmail.com> wrote:
> Dear list,
>
> I have created a sequence file like this:
>
> seqWriter = SequenceFile.createWriter(fs, getConf(), new Path(hdfsPath),
> IntWritable.class, BytesWritable.class, SequenceFile.CompressionType.NONE);
> seqWriter.append(new IntWritable(index++), new BytesWritable(buf));
>
> (with buf a byte array.)
>
> Now, when reading the same sequence file in a map reduce job, I specify the
> mapper like this:
>
> public static class NoOfMovesMapper
> extends Mapper<IntWritable, BytesWritable, IntWritable, IntWritable>
> {
>
> and configure the SequenceFile as:
>
> SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
> Path(args[i]));
>
> This job fails with:
>
> java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
> be cast to org.apache.hadoop.io.IntWritable
> at
> org.gostats.hadoop.NoOfMoves$NoOfMovesMapper.map(NoOfMoves.java:1)
>
> I have to specify the mapper as
>
> extends Mapper<LongWritable, Text, IntWritable, IntWritable> {
>
> to read the sequence file. But then the number of records and invocations of
> the map is much larger than I would expect. I thought that I will have as
> many invocations of map as records in the sequence file.
>
> What am I doing wrong? Were am I wrong?
>
> Thanks in advance,
>
> Jens
--
Harsh J
Re: Types and SequenceFiles
Posted by Harsh J <ha...@cloudera.com>.
Hi Jens,
Please read this old thread at http://search-hadoop.com/m/WHvZDCfVsD
which covers the issue, the solution and more.
On Fri, May 31, 2013 at 1:39 AM, Jens Scheidtmann
<je...@gmail.com> wrote:
> Dear list,
>
> I have created a sequence file like this:
>
> seqWriter = SequenceFile.createWriter(fs, getConf(), new Path(hdfsPath),
> IntWritable.class, BytesWritable.class, SequenceFile.CompressionType.NONE);
> seqWriter.append(new IntWritable(index++), new BytesWritable(buf));
>
> (with buf a byte array.)
>
> Now, when reading the same sequence file in a map reduce job, I specify the
> mapper like this:
>
> public static class NoOfMovesMapper
> extends Mapper<IntWritable, BytesWritable, IntWritable, IntWritable>
> {
>
> and configure the SequenceFile as:
>
> SequenceFileAsBinaryInputFormat.addInputPath(jobConf, new
> Path(args[i]));
>
> This job fails with:
>
> java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
> be cast to org.apache.hadoop.io.IntWritable
> at
> org.gostats.hadoop.NoOfMoves$NoOfMovesMapper.map(NoOfMoves.java:1)
>
> I have to specify the mapper as
>
> extends Mapper<LongWritable, Text, IntWritable, IntWritable> {
>
> to read the sequence file. But then the number of records and invocations of
> the map is much larger than I would expect. I thought that I will have as
> many invocations of map as records in the sequence file.
>
> What am I doing wrong? Were am I wrong?
>
> Thanks in advance,
>
> Jens
--
Harsh J