You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Tim Fletcher <zi...@gmail.com> on 2011/08/19 14:31:31 UTC
SequenceFileInputFormat doesn't return whole records
Hi all,
I am having issues using SequenceFileInputFormat to retrieve whole records
I have 1 job that is used to write to a SequenceFile
SequenceFileOutputFormat.setOutputPath(job, new Path("out/data"));
SequenceFileOutputFormat.setOutputCompressionType(job,
SequenceFile.CompressionType.NONE);
I then have a second job that is ment to read the file for processing
SequenceFileInputFormat.addInputPath(job, new Path("out/data"));
However, the values that i get as the arguments to the Map part of my job
only seems to contain parts of the record. I am sure that i am missing
something rather fundamental as to how Hadoop splits inputs to the Mapper,
but can't seem to find a way to stop the records being split.
Any help (or a pointer to a specific page in the doc) would be greatly
appreciated
Regards,
Tim
Re: SequenceFileInputFormat doesn't return whole records
Posted by Tim Fletcher <zi...@gmail.com>.
Harsh, that was exactly the issue!
Thanks very much for your help
Tim
On 19 August 2011 15:15, Harsh J <ha...@cloudera.com> wrote:
> Tim,
>
> Do you also set your I/O formats explicitly to SequenceFileInputFormat
> and SequenceFileOutputFormat? Via job.setInputFormat/setOutputFormat I
> mean.
>
> Hadoop should not be splitting records across maps/mappers. There are
> specific test cases that ensure this does not happen, so it would seem
> strange if it does this.
>
> On Fri, Aug 19, 2011 at 6:01 PM, Tim Fletcher <zi...@gmail.com> wrote:
> > Hi all,
> > I am having issues using SequenceFileInputFormat to retrieve whole
> records
> > I have 1 job that is used to write to a SequenceFile
> > SequenceFileOutputFormat.setOutputPath(job, new Path("out/data"));
> > SequenceFileOutputFormat.setOutputCompressionType(job,
> > SequenceFile.CompressionType.NONE);
> > I then have a second job that is ment to read the file for processing
> > SequenceFileInputFormat.addInputPath(job, new Path("out/data"));
> > However, the values that i get as the arguments to the Map part of my job
> > only seems to contain parts of the record. I am sure that i am missing
> > something rather fundamental as to how Hadoop splits inputs to the
> Mapper,
> > but can't seem to find a way to stop the records being split.
> > Any help (or a pointer to a specific page in the doc) would be greatly
> > appreciated
> > Regards,
> > Tim
>
>
>
> --
> Harsh J
>
Re: SequenceFileInputFormat doesn't return whole records
Posted by Harsh J <ha...@cloudera.com>.
Tim,
Do you also set your I/O formats explicitly to SequenceFileInputFormat
and SequenceFileOutputFormat? Via job.setInputFormat/setOutputFormat I
mean.
Hadoop should not be splitting records across maps/mappers. There are
specific test cases that ensure this does not happen, so it would seem
strange if it does this.
On Fri, Aug 19, 2011 at 6:01 PM, Tim Fletcher <zi...@gmail.com> wrote:
> Hi all,
> I am having issues using SequenceFileInputFormat to retrieve whole records
> I have 1 job that is used to write to a SequenceFile
> SequenceFileOutputFormat.setOutputPath(job, new Path("out/data"));
> SequenceFileOutputFormat.setOutputCompressionType(job,
> SequenceFile.CompressionType.NONE);
> I then have a second job that is ment to read the file for processing
> SequenceFileInputFormat.addInputPath(job, new Path("out/data"));
> However, the values that i get as the arguments to the Map part of my job
> only seems to contain parts of the record. I am sure that i am missing
> something rather fundamental as to how Hadoop splits inputs to the Mapper,
> but can't seem to find a way to stop the records being split.
> Any help (or a pointer to a specific page in the doc) would be greatly
> appreciated
> Regards,
> Tim
--
Harsh J