TextInputFormat to SequenceFile Output format question

[Mapred Learn <ma...@gmail.com>]

Hi guys,
If I have a text file of 10 GB and I want to convert it to sequence file
using map-reduce and make filesplits of 1 GB each so that 10 mappers work in
parallel on it and convert it to Sequence file output.
Can I combine these 10 mapper outputs into 1 sequence file of 10 GB size in
reduce stage ? Is it possible ? If yes, would it be a very slow operation ?

-thanks
JJ

Re: TextInputFormat to SequenceFile Output format question

[Mapred Learn <ma...@gmail.com>]

Hey Harsh,
I was trying to use parallelism of mappers to do it quickly.

If I don't use map-reduce, a 10 GB text file to sequence file conversion
would be very slow. Isn't it so ?

Also what kind of transformation in mapper phase are you referring to ?

-JJ

On Fri, Feb 25, 2011 at 9:45 PM, Harsh J <qw...@gmail.com> wrote:

> Unless some transformation can additionally be applied in the Mapper
> phases for your text input, this operation could be done without
> MapReduce itself; since your requirement of a single file output will
> incur unnecessary intermediate-phase costs.
>
> On Sat, Feb 26, 2011 at 7:22 AM, Mapred Learn <ma...@gmail.com>
> wrote:
> > Hi guys,
> > If I have a text file of 10 GB and I want to convert it to sequence file
> > using map-reduce and make filesplits of 1 GB each so that 10 mappers work
> in
> > parallel on it and convert it to Sequence file output.
> > Can I combine these 10 mapper outputs into 1 sequence file of 10 GB size
> in
> > reduce stage ? Is it possible ? If yes, would it be a very slow operation
> ?
> >
> > -thanks
> > JJ
>
>
>
> --
> Harsh J
> www.harshj.com
>

Re: TextInputFormat to SequenceFile Output format question

[Harsh J <qw...@gmail.com>]

Unless some transformation can additionally be applied in the Mapper
phases for your text input, this operation could be done without
MapReduce itself; since your requirement of a single file output will
incur unnecessary intermediate-phase costs.

On Sat, Feb 26, 2011 at 7:22 AM, Mapred Learn <ma...@gmail.com> wrote:
> Hi guys,
> If I have a text file of 10 GB and I want to convert it to sequence file
> using map-reduce and make filesplits of 1 GB each so that 10 mappers work in
> parallel on it and convert it to Sequence file output.
> Can I combine these 10 mapper outputs into 1 sequence file of 10 GB size in
> reduce stage ? Is it possible ? If yes, would it be a very slow operation ?
>
> -thanks
> JJ



-- 
Harsh J
www.harshj.com