You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Sridhar Raman <sr...@gmail.com> on 2008/04/18 08:20:12 UTC

Input and Output types?

Hi

I am new to MapReduce and Hadoop, and I have managed to find my way through
with a few programs.  But I still have some doubts that are constantly
clinging onto me.  I am not too sure whether these are basic doubts, or just
some documentation that I missed somewhere.

1)  Should my input _always_ be text files?  What if my input is in the form
of Java objects?  Where do I handle this conversion?

2)  How do I control how the output is written?  For example, if I want to
output in a format that is my own, how do I do it?

Thanks,
Sridhar

Re: Input and Output types?

Posted by Sridhar Raman <sr...@gmail.com>.
Thanks.

On Fri, Apr 18, 2008 at 9:14 PM, Owen O'Malley <oo...@yahoo-inc.com> wrote:

>
> On Apr 17, 2008, at 11:20 PM, Sridhar Raman wrote:
>
>  I am new to MapReduce and Hadoop, and I have managed to find my way
> > through
> > with a few programs.  But I still have some doubts that are constantly
> > clinging onto me.  I am not too sure whether these are basic doubts, or
> > just
> > some documentation that I missed somewhere.
> >
>
> Take a look at  http://tinyurl.com/4y7776 under InputFormats.
>
>  1)  Should my input _always_ be text files?  What if my input is in the
> > form
> > of Java objects?  Where do I handle this conversion?
> >
>
> You can define your own InputFormat that reads an arbitrary format, or use
> SequenceFileInputFormat that reads SequenceFiles. SequenceFiles are a file
> format defined by Hadoop to hold binary data consisting of Writable keys and
> values.
>
>  2)  How do I control how the output is written?  For example, if I want
> > to
> > output in a format that is my own, how do I do it?
> >
>
> That is controlled by the OutputFormat. It defaults to TextOutputFormat,
> but you can either use SequenceFileOutputFormat or make your own.
>
> -- Owen
>

Re: Input and Output types?

Posted by Owen O'Malley <oo...@yahoo-inc.com>.
On Apr 17, 2008, at 11:20 PM, Sridhar Raman wrote:

> I am new to MapReduce and Hadoop, and I have managed to find my way  
> through
> with a few programs.  But I still have some doubts that are constantly
> clinging onto me.  I am not too sure whether these are basic  
> doubts, or just
> some documentation that I missed somewhere.

Take a look at  http://tinyurl.com/4y7776 under InputFormats.

> 1)  Should my input _always_ be text files?  What if my input is in  
> the form
> of Java objects?  Where do I handle this conversion?

You can define your own InputFormat that reads an arbitrary format,  
or use SequenceFileInputFormat that reads SequenceFiles.  
SequenceFiles are a file format defined by Hadoop to hold binary data  
consisting of Writable keys and values.

> 2)  How do I control how the output is written?  For example, if I  
> want to
> output in a format that is my own, how do I do it?

That is controlled by the OutputFormat. It defaults to  
TextOutputFormat, but you can either use SequenceFileOutputFormat or  
make your own.

-- Owen