Import the results into SimpleDB

[Mark Kerzner <ma...@gmail.com>]

Hi,

I want a Hadoop job that will simply take each line of the input text file
and store it (after parsing) in a database, like SimpleDB.

Can I put this code into Mapper, make no call to "collect" in it, and have
no reducers at all? Do I set the reduce class to
null, conf.setReducerClass(null)? or not set it at all?

Thank you,
Mark

Re: Import the results into SimpleDB

[Amandeep Khurana <am...@gmail.com>]

>
> Might as well not use Hadoop then...


Hadoop makes it easy to parallelize the work... Makes perfect sense to use
it!

Re: Import the results into SimpleDB

[Mark Kerzner <ma...@gmail.com>]

:)

I create this text file in Hadoop. Only I want to make the db import a
separate Hadoop job, run it in Amazon EMR, and make it fast by running
sufficient number of nodes.

Mark

On Tue, May 11, 2010 at 9:13 PM, Darren Govoni <da...@ontrenet.com> wrote:

> Might as well not use Hadoop then...
>
> On Tue, 2010-05-11 at 21:02 -0500, Mark Kerzner wrote:
>
> > Hi,
> >
> > I want a Hadoop job that will simply take each line of the input text
> file
> > and store it (after parsing) in a database, like SimpleDB.
> >
> > Can I put this code into Mapper, make no call to "collect" in it, and
> have
> > no reducers at all? Do I set the reduce class to
> > null, conf.setReducerClass(null)? or not set it at all?
> >
> > Thank you,
> > Mark
>
>
>

Re: Import the results into SimpleDB

[Darren Govoni <da...@ontrenet.com>]

Might as well not use Hadoop then...

On Tue, 2010-05-11 at 21:02 -0500, Mark Kerzner wrote:

> Hi,
> 
> I want a Hadoop job that will simply take each line of the input text file
> and store it (after parsing) in a database, like SimpleDB.
> 
> Can I put this code into Mapper, make no call to "collect" in it, and have
> no reducers at all? Do I set the reduce class to
> null, conf.setReducerClass(null)? or not set it at all?
> 
> Thank you,
> Mark