You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mark Kerzner <ma...@gmail.com> on 2009/10/14 00:12:52 UTC
Database to use with Hadoop
Hi,
I run Elastic MapReduce. The output of my application is a text file, where
each line is essentially a set of fields. It will fit very nicely into a
simple database, but which database
1. Is persistent after cluster shutdown;
2. Can be written to by many reducers?
Amazon SimpleDB could do - but does it work with Hadoop?
Thank you,
Mark
Re: Database to use with Hadoop
Posted by Mark Kerzner <ma...@gmail.com>.
Thank you, all. It looks like SimpleDB may be good enough for my needs. The
forums claim that you can write to it from all reducers at once, being that
it is highly optimized for concurrent access.
On Tue, Oct 13, 2009 at 5:30 PM, Jeff Hammerbacher <ha...@cloudera.com>wrote:
> Hey Mark,
>
> You will probably get some mileage from
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2571.
>
> Regards,
> Jeff
>
> On Tue, Oct 13, 2009 at 3:19 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
>
> > You can put into Hbase. Or you can use the DBOutputFormat and interface
> > with
> > an RDBMS.
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Tue, Oct 13, 2009 at 3:12 PM, Mark Kerzner <ma...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I run Elastic MapReduce. The output of my application is a text file,
> > where
> > > each line is essentially a set of fields. It will fit very nicely into
> a
> > > simple database, but which database
> > >
> > > 1. Is persistent after cluster shutdown;
> > > 2. Can be written to by many reducers?
> > >
> > > Amazon SimpleDB could do - but does it work with Hadoop?
> > >
> > > Thank you,
> > > Mark
> > >
> >
>
Re: Database to use with Hadoop
Posted by Jeff Hammerbacher <ha...@cloudera.com>.
Hey Mark,
You will probably get some mileage from
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2571.
Regards,
Jeff
On Tue, Oct 13, 2009 at 3:19 PM, Amandeep Khurana <am...@gmail.com> wrote:
> You can put into Hbase. Or you can use the DBOutputFormat and interface
> with
> an RDBMS.
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Tue, Oct 13, 2009 at 3:12 PM, Mark Kerzner <ma...@gmail.com>
> wrote:
>
> > Hi,
> > I run Elastic MapReduce. The output of my application is a text file,
> where
> > each line is essentially a set of fields. It will fit very nicely into a
> > simple database, but which database
> >
> > 1. Is persistent after cluster shutdown;
> > 2. Can be written to by many reducers?
> >
> > Amazon SimpleDB could do - but does it work with Hadoop?
> >
> > Thank you,
> > Mark
> >
>
Re: Database to use with Hadoop
Posted by Amandeep Khurana <am...@gmail.com>.
You can put into Hbase. Or you can use the DBOutputFormat and interface with
an RDBMS.
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Tue, Oct 13, 2009 at 3:12 PM, Mark Kerzner <ma...@gmail.com> wrote:
> Hi,
> I run Elastic MapReduce. The output of my application is a text file, where
> each line is essentially a set of fields. It will fit very nicely into a
> simple database, but which database
>
> 1. Is persistent after cluster shutdown;
> 2. Can be written to by many reducers?
>
> Amazon SimpleDB could do - but does it work with Hadoop?
>
> Thank you,
> Mark
>