Question about database application in MapReduce framework

[Ming Yang <mi...@gmail.com>]

Hi,

I am a new learner to MapReduce framework and the Hadoop implementation.
Currently I am trying to rewrite a application to utilize the infrastructure,
However the application need to access the database, which is not clustered,
very frequently, and it seems not to be very suitable to fit in the MapReduce
framework, since one advantage that MapReduce provides is to improve data
locality by distributed file system. Is there any practical way to
migrate existing
applications which is using non-distributed database to the MapReduce framework
or to use Hadoop?

Thank you,

Ming Yang

RE: Question about database application in MapReduce framework

[Devaraj Das <dd...@yahoo-inc.com>]

In short, the answer is, you could use hadoop to process input data and also
update the centralized database during the data processing. However, your
centralized database might become the bottleneck to scaling, since it might
start choking when the number of concurrent accesses goes beyond a certain
number.

> -----Original Message-----
> From: Ming Yang [mailto:minghsien@gmail.com] 
> Sent: Friday, September 21, 2007 10:13 AM
> To: hadoop-user@lucene.apache.org
> Subject: Question about database application in MapReduce framework
> 
> Hi,
> 
> I am a new learner to MapReduce framework and the Hadoop 
> implementation.
> Currently I am trying to rewrite a application to utilize the 
> infrastructure, However the application need to access the 
> database, which is not clustered, very frequently, and it 
> seems not to be very suitable to fit in the MapReduce 
> framework, since one advantage that MapReduce provides is to 
> improve data locality by distributed file system. Is there 
> any practical way to migrate existing applications which is 
> using non-distributed database to the MapReduce framework or 
> to use Hadoop?
> 
> Thank you,
> 
> Ming Yang
>