You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by bayee <ba...@ownskin.com> on 2009/03/17 05:33:16 UTC
hadoop migration
Hi,
We are running a website with quiet a lot of traffic. At the moment we
are using about 20 sql servers and about 60 application servers/file
servers. We are thinking of porting everything to hadoop. My question is
does 80 nodes of hadoop can perform much better than 20 sql server + 60
native file servers?
We have tried setup 1 hadoop server and run a simple grep example, and
the speed is very slow. Does hadoop can only perform under a lot of
nodes? What is the minimum of nodes do we need to replace our current 20
sql server + 60 app/file servers?
Best Wishes,
Hsin Yee
Re: hadoop migration
Posted by Stuart Sierra <th...@gmail.com>.
On Sun, Mar 22, 2009 at 2:17 PM, nitesh bhatia
<ni...@gmail.com> wrote:
> Bigtable ???
> Is it opensource ? I am not sure if google has released any code of
> bigtable. So far only 1 research paper is available.
No, Google has never released source code for MapReduce and BigTable.
Hadoop and HBase attempt to fill that gap.
-Stuart
Re: hadoop migration
Posted by "Philip M. White" <pm...@qnan.org>.
On Sun, Mar 22, 2009 at 11:47:35PM +0530, nitesh bhatia wrote:
> Bigtable ???
> Is it opensource ? I am not sure if google has released any code of
> bigtable. So far only 1 research paper is available.
HBase is an implementation of BigTable.
--
Philip
Re: hadoop migration
Posted by nitesh bhatia <ni...@gmail.com>.
Bigtable ???
Is it opensource ? I am not sure if google has released any code of
bigtable. So far only 1 research paper is available.
--nitesh
On Tue, Mar 17, 2009 at 11:01 AM, Amandeep Khurana <am...@gmail.com> wrote:
> AFAIK, Google uses BigTable for pretty much most of their backend stuff.
> The
> thing to note here is that BigTable is much more mature than Hbase.
>
> You can try it out and see how it works out for you. Do share your results
> on the mailing list...
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
> On Mon, Mar 16, 2009 at 10:28 PM, W <wi...@gmail.com> wrote:
>
> > Thanks for the quick response Aman,
> >
> > Ok .., i see the point now.
> >
> > currently i'm doing some research on creating a google books like
> > application using hbase as
> > a backend for storing the files and solr as indexer. From this
> > prototype, my be i can measure how fast
> > is hbase on serving data to the client ... (google using bigTable for
> > their books.google.com right ?)
> >
> > Thanks!
> >
> > Regards,
> > Wildan
> >
> > On Tue, Mar 17, 2009 at 12:13 PM, Amandeep Khurana <am...@gmail.com>
> > wrote:
> > > Hypertable is not as mature as Hbase yet. The next release of Hbase,
> > 0.20.0,
> > > includes some patches which reduce the latency of responses and makes
> it
> > > suitable to be used as a backend for a webapp. However the current
> > release
> > > isnt optimized for this purpose.
> > >
> > > The idea behind Hadoop and the rest of the tools around it is more of a
> > data
> > > processing system than a backend datastore for a website. The output of
> > the
> > > processing that Hadoop does is typically taken into a MySQL cluster
> which
> > > feeds a website.
> > >
> > >
> > >
> >
> >
> > --
> > ---
> > OpenThink Labs
> > www.tobethink.com
> >
> > Aligning IT and Education
> >
> > >> 021-99325243
> > Y! : hawking_123
> > Linkedln : http://www.linkedin.com/in/wildanmaulana
> >
>
--
Nitesh Bhatia
Dhirubhai Ambani Institute of Information & Communication Technology
Gandhinagar
Gujarat
"Life is never perfect. It just depends where you draw the line."
visit:
http://www.awaaaz.com - connecting through music
http://www.volstreet.com - lets volunteer for better tomorrow
http://www.instibuzz.com - Voice opinions, Transact easily, Have fun
Re: hadoop migration
Posted by Amandeep Khurana <am...@gmail.com>.
AFAIK, Google uses BigTable for pretty much most of their backend stuff. The
thing to note here is that BigTable is much more mature than Hbase.
You can try it out and see how it works out for you. Do share your results
on the mailing list...
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Mon, Mar 16, 2009 at 10:28 PM, W <wi...@gmail.com> wrote:
> Thanks for the quick response Aman,
>
> Ok .., i see the point now.
>
> currently i'm doing some research on creating a google books like
> application using hbase as
> a backend for storing the files and solr as indexer. From this
> prototype, my be i can measure how fast
> is hbase on serving data to the client ... (google using bigTable for
> their books.google.com right ?)
>
> Thanks!
>
> Regards,
> Wildan
>
> On Tue, Mar 17, 2009 at 12:13 PM, Amandeep Khurana <am...@gmail.com>
> wrote:
> > Hypertable is not as mature as Hbase yet. The next release of Hbase,
> 0.20.0,
> > includes some patches which reduce the latency of responses and makes it
> > suitable to be used as a backend for a webapp. However the current
> release
> > isnt optimized for this purpose.
> >
> > The idea behind Hadoop and the rest of the tools around it is more of a
> data
> > processing system than a backend datastore for a website. The output of
> the
> > processing that Hadoop does is typically taken into a MySQL cluster which
> > feeds a website.
> >
> >
> >
>
>
> --
> ---
> OpenThink Labs
> www.tobethink.com
>
> Aligning IT and Education
>
> >> 021-99325243
> Y! : hawking_123
> Linkedln : http://www.linkedin.com/in/wildanmaulana
>
Re: hadoop migration
Posted by W <wi...@gmail.com>.
Thanks for the quick response Aman,
Ok .., i see the point now.
currently i'm doing some research on creating a google books like
application using hbase as
a backend for storing the files and solr as indexer. From this
prototype, my be i can measure how fast
is hbase on serving data to the client ... (google using bigTable for
their books.google.com right ?)
Thanks!
Regards,
Wildan
On Tue, Mar 17, 2009 at 12:13 PM, Amandeep Khurana <am...@gmail.com> wrote:
> Hypertable is not as mature as Hbase yet. The next release of Hbase, 0.20.0,
> includes some patches which reduce the latency of responses and makes it
> suitable to be used as a backend for a webapp. However the current release
> isnt optimized for this purpose.
>
> The idea behind Hadoop and the rest of the tools around it is more of a data
> processing system than a backend datastore for a website. The output of the
> processing that Hadoop does is typically taken into a MySQL cluster which
> feeds a website.
>
>
>
--
---
OpenThink Labs
www.tobethink.com
Aligning IT and Education
>> 021-99325243
Y! : hawking_123
Linkedln : http://www.linkedin.com/in/wildanmaulana
Re: hadoop migration
Posted by Amandeep Khurana <am...@gmail.com>.
Hypertable is not as mature as Hbase yet. The next release of Hbase, 0.20.0,
includes some patches which reduce the latency of responses and makes it
suitable to be used as a backend for a webapp. However the current release
isnt optimized for this purpose.
The idea behind Hadoop and the rest of the tools around it is more of a data
processing system than a backend datastore for a website. The output of the
processing that Hadoop does is typically taken into a MySQL cluster which
feeds a website.
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Mon, Mar 16, 2009 at 10:05 PM, W <wi...@gmail.com> wrote:
> > Of course, There is a storage solution called HBase for Hadoop. But,
> > In my experience, not applicable for online data access yet.
> >
>
> I see.., how about hypertable ? does it mature enough to be used in
> production ? , i read that
> hypertable can be integrated with hadoop, or is there any other
> alternative other than hbase ?
>
> Thanks!
>
> Regards,
> Wildan
>
> --
> ---
> OpenThink Labs
> www.tobethink.com
>
> Aligning IT and Education
>
> >> 021-99325243
> Y! : hawking_123
> Linkedln : http://www.linkedin.com/in/wildanmaulana
>
Re: hadoop migration
Posted by W <wi...@gmail.com>.
> Of course, There is a storage solution called HBase for Hadoop. But,
> In my experience, not applicable for online data access yet.
>
I see.., how about hypertable ? does it mature enough to be used in
production ? , i read that
hypertable can be integrated with hadoop, or is there any other
alternative other than hbase ?
Thanks!
Regards,
Wildan
--
---
OpenThink Labs
www.tobethink.com
Aligning IT and Education
>> 021-99325243
Y! : hawking_123
Linkedln : http://www.linkedin.com/in/wildanmaulana
Re: hadoop migration
Posted by "Edward J. Yoon" <ed...@apache.org>.
Hi,
Your SQL servers seems database accessible by the internet. Hadoop is
a distributed file-system And, it's quite different with (Database +
SAN storage) cluster architecture.
Of course, There is a storage solution called HBase for Hadoop. But,
In my experience, not applicable for online data access yet.
On Tue, Mar 17, 2009 at 1:33 PM, bayee <ba...@ownskin.com> wrote:
> Hi,
>
> We are running a website with quiet a lot of traffic. At the moment we are
> using about 20 sql servers and about 60 application servers/file servers. We
> are thinking of porting everything to hadoop. My question is does 80 nodes
> of hadoop can perform much better than 20 sql server + 60 native file
> servers?
>
> We have tried setup 1 hadoop server and run a simple grep example, and the
> speed is very slow. Does hadoop can only perform under a lot of nodes? What
> is the minimum of nodes do we need to replace our current 20 sql server + 60
> app/file servers?
>
> Best Wishes,
> Hsin Yee
>
--
Best Regards, Edward J. Yoon
edwardyoon@apache.org
http://blog.udanax.org