You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jean-Daniel Cryans <jd...@apache.org> on 2009/10/07 23:21:02 UTC
On storing HBase data in AWS S3
Hi users,
I've recently helped debugging a 0.19 HBase setup that was using S3 as
its DFS (one of the problem is discussed in another thread) and I
think I've gathered enough information to guide new users on whether
this is a valuable solution.
Short answer: don't use it for user-facing apps, consider it for
elastic EC2 clusters.
Long answer:
The main reason why you would want to store your data inside S3 would
be because of the marketed high availability and infinite scalability.
As the website says: "It gives any developer access to the same highly
scalable, reliable, fast, inexpensive data storage infrastructure that
Amazon uses to run its own global network of web sites. The service
aims to maximize benefits of scale and to pass those benefits on to
developers." BTW I don't refute any of this as in my experience this
has been mostly true.
HBase can use any filesystem supported in Hadoop, including S3, so it
seems like a no brainer to use it instead of having to setup Hadoop.
Yes indeed, but...
- You absolutely have to deploy your region servers in EC2 because of
the obvious latency and bandwidth every filesystem access will occur.
- The way the S3 code works in Hadoop, it writes on disk every inbound
and outbound file. Apart from slowing down even more every operation,
if you didn't change the hadoop.tmp.dir it will write in /tmp and that
volume on EC2 is always very very small. In fact, the first thing I
had to debug was a "No space left on device" which seems weird since
S3 should have infinite storage, but the error was really given when
data was written in the tmp folder.
- There are some unknown interactions because HBase has a very
different file usage pattern than MapReduce jobs and was optimized for
HDFS, not distant networked storage.
So if you need speed, simply don't use S3 with HBase as it will be too
slow . You can consider using it for elastic MapReduce jobs the same
way people use it with Hadoop because you don't have to keep all the
nodes up all the time.
J-D
Re: On storing HBase data in AWS S3
Posted by "Ananth T. Sarathy" <an...@gmail.com>.
Yeah, first of all a lot of thanks for your help. We are moving over to
version .20.0 now, and we'll let you know how that goes. We've learned a
few things in the process so we hope to get that ready soon.
It seems to work fairly well inside the cloud. As far as apps outside the
cloud, we have noticed a lot of weirdness with configuring it (which has
more to do with us I am sure) which includes region servers binding to the
internal IP address rather than the external which makes it hard to test and
develop against from outside the cloud.
Additionally a couple of other pointers you need to download the jets3t and
the commons-codec to your lib directory to connect to s3, and you need your
core-site.xml to have all your s3 login info.
Ananth T Sarathy
On Wed, Oct 7, 2009 at 5:21 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Hi users,
>
> I've recently helped debugging a 0.19 HBase setup that was using S3 as
> its DFS (one of the problem is discussed in another thread) and I
> think I've gathered enough information to guide new users on whether
> this is a valuable solution.
>
> Short answer: don't use it for user-facing apps, consider it for
> elastic EC2 clusters.
>
> Long answer:
>
> The main reason why you would want to store your data inside S3 would
> be because of the marketed high availability and infinite scalability.
> As the website says: "It gives any developer access to the same highly
> scalable, reliable, fast, inexpensive data storage infrastructure that
> Amazon uses to run its own global network of web sites. The service
> aims to maximize benefits of scale and to pass those benefits on to
> developers." BTW I don't refute any of this as in my experience this
> has been mostly true.
>
> HBase can use any filesystem supported in Hadoop, including S3, so it
> seems like a no brainer to use it instead of having to setup Hadoop.
> Yes indeed, but...
>
> - You absolutely have to deploy your region servers in EC2 because of
> the obvious latency and bandwidth every filesystem access will occur.
> - The way the S3 code works in Hadoop, it writes on disk every inbound
> and outbound file. Apart from slowing down even more every operation,
> if you didn't change the hadoop.tmp.dir it will write in /tmp and that
> volume on EC2 is always very very small. In fact, the first thing I
> had to debug was a "No space left on device" which seems weird since
> S3 should have infinite storage, but the error was really given when
> data was written in the tmp folder.
> - There are some unknown interactions because HBase has a very
> different file usage pattern than MapReduce jobs and was optimized for
> HDFS, not distant networked storage.
>
> So if you need speed, simply don't use S3 with HBase as it will be too
> slow . You can consider using it for elastic MapReduce jobs the same
> way people use it with Hadoop because you don't have to keep all the
> nodes up all the time.
>
> J-D
>
Re: On storing HBase data in AWS S3
Posted by Keith Thomas <ke...@gmail.com>.
A big thanks to everyone who contributed to this post. The discussion above
has been very helpful to my efforts in this area.
--
View this message in context: http://www.nabble.com/On-storing-HBase-data-in-AWS-S3-tp25794704p25902203.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: On storing HBase data in AWS S3
Posted by Tatsuya Kawano <ta...@snowcocoa.info>.
Hi Keith,
> On Wed, Oct 14, 2009 at 11:58 AM, Keith Thomas wrote:
>> Am I correct in understanding that a farm of EC2 instances with Hadoop and
>> HBase installed and configured individually by myself are the quickest and
>> most effective way to progress with this effort?
On Wed, Oct 14, 2009 at 12:31 AM, Tatsuya Kawano wrote:
> Well, you're not wrong. To run HBase on Amazon Web Services, you
> should use EC2 instances and configure them by yourself. Make sure you
> pick Extra Large instances from EC2 (see:
> http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A8), and you may
> also want EBS volumes as the storage devices rather than S3. (S3 is
> good for archiving data)
I checked this again and realized the easiest way to set up your EC2
instances could be using Cloudera's pre-built disk images and client
scripts.
HBase added to Cloudera distribution (hbase-user announcement by Stack)
http://bit.ly/1dDutR
Getting Started -- Cloudera Distribution AMI for Hadoop:
http://bit.ly/4ma9Tv
As I said in the previous post, use EBS (not S3) as the underling
storage for HDFS.
An correction from my last post.
On Thu, Oct 15, 2009 at 2:25 AM, Tatsuya Kawano wrote:
> Note that an Extra Large instance of Amazon EC2 is equipped with 4
> virtual CPU cores and 7.9GB RAM.
-- A High-CPU Extra Large (c1.xlarge) instance is equipped with 8
virtual CPU cores and 7 GB RAM.
-- An Extra Large (m1.xlarge) instance is equipped with 4 virtual CPU
cores and 15 GB RAM.
Thanks,
--
Tatsuya Kawano (Mr.)
Tokyo, Japan
Re: On storing HBase data in AWS S3
Posted by Tatsuya Kawano <ta...@snowcocoa.info>.
Hi Ryan,
Thanks for correcting me, and also thanks for the great work since 0.20.
So, the facts are:
1. HBase (0.20.0 --) is capable to handle realtime queries. (e.g. stumbleupon)
2. HBase will fit in certain parts of realtime query use cases, where
transaction isolation is not so important but flexible data structure
is essential.
3. The minimum HBase configuration requires more servers than MySQL
does, but HBase can scale beyond anything MySQL can do.
4. There is a Grails plugin for HBase. (Sorry Keith, I didn't know you
already wrote it.)
One more question:
Server Layout
So I can fit HBase processes in 4 EC2 instances (1 master + 3
regions). How about 3 ZooKeepers and DFS Name Node / Secondary Name
None? Where should they go?
Maybe...?
Server 1. DFS Name Node / ZooKeeper #1
Server 2. HBase Master / ZooKeeper #2
Server 3. DFS Secondary Name Node / HBase Master (Backup) / ZooKeeper #3
Server 4. HBase Region Server / DFS Data Node
Server 5. HBase Region Server / DFS Data Node
Server 6. HBase Region Server / DFS Data Node
So do I need 6 EC2 Extra Large instances? Or are you suggesting to
squeeze ZKs and name nodes into the same 4 instances?
Note that an Extra Large instance of Amazon EC2 is equipped with 4
virtual CPU cores and 7.9GB RAM.
Thanks,
--
Tatsuya Kawano (Mr.)
Tokyo, Japan
On Wed, Oct 14, 2009 at 5:02 PM, Ryan Rawson <ry...@gmail.com> wrote:
> Hey!
>
> I strongly disagree with Tatsaya's assessment of HBase, specifically below:
>
>
> On Wed, Oct 14, 2009 at 12:31 AM, Tatsuya Kawano
> <ta...@snowcocoa.info> wrote:
>> HI Keith,
>>
>> On Wed, Oct 14, 2009 at 11:58 AM, Keith Thomas <ke...@gmail.com> wrote:
>>> Am I correct in understanding that a farm of EC2 instances with Hadoop and
>>> HBase installed and configured individually by myself are the quickest and
>>> most effective way to progress with this effort?
>>
>> Well, you're not wrong. To run HBase on Amazon Web Services, you
>> should use EC2 instances and configure them by yourself. Make sure you
>> pick Extra Large instances from EC2 (see:
>> http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A8), and you may
>> also want EBS volumes as the storage devices rather than S3. (S3 is
>> good for archiving data)
>>
>>
>> But...
>>
>> Are you really sure you want to use HBase for your Grail based web
>> application on the cloud? I would definitely recommend MySQL which
>> should be more suitable for both web applications and Amazon Web
>> Services environment. HBase is not a cloud database and is currently
>> more suitable for batch processing with billions of records.
>
> This is not a correct assessment - first off, what does it mean to be
> a "cloud database". And secondly, HBase is suitable for storing real
> time queries, and it is a major use case that we have here at
> stumbleupon.
>
>>
>> If you use HBase for this purpose, you will
>>
>> -- loose the Object Relational Mapping support from Grails.
>> -- have to take care of database transactions and secondary indices by yourself.
>
> You do "lose" the transactions (if you even used them) and you may
> have to maintain secondary indexes, but you gain a flexible
> schema-less column-oriented datastore that scales far beyond anything
> mysql can do.
>
>> -- likely suffered from a latency of data retrieval, unless you use memcached.
>
> This is not correct - HBase has good caching built in, and takes full
> advantage of linux's disk buffer cache. Much more effective than MySQL
> because it is easier to get more ram across 10-20 machines (or more)
> than ram in 1-2 machines.
>
>
>> -- need more server resources than MySQL. MySQL can run on 1 EC2
>> instance, while HBase requires about 12 EC2 instances (2 for masters
>> and DFS namenodes, 5 for region servers and DFS datanodes, 5 for
>> ZooKeeper)
>
> Again, this is not entirely correct, you are overspecing quite a bit.
> 3 ZK nodes is fine, and they should be able to run on the "master"
> nodes. And you also reveal your misunderstanding, suggesting to the OP
> that you can run namenode on 2 hosts and that is that. The situation
> for HDFS is (unfortunately) more complicated than that.
>
> It is totally possible for a HBase cluster to be run on 4 EC2
> instances, 1 master, 3 datanodes. Maybe even less, but you are
> sacrificing data reliability.
>
>
> i appreciate your enthusiasm for HBase, but please don't mislead our
> users so badly!
>
> Thanks,
> -ryan
>
>>
>>
>> Is there any special reason to use HBase for you web application?
>>
>> Thanks,
>>
>> --
>> Tatsuya Kawano (Mr.)
>> Tokyo, Japan
>>
RE: On storing HBase data in AWS S3
Posted by Sanjay Sharma <sa...@impetus.co.in>.
Hi Keith,
"I am looking for the easiest way to bring up an HBase and Hadoop environment as the persistence mechanism for a Grails based web application."
I think Ryan has already cleared the doubts around using HBase for live applications.
You might start looking at a new grails plugin- http://grails.org/plugin/gorm-hbase to get a head start.
-sanjay
-----Original Message-----
From: Ryan Rawson [mailto:ryanobjc@gmail.com]
Sent: Wednesday, October 14, 2009 1:32 PM
To: hbase-user@hadoop.apache.org
Subject: Re: On storing HBase data in AWS S3
Hey!
I strongly disagree with Tatsaya's assessment of HBase, specifically below:
On Wed, Oct 14, 2009 at 12:31 AM, Tatsuya Kawano
<ta...@snowcocoa.info> wrote:
> HI Keith,
>
> On Wed, Oct 14, 2009 at 11:58 AM, Keith Thomas <ke...@gmail.com> wrote:
>> Am I correct in understanding that a farm of EC2 instances with Hadoop and
>> HBase installed and configured individually by myself are the quickest and
>> most effective way to progress with this effort?
>
> Well, you're not wrong. To run HBase on Amazon Web Services, you
> should use EC2 instances and configure them by yourself. Make sure you
> pick Extra Large instances from EC2 (see:
> http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A8), and you may
> also want EBS volumes as the storage devices rather than S3. (S3 is
> good for archiving data)
>
>
> But...
>
> Are you really sure you want to use HBase for your Grail based web
> application on the cloud? I would definitely recommend MySQL which
> should be more suitable for both web applications and Amazon Web
> Services environment. HBase is not a cloud database and is currently
> more suitable for batch processing with billions of records.
This is not a correct assessment - first off, what does it mean to be
a "cloud database". And secondly, HBase is suitable for storing real
time queries, and it is a major use case that we have here at
stumbleupon.
>
> If you use HBase for this purpose, you will
>
> -- loose the Object Relational Mapping support from Grails.
> -- have to take care of database transactions and secondary indices by yourself.
You do "lose" the transactions (if you even used them) and you may
have to maintain secondary indexes, but you gain a flexible
schema-less column-oriented datastore that scales far beyond anything
mysql can do.
> -- likely suffered from a latency of data retrieval, unless you use memcached.
This is not correct - HBase has good caching built in, and takes full
advantage of linux's disk buffer cache. Much more effective than MySQL
because it is easier to get more ram across 10-20 machines (or more)
than ram in 1-2 machines.
> -- need more server resources than MySQL. MySQL can run on 1 EC2
> instance, while HBase requires about 12 EC2 instances (2 for masters
> and DFS namenodes, 5 for region servers and DFS datanodes, 5 for
> ZooKeeper)
Again, this is not entirely correct, you are overspecing quite a bit.
3 ZK nodes is fine, and they should be able to run on the "master"
nodes. And you also reveal your misunderstanding, suggesting to the OP
that you can run namenode on 2 hosts and that is that. The situation
for HDFS is (unfortunately) more complicated than that.
It is totally possible for a HBase cluster to be run on 4 EC2
instances, 1 master, 3 datanodes. Maybe even less, but you are
sacrificing data reliability.
i appreciate your enthusiasm for HBase, but please don't mislead our
users so badly!
Thanks,
-ryan
>
>
> Is there any special reason to use HBase for you web application?
>
> Thanks,
>
> --
> Tatsuya Kawano (Mr.)
> Tokyo, Japan
>
Follow us on Twitter- https://twitter.com/impetuscalling.
*Impetus Celebrates Green Diwali.
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Re: On storing HBase data in AWS S3
Posted by Ryan Rawson <ry...@gmail.com>.
Hey!
I strongly disagree with Tatsaya's assessment of HBase, specifically below:
On Wed, Oct 14, 2009 at 12:31 AM, Tatsuya Kawano
<ta...@snowcocoa.info> wrote:
> HI Keith,
>
> On Wed, Oct 14, 2009 at 11:58 AM, Keith Thomas <ke...@gmail.com> wrote:
>> Am I correct in understanding that a farm of EC2 instances with Hadoop and
>> HBase installed and configured individually by myself are the quickest and
>> most effective way to progress with this effort?
>
> Well, you're not wrong. To run HBase on Amazon Web Services, you
> should use EC2 instances and configure them by yourself. Make sure you
> pick Extra Large instances from EC2 (see:
> http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A8), and you may
> also want EBS volumes as the storage devices rather than S3. (S3 is
> good for archiving data)
>
>
> But...
>
> Are you really sure you want to use HBase for your Grail based web
> application on the cloud? I would definitely recommend MySQL which
> should be more suitable for both web applications and Amazon Web
> Services environment. HBase is not a cloud database and is currently
> more suitable for batch processing with billions of records.
This is not a correct assessment - first off, what does it mean to be
a "cloud database". And secondly, HBase is suitable for storing real
time queries, and it is a major use case that we have here at
stumbleupon.
>
> If you use HBase for this purpose, you will
>
> -- loose the Object Relational Mapping support from Grails.
> -- have to take care of database transactions and secondary indices by yourself.
You do "lose" the transactions (if you even used them) and you may
have to maintain secondary indexes, but you gain a flexible
schema-less column-oriented datastore that scales far beyond anything
mysql can do.
> -- likely suffered from a latency of data retrieval, unless you use memcached.
This is not correct - HBase has good caching built in, and takes full
advantage of linux's disk buffer cache. Much more effective than MySQL
because it is easier to get more ram across 10-20 machines (or more)
than ram in 1-2 machines.
> -- need more server resources than MySQL. MySQL can run on 1 EC2
> instance, while HBase requires about 12 EC2 instances (2 for masters
> and DFS namenodes, 5 for region servers and DFS datanodes, 5 for
> ZooKeeper)
Again, this is not entirely correct, you are overspecing quite a bit.
3 ZK nodes is fine, and they should be able to run on the "master"
nodes. And you also reveal your misunderstanding, suggesting to the OP
that you can run namenode on 2 hosts and that is that. The situation
for HDFS is (unfortunately) more complicated than that.
It is totally possible for a HBase cluster to be run on 4 EC2
instances, 1 master, 3 datanodes. Maybe even less, but you are
sacrificing data reliability.
i appreciate your enthusiasm for HBase, but please don't mislead our
users so badly!
Thanks,
-ryan
>
>
> Is there any special reason to use HBase for you web application?
>
> Thanks,
>
> --
> Tatsuya Kawano (Mr.)
> Tokyo, Japan
>
Re: On storing HBase data in AWS S3
Posted by Tatsuya Kawano <ta...@snowcocoa.info>.
HI Keith,
On Wed, Oct 14, 2009 at 11:58 AM, Keith Thomas <ke...@gmail.com> wrote:
> Am I correct in understanding that a farm of EC2 instances with Hadoop and
> HBase installed and configured individually by myself are the quickest and
> most effective way to progress with this effort?
Well, you're not wrong. To run HBase on Amazon Web Services, you
should use EC2 instances and configure them by yourself. Make sure you
pick Extra Large instances from EC2 (see:
http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A8), and you may
also want EBS volumes as the storage devices rather than S3. (S3 is
good for archiving data)
But...
Are you really sure you want to use HBase for your Grail based web
application on the cloud? I would definitely recommend MySQL which
should be more suitable for both web applications and Amazon Web
Services environment. HBase is not a cloud database and is currently
more suitable for batch processing with billions of records.
If you use HBase for this purpose, you will
-- loose the Object Relational Mapping support from Grails.
-- have to take care of database transactions and secondary indices by yourself.
-- likely suffered from a latency of data retrieval, unless you use memcached.
-- need more server resources than MySQL. MySQL can run on 1 EC2
instance, while HBase requires about 12 EC2 instances (2 for masters
and DFS namenodes, 5 for region servers and DFS datanodes, 5 for
ZooKeeper)
Is there any special reason to use HBase for you web application?
Thanks,
--
Tatsuya Kawano (Mr.)
Tokyo, Japan
Re: On storing HBase data in AWS S3
Posted by Keith Thomas <ke...@gmail.com>.
I am a complete newbie to the wonderful world of Amazon services so I
apologize if I am asking a question that has already been answered.
I am looking for the easiest way to bring up an HBase and Hadoop environment
as the persistence mechanism for a Grails based web application. I was not
entirely clear which of the myriad of services offered provides the best
approach - EC2, S3, Elastic Map/Reduce, etc etc - until the previous post
pointed me towards EC2 over S3.
Am I correct in understanding that a farm of EC2 instances with Hadoop and
HBase installed and configured individually by myself are the quickest and
most effective way to progress with this effort?
Jean-Daniel Cryans-2 wrote:
>
> Hi users,
>
> I've recently helped debugging a 0.19 HBase setup that was using S3 as
> its DFS (one of the problem is discussed in another thread) and I
> think I've gathered enough information to guide new users on whether
> this is a valuable solution.
>
> Short answer: don't use it for user-facing apps, consider it for
> elastic EC2 clusters.
>
> Long answer:
>
> The main reason why you would want to store your data inside S3 would
> be because of the marketed high availability and infinite scalability.
> As the website says: "It gives any developer access to the same highly
> scalable, reliable, fast, inexpensive data storage infrastructure that
> Amazon uses to run its own global network of web sites. The service
> aims to maximize benefits of scale and to pass those benefits on to
> developers." BTW I don't refute any of this as in my experience this
> has been mostly true.
>
> HBase can use any filesystem supported in Hadoop, including S3, so it
> seems like a no brainer to use it instead of having to setup Hadoop.
> Yes indeed, but...
>
> - You absolutely have to deploy your region servers in EC2 because of
> the obvious latency and bandwidth every filesystem access will occur.
> - The way the S3 code works in Hadoop, it writes on disk every inbound
> and outbound file. Apart from slowing down even more every operation,
> if you didn't change the hadoop.tmp.dir it will write in /tmp and that
> volume on EC2 is always very very small. In fact, the first thing I
> had to debug was a "No space left on device" which seems weird since
> S3 should have infinite storage, but the error was really given when
> data was written in the tmp folder.
> - There are some unknown interactions because HBase has a very
> different file usage pattern than MapReduce jobs and was optimized for
> HDFS, not distant networked storage.
>
> So if you need speed, simply don't use S3 with HBase as it will be too
> slow . You can consider using it for elastic MapReduce jobs the same
> way people use it with Hadoop because you don't have to keep all the
> nodes up all the time.
>
> J-D
>
>
--
View this message in context: http://www.nabble.com/On-storing-HBase-data-in-AWS-S3-tp25794704p25884592.html
Sent from the HBase User mailing list archive at Nabble.com.