Re: How to improve HBase throughput with YCSB?

[Harold Lim <ro...@yahoo.com>]

Hi Andrew,

I tried running on c1.xlarge instances and the performance improved a little bit but the throughput is still low. I can now get throughput of 700+ read operations per second (up from 400-500+). I was hoping to get throughput in the order of thousands.

I was wondering if there is something wrong with my set-up or is it normal for HBase running on ec2 instances to get low throughput numbers?


-Harold


--- On Mon, 5/30/11, Andrew Purtell <ap...@apache.org> wrote:

> From: Andrew Purtell <ap...@apache.org>
> Subject: Re: How to improve HBase throughput with YCSB?
> To: user@hbase.apache.org
> Date: Monday, May 30, 2011, 8:33 PM
> The hypervisor steals a lot of CPU
> time from m1.large instances.  You should be using
> c1.xlarge instances.
> 
> Are you using local storage or EBS?
> 
> Be aware that I/O performance on EC2 for any system is
> lower than if you are using real hardware, significantly so
> if not using one of the instance types with I/O performance
> listed as "high". 
> 
> > 2011/5/30 Harold Lim <ro...@yahoo.com>
> > 
> > > Hi All,
> > >
> > > I have an HBase cluster on ec2 m1.large instance
> (10
> > > region servers).
> 
> 

Re: How to improve HBase throughput with YCSB?

[Jeff Whiting <je...@qualtrics.com>]

I saw large increases in performance with YCSB by enabling bloom filters. (Sorry it has been too 
long to remember how much, but it was significant).

~Jeff

On 6/1/2011 8:41 AM, Ted Dunning wrote:
> y need to read several locations to access your data
> since it effectively overlays multiple hfiles.

-- 
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com

Re: How to improve HBase throughput with YCSB?

[Ted Dunning <td...@maprtech.com>]

Answers in-line.

On Wed, Jun 1, 2011 at 12:42 AM, Harold Lim <ro...@yahoo.com> wrote:

> Hi Ted,
>
> > You appear to be running on about 10 disks total.
> > Each disk should be
> > capable of about 100 ops per second but they appear to be
> > doing about 70.
> >  This is plausible overhead.
>
>
> Each c1.xlarge instance has 4 ephemeral disk. However, I forgot to modify
> my script to mount the other 2 ephemeral disk and add them to dfs.data.dir.
> So, it should be running on 20 disks total. That would make it 100 ops per
> second vs 35 ops per second? Is that still a plausible overhead?
>

Potentially.  Hbase may need to read several locations to access your data
since it effectively overlays multiple hfiles.


> Is there a difference to the performance if I add the 4 disks to the
> dfs.data.dir vs. setting a raid-0 of the 4 ephemeral disks and have a single
> location for dfs.data.dir?
>

I would avoid raid-0

> > Uniform random can be a reasonably good approximation if
> > you are running
> > behind a cache large enough to cache all repeated
> > accesses.  If you aren't
> > behind a cache, uniform access might be very unrealistic
> > (and pessimistic).
> >
> > Do you have logs that you can use to model your actual read
> > behaviors?
> >
>
> Right now, I'm just playing with completely uniformly random. However, I
> have also tried a Zipf distribution and the throughput seems to saturate at
> around 1.2k ops per second.
>

Harumph.

What about data that prefers recently accessed keys?

Re: How to improve HBase throughput with YCSB?

[Harold Lim <ro...@yahoo.com>]

Hi Ted,

> You appear to be running on about 10 disks total. 
> Each disk should be
> capable of about 100 ops per second but they appear to be
> doing about 70.
>  This is plausible overhead.


Each c1.xlarge instance has 4 ephemeral disk. However, I forgot to modify my script to mount the other 2 ephemeral disk and add them to dfs.data.dir. So, it should be running on 20 disks total. That would make it 100 ops per second vs 35 ops per second? Is that still a plausible overhead?

Is there a difference to the performance if I add the 4 disks to the dfs.data.dir vs. setting a raid-0 of the 4 ephemeral disks and have a single location for dfs.data.dir?


I'll also try your suggestion of using multiple ebs stores.


> 
> Is your actual load going to be completely uniformly
> random?  Or will there
> be a Zipf distribution?  Will there be burst of
> repeated accesses?
> 
> Uniform random can be a reasonably good approximation if
> you are running
> behind a cache large enough to cache all repeated
> accesses.  If you aren't
> behind a cache, uniform access might be very unrealistic
> (and pessimistic).
> 
> Do you have logs that you can use to model your actual read
> behaviors?
> 

Right now, I'm just playing with completely uniformly random. However, I have also tried a Zipf distribution and the throughput seems to saturate at around 1.2k ops per second.

I actually don't have logs to model my read behaviors. I'm using HBase as part of my research project.

Thanks,
Harold  

Re: How to improve HBase throughput with YCSB?

[Ted Dunning <td...@maprtech.com>]

Woof.

Of course.

Harold,

You appear to be running on about 10 disks total.  Each disk should be
capable of about 100 ops per second but they appear to be doing about 70.
 This is plausible overhead.

Try attaching 5 or 10 small EBS partitions to each of your nodes and use
them in HDFS.  That may substantially increase your maximum IOP rate and
thus your read rate.

Is your actual load going to be completely uniformly random?  Or will there
be a Zipf distribution?  Will there be burst of repeated accesses?

Uniform random can be a reasonably good approximation if you are running
behind a cache large enough to cache all repeated accesses.  If you aren't
behind a cache, uniform access might be very unrealistic (and pessimistic).

Do you have logs that you can use to model your actual read behaviors?


On Tue, May 31, 2011 at 10:00 PM, Harold Lim <ro...@yahoo.com> wrote:

> Hi Andrew,
>
> I tried running on c1.xlarge instances and the performance improved a
> little bit but the throughput is still low. I can now get throughput of 700+
> read operations per second (up from 400-500+). I was hoping to get
> throughput in the order of thousands.
>
> I was wondering if there is something wrong with my set-up or is it normal
> for HBase running on ec2 instances to get low throughput numbers?
>
>
> -Harold
>
>
> --- On Mon, 5/30/11, Andrew Purtell <ap...@apache.org> wrote:
>
> > From: Andrew Purtell <ap...@apache.org>
> > Subject: Re: How to improve HBase throughput with YCSB?
> > To: user@hbase.apache.org
> > Date: Monday, May 30, 2011, 8:33 PM
> > The hypervisor steals a lot of CPU
> > time from m1.large instances.  You should be using
> > c1.xlarge instances.
> >
> > Are you using local storage or EBS?
> >
> > Be aware that I/O performance on EC2 for any system is
> > lower than if you are using real hardware, significantly so
> > if not using one of the instance types with I/O performance
> > listed as "high".
> >
> > > 2011/5/30 Harold Lim <ro...@yahoo.com>
> > >
> > > > Hi All,
> > > >
> > > > I have an HBase cluster on ec2 m1.large instance
> > (10
> > > > region servers).
> >
> >
>