performance testing

[Dave Latham <la...@davelink.net>]

While working on HBASE-3894 I was wondering what the best way to test
performance sensitive patches to HBase was.  There is the
o.a.h.h.PerformanceEvaluation class, but it doesn't appear have been really
touched in close to a year, and I haven't heard much mention of it on the
mailing lists in awhile.  (The numbers at
http://wiki.apache.org/hadoop/Hbase/PerformanceEvaluation are even further
out of date).  Does anyone have a good sense of the usefulness of this tool?

There's also YCSB, which I've seen more people mention recently.  However,
the book says "TODO: Describe how YCSB is poor for putting up a decent
cluster load." ( http://hbase.apache.org/book.html#d470e4848 ) so I get the
sense that it's not a great test for HBase.

Are there any others?

Does anyone have thoughts about the best way to run hbase through some
benchmarks to evaluate how patches may affect performance?  Would it be
worth trying to build a new set of benchmarks for hbase, or taking one of
these existing ones and improving it some way?

Dave

Re: performance testing

[Stack <st...@duboce.net>]

We used to use PE.  Pluses were that you could run it as a
multi-threaded client or as clients hosted in a MR job.  Negatives
were that the loading was uniform and therefore unreal.  When YCSB
came out, we all went gah-gah for it because it seemed at first better
than PE in that you could put up different loading types via
configuration only, etc., but since then, the bloom as gone off the
rose.  Various complain that they can't get YCSB to put up a decent
loading and so it seems we again have an opening for a decent loading
tool.

St.Ack

On Tue, May 24, 2011 at 9:14 PM, Dave Latham <la...@davelink.net> wrote:
> While working on HBASE-3894 I was wondering what the best way to test
> performance sensitive patches to HBase was.  There is the
> o.a.h.h.PerformanceEvaluation class, but it doesn't appear have been really
> touched in close to a year, and I haven't heard much mention of it on the
> mailing lists in awhile.  (The numbers at
> http://wiki.apache.org/hadoop/Hbase/PerformanceEvaluation are even further
> out of date).  Does anyone have a good sense of the usefulness of this tool?
>
> There's also YCSB, which I've seen more people mention recently.  However,
> the book says "TODO: Describe how YCSB is poor for putting up a decent
> cluster load." ( http://hbase.apache.org/book.html#d470e4848 ) so I get the
> sense that it's not a great test for HBase.
>
> Are there any others?
>
> Does anyone have thoughts about the best way to run hbase through some
> benchmarks to evaluate how patches may affect performance?  Would it be
> worth trying to build a new set of benchmarks for hbase, or taking one of
> these existing ones and improving it some way?
>
> Dave
>