You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Jean-Marc Spaggiari <je...@spaggiari.org> on 2013/03/08 14:58:27 UTC
Performances Tests
Hi,
In HBase we already have PerformanceEvaluation which gives us a good
way to validate that nothing broke HBase speed in the recent updates.
I can see in the JIRAs many improvements coming, like for the lazy
seeks, the bloom filters, etc. however, there is no tests for those
improvements.
Will it not be good to ask people to add some new tests in
PerformanceEvaluation when they are introducing an improvement which
is not covered there?
We should not touch existing tests because we need to have a way to
compare the baseline between the different versions, but we can still
add some new. Like in addition to RandomSeekScanTest we can add
RandomSeekScanBloomEnabledTest and so on. And even better if we can
back port those new tests to previous version.
The same way we add a test class when we introduce a new feature,
should we add a performance test method to test it too?
JM
Re: Performances Tests
Posted by Ted Yu <yu...@gmail.com>.
Thanks for sharing this result, Jean-Marc.
For randomRead, looks like the degradation happened between 0.94.0 and
0.94.1
Cheers
On Sat, Mar 16, 2013 at 7:03 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi Enis,
>
> "interesting" in the positive way ;)
>
> Results are there:
>
> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>
> The improvment on scan are impressive. sequentialRead and randomScan went
> down.
>
> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>
> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
> with it on previous HBase version...
>
> JM
>
> 2013/3/12 Enis Söztutar <en...@gmail.com>:
> >> I just finished to run all the PerformanceEvaluation tests on a
> > dedicated computer with all 0.9x.x HBase versions, and I found results
> > interesting.
> > Can you please provide your numbers if you can. What is interesting from
> > your findings?
> >
> > Enis
> >
> >
> >
> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> If you run only 1 client with PerformanceEvaluation, it's not running
> >> it over MapReduce, so you don't have this overhead. But you can still
> >> run it if you want to have something more distributed. Might be useful
> >> to have the 2 options. But at the end, LoadTestTool or
> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
> >> those tests.
> >>
> >> I just finished to run all the PerformanceEvaluation tests on a
> >> dedicated computer with all 0.9x.x HBase versions, and I found results
> >> interesting. That gives us a good baseline to see if new HBase
> >> improvements are really improving performances.
> >>
> >> JM
> >>
> >> 2013/3/8 Andrew Purtell <ap...@apache.org>:
> >> > Tangentally: I think I prefer LoadTestTool over
> PerformanceEvaluation, it
> >> > doesn't depend on nor is influenced by MapReduce job startup.
> >> >
> >> >
> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
> >> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >> >
> >> >> @JM
> >> >> I agree with you. Mainly the perf improvement changes needs some
> >> >> testcases.
> >> >> But sometimes the scenario on which the perf improvments happens are
> bit
> >> >> difficult to generate and we will be able to do in a standalone case
> >> only.
> >> >> May be overall if we need to get that perf improvment result we
> need a
> >> >> real cluster with suitable data. That is what i have experienced.
> Just
> >> >> telling.
> >> >>
> >> >> Regards
> >> >> Ram
> >> >>
> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
> >> >> jean-marc@spaggiari.org
> >> >> > wrote:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > In HBase we already have PerformanceEvaluation which gives us a
> good
> >> >> > way to validate that nothing broke HBase speed in the recent
> updates.
> >> >> >
> >> >> > I can see in the JIRAs many improvements coming, like for the lazy
> >> >> > seeks, the bloom filters, etc. however, there is no tests for those
> >> >> > improvements.
> >> >> >
> >> >> > Will it not be good to ask people to add some new tests in
> >> >> > PerformanceEvaluation when they are introducing an improvement
> which
> >> >> > is not covered there?
> >> >> >
> >> >> > We should not touch existing tests because we need to have a way to
> >> >> > compare the baseline between the different versions, but we can
> still
> >> >> > add some new. Like in addition to RandomSeekScanTest we can add
> >> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
> >> >> > back port those new tests to previous version.
> >> >> >
> >> >> > The same way we add a test class when we introduce a new feature,
> >> >> > should we add a performance test method to test it too?
> >> >> >
> >> >> > JM
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> >
> >> > - Andy
> >> >
> >> > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> >> > (via Tom White)
> >>
>
Re: Performances Tests
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Your welcome Enis.
Here is the update file with Trunk results added
http://www.spaggiari.org/media/blogs/hbase/pictures/performances_20130321.pdf
I have also added the Y scale label as asked. It's now all in
rows/seconds or rows/minutes when it's too slow.
The todo is now:
- Re-download all HBase version to make sure PE is using the right one (WIP);
- Re-run the scanRange100 tests to validate the values already found (Next);
- Add 0.95;
- Add LoadTestTool;
- Add HFilePerformanceEvaluation.
2013/3/20 Enis Söztutar <en...@gmail.com>:
> Thanks so much for doing this J-M.
>
> Enis
>
>
> On Wed, Mar 20, 2013 at 11:44 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> Hi Ted,
>>
>> I will try to build the trunk version and add it in the list....
>>
>> So I have to:
>> - Re-download all HBase version to make sure PE is using the right one;
>> - Re-run the scanRange100 tests to validate the values already found;
>> - Add the trunk;
>> - Add LoadTestTool
>>
>> Than will keep my free time buzy ;)
>>
>> I keep you all posted as soon as it's done.
>>
>> JM
>>
>> 2013/3/20 Ted Yu <yu...@gmail.com>:
>> > I am curious to know how trunk stands in the performance comparison.
>> > There have been many optimizations going into trunk. Getting hold of
>> > overall improvement would be nice.
>> >
>> > Cheers
>> >
>> > On Wed, Mar 20, 2013 at 5:02 AM, Jean-Marc Spaggiari <
>> > jean-marc@spaggiari.org> wrote:
>> >
>> >> Hi Lars,
>> >>
>> >> Can you share the code you are using so I can compate with PE? Also, I
>> >> will re-run all for my scanRange100 tests today and update the
>> >> spreadsheet again to make sure it's correct. Also also re-download all
>> >> the HBase versions to make sure they are all clean. I'm not doing any
>> >> configuration with them. Simply reducing the logs and tmp pointing to
>> >> memory file system.
>> >>
>> >> I will keep you posted when it's done.
>> >>
>> >> Hi Jonathan,
>> >>
>> >> It's usually rows per seconds, but with a factor 10. Sometime I had to
>> >> divide by 100000, sometime to multiply to get numbers bigger... I will
>> >> take a look at th formulas and add the legend for each of the charts.
>> >>
>> >> JM
>> >>
>> >> 2013/3/19 Jonathan Hsieh <jo...@cloudera.com>:
>> >> > What is the y axis's unit? seconds or operations per second etc?
>> (nit:
>> >> > would be nice to have on the axis.. )
>> >> >
>> >> > Based on the context, I believe it is ops/s.
>> >> >
>> >> > Jon.
>> >> >
>> >> > On Sat, Mar 16, 2013 at 7:03 PM, Jean-Marc Spaggiari <
>> >> > jean-marc@spaggiari.org> wrote:
>> >> >
>> >> >> Hi Enis,
>> >> >>
>> >> >> "interesting" in the positive way ;)
>> >> >>
>> >> >> Results are there:
>> >> >>
>> >> >>
>> >>
>> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>> >> >>
>> >> >> The improvment on scan are impressive. sequentialRead and randomScan
>> >> went
>> >> >> down.
>> >> >>
>> >> >> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>> >> >>
>> >> >> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
>> >> >> with it on previous HBase version...
>> >> >>
>> >> >> JM
>> >> >>
>> >> >> 2013/3/12 Enis Söztutar <en...@gmail.com>:
>> >> >> >> I just finished to run all the PerformanceEvaluation tests on a
>> >> >> > dedicated computer with all 0.9x.x HBase versions, and I found
>> results
>> >> >> > interesting.
>> >> >> > Can you please provide your numbers if you can. What is interesting
>> >> from
>> >> >> > your findings?
>> >> >> >
>> >> >> > Enis
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
>> >> >> > jean-marc@spaggiari.org> wrote:
>> >> >> >
>> >> >> >> If you run only 1 client with PerformanceEvaluation, it's not
>> running
>> >> >> >> it over MapReduce, so you don't have this overhead. But you can
>> still
>> >> >> >> run it if you want to have something more distributed. Might be
>> >> useful
>> >> >> >> to have the 2 options. But at the end, LoadTestTool or
>> >> >> >> PerformanceEvaluation, any of the 2 is good as long as we are
>> adding
>> >> >> >> those tests.
>> >> >> >>
>> >> >> >> I just finished to run all the PerformanceEvaluation tests on a
>> >> >> >> dedicated computer with all 0.9x.x HBase versions, and I found
>> >> results
>> >> >> >> interesting. That gives us a good baseline to see if new HBase
>> >> >> >> improvements are really improving performances.
>> >> >> >>
>> >> >> >> JM
>> >> >> >>
>> >> >> >> 2013/3/8 Andrew Purtell <ap...@apache.org>:
>> >> >> >> > Tangentally: I think I prefer LoadTestTool over
>> >> >> PerformanceEvaluation, it
>> >> >> >> > doesn't depend on nor is influenced by MapReduce job startup.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
>> >> >> >> > ramkrishna.s.vasudevan@gmail.com> wrote:
>> >> >> >> >
>> >> >> >> >> @JM
>> >> >> >> >> I agree with you. Mainly the perf improvement changes needs
>> some
>> >> >> >> >> testcases.
>> >> >> >> >> But sometimes the scenario on which the perf improvments
>> happens
>> >> are
>> >> >> bit
>> >> >> >> >> difficult to generate and we will be able to do in a standalone
>> >> case
>> >> >> >> only.
>> >> >> >> >> May be overall if we need to get that perf improvment result
>> we
>> >> >> need a
>> >> >> >> >> real cluster with suitable data. That is what i have
>> experienced.
>> >> >> Just
>> >> >> >> >> telling.
>> >> >> >> >>
>> >> >> >> >> Regards
>> >> >> >> >> Ram
>> >> >> >> >>
>> >> >> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
>> >> >> >> >> jean-marc@spaggiari.org
>> >> >> >> >> > wrote:
>> >> >> >> >>
>> >> >> >> >> > Hi,
>> >> >> >> >> >
>> >> >> >> >> > In HBase we already have PerformanceEvaluation which gives
>> us a
>> >> >> good
>> >> >> >> >> > way to validate that nothing broke HBase speed in the recent
>> >> >> updates.
>> >> >> >> >> >
>> >> >> >> >> > I can see in the JIRAs many improvements coming, like for the
>> >> lazy
>> >> >> >> >> > seeks, the bloom filters, etc. however, there is no tests for
>> >> those
>> >> >> >> >> > improvements.
>> >> >> >> >> >
>> >> >> >> >> > Will it not be good to ask people to add some new tests in
>> >> >> >> >> > PerformanceEvaluation when they are introducing an
>> improvement
>> >> >> which
>> >> >> >> >> > is not covered there?
>> >> >> >> >> >
>> >> >> >> >> > We should not touch existing tests because we need to have a
>> >> way to
>> >> >> >> >> > compare the baseline between the different versions, but we
>> can
>> >> >> still
>> >> >> >> >> > add some new. Like in addition to RandomSeekScanTest we can
>> add
>> >> >> >> >> > RandomSeekScanBloomEnabledTest and so on. And even better if
>> we
>> >> can
>> >> >> >> >> > back port those new tests to previous version.
>> >> >> >> >> >
>> >> >> >> >> > The same way we add a test class when we introduce a new
>> >> feature,
>> >> >> >> >> > should we add a performance test method to test it too?
>> >> >> >> >> >
>> >> >> >> >> > JM
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > Best regards,
>> >> >> >> >
>> >> >> >> > - Andy
>> >> >> >> >
>> >> >> >> > Problems worthy of attack prove their worth by hitting back. -
>> Piet
>> >> >> Hein
>> >> >> >> > (via Tom White)
>> >> >> >>
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > // Jonathan Hsieh (shay)
>> >> > // Software Engineer, Cloudera
>> >> > // jon@cloudera.com
>> >>
>>
Re: Performances Tests
Posted by Enis Söztutar <en...@gmail.com>.
Thanks so much for doing this J-M.
Enis
On Wed, Mar 20, 2013 at 11:44 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi Ted,
>
> I will try to build the trunk version and add it in the list....
>
> So I have to:
> - Re-download all HBase version to make sure PE is using the right one;
> - Re-run the scanRange100 tests to validate the values already found;
> - Add the trunk;
> - Add LoadTestTool
>
> Than will keep my free time buzy ;)
>
> I keep you all posted as soon as it's done.
>
> JM
>
> 2013/3/20 Ted Yu <yu...@gmail.com>:
> > I am curious to know how trunk stands in the performance comparison.
> > There have been many optimizations going into trunk. Getting hold of
> > overall improvement would be nice.
> >
> > Cheers
> >
> > On Wed, Mar 20, 2013 at 5:02 AM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> Hi Lars,
> >>
> >> Can you share the code you are using so I can compate with PE? Also, I
> >> will re-run all for my scanRange100 tests today and update the
> >> spreadsheet again to make sure it's correct. Also also re-download all
> >> the HBase versions to make sure they are all clean. I'm not doing any
> >> configuration with them. Simply reducing the logs and tmp pointing to
> >> memory file system.
> >>
> >> I will keep you posted when it's done.
> >>
> >> Hi Jonathan,
> >>
> >> It's usually rows per seconds, but with a factor 10. Sometime I had to
> >> divide by 100000, sometime to multiply to get numbers bigger... I will
> >> take a look at th formulas and add the legend for each of the charts.
> >>
> >> JM
> >>
> >> 2013/3/19 Jonathan Hsieh <jo...@cloudera.com>:
> >> > What is the y axis's unit? seconds or operations per second etc?
> (nit:
> >> > would be nice to have on the axis.. )
> >> >
> >> > Based on the context, I believe it is ops/s.
> >> >
> >> > Jon.
> >> >
> >> > On Sat, Mar 16, 2013 at 7:03 PM, Jean-Marc Spaggiari <
> >> > jean-marc@spaggiari.org> wrote:
> >> >
> >> >> Hi Enis,
> >> >>
> >> >> "interesting" in the positive way ;)
> >> >>
> >> >> Results are there:
> >> >>
> >> >>
> >>
> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
> >> >>
> >> >> The improvment on scan are impressive. sequentialRead and randomScan
> >> went
> >> >> down.
> >> >>
> >> >> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
> >> >>
> >> >> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
> >> >> with it on previous HBase version...
> >> >>
> >> >> JM
> >> >>
> >> >> 2013/3/12 Enis Söztutar <en...@gmail.com>:
> >> >> >> I just finished to run all the PerformanceEvaluation tests on a
> >> >> > dedicated computer with all 0.9x.x HBase versions, and I found
> results
> >> >> > interesting.
> >> >> > Can you please provide your numbers if you can. What is interesting
> >> from
> >> >> > your findings?
> >> >> >
> >> >> > Enis
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> >> >> > jean-marc@spaggiari.org> wrote:
> >> >> >
> >> >> >> If you run only 1 client with PerformanceEvaluation, it's not
> running
> >> >> >> it over MapReduce, so you don't have this overhead. But you can
> still
> >> >> >> run it if you want to have something more distributed. Might be
> >> useful
> >> >> >> to have the 2 options. But at the end, LoadTestTool or
> >> >> >> PerformanceEvaluation, any of the 2 is good as long as we are
> adding
> >> >> >> those tests.
> >> >> >>
> >> >> >> I just finished to run all the PerformanceEvaluation tests on a
> >> >> >> dedicated computer with all 0.9x.x HBase versions, and I found
> >> results
> >> >> >> interesting. That gives us a good baseline to see if new HBase
> >> >> >> improvements are really improving performances.
> >> >> >>
> >> >> >> JM
> >> >> >>
> >> >> >> 2013/3/8 Andrew Purtell <ap...@apache.org>:
> >> >> >> > Tangentally: I think I prefer LoadTestTool over
> >> >> PerformanceEvaluation, it
> >> >> >> > doesn't depend on nor is influenced by MapReduce job startup.
> >> >> >> >
> >> >> >> >
> >> >> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
> >> >> >> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >> >> >> >
> >> >> >> >> @JM
> >> >> >> >> I agree with you. Mainly the perf improvement changes needs
> some
> >> >> >> >> testcases.
> >> >> >> >> But sometimes the scenario on which the perf improvments
> happens
> >> are
> >> >> bit
> >> >> >> >> difficult to generate and we will be able to do in a standalone
> >> case
> >> >> >> only.
> >> >> >> >> May be overall if we need to get that perf improvment result
> we
> >> >> need a
> >> >> >> >> real cluster with suitable data. That is what i have
> experienced.
> >> >> Just
> >> >> >> >> telling.
> >> >> >> >>
> >> >> >> >> Regards
> >> >> >> >> Ram
> >> >> >> >>
> >> >> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
> >> >> >> >> jean-marc@spaggiari.org
> >> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> > Hi,
> >> >> >> >> >
> >> >> >> >> > In HBase we already have PerformanceEvaluation which gives
> us a
> >> >> good
> >> >> >> >> > way to validate that nothing broke HBase speed in the recent
> >> >> updates.
> >> >> >> >> >
> >> >> >> >> > I can see in the JIRAs many improvements coming, like for the
> >> lazy
> >> >> >> >> > seeks, the bloom filters, etc. however, there is no tests for
> >> those
> >> >> >> >> > improvements.
> >> >> >> >> >
> >> >> >> >> > Will it not be good to ask people to add some new tests in
> >> >> >> >> > PerformanceEvaluation when they are introducing an
> improvement
> >> >> which
> >> >> >> >> > is not covered there?
> >> >> >> >> >
> >> >> >> >> > We should not touch existing tests because we need to have a
> >> way to
> >> >> >> >> > compare the baseline between the different versions, but we
> can
> >> >> still
> >> >> >> >> > add some new. Like in addition to RandomSeekScanTest we can
> add
> >> >> >> >> > RandomSeekScanBloomEnabledTest and so on. And even better if
> we
> >> can
> >> >> >> >> > back port those new tests to previous version.
> >> >> >> >> >
> >> >> >> >> > The same way we add a test class when we introduce a new
> >> feature,
> >> >> >> >> > should we add a performance test method to test it too?
> >> >> >> >> >
> >> >> >> >> > JM
> >> >> >> >> >
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> > Best regards,
> >> >> >> >
> >> >> >> > - Andy
> >> >> >> >
> >> >> >> > Problems worthy of attack prove their worth by hitting back. -
> Piet
> >> >> Hein
> >> >> >> > (via Tom White)
> >> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > // Jonathan Hsieh (shay)
> >> > // Software Engineer, Cloudera
> >> > // jon@cloudera.com
> >>
>
Re: Performances Tests
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Ted,
I will try to build the trunk version and add it in the list....
So I have to:
- Re-download all HBase version to make sure PE is using the right one;
- Re-run the scanRange100 tests to validate the values already found;
- Add the trunk;
- Add LoadTestTool
Than will keep my free time buzy ;)
I keep you all posted as soon as it's done.
JM
2013/3/20 Ted Yu <yu...@gmail.com>:
> I am curious to know how trunk stands in the performance comparison.
> There have been many optimizations going into trunk. Getting hold of
> overall improvement would be nice.
>
> Cheers
>
> On Wed, Mar 20, 2013 at 5:02 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> Hi Lars,
>>
>> Can you share the code you are using so I can compate with PE? Also, I
>> will re-run all for my scanRange100 tests today and update the
>> spreadsheet again to make sure it's correct. Also also re-download all
>> the HBase versions to make sure they are all clean. I'm not doing any
>> configuration with them. Simply reducing the logs and tmp pointing to
>> memory file system.
>>
>> I will keep you posted when it's done.
>>
>> Hi Jonathan,
>>
>> It's usually rows per seconds, but with a factor 10. Sometime I had to
>> divide by 100000, sometime to multiply to get numbers bigger... I will
>> take a look at th formulas and add the legend for each of the charts.
>>
>> JM
>>
>> 2013/3/19 Jonathan Hsieh <jo...@cloudera.com>:
>> > What is the y axis's unit? seconds or operations per second etc? (nit:
>> > would be nice to have on the axis.. )
>> >
>> > Based on the context, I believe it is ops/s.
>> >
>> > Jon.
>> >
>> > On Sat, Mar 16, 2013 at 7:03 PM, Jean-Marc Spaggiari <
>> > jean-marc@spaggiari.org> wrote:
>> >
>> >> Hi Enis,
>> >>
>> >> "interesting" in the positive way ;)
>> >>
>> >> Results are there:
>> >>
>> >>
>> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>> >>
>> >> The improvment on scan are impressive. sequentialRead and randomScan
>> went
>> >> down.
>> >>
>> >> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>> >>
>> >> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
>> >> with it on previous HBase version...
>> >>
>> >> JM
>> >>
>> >> 2013/3/12 Enis Söztutar <en...@gmail.com>:
>> >> >> I just finished to run all the PerformanceEvaluation tests on a
>> >> > dedicated computer with all 0.9x.x HBase versions, and I found results
>> >> > interesting.
>> >> > Can you please provide your numbers if you can. What is interesting
>> from
>> >> > your findings?
>> >> >
>> >> > Enis
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
>> >> > jean-marc@spaggiari.org> wrote:
>> >> >
>> >> >> If you run only 1 client with PerformanceEvaluation, it's not running
>> >> >> it over MapReduce, so you don't have this overhead. But you can still
>> >> >> run it if you want to have something more distributed. Might be
>> useful
>> >> >> to have the 2 options. But at the end, LoadTestTool or
>> >> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
>> >> >> those tests.
>> >> >>
>> >> >> I just finished to run all the PerformanceEvaluation tests on a
>> >> >> dedicated computer with all 0.9x.x HBase versions, and I found
>> results
>> >> >> interesting. That gives us a good baseline to see if new HBase
>> >> >> improvements are really improving performances.
>> >> >>
>> >> >> JM
>> >> >>
>> >> >> 2013/3/8 Andrew Purtell <ap...@apache.org>:
>> >> >> > Tangentally: I think I prefer LoadTestTool over
>> >> PerformanceEvaluation, it
>> >> >> > doesn't depend on nor is influenced by MapReduce job startup.
>> >> >> >
>> >> >> >
>> >> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
>> >> >> > ramkrishna.s.vasudevan@gmail.com> wrote:
>> >> >> >
>> >> >> >> @JM
>> >> >> >> I agree with you. Mainly the perf improvement changes needs some
>> >> >> >> testcases.
>> >> >> >> But sometimes the scenario on which the perf improvments happens
>> are
>> >> bit
>> >> >> >> difficult to generate and we will be able to do in a standalone
>> case
>> >> >> only.
>> >> >> >> May be overall if we need to get that perf improvment result we
>> >> need a
>> >> >> >> real cluster with suitable data. That is what i have experienced.
>> >> Just
>> >> >> >> telling.
>> >> >> >>
>> >> >> >> Regards
>> >> >> >> Ram
>> >> >> >>
>> >> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
>> >> >> >> jean-marc@spaggiari.org
>> >> >> >> > wrote:
>> >> >> >>
>> >> >> >> > Hi,
>> >> >> >> >
>> >> >> >> > In HBase we already have PerformanceEvaluation which gives us a
>> >> good
>> >> >> >> > way to validate that nothing broke HBase speed in the recent
>> >> updates.
>> >> >> >> >
>> >> >> >> > I can see in the JIRAs many improvements coming, like for the
>> lazy
>> >> >> >> > seeks, the bloom filters, etc. however, there is no tests for
>> those
>> >> >> >> > improvements.
>> >> >> >> >
>> >> >> >> > Will it not be good to ask people to add some new tests in
>> >> >> >> > PerformanceEvaluation when they are introducing an improvement
>> >> which
>> >> >> >> > is not covered there?
>> >> >> >> >
>> >> >> >> > We should not touch existing tests because we need to have a
>> way to
>> >> >> >> > compare the baseline between the different versions, but we can
>> >> still
>> >> >> >> > add some new. Like in addition to RandomSeekScanTest we can add
>> >> >> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we
>> can
>> >> >> >> > back port those new tests to previous version.
>> >> >> >> >
>> >> >> >> > The same way we add a test class when we introduce a new
>> feature,
>> >> >> >> > should we add a performance test method to test it too?
>> >> >> >> >
>> >> >> >> > JM
>> >> >> >> >
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Best regards,
>> >> >> >
>> >> >> > - Andy
>> >> >> >
>> >> >> > Problems worthy of attack prove their worth by hitting back. - Piet
>> >> Hein
>> >> >> > (via Tom White)
>> >> >>
>> >>
>> >
>> >
>> >
>> > --
>> > // Jonathan Hsieh (shay)
>> > // Software Engineer, Cloudera
>> > // jon@cloudera.com
>>
Re: Performances Tests
Posted by Ted Yu <yu...@gmail.com>.
I am curious to know how trunk stands in the performance comparison.
There have been many optimizations going into trunk. Getting hold of
overall improvement would be nice.
Cheers
On Wed, Mar 20, 2013 at 5:02 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi Lars,
>
> Can you share the code you are using so I can compate with PE? Also, I
> will re-run all for my scanRange100 tests today and update the
> spreadsheet again to make sure it's correct. Also also re-download all
> the HBase versions to make sure they are all clean. I'm not doing any
> configuration with them. Simply reducing the logs and tmp pointing to
> memory file system.
>
> I will keep you posted when it's done.
>
> Hi Jonathan,
>
> It's usually rows per seconds, but with a factor 10. Sometime I had to
> divide by 100000, sometime to multiply to get numbers bigger... I will
> take a look at th formulas and add the legend for each of the charts.
>
> JM
>
> 2013/3/19 Jonathan Hsieh <jo...@cloudera.com>:
> > What is the y axis's unit? seconds or operations per second etc? (nit:
> > would be nice to have on the axis.. )
> >
> > Based on the context, I believe it is ops/s.
> >
> > Jon.
> >
> > On Sat, Mar 16, 2013 at 7:03 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> Hi Enis,
> >>
> >> "interesting" in the positive way ;)
> >>
> >> Results are there:
> >>
> >>
> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
> >>
> >> The improvment on scan are impressive. sequentialRead and randomScan
> went
> >> down.
> >>
> >> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
> >>
> >> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
> >> with it on previous HBase version...
> >>
> >> JM
> >>
> >> 2013/3/12 Enis Söztutar <en...@gmail.com>:
> >> >> I just finished to run all the PerformanceEvaluation tests on a
> >> > dedicated computer with all 0.9x.x HBase versions, and I found results
> >> > interesting.
> >> > Can you please provide your numbers if you can. What is interesting
> from
> >> > your findings?
> >> >
> >> > Enis
> >> >
> >> >
> >> >
> >> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> >> > jean-marc@spaggiari.org> wrote:
> >> >
> >> >> If you run only 1 client with PerformanceEvaluation, it's not running
> >> >> it over MapReduce, so you don't have this overhead. But you can still
> >> >> run it if you want to have something more distributed. Might be
> useful
> >> >> to have the 2 options. But at the end, LoadTestTool or
> >> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
> >> >> those tests.
> >> >>
> >> >> I just finished to run all the PerformanceEvaluation tests on a
> >> >> dedicated computer with all 0.9x.x HBase versions, and I found
> results
> >> >> interesting. That gives us a good baseline to see if new HBase
> >> >> improvements are really improving performances.
> >> >>
> >> >> JM
> >> >>
> >> >> 2013/3/8 Andrew Purtell <ap...@apache.org>:
> >> >> > Tangentally: I think I prefer LoadTestTool over
> >> PerformanceEvaluation, it
> >> >> > doesn't depend on nor is influenced by MapReduce job startup.
> >> >> >
> >> >> >
> >> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
> >> >> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >> >> >
> >> >> >> @JM
> >> >> >> I agree with you. Mainly the perf improvement changes needs some
> >> >> >> testcases.
> >> >> >> But sometimes the scenario on which the perf improvments happens
> are
> >> bit
> >> >> >> difficult to generate and we will be able to do in a standalone
> case
> >> >> only.
> >> >> >> May be overall if we need to get that perf improvment result we
> >> need a
> >> >> >> real cluster with suitable data. That is what i have experienced.
> >> Just
> >> >> >> telling.
> >> >> >>
> >> >> >> Regards
> >> >> >> Ram
> >> >> >>
> >> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
> >> >> >> jean-marc@spaggiari.org
> >> >> >> > wrote:
> >> >> >>
> >> >> >> > Hi,
> >> >> >> >
> >> >> >> > In HBase we already have PerformanceEvaluation which gives us a
> >> good
> >> >> >> > way to validate that nothing broke HBase speed in the recent
> >> updates.
> >> >> >> >
> >> >> >> > I can see in the JIRAs many improvements coming, like for the
> lazy
> >> >> >> > seeks, the bloom filters, etc. however, there is no tests for
> those
> >> >> >> > improvements.
> >> >> >> >
> >> >> >> > Will it not be good to ask people to add some new tests in
> >> >> >> > PerformanceEvaluation when they are introducing an improvement
> >> which
> >> >> >> > is not covered there?
> >> >> >> >
> >> >> >> > We should not touch existing tests because we need to have a
> way to
> >> >> >> > compare the baseline between the different versions, but we can
> >> still
> >> >> >> > add some new. Like in addition to RandomSeekScanTest we can add
> >> >> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we
> can
> >> >> >> > back port those new tests to previous version.
> >> >> >> >
> >> >> >> > The same way we add a test class when we introduce a new
> feature,
> >> >> >> > should we add a performance test method to test it too?
> >> >> >> >
> >> >> >> > JM
> >> >> >> >
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Best regards,
> >> >> >
> >> >> > - Andy
> >> >> >
> >> >> > Problems worthy of attack prove their worth by hitting back. - Piet
> >> Hein
> >> >> > (via Tom White)
> >> >>
> >>
> >
> >
> >
> > --
> > // Jonathan Hsieh (shay)
> > // Software Engineer, Cloudera
> > // jon@cloudera.com
>
Re: Performances Tests
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Lars,
Can you share the code you are using so I can compate with PE? Also, I
will re-run all for my scanRange100 tests today and update the
spreadsheet again to make sure it's correct. Also also re-download all
the HBase versions to make sure they are all clean. I'm not doing any
configuration with them. Simply reducing the logs and tmp pointing to
memory file system.
I will keep you posted when it's done.
Hi Jonathan,
It's usually rows per seconds, but with a factor 10. Sometime I had to
divide by 100000, sometime to multiply to get numbers bigger... I will
take a look at th formulas and add the legend for each of the charts.
JM
2013/3/19 Jonathan Hsieh <jo...@cloudera.com>:
> What is the y axis's unit? seconds or operations per second etc? (nit:
> would be nice to have on the axis.. )
>
> Based on the context, I believe it is ops/s.
>
> Jon.
>
> On Sat, Mar 16, 2013 at 7:03 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> Hi Enis,
>>
>> "interesting" in the positive way ;)
>>
>> Results are there:
>>
>> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>>
>> The improvment on scan are impressive. sequentialRead and randomScan went
>> down.
>>
>> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>>
>> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
>> with it on previous HBase version...
>>
>> JM
>>
>> 2013/3/12 Enis Söztutar <en...@gmail.com>:
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> > dedicated computer with all 0.9x.x HBase versions, and I found results
>> > interesting.
>> > Can you please provide your numbers if you can. What is interesting from
>> > your findings?
>> >
>> > Enis
>> >
>> >
>> >
>> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
>> > jean-marc@spaggiari.org> wrote:
>> >
>> >> If you run only 1 client with PerformanceEvaluation, it's not running
>> >> it over MapReduce, so you don't have this overhead. But you can still
>> >> run it if you want to have something more distributed. Might be useful
>> >> to have the 2 options. But at the end, LoadTestTool or
>> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
>> >> those tests.
>> >>
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> >> dedicated computer with all 0.9x.x HBase versions, and I found results
>> >> interesting. That gives us a good baseline to see if new HBase
>> >> improvements are really improving performances.
>> >>
>> >> JM
>> >>
>> >> 2013/3/8 Andrew Purtell <ap...@apache.org>:
>> >> > Tangentally: I think I prefer LoadTestTool over
>> PerformanceEvaluation, it
>> >> > doesn't depend on nor is influenced by MapReduce job startup.
>> >> >
>> >> >
>> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
>> >> > ramkrishna.s.vasudevan@gmail.com> wrote:
>> >> >
>> >> >> @JM
>> >> >> I agree with you. Mainly the perf improvement changes needs some
>> >> >> testcases.
>> >> >> But sometimes the scenario on which the perf improvments happens are
>> bit
>> >> >> difficult to generate and we will be able to do in a standalone case
>> >> only.
>> >> >> May be overall if we need to get that perf improvment result we
>> need a
>> >> >> real cluster with suitable data. That is what i have experienced.
>> Just
>> >> >> telling.
>> >> >>
>> >> >> Regards
>> >> >> Ram
>> >> >>
>> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
>> >> >> jean-marc@spaggiari.org
>> >> >> > wrote:
>> >> >>
>> >> >> > Hi,
>> >> >> >
>> >> >> > In HBase we already have PerformanceEvaluation which gives us a
>> good
>> >> >> > way to validate that nothing broke HBase speed in the recent
>> updates.
>> >> >> >
>> >> >> > I can see in the JIRAs many improvements coming, like for the lazy
>> >> >> > seeks, the bloom filters, etc. however, there is no tests for those
>> >> >> > improvements.
>> >> >> >
>> >> >> > Will it not be good to ask people to add some new tests in
>> >> >> > PerformanceEvaluation when they are introducing an improvement
>> which
>> >> >> > is not covered there?
>> >> >> >
>> >> >> > We should not touch existing tests because we need to have a way to
>> >> >> > compare the baseline between the different versions, but we can
>> still
>> >> >> > add some new. Like in addition to RandomSeekScanTest we can add
>> >> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
>> >> >> > back port those new tests to previous version.
>> >> >> >
>> >> >> > The same way we add a test class when we introduce a new feature,
>> >> >> > should we add a performance test method to test it too?
>> >> >> >
>> >> >> > JM
>> >> >> >
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Best regards,
>> >> >
>> >> > - Andy
>> >> >
>> >> > Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein
>> >> > (via Tom White)
>> >>
>>
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
Re: Performances Tests
Posted by Jonathan Hsieh <jo...@cloudera.com>.
What is the y axis's unit? seconds or operations per second etc? (nit:
would be nice to have on the axis.. )
Based on the context, I believe it is ops/s.
Jon.
On Sat, Mar 16, 2013 at 7:03 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi Enis,
>
> "interesting" in the positive way ;)
>
> Results are there:
>
> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>
> The improvment on scan are impressive. sequentialRead and randomScan went
> down.
>
> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>
> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
> with it on previous HBase version...
>
> JM
>
> 2013/3/12 Enis Söztutar <en...@gmail.com>:
> >> I just finished to run all the PerformanceEvaluation tests on a
> > dedicated computer with all 0.9x.x HBase versions, and I found results
> > interesting.
> > Can you please provide your numbers if you can. What is interesting from
> > your findings?
> >
> > Enis
> >
> >
> >
> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> If you run only 1 client with PerformanceEvaluation, it's not running
> >> it over MapReduce, so you don't have this overhead. But you can still
> >> run it if you want to have something more distributed. Might be useful
> >> to have the 2 options. But at the end, LoadTestTool or
> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
> >> those tests.
> >>
> >> I just finished to run all the PerformanceEvaluation tests on a
> >> dedicated computer with all 0.9x.x HBase versions, and I found results
> >> interesting. That gives us a good baseline to see if new HBase
> >> improvements are really improving performances.
> >>
> >> JM
> >>
> >> 2013/3/8 Andrew Purtell <ap...@apache.org>:
> >> > Tangentally: I think I prefer LoadTestTool over
> PerformanceEvaluation, it
> >> > doesn't depend on nor is influenced by MapReduce job startup.
> >> >
> >> >
> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
> >> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >> >
> >> >> @JM
> >> >> I agree with you. Mainly the perf improvement changes needs some
> >> >> testcases.
> >> >> But sometimes the scenario on which the perf improvments happens are
> bit
> >> >> difficult to generate and we will be able to do in a standalone case
> >> only.
> >> >> May be overall if we need to get that perf improvment result we
> need a
> >> >> real cluster with suitable data. That is what i have experienced.
> Just
> >> >> telling.
> >> >>
> >> >> Regards
> >> >> Ram
> >> >>
> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
> >> >> jean-marc@spaggiari.org
> >> >> > wrote:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > In HBase we already have PerformanceEvaluation which gives us a
> good
> >> >> > way to validate that nothing broke HBase speed in the recent
> updates.
> >> >> >
> >> >> > I can see in the JIRAs many improvements coming, like for the lazy
> >> >> > seeks, the bloom filters, etc. however, there is no tests for those
> >> >> > improvements.
> >> >> >
> >> >> > Will it not be good to ask people to add some new tests in
> >> >> > PerformanceEvaluation when they are introducing an improvement
> which
> >> >> > is not covered there?
> >> >> >
> >> >> > We should not touch existing tests because we need to have a way to
> >> >> > compare the baseline between the different versions, but we can
> still
> >> >> > add some new. Like in addition to RandomSeekScanTest we can add
> >> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
> >> >> > back port those new tests to previous version.
> >> >> >
> >> >> > The same way we add a test class when we introduce a new feature,
> >> >> > should we add a performance test method to test it too?
> >> >> >
> >> >> > JM
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> >
> >> > - Andy
> >> >
> >> > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> >> > (via Tom White)
> >>
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: Performances Tests
Posted by Andrew Purtell <ap...@apache.org>.
Anything at all would be great JM. Looking forward to it if you have the
time and interest. Thanks!
On Sunday, March 17, 2013, Jean-Marc Spaggiari wrote:
> @Lars: here is an example of what I use
> for i in {1..10}; do echo; echo -n $i ; rm -rf /tmp/*;
> bin/start-hbase.sh; sleep 60; bin/hbase
> org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 1;
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=100
> filterScan 1; bin/stop-hbase.sh; done &>> output.txt
>
> Giving only 1 as the readers allow to not launch the test as a MR. I
> have not try the same tests with a bigger number, but I can.
>
> @Andy: LoadTestTool is also on my list, but for YCSB it's a but harder
> since I will need more than one dedicated computer. But I will look at
> it too. I might be able to remove 1 node from my cluster and dedicate
> it to the tests when required...
>
>
> I will be pretty buzy next week, but before next week-end I will try
> to prepare the LoadTestTool scripts to run something similar. With
> multiple scenarios, like with and whithout bloom, etc.
>
> If yu have any recommandation/request, feel free! I will come back
> soon with some additionnal numbers.
>
> JM
>
> 2013/3/17 Andrew Purtell <apurtell@apache.org <javascript:;>>:
> > I'm not sure I trust the results of PerformanceEvauation.
> >
> > LoadTestTool and YCSB have their own issues but seem to produce more
> > consistent results. I've been thinking on adding scanning and filtering
> > tests to LoadTestTool.
> >
> > On Sunday, March 17, 2013, lars hofhansl wrote:
> >
> >> Cool. The 0.94.3 scanning improvements seems almost unbelievable
> >> (especially since many of my improvements to reduce the internal
> friction
> >> went into 0.94.4).
> >> I would like to track down the random read regression.
> >>
> >> Can you send the commands you ran? Are you running this as M/R job or
> >> standalone client?
> >>
> >> Thanks for doing this J-M.
> >>
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ________________________________
> >> From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;><javascript:;>>
> >> To: dev@hbase.apache.org <javascript:;> <javascript:;>
> >> Sent: Saturday, March 16, 2013 7:03 PM
> >> Subject: Re: Performances Tests
> >>
> >> Hi Enis,
> >>
> >> "interesting" in the positive way ;)
> >>
> >> Results are there:
> >>
> >>
> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
> >>
> >> The improvment on scan are impressive. sequentialRead and randomScan
> went
> >> down.
> >>
> >> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
> >>
> >> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
> >> with it on previous HBase version...
> >>
> >> JM
> >>
> >> 2013/3/12 Enis Söztutar <en...@gmail.com>:
> >> >> I just finished to run all the PerformanceEvaluation tests on a
> >> > dedicated computer with all 0.9x.x HBase versions, and I found results
> >> > interesting.
> >> > Can you please provide your numbers if you can. What is interesting
> from
> >> > your findings?
> >> >
> >> > Enis
> >> >
> >> >
> >> >
> >> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> >> > jean-marc@spaggiari.org> wrote:
> >> >
> >> >> If you run only 1 client with PerformanceEvaluation, it's not running
> >> >> it over MapReduce, so you don't have this overhead. But you can still
> >> >> run it if you want to have something more distributed. Might be
> useful
> >> >> to have the 2 options. But at the end, LoadTestTool or
> >> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
> >> >> those tests.
> >> >>
> >> >> I just finished to run all the PerformanceEvaluation tests on a
> >> >> dedicated computer with all 0.9x.x HBase versions, and I found
> results
> >> >> interesting. That gives us a good baseline to see if new HBase
> >> >> improvements are really improving performances.
> >> >>
> >> >> JM
> >> >>
> >> >> 2013/3/8 Andrew Purtell <ap...@apache.org>:
> >> >> > Tangentally: I think I prefer LoadTestTool over
> >> PerformanceEvaluation, it
> >> >> > doesn't depend on nor is influenced by MapReduce job startup.
> >> >> >
> >> >> >
> >> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
> >> >> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >> >> >
> >> >> >> @JM
> >> >> >> I agree with you. Mainly the perf improvement changes needs some
> >> >> >> testcases.
> >> >> >> But sometimes the scenario on which the perf improvments happens
> are
> >> bit
> >> >> >> difficult to generate and we will be able to do in a standalone
> case
> >> >> only.
> >> >> >> May be overall if we need to get that perf improvment result we
> >> need a
> >> >> >> real cluster with suitable data. That is what i have experienced.
> >> Just
> >> >> >> telling.
> >> >> >>
> >> >> >> Regards
> >> >> >> Ram
> >> >> >>
> >> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
> >> >> >> jean-marc@spaggiari.org
> >> >> >> > wrote:
> >> >> >>
> >> >> >> > Hi,
> >> >> >> >
> >> >> >> > In HBase we already have PerformanceEvaluation which gives us a
> >> good
> >> >> >> > way to validate that nothing broke HBase speed in the recent
> >> updates.
> >> >> >> >
> >> >> >> > I can see in the JIRAs many improvements coming, like for the
> lazy
> >> >> >> > seeks, the bloom filters, etc.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Performances Tests
Posted by lars hofhansl <la...@apache.org>.
I did my own tests (single threaded hacked together test code - not PE or LT) and I do not measure scan perf improvements between 0.94.2 and 0.94.3.
But I do see improvements between 0.94.3 and 0.94.4.
So I am starting to either:
1. Not trust my test code
2. Not trust PE
3. Not trust your methodology
Or maybe PE exercises some code that I don't.
;)
-- Lars
________________________________
From: Jean-Marc Spaggiari <je...@spaggiari.org>
To: dev@hbase.apache.org
Sent: Sunday, March 17, 2013 9:47 AM
Subject: Re: Performances Tests
@Lars: here is an example of what I use
for i in {1..10}; do echo; echo -n $i ; rm -rf /tmp/*;
bin/start-hbase.sh; sleep 60; bin/hbase
org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 1;
bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=100
filterScan 1; bin/stop-hbase.sh; done &>> output.txt
Giving only 1 as the readers allow to not launch the test as a MR. I
have not try the same tests with a bigger number, but I can.
@Andy: LoadTestTool is also on my list, but for YCSB it's a but harder
since I will need more than one dedicated computer. But I will look at
it too. I might be able to remove 1 node from my cluster and dedicate
it to the tests when required...
I will be pretty buzy next week, but before next week-end I will try
to prepare the LoadTestTool scripts to run something similar. With
multiple scenarios, like with and whithout bloom, etc.
If yu have any recommandation/request, feel free! I will come back
soon with some additionnal numbers.
JM
2013/3/17 Andrew Purtell <ap...@apache.org>:
> I'm not sure I trust the results of PerformanceEvauation.
>
> LoadTestTool and YCSB have their own issues but seem to produce more
> consistent results. I've been thinking on adding scanning and filtering
> tests to LoadTestTool.
>
> On Sunday, March 17, 2013, lars hofhansl wrote:
>
>> Cool. The 0.94.3 scanning improvements seems almost unbelievable
>> (especially since many of my improvements to reduce the internal friction
>> went into 0.94.4).
>> I would like to track down the random read regression.
>>
>> Can you send the commands you ran? Are you running this as M/R job or
>> standalone client?
>>
>> Thanks for doing this J-M.
>>
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
>> To: dev@hbase.apache.org <javascript:;>
>> Sent: Saturday, March 16, 2013 7:03 PM
>> Subject: Re: Performances Tests
>>
>> Hi Enis,
>>
>> "interesting" in the positive way ;)
>>
>> Results are there:
>>
>> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>>
>> The improvment on scan are impressive. sequentialRead and randomScan went
>> down.
>>
>> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>>
>> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
>> with it on previous HBase version...
>>
>> JM
>>
>> 2013/3/12 Enis Söztutar <en...@gmail.com>:
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> > dedicated computer with all 0.9x.x HBase versions, and I found results
>> > interesting.
>> > Can you please provide your numbers if you can. What is interesting from
>> > your findings?
>> >
>> > Enis
>> >
>> >
>> >
>> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
>> > jean-marc@spaggiari.org> wrote:
>> >
>> >> If you run only 1 client with PerformanceEvaluation, it's not running
>> >> it over MapReduce, so you don't have this overhead. But you can still
>> >> run it if you want to have something more distributed. Might be useful
>> >> to have the 2 options. But at the end, LoadTestTool or
>> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
>> >> those tests.
>> >>
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> >> dedicated computer with all 0.9x.x HBase versions, and I found results
>> >> interesting. That gives us a good baseline to see if new HBase
>> >> improvements are really improving performances.
>> >>
>> >> JM
>> >>
>> >> 2013/3/8 Andrew Purtell <ap...@apache.org>:
>> >> > Tangentally: I think I prefer LoadTestTool over
>> PerformanceEvaluation, it
>> >> > doesn't depend on nor is influenced by MapReduce job startup.
>> >> >
>> >> >
>> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
>> >> > ramkrishna.s.vasudevan@gmail.com> wrote:
>> >> >
>> >> >> @JM
>> >> >> I agree with you. Mainly the perf improvement changes needs some
>> >> >> testcases.
>> >> >> But sometimes the scenario on which the perf improvments happens are
>> bit
>> >> >> difficult to generate and we will be able to do in a standalone case
>> >> only.
>> >> >> May be overall if we need to get that perf improvment result we
>> need a
>> >> >> real cluster with suitable data. That is what i have experienced.
>> Just
>> >> >> telling.
>> >> >>
>> >> >> Regards
>> >> >> Ram
>> >> >>
>> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
>> >> >> jean-marc@spaggiari.org
>> >> >> > wrote:
>> >> >>
>> >> >> > Hi,
>> >> >> >
>> >> >> > In HBase we already have PerformanceEvaluation which gives us a
>> good
>> >> >> > way to validate that nothing broke HBase speed in the recent
>> updates.
>> >> >> >
>> >> >> > I can see in the JIRAs many improvements coming, like for the lazy
>> >> >> > seeks, the bloom filters, etc. however, there is no tests for those
>> >> >> > improvements.
>> >> >> >
>> >> >> > Will it not be good to ask people to add some new tests in
>> >> >> > PerformanceEvaluation when they are introducing an improvement
>> which
>> >> >> > is not covered there?
>> >> >> >
>> >> >> > We should not touch existing tests because we need to have a way to
>> >> >> > compare the baseline between the different versions, but we can
>> still
>> >> >> > add some new. Like in addition to RandomSeekScanTest we can add
>> >> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
>> >> >> > back port those new tests to previous version.
>> >> >> >
>> >> >> > The s
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: Performances Tests
Posted by lars hofhansl <la...@apache.org>.
Thanks J-M, so that should be easy to reproduce. I'll try.
-- Lars
________________________________
From: Jean-Marc Spaggiari <je...@spaggiari.org>
To: dev@hbase.apache.org
Sent: Sunday, March 17, 2013 9:47 AM
Subject: Re: Performances Tests
@Lars: here is an example of what I use
for i in {1..10}; do echo; echo -n $i ; rm -rf /tmp/*;
bin/start-hbase.sh; sleep 60; bin/hbase
org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 1;
bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=100
filterScan 1; bin/stop-hbase.sh; done &>> output.txt
Giving only 1 as the readers allow to not launch the test as a MR. I
have not try the same tests with a bigger number, but I can.
@Andy: LoadTestTool is also on my list, but for YCSB it's a but harder
since I will need more than one dedicated computer. But I will look at
it too. I might be able to remove 1 node from my cluster and dedicate
it to the tests when required...
I will be pretty buzy next week, but before next week-end I will try
to prepare the LoadTestTool scripts to run something similar. With
multiple scenarios, like with and whithout bloom, etc.
If yu have any recommandation/request, feel free! I will come back
soon with some additionnal numbers.
JM
2013/3/17 Andrew Purtell <ap...@apache.org>:
> I'm not sure I trust the results of PerformanceEvauation.
>
> LoadTestTool and YCSB have their own issues but seem to produce more
> consistent results. I've been thinking on adding scanning and filtering
> tests to LoadTestTool.
>
> On Sunday, March 17, 2013, lars hofhansl wrote:
>
>> Cool. The 0.94.3 scanning improvements seems almost unbelievable
>> (especially since many of my improvements to reduce the internal friction
>> went into 0.94.4).
>> I would like to track down the random read regression.
>>
>> Can you send the commands you ran? Are you running this as M/R job or
>> standalone client?
>>
>> Thanks for doing this J-M.
>>
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
>> To: dev@hbase.apache.org <javascript:;>
>> Sent: Saturday, March 16, 2013 7:03 PM
>> Subject: Re: Performances Tests
>>
>> Hi Enis,
>>
>> "interesting" in the positive way ;)
>>
>> Results are there:
>>
>> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>>
>> The improvment on scan are impressive. sequentialRead and randomScan went
>> down.
>>
>> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>>
>> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
>> with it on previous HBase version...
>>
>> JM
>>
>> 2013/3/12 Enis Söztutar <en...@gmail.com>:
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> > dedicated computer with all 0.9x.x HBase versions, and I found results
>> > interesting.
>> > Can you please provide your numbers if you can. What is interesting from
>> > your findings?
>> >
>> > Enis
>> >
>> >
>> >
>> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
>> > jean-marc@spaggiari.org> wrote:
>> >
>> >> If you run only 1 client with PerformanceEvaluation, it's not running
>> >> it over MapReduce, so you don't have this overhead. But you can still
>> >> run it if you want to have something more distributed. Might be useful
>> >> to have the 2 options. But at the end, LoadTestTool or
>> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
>> >> those tests.
>> >>
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> >> dedicated computer with all 0.9x.x HBase versions, and I found results
>> >> interesting. That gives us a good baseline to see if new HBase
>> >> improvements are really improving performances.
>> >>
>> >> JM
>> >>
>> >> 2013/3/8 Andrew Purtell <ap...@apache.org>:
>> >> > Tangentally: I think I prefer LoadTestTool over
>> PerformanceEvaluation, it
>> >> > doesn't depend on nor is influenced by MapReduce job startup.
>> >> >
>> >> >
>> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
>> >> > ramkrishna.s.vasudevan@gmail.com> wrote:
>> >> >
>> >> >> @JM
>> >> >> I agree with you. Mainly the perf improvement changes needs some
>> >> >> testcases.
>> >> >> But sometimes the scenario on which the perf improvments happens are
>> bit
>> >> >> difficult to generate and we will be able to do in a standalone case
>> >> only.
>> >> >> May be overall if we need to get that perf improvment result we
>> need a
>> >> >> real cluster with suitable data. That is what i have experienced.
>> Just
>> >> >> telling.
>> >> >>
>> >> >> Regards
>> >> >> Ram
>> >> >>
>> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
>> >> >> jean-marc@spaggiari.org
>> >> >> > wrote:
>> >> >>
>> >> >> > Hi,
>> >> >> >
>> >> >> > In HBase we already have PerformanceEvaluation which gives us a
>> good
>> >> >> > way to validate that nothing broke HBase speed in the recent
>> updates.
>> >> >> >
>> >> >> > I can see in the JIRAs many improvements coming, like for the lazy
>> >> >> > seeks, the bloom filters, etc. however, there is no tests for those
>> >> >> > improvements.
>> >> >> >
>> >> >> > Will it not be good to ask people to add some new tests in
>> >> >> > PerformanceEvaluation when they are introducing an improvement
>> which
>> >> >> > is not covered there?
>> >> >> >
>> >> >> > We should not touch existing tests because we need to have a way to
>> >> >> > compare the baseline between the different versions, but we can
>> still
>> >> >> > add some new. Like in addition to RandomSeekScanTest we can add
>> >> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
>> >> >> > back port those new tests to previous version.
>> >> >> >
>> >> >> > The s
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: Performances Tests
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
@Lars: here is an example of what I use
for i in {1..10}; do echo; echo -n $i ; rm -rf /tmp/*;
bin/start-hbase.sh; sleep 60; bin/hbase
org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 1;
bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=100
filterScan 1; bin/stop-hbase.sh; done &>> output.txt
Giving only 1 as the readers allow to not launch the test as a MR. I
have not try the same tests with a bigger number, but I can.
@Andy: LoadTestTool is also on my list, but for YCSB it's a but harder
since I will need more than one dedicated computer. But I will look at
it too. I might be able to remove 1 node from my cluster and dedicate
it to the tests when required...
I will be pretty buzy next week, but before next week-end I will try
to prepare the LoadTestTool scripts to run something similar. With
multiple scenarios, like with and whithout bloom, etc.
If yu have any recommandation/request, feel free! I will come back
soon with some additionnal numbers.
JM
2013/3/17 Andrew Purtell <ap...@apache.org>:
> I'm not sure I trust the results of PerformanceEvauation.
>
> LoadTestTool and YCSB have their own issues but seem to produce more
> consistent results. I've been thinking on adding scanning and filtering
> tests to LoadTestTool.
>
> On Sunday, March 17, 2013, lars hofhansl wrote:
>
>> Cool. The 0.94.3 scanning improvements seems almost unbelievable
>> (especially since many of my improvements to reduce the internal friction
>> went into 0.94.4).
>> I would like to track down the random read regression.
>>
>> Can you send the commands you ran? Are you running this as M/R job or
>> standalone client?
>>
>> Thanks for doing this J-M.
>>
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
>> To: dev@hbase.apache.org <javascript:;>
>> Sent: Saturday, March 16, 2013 7:03 PM
>> Subject: Re: Performances Tests
>>
>> Hi Enis,
>>
>> "interesting" in the positive way ;)
>>
>> Results are there:
>>
>> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>>
>> The improvment on scan are impressive. sequentialRead and randomScan went
>> down.
>>
>> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>>
>> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
>> with it on previous HBase version...
>>
>> JM
>>
>> 2013/3/12 Enis Söztutar <en...@gmail.com>:
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> > dedicated computer with all 0.9x.x HBase versions, and I found results
>> > interesting.
>> > Can you please provide your numbers if you can. What is interesting from
>> > your findings?
>> >
>> > Enis
>> >
>> >
>> >
>> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
>> > jean-marc@spaggiari.org> wrote:
>> >
>> >> If you run only 1 client with PerformanceEvaluation, it's not running
>> >> it over MapReduce, so you don't have this overhead. But you can still
>> >> run it if you want to have something more distributed. Might be useful
>> >> to have the 2 options. But at the end, LoadTestTool or
>> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
>> >> those tests.
>> >>
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> >> dedicated computer with all 0.9x.x HBase versions, and I found results
>> >> interesting. That gives us a good baseline to see if new HBase
>> >> improvements are really improving performances.
>> >>
>> >> JM
>> >>
>> >> 2013/3/8 Andrew Purtell <ap...@apache.org>:
>> >> > Tangentally: I think I prefer LoadTestTool over
>> PerformanceEvaluation, it
>> >> > doesn't depend on nor is influenced by MapReduce job startup.
>> >> >
>> >> >
>> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
>> >> > ramkrishna.s.vasudevan@gmail.com> wrote:
>> >> >
>> >> >> @JM
>> >> >> I agree with you. Mainly the perf improvement changes needs some
>> >> >> testcases.
>> >> >> But sometimes the scenario on which the perf improvments happens are
>> bit
>> >> >> difficult to generate and we will be able to do in a standalone case
>> >> only.
>> >> >> May be overall if we need to get that perf improvment result we
>> need a
>> >> >> real cluster with suitable data. That is what i have experienced.
>> Just
>> >> >> telling.
>> >> >>
>> >> >> Regards
>> >> >> Ram
>> >> >>
>> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
>> >> >> jean-marc@spaggiari.org
>> >> >> > wrote:
>> >> >>
>> >> >> > Hi,
>> >> >> >
>> >> >> > In HBase we already have PerformanceEvaluation which gives us a
>> good
>> >> >> > way to validate that nothing broke HBase speed in the recent
>> updates.
>> >> >> >
>> >> >> > I can see in the JIRAs many improvements coming, like for the lazy
>> >> >> > seeks, the bloom filters, etc. however, there is no tests for those
>> >> >> > improvements.
>> >> >> >
>> >> >> > Will it not be good to ask people to add some new tests in
>> >> >> > PerformanceEvaluation when they are introducing an improvement
>> which
>> >> >> > is not covered there?
>> >> >> >
>> >> >> > We should not touch existing tests because we need to have a way to
>> >> >> > compare the baseline between the different versions, but we can
>> still
>> >> >> > add some new. Like in addition to RandomSeekScanTest we can add
>> >> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
>> >> >> > back port those new tests to previous version.
>> >> >> >
>> >> >> > The s
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: Performances Tests
Posted by Andrew Purtell <ap...@apache.org>.
I'm not sure I trust the results of PerformanceEvauation.
LoadTestTool and YCSB have their own issues but seem to produce more
consistent results. I've been thinking on adding scanning and filtering
tests to LoadTestTool.
On Sunday, March 17, 2013, lars hofhansl wrote:
> Cool. The 0.94.3 scanning improvements seems almost unbelievable
> (especially since many of my improvements to reduce the internal friction
> went into 0.94.4).
> I would like to track down the random read regression.
>
> Can you send the commands you ran? Are you running this as M/R job or
> standalone client?
>
> Thanks for doing this J-M.
>
>
> -- Lars
>
>
>
> ________________________________
> From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
> To: dev@hbase.apache.org <javascript:;>
> Sent: Saturday, March 16, 2013 7:03 PM
> Subject: Re: Performances Tests
>
> Hi Enis,
>
> "interesting" in the positive way ;)
>
> Results are there:
>
> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>
> The improvment on scan are impressive. sequentialRead and randomScan went
> down.
>
> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>
> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
> with it on previous HBase version...
>
> JM
>
> 2013/3/12 Enis Söztutar <en...@gmail.com>:
> >> I just finished to run all the PerformanceEvaluation tests on a
> > dedicated computer with all 0.9x.x HBase versions, and I found results
> > interesting.
> > Can you please provide your numbers if you can. What is interesting from
> > your findings?
> >
> > Enis
> >
> >
> >
> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> If you run only 1 client with PerformanceEvaluation, it's not running
> >> it over MapReduce, so you don't have this overhead. But you can still
> >> run it if you want to have something more distributed. Might be useful
> >> to have the 2 options. But at the end, LoadTestTool or
> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
> >> those tests.
> >>
> >> I just finished to run all the PerformanceEvaluation tests on a
> >> dedicated computer with all 0.9x.x HBase versions, and I found results
> >> interesting. That gives us a good baseline to see if new HBase
> >> improvements are really improving performances.
> >>
> >> JM
> >>
> >> 2013/3/8 Andrew Purtell <ap...@apache.org>:
> >> > Tangentally: I think I prefer LoadTestTool over
> PerformanceEvaluation, it
> >> > doesn't depend on nor is influenced by MapReduce job startup.
> >> >
> >> >
> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
> >> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >> >
> >> >> @JM
> >> >> I agree with you. Mainly the perf improvement changes needs some
> >> >> testcases.
> >> >> But sometimes the scenario on which the perf improvments happens are
> bit
> >> >> difficult to generate and we will be able to do in a standalone case
> >> only.
> >> >> May be overall if we need to get that perf improvment result we
> need a
> >> >> real cluster with suitable data. That is what i have experienced.
> Just
> >> >> telling.
> >> >>
> >> >> Regards
> >> >> Ram
> >> >>
> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
> >> >> jean-marc@spaggiari.org
> >> >> > wrote:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > In HBase we already have PerformanceEvaluation which gives us a
> good
> >> >> > way to validate that nothing broke HBase speed in the recent
> updates.
> >> >> >
> >> >> > I can see in the JIRAs many improvements coming, like for the lazy
> >> >> > seeks, the bloom filters, etc. however, there is no tests for those
> >> >> > improvements.
> >> >> >
> >> >> > Will it not be good to ask people to add some new tests in
> >> >> > PerformanceEvaluation when they are introducing an improvement
> which
> >> >> > is not covered there?
> >> >> >
> >> >> > We should not touch existing tests because we need to have a way to
> >> >> > compare the baseline between the different versions, but we can
> still
> >> >> > add some new. Like in addition to RandomSeekScanTest we can add
> >> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
> >> >> > back port those new tests to previous version.
> >> >> >
> >> >> > The s
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Performances Tests
Posted by lars hofhansl <la...@apache.org>.
Cool. The 0.94.3 scanning improvements seems almost unbelievable (especially since many of my improvements to reduce the internal friction went into 0.94.4).
I would like to track down the random read regression.
Can you send the commands you ran? Are you running this as M/R job or standalone client?
Thanks for doing this J-M.
-- Lars
________________________________
From: Jean-Marc Spaggiari <je...@spaggiari.org>
To: dev@hbase.apache.org
Sent: Saturday, March 16, 2013 7:03 PM
Subject: Re: Performances Tests
Hi Enis,
"interesting" in the positive way ;)
Results are there:
http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
The improvment on scan are impressive. sequentialRead and randomScan went down.
In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
I will add HFilePerformanceEvaluation soon but I'm facinf some issues
with it on previous HBase version...
JM
2013/3/12 Enis Söztutar <en...@gmail.com>:
>> I just finished to run all the PerformanceEvaluation tests on a
> dedicated computer with all 0.9x.x HBase versions, and I found results
> interesting.
> Can you please provide your numbers if you can. What is interesting from
> your findings?
>
> Enis
>
>
>
> On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> If you run only 1 client with PerformanceEvaluation, it's not running
>> it over MapReduce, so you don't have this overhead. But you can still
>> run it if you want to have something more distributed. Might be useful
>> to have the 2 options. But at the end, LoadTestTool or
>> PerformanceEvaluation, any of the 2 is good as long as we are adding
>> those tests.
>>
>> I just finished to run all the PerformanceEvaluation tests on a
>> dedicated computer with all 0.9x.x HBase versions, and I found results
>> interesting. That gives us a good baseline to see if new HBase
>> improvements are really improving performances.
>>
>> JM
>>
>> 2013/3/8 Andrew Purtell <ap...@apache.org>:
>> > Tangentally: I think I prefer LoadTestTool over PerformanceEvaluation, it
>> > doesn't depend on nor is influenced by MapReduce job startup.
>> >
>> >
>> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
>> > ramkrishna.s.vasudevan@gmail.com> wrote:
>> >
>> >> @JM
>> >> I agree with you. Mainly the perf improvement changes needs some
>> >> testcases.
>> >> But sometimes the scenario on which the perf improvments happens are bit
>> >> difficult to generate and we will be able to do in a standalone case
>> only.
>> >> May be overall if we need to get that perf improvment result we need a
>> >> real cluster with suitable data. That is what i have experienced. Just
>> >> telling.
>> >>
>> >> Regards
>> >> Ram
>> >>
>> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
>> >> jean-marc@spaggiari.org
>> >> > wrote:
>> >>
>> >> > Hi,
>> >> >
>> >> > In HBase we already have PerformanceEvaluation which gives us a good
>> >> > way to validate that nothing broke HBase speed in the recent updates.
>> >> >
>> >> > I can see in the JIRAs many improvements coming, like for the lazy
>> >> > seeks, the bloom filters, etc. however, there is no tests for those
>> >> > improvements.
>> >> >
>> >> > Will it not be good to ask people to add some new tests in
>> >> > PerformanceEvaluation when they are introducing an improvement which
>> >> > is not covered there?
>> >> >
>> >> > We should not touch existing tests because we need to have a way to
>> >> > compare the baseline between the different versions, but we can still
>> >> > add some new. Like in addition to RandomSeekScanTest we can add
>> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
>> >> > back port those new tests to previous version.
>> >> >
>> >> > The same way we add a test class when we introduce a new feature,
>> >> > should we add a performance test method to test it too?
>> >> >
>> >> > JM
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > Best regards,
>> >
>> > - Andy
>> >
>> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> > (via Tom White)
>>
Re: Performances Tests
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Enis,
"interesting" in the positive way ;)
Results are there:
http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
The improvment on scan are impressive. sequentialRead and randomScan went down.
In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
I will add HFilePerformanceEvaluation soon but I'm facinf some issues
with it on previous HBase version...
JM
2013/3/12 Enis Söztutar <en...@gmail.com>:
>> I just finished to run all the PerformanceEvaluation tests on a
> dedicated computer with all 0.9x.x HBase versions, and I found results
> interesting.
> Can you please provide your numbers if you can. What is interesting from
> your findings?
>
> Enis
>
>
>
> On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> If you run only 1 client with PerformanceEvaluation, it's not running
>> it over MapReduce, so you don't have this overhead. But you can still
>> run it if you want to have something more distributed. Might be useful
>> to have the 2 options. But at the end, LoadTestTool or
>> PerformanceEvaluation, any of the 2 is good as long as we are adding
>> those tests.
>>
>> I just finished to run all the PerformanceEvaluation tests on a
>> dedicated computer with all 0.9x.x HBase versions, and I found results
>> interesting. That gives us a good baseline to see if new HBase
>> improvements are really improving performances.
>>
>> JM
>>
>> 2013/3/8 Andrew Purtell <ap...@apache.org>:
>> > Tangentally: I think I prefer LoadTestTool over PerformanceEvaluation, it
>> > doesn't depend on nor is influenced by MapReduce job startup.
>> >
>> >
>> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
>> > ramkrishna.s.vasudevan@gmail.com> wrote:
>> >
>> >> @JM
>> >> I agree with you. Mainly the perf improvement changes needs some
>> >> testcases.
>> >> But sometimes the scenario on which the perf improvments happens are bit
>> >> difficult to generate and we will be able to do in a standalone case
>> only.
>> >> May be overall if we need to get that perf improvment result we need a
>> >> real cluster with suitable data. That is what i have experienced. Just
>> >> telling.
>> >>
>> >> Regards
>> >> Ram
>> >>
>> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
>> >> jean-marc@spaggiari.org
>> >> > wrote:
>> >>
>> >> > Hi,
>> >> >
>> >> > In HBase we already have PerformanceEvaluation which gives us a good
>> >> > way to validate that nothing broke HBase speed in the recent updates.
>> >> >
>> >> > I can see in the JIRAs many improvements coming, like for the lazy
>> >> > seeks, the bloom filters, etc. however, there is no tests for those
>> >> > improvements.
>> >> >
>> >> > Will it not be good to ask people to add some new tests in
>> >> > PerformanceEvaluation when they are introducing an improvement which
>> >> > is not covered there?
>> >> >
>> >> > We should not touch existing tests because we need to have a way to
>> >> > compare the baseline between the different versions, but we can still
>> >> > add some new. Like in addition to RandomSeekScanTest we can add
>> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
>> >> > back port those new tests to previous version.
>> >> >
>> >> > The same way we add a test class when we introduce a new feature,
>> >> > should we add a performance test method to test it too?
>> >> >
>> >> > JM
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > Best regards,
>> >
>> > - Andy
>> >
>> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> > (via Tom White)
>>
Re: Performances Tests
Posted by Enis Söztutar <en...@gmail.com>.
> I just finished to run all the PerformanceEvaluation tests on a
dedicated computer with all 0.9x.x HBase versions, and I found results
interesting.
Can you please provide your numbers if you can. What is interesting from
your findings?
Enis
On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> If you run only 1 client with PerformanceEvaluation, it's not running
> it over MapReduce, so you don't have this overhead. But you can still
> run it if you want to have something more distributed. Might be useful
> to have the 2 options. But at the end, LoadTestTool or
> PerformanceEvaluation, any of the 2 is good as long as we are adding
> those tests.
>
> I just finished to run all the PerformanceEvaluation tests on a
> dedicated computer with all 0.9x.x HBase versions, and I found results
> interesting. That gives us a good baseline to see if new HBase
> improvements are really improving performances.
>
> JM
>
> 2013/3/8 Andrew Purtell <ap...@apache.org>:
> > Tangentally: I think I prefer LoadTestTool over PerformanceEvaluation, it
> > doesn't depend on nor is influenced by MapReduce job startup.
> >
> >
> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >
> >> @JM
> >> I agree with you. Mainly the perf improvement changes needs some
> >> testcases.
> >> But sometimes the scenario on which the perf improvments happens are bit
> >> difficult to generate and we will be able to do in a standalone case
> only.
> >> May be overall if we need to get that perf improvment result we need a
> >> real cluster with suitable data. That is what i have experienced. Just
> >> telling.
> >>
> >> Regards
> >> Ram
> >>
> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
> >> jean-marc@spaggiari.org
> >> > wrote:
> >>
> >> > Hi,
> >> >
> >> > In HBase we already have PerformanceEvaluation which gives us a good
> >> > way to validate that nothing broke HBase speed in the recent updates.
> >> >
> >> > I can see in the JIRAs many improvements coming, like for the lazy
> >> > seeks, the bloom filters, etc. however, there is no tests for those
> >> > improvements.
> >> >
> >> > Will it not be good to ask people to add some new tests in
> >> > PerformanceEvaluation when they are introducing an improvement which
> >> > is not covered there?
> >> >
> >> > We should not touch existing tests because we need to have a way to
> >> > compare the baseline between the different versions, but we can still
> >> > add some new. Like in addition to RandomSeekScanTest we can add
> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
> >> > back port those new tests to previous version.
> >> >
> >> > The same way we add a test class when we introduce a new feature,
> >> > should we add a performance test method to test it too?
> >> >
> >> > JM
> >> >
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
>
Re: Performances Tests
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
If you run only 1 client with PerformanceEvaluation, it's not running
it over MapReduce, so you don't have this overhead. But you can still
run it if you want to have something more distributed. Might be useful
to have the 2 options. But at the end, LoadTestTool or
PerformanceEvaluation, any of the 2 is good as long as we are adding
those tests.
I just finished to run all the PerformanceEvaluation tests on a
dedicated computer with all 0.9x.x HBase versions, and I found results
interesting. That gives us a good baseline to see if new HBase
improvements are really improving performances.
JM
2013/3/8 Andrew Purtell <ap...@apache.org>:
> Tangentally: I think I prefer LoadTestTool over PerformanceEvaluation, it
> doesn't depend on nor is influenced by MapReduce job startup.
>
>
> On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
>> @JM
>> I agree with you. Mainly the perf improvement changes needs some
>> testcases.
>> But sometimes the scenario on which the perf improvments happens are bit
>> difficult to generate and we will be able to do in a standalone case only.
>> May be overall if we need to get that perf improvment result we need a
>> real cluster with suitable data. That is what i have experienced. Just
>> telling.
>>
>> Regards
>> Ram
>>
>> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org
>> > wrote:
>>
>> > Hi,
>> >
>> > In HBase we already have PerformanceEvaluation which gives us a good
>> > way to validate that nothing broke HBase speed in the recent updates.
>> >
>> > I can see in the JIRAs many improvements coming, like for the lazy
>> > seeks, the bloom filters, etc. however, there is no tests for those
>> > improvements.
>> >
>> > Will it not be good to ask people to add some new tests in
>> > PerformanceEvaluation when they are introducing an improvement which
>> > is not covered there?
>> >
>> > We should not touch existing tests because we need to have a way to
>> > compare the baseline between the different versions, but we can still
>> > add some new. Like in addition to RandomSeekScanTest we can add
>> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
>> > back port those new tests to previous version.
>> >
>> > The same way we add a test class when we introduce a new feature,
>> > should we add a performance test method to test it too?
>> >
>> > JM
>> >
>>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: Performances Tests
Posted by Andrew Purtell <ap...@apache.org>.
Tangentally: I think I prefer LoadTestTool over PerformanceEvaluation, it
doesn't depend on nor is influenced by MapReduce job startup.
On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> @JM
> I agree with you. Mainly the perf improvement changes needs some
> testcases.
> But sometimes the scenario on which the perf improvments happens are bit
> difficult to generate and we will be able to do in a standalone case only.
> May be overall if we need to get that perf improvment result we need a
> real cluster with suitable data. That is what i have experienced. Just
> telling.
>
> Regards
> Ram
>
> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org
> > wrote:
>
> > Hi,
> >
> > In HBase we already have PerformanceEvaluation which gives us a good
> > way to validate that nothing broke HBase speed in the recent updates.
> >
> > I can see in the JIRAs many improvements coming, like for the lazy
> > seeks, the bloom filters, etc. however, there is no tests for those
> > improvements.
> >
> > Will it not be good to ask people to add some new tests in
> > PerformanceEvaluation when they are introducing an improvement which
> > is not covered there?
> >
> > We should not touch existing tests because we need to have a way to
> > compare the baseline between the different versions, but we can still
> > add some new. Like in addition to RandomSeekScanTest we can add
> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
> > back port those new tests to previous version.
> >
> > The same way we add a test class when we introduce a new feature,
> > should we add a performance test method to test it too?
> >
> > JM
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Performances Tests
Posted by ramkrishna vasudevan <ra...@gmail.com>.
@JM
I agree with you. Mainly the perf improvement changes needs some
testcases.
But sometimes the scenario on which the perf improvments happens are bit
difficult to generate and we will be able to do in a standalone case only.
May be overall if we need to get that perf improvment result we need a
real cluster with suitable data. That is what i have experienced. Just
telling.
Regards
Ram
On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <jean-marc@spaggiari.org
> wrote:
> Hi,
>
> In HBase we already have PerformanceEvaluation which gives us a good
> way to validate that nothing broke HBase speed in the recent updates.
>
> I can see in the JIRAs many improvements coming, like for the lazy
> seeks, the bloom filters, etc. however, there is no tests for those
> improvements.
>
> Will it not be good to ask people to add some new tests in
> PerformanceEvaluation when they are introducing an improvement which
> is not covered there?
>
> We should not touch existing tests because we need to have a way to
> compare the baseline between the different versions, but we can still
> add some new. Like in addition to RandomSeekScanTest we can add
> RandomSeekScanBloomEnabledTest and so on. And even better if we can
> back port those new tests to previous version.
>
> The same way we add a test class when we introduce a new feature,
> should we add a performance test method to test it too?
>
> JM
>