You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by lars hofhansl <la...@apache.org> on 2013/06/22 00:46:31 UTC
30% random performance in 0.95+
Is anybody ware of what set of changes would be responsible for giving 0.95+ a 30% boost in random read performance over 0.94?
Thanks.
-- Lars
Re: 30% random performance in 0.95+
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
If we can't rely on PE, then why do we keep it? ;)
I think that even if it loads the local hard drive, we can still
compare the 2 results because it does the same thing against the 2
versions. So if one version uses more the hard drive than another one,
then even with 12 drives and 100 nodes, it will still use more the
hard drive than the previous version. Goal here is really to compare 2
versions together, and not the performances.
Now, to make sure that I can run comparisons as good as possible, what
is the minimum hardware requirement? Are 3 dedicated servers enought?
(1RS+MS and 2RS) Or should I really go with 1 master + 3RS as a
minimum to have relevant results? I want to build something we will be
able to rely.
JM
2013/6/29 Varun Sharma <va...@pinterest.com>:
> Actually sending in a separate thread - since it does not really compare
> different versions of HBase but one version of Block Cache vs FS
> Cache(through hdfs).
>
>
> On Sat, Jun 29, 2013 at 12:00 PM, Varun Sharma <va...@pinterest.com> wrote:
>
>> I did some tests yesterday, on this. I will send them in a separate thread.
>>
>>
>> On Sat, Jun 29, 2013 at 5:10 AM, lars hofhansl <la...@apache.org> wrote:
>>
>>> In my measurements 0.94 has been getting faster with each release in both
>>> read and write performance.
>>> I wonder how representative PE is after all; it only tests via the local
>>> FS layer (not HDFS), among other issues.
>>>
>>> -- Lars
>>>
>>>
>>>
>>> ----- Original Message -----
>>> From: Jean-Marc Spaggiari <je...@spaggiari.org>
>>> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
>>> Cc:
>>> Sent: Friday, June 28, 2013 8:03 PM
>>> Subject: Re: 30% random performance in 0.95+
>>>
>>> I think we should do that on 0.94 as well. I don't see any good reason
>>> to not do it.
>>>
>>> JM
>>>
>>> 2013/6/28 lars hofhansl <la...@apache.org>:
>>> > Yep.
>>> > Now the question is: Make these changes to 0.94 as well? Or just
>>> document these better.
>>> >
>>> > -- Lars
>>> >
>>> >
>>> >
>>> > ----- Original Message -----
>>> > From: Andrew Purtell <ap...@apache.org>
>>> > To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <
>>> larsh@apache.org>
>>> > Cc:
>>> > Sent: Friday, June 28, 2013 2:08 PM
>>> > Subject: Re: 30% random performance in 0.95+
>>> >
>>> > I've been thinking about how to periodically search through some of our
>>> > parameter space to see what changes to defaults are better all the way
>>> > around. Probably will so something based on Bigtop.
>>> >
>>> >
>>> > On Friday, June 28, 2013, lars hofhansl wrote:
>>> >
>>> >> And indeed just this makes a tremendous difference. Unpatched 0.94 with
>>> >> 40% block cache configured is actually faster than 0.95 with the same
>>> block
>>> >> cache size.
>>> >>
>>> >> -- Lars
>>> >>
>>> >>
>>> >>
>>> >> ----- Original Message -----
>>> >> From: lars hofhansl <larsh@apache.org <javascript:;>>
>>> >> To: "dev@hbase.apache.org <javascript:;>" <dev@hbase.apache.org
>>> <javascript:;>
>>> >> >
>>> >> Cc:
>>> >> Sent: Friday, June 28, 2013 1:34 PM
>>> >> Subject: Re: 30% random performance in 0.95+
>>> >>
>>> >> Thanks JM,
>>> >>
>>> >> HBASE-8450 (r1485562) is interesting. It increases (among other things)
>>> >> the block cache percentage from 24 to 40%, which would lead to a higher
>>> >> probability of a future random read to hit an already cached block.
>>> >>
>>> >>
>>> >> -- Lars
>>> >>
>>> >>
>>> >>
>>> >> ----- Original Message -----
>>> >> From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
>>> >> To: dev@hbase.apache.org <javascript:;>; lars hofhansl <
>>> larsh@apache.org<javascript:;>
>>> >> >
>>> >> Cc:
>>> >> Sent: Friday, June 28, 2013 1:18 PM
>>> >> Subject: Re: 30% random performance in 0.95+
>>> >>
>>> >> I have the script done to run over a list of "svn releases", so if
>>> >> required, just give me a bunch of them or a range and I can restart.
>>> >> Just keep me posted.
>>> >>
>>> >> JM
>>> >>
>>> >> 2013/6/28 lars hofhansl <larsh@apache.org <javascript:;>>:
>>> >> > I did a few more test (on my laptop, which is not quite
>>> representative),
>>> >> and found only a 2-3% improvement from HBASE-8001+HBASE-8012 in the
>>> end.
>>> >> > I'll look through the issues that you identified.
>>> >> >
>>> >> > -- Lars
>>> >> >
>>> >> >
>>> >> >
>>> >> > ----- Original Message -----
>>> >> > From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
>>> >> > To: dev@hbase.apache.org <javascript:;>
>>> >> > Cc:
>>> >> > Sent: Friday, June 28, 2013 12:51 PM
>>> >> > Subject: Re: 30% random performance in 0.95+
>>> >> >
>>> >> > Sorry folks,
>>> >> >
>>> >> > I'm a bit late to run the tests... 0.94.8 and 0.94.9 are currently
>>> >> > running, but here is what I have been able to capture so far for 0.95
>>> >> > over the last year:
>>> >> > r1357480 1513196
>>> >> > r1367009 1440244.4
>>> >> > r1375812 1287143.5
>>> >> > r1381671 1287200.2
>>> >> > r1388620 1295262.6
>>> >> > r1394335 1022140.2
>>> >> > r1403898 884171.9
>>> >> > r1410631 804229.9
>>> >> > r1419787 846816.9
>>> >> > r1426557 853535.3
>>> >> > r1433514 873265.1
>>> >> > r1438972 840666.9
>>> >> > r1446106 877432.2
>>> >> > r1452661 883974.8
>>> >> > r1458421 882233.3
>>> >> > r1464267 847000.8
>>> >> > r1478964 877433.5
>>> >> > r1485868 744905.5
>>> >> > r1494869 765105.9
>>> >> >
>>> >> > So seems that there was some improvements between r1367009 and
>>> >> > r1403898 but they are old. Also another major improvement between
>>> >> > r1478964 and r1485868...
>>> >> >
>>> >> > Let me know if you want me to dig further and I will be very happy
>>> to do
>>> >> so.
>>> >> >
>>> >> > JM
>>> >> >
>>> >> > 2013/6/28 Stack <stack@duboce.net <javascript:;>>:
>>> >> >> On Fri, Jun 28, 2013 at 10:53 AM, lars hofhansl <larsh@apache.org
>>> <javascript:;>>
>>> >> wrote:
>>> >> >>
>>> >> >>> I partially tracked this down to HBASE-8001 and HBASE-8012 by
>>> looking
>>> >> at
>>> >> >>> the call stacks in a profiling session.
>>> >> >>> HBASE-8767 is a backport of both patched to 0.94.
>>> >> >>>
>>> >> >>
>>> >> >> Sounds like nice work by Raymond Liu...
>>> >> >> St.Ack
>>> >> >
>>> >>
>>> >
>>> >
>>> > --
>>> > Best regards,
>>> >
>>> > - Andy
>>> >
>>> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>> > (via Tom White)
>>> >
>>>
>>>
>>
Re: 30% random performance in 0.95+
Posted by Varun Sharma <va...@pinterest.com>.
Actually sending in a separate thread - since it does not really compare
different versions of HBase but one version of Block Cache vs FS
Cache(through hdfs).
On Sat, Jun 29, 2013 at 12:00 PM, Varun Sharma <va...@pinterest.com> wrote:
> I did some tests yesterday, on this. I will send them in a separate thread.
>
>
> On Sat, Jun 29, 2013 at 5:10 AM, lars hofhansl <la...@apache.org> wrote:
>
>> In my measurements 0.94 has been getting faster with each release in both
>> read and write performance.
>> I wonder how representative PE is after all; it only tests via the local
>> FS layer (not HDFS), among other issues.
>>
>> -- Lars
>>
>>
>>
>> ----- Original Message -----
>> From: Jean-Marc Spaggiari <je...@spaggiari.org>
>> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
>> Cc:
>> Sent: Friday, June 28, 2013 8:03 PM
>> Subject: Re: 30% random performance in 0.95+
>>
>> I think we should do that on 0.94 as well. I don't see any good reason
>> to not do it.
>>
>> JM
>>
>> 2013/6/28 lars hofhansl <la...@apache.org>:
>> > Yep.
>> > Now the question is: Make these changes to 0.94 as well? Or just
>> document these better.
>> >
>> > -- Lars
>> >
>> >
>> >
>> > ----- Original Message -----
>> > From: Andrew Purtell <ap...@apache.org>
>> > To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <
>> larsh@apache.org>
>> > Cc:
>> > Sent: Friday, June 28, 2013 2:08 PM
>> > Subject: Re: 30% random performance in 0.95+
>> >
>> > I've been thinking about how to periodically search through some of our
>> > parameter space to see what changes to defaults are better all the way
>> > around. Probably will so something based on Bigtop.
>> >
>> >
>> > On Friday, June 28, 2013, lars hofhansl wrote:
>> >
>> >> And indeed just this makes a tremendous difference. Unpatched 0.94 with
>> >> 40% block cache configured is actually faster than 0.95 with the same
>> block
>> >> cache size.
>> >>
>> >> -- Lars
>> >>
>> >>
>> >>
>> >> ----- Original Message -----
>> >> From: lars hofhansl <larsh@apache.org <javascript:;>>
>> >> To: "dev@hbase.apache.org <javascript:;>" <dev@hbase.apache.org
>> <javascript:;>
>> >> >
>> >> Cc:
>> >> Sent: Friday, June 28, 2013 1:34 PM
>> >> Subject: Re: 30% random performance in 0.95+
>> >>
>> >> Thanks JM,
>> >>
>> >> HBASE-8450 (r1485562) is interesting. It increases (among other things)
>> >> the block cache percentage from 24 to 40%, which would lead to a higher
>> >> probability of a future random read to hit an already cached block.
>> >>
>> >>
>> >> -- Lars
>> >>
>> >>
>> >>
>> >> ----- Original Message -----
>> >> From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
>> >> To: dev@hbase.apache.org <javascript:;>; lars hofhansl <
>> larsh@apache.org<javascript:;>
>> >> >
>> >> Cc:
>> >> Sent: Friday, June 28, 2013 1:18 PM
>> >> Subject: Re: 30% random performance in 0.95+
>> >>
>> >> I have the script done to run over a list of "svn releases", so if
>> >> required, just give me a bunch of them or a range and I can restart.
>> >> Just keep me posted.
>> >>
>> >> JM
>> >>
>> >> 2013/6/28 lars hofhansl <larsh@apache.org <javascript:;>>:
>> >> > I did a few more test (on my laptop, which is not quite
>> representative),
>> >> and found only a 2-3% improvement from HBASE-8001+HBASE-8012 in the
>> end.
>> >> > I'll look through the issues that you identified.
>> >> >
>> >> > -- Lars
>> >> >
>> >> >
>> >> >
>> >> > ----- Original Message -----
>> >> > From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
>> >> > To: dev@hbase.apache.org <javascript:;>
>> >> > Cc:
>> >> > Sent: Friday, June 28, 2013 12:51 PM
>> >> > Subject: Re: 30% random performance in 0.95+
>> >> >
>> >> > Sorry folks,
>> >> >
>> >> > I'm a bit late to run the tests... 0.94.8 and 0.94.9 are currently
>> >> > running, but here is what I have been able to capture so far for 0.95
>> >> > over the last year:
>> >> > r1357480 1513196
>> >> > r1367009 1440244.4
>> >> > r1375812 1287143.5
>> >> > r1381671 1287200.2
>> >> > r1388620 1295262.6
>> >> > r1394335 1022140.2
>> >> > r1403898 884171.9
>> >> > r1410631 804229.9
>> >> > r1419787 846816.9
>> >> > r1426557 853535.3
>> >> > r1433514 873265.1
>> >> > r1438972 840666.9
>> >> > r1446106 877432.2
>> >> > r1452661 883974.8
>> >> > r1458421 882233.3
>> >> > r1464267 847000.8
>> >> > r1478964 877433.5
>> >> > r1485868 744905.5
>> >> > r1494869 765105.9
>> >> >
>> >> > So seems that there was some improvements between r1367009 and
>> >> > r1403898 but they are old. Also another major improvement between
>> >> > r1478964 and r1485868...
>> >> >
>> >> > Let me know if you want me to dig further and I will be very happy
>> to do
>> >> so.
>> >> >
>> >> > JM
>> >> >
>> >> > 2013/6/28 Stack <stack@duboce.net <javascript:;>>:
>> >> >> On Fri, Jun 28, 2013 at 10:53 AM, lars hofhansl <larsh@apache.org
>> <javascript:;>>
>> >> wrote:
>> >> >>
>> >> >>> I partially tracked this down to HBASE-8001 and HBASE-8012 by
>> looking
>> >> at
>> >> >>> the call stacks in a profiling session.
>> >> >>> HBASE-8767 is a backport of both patched to 0.94.
>> >> >>>
>> >> >>
>> >> >> Sounds like nice work by Raymond Liu...
>> >> >> St.Ack
>> >> >
>> >>
>> >
>> >
>> > --
>> > Best regards,
>> >
>> > - Andy
>> >
>> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> > (via Tom White)
>> >
>>
>>
>
Re: 30% random performance in 0.95+
Posted by Varun Sharma <va...@pinterest.com>.
I did some tests yesterday, on this. I will send them in a separate thread.
On Sat, Jun 29, 2013 at 5:10 AM, lars hofhansl <la...@apache.org> wrote:
> In my measurements 0.94 has been getting faster with each release in both
> read and write performance.
> I wonder how representative PE is after all; it only tests via the local
> FS layer (not HDFS), among other issues.
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: Jean-Marc Spaggiari <je...@spaggiari.org>
> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> Cc:
> Sent: Friday, June 28, 2013 8:03 PM
> Subject: Re: 30% random performance in 0.95+
>
> I think we should do that on 0.94 as well. I don't see any good reason
> to not do it.
>
> JM
>
> 2013/6/28 lars hofhansl <la...@apache.org>:
> > Yep.
> > Now the question is: Make these changes to 0.94 as well? Or just
> document these better.
> >
> > -- Lars
> >
> >
> >
> > ----- Original Message -----
> > From: Andrew Purtell <ap...@apache.org>
> > To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <
> larsh@apache.org>
> > Cc:
> > Sent: Friday, June 28, 2013 2:08 PM
> > Subject: Re: 30% random performance in 0.95+
> >
> > I've been thinking about how to periodically search through some of our
> > parameter space to see what changes to defaults are better all the way
> > around. Probably will so something based on Bigtop.
> >
> >
> > On Friday, June 28, 2013, lars hofhansl wrote:
> >
> >> And indeed just this makes a tremendous difference. Unpatched 0.94 with
> >> 40% block cache configured is actually faster than 0.95 with the same
> block
> >> cache size.
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ----- Original Message -----
> >> From: lars hofhansl <larsh@apache.org <javascript:;>>
> >> To: "dev@hbase.apache.org <javascript:;>" <dev@hbase.apache.org
> <javascript:;>
> >> >
> >> Cc:
> >> Sent: Friday, June 28, 2013 1:34 PM
> >> Subject: Re: 30% random performance in 0.95+
> >>
> >> Thanks JM,
> >>
> >> HBASE-8450 (r1485562) is interesting. It increases (among other things)
> >> the block cache percentage from 24 to 40%, which would lead to a higher
> >> probability of a future random read to hit an already cached block.
> >>
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ----- Original Message -----
> >> From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
> >> To: dev@hbase.apache.org <javascript:;>; lars hofhansl <
> larsh@apache.org<javascript:;>
> >> >
> >> Cc:
> >> Sent: Friday, June 28, 2013 1:18 PM
> >> Subject: Re: 30% random performance in 0.95+
> >>
> >> I have the script done to run over a list of "svn releases", so if
> >> required, just give me a bunch of them or a range and I can restart.
> >> Just keep me posted.
> >>
> >> JM
> >>
> >> 2013/6/28 lars hofhansl <larsh@apache.org <javascript:;>>:
> >> > I did a few more test (on my laptop, which is not quite
> representative),
> >> and found only a 2-3% improvement from HBASE-8001+HBASE-8012 in the end.
> >> > I'll look through the issues that you identified.
> >> >
> >> > -- Lars
> >> >
> >> >
> >> >
> >> > ----- Original Message -----
> >> > From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
> >> > To: dev@hbase.apache.org <javascript:;>
> >> > Cc:
> >> > Sent: Friday, June 28, 2013 12:51 PM
> >> > Subject: Re: 30% random performance in 0.95+
> >> >
> >> > Sorry folks,
> >> >
> >> > I'm a bit late to run the tests... 0.94.8 and 0.94.9 are currently
> >> > running, but here is what I have been able to capture so far for 0.95
> >> > over the last year:
> >> > r1357480 1513196
> >> > r1367009 1440244.4
> >> > r1375812 1287143.5
> >> > r1381671 1287200.2
> >> > r1388620 1295262.6
> >> > r1394335 1022140.2
> >> > r1403898 884171.9
> >> > r1410631 804229.9
> >> > r1419787 846816.9
> >> > r1426557 853535.3
> >> > r1433514 873265.1
> >> > r1438972 840666.9
> >> > r1446106 877432.2
> >> > r1452661 883974.8
> >> > r1458421 882233.3
> >> > r1464267 847000.8
> >> > r1478964 877433.5
> >> > r1485868 744905.5
> >> > r1494869 765105.9
> >> >
> >> > So seems that there was some improvements between r1367009 and
> >> > r1403898 but they are old. Also another major improvement between
> >> > r1478964 and r1485868...
> >> >
> >> > Let me know if you want me to dig further and I will be very happy to
> do
> >> so.
> >> >
> >> > JM
> >> >
> >> > 2013/6/28 Stack <stack@duboce.net <javascript:;>>:
> >> >> On Fri, Jun 28, 2013 at 10:53 AM, lars hofhansl <larsh@apache.org
> <javascript:;>>
> >> wrote:
> >> >>
> >> >>> I partially tracked this down to HBASE-8001 and HBASE-8012 by
> looking
> >> at
> >> >>> the call stacks in a profiling session.
> >> >>> HBASE-8767 is a backport of both patched to 0.94.
> >> >>>
> >> >>
> >> >> Sounds like nice work by Raymond Liu...
> >> >> St.Ack
> >> >
> >>
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
Re: 30% random performance in 0.95+
Posted by lars hofhansl <la...@apache.org>.
In my measurements 0.94 has been getting faster with each release in both read and write performance.
I wonder how representative PE is after all; it only tests via the local FS layer (not HDFS), among other issues.
-- Lars
----- Original Message -----
From: Jean-Marc Spaggiari <je...@spaggiari.org>
To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
Cc:
Sent: Friday, June 28, 2013 8:03 PM
Subject: Re: 30% random performance in 0.95+
I think we should do that on 0.94 as well. I don't see any good reason
to not do it.
JM
2013/6/28 lars hofhansl <la...@apache.org>:
> Yep.
> Now the question is: Make these changes to 0.94 as well? Or just document these better.
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: Andrew Purtell <ap...@apache.org>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <la...@apache.org>
> Cc:
> Sent: Friday, June 28, 2013 2:08 PM
> Subject: Re: 30% random performance in 0.95+
>
> I've been thinking about how to periodically search through some of our
> parameter space to see what changes to defaults are better all the way
> around. Probably will so something based on Bigtop.
>
>
> On Friday, June 28, 2013, lars hofhansl wrote:
>
>> And indeed just this makes a tremendous difference. Unpatched 0.94 with
>> 40% block cache configured is actually faster than 0.95 with the same block
>> cache size.
>>
>> -- Lars
>>
>>
>>
>> ----- Original Message -----
>> From: lars hofhansl <larsh@apache.org <javascript:;>>
>> To: "dev@hbase.apache.org <javascript:;>" <dev@hbase.apache.org<javascript:;>
>> >
>> Cc:
>> Sent: Friday, June 28, 2013 1:34 PM
>> Subject: Re: 30% random performance in 0.95+
>>
>> Thanks JM,
>>
>> HBASE-8450 (r1485562) is interesting. It increases (among other things)
>> the block cache percentage from 24 to 40%, which would lead to a higher
>> probability of a future random read to hit an already cached block.
>>
>>
>> -- Lars
>>
>>
>>
>> ----- Original Message -----
>> From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
>> To: dev@hbase.apache.org <javascript:;>; lars hofhansl <larsh@apache.org<javascript:;>
>> >
>> Cc:
>> Sent: Friday, June 28, 2013 1:18 PM
>> Subject: Re: 30% random performance in 0.95+
>>
>> I have the script done to run over a list of "svn releases", so if
>> required, just give me a bunch of them or a range and I can restart.
>> Just keep me posted.
>>
>> JM
>>
>> 2013/6/28 lars hofhansl <larsh@apache.org <javascript:;>>:
>> > I did a few more test (on my laptop, which is not quite representative),
>> and found only a 2-3% improvement from HBASE-8001+HBASE-8012 in the end.
>> > I'll look through the issues that you identified.
>> >
>> > -- Lars
>> >
>> >
>> >
>> > ----- Original Message -----
>> > From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
>> > To: dev@hbase.apache.org <javascript:;>
>> > Cc:
>> > Sent: Friday, June 28, 2013 12:51 PM
>> > Subject: Re: 30% random performance in 0.95+
>> >
>> > Sorry folks,
>> >
>> > I'm a bit late to run the tests... 0.94.8 and 0.94.9 are currently
>> > running, but here is what I have been able to capture so far for 0.95
>> > over the last year:
>> > r1357480 1513196
>> > r1367009 1440244.4
>> > r1375812 1287143.5
>> > r1381671 1287200.2
>> > r1388620 1295262.6
>> > r1394335 1022140.2
>> > r1403898 884171.9
>> > r1410631 804229.9
>> > r1419787 846816.9
>> > r1426557 853535.3
>> > r1433514 873265.1
>> > r1438972 840666.9
>> > r1446106 877432.2
>> > r1452661 883974.8
>> > r1458421 882233.3
>> > r1464267 847000.8
>> > r1478964 877433.5
>> > r1485868 744905.5
>> > r1494869 765105.9
>> >
>> > So seems that there was some improvements between r1367009 and
>> > r1403898 but they are old. Also another major improvement between
>> > r1478964 and r1485868...
>> >
>> > Let me know if you want me to dig further and I will be very happy to do
>> so.
>> >
>> > JM
>> >
>> > 2013/6/28 Stack <stack@duboce.net <javascript:;>>:
>> >> On Fri, Jun 28, 2013 at 10:53 AM, lars hofhansl <larsh@apache.org<javascript:;>>
>> wrote:
>> >>
>> >>> I partially tracked this down to HBASE-8001 and HBASE-8012 by looking
>> at
>> >>> the call stacks in a profiling session.
>> >>> HBASE-8767 is a backport of both patched to 0.94.
>> >>>
>> >>
>> >> Sounds like nice work by Raymond Liu...
>> >> St.Ack
>> >
>>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: 30% random performance in 0.95+
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
I think we should do that on 0.94 as well. I don't see any good reason
to not do it.
JM
2013/6/28 lars hofhansl <la...@apache.org>:
> Yep.
> Now the question is: Make these changes to 0.94 as well? Or just document these better.
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: Andrew Purtell <ap...@apache.org>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <la...@apache.org>
> Cc:
> Sent: Friday, June 28, 2013 2:08 PM
> Subject: Re: 30% random performance in 0.95+
>
> I've been thinking about how to periodically search through some of our
> parameter space to see what changes to defaults are better all the way
> around. Probably will so something based on Bigtop.
>
>
> On Friday, June 28, 2013, lars hofhansl wrote:
>
>> And indeed just this makes a tremendous difference. Unpatched 0.94 with
>> 40% block cache configured is actually faster than 0.95 with the same block
>> cache size.
>>
>> -- Lars
>>
>>
>>
>> ----- Original Message -----
>> From: lars hofhansl <larsh@apache.org <javascript:;>>
>> To: "dev@hbase.apache.org <javascript:;>" <dev@hbase.apache.org<javascript:;>
>> >
>> Cc:
>> Sent: Friday, June 28, 2013 1:34 PM
>> Subject: Re: 30% random performance in 0.95+
>>
>> Thanks JM,
>>
>> HBASE-8450 (r1485562) is interesting. It increases (among other things)
>> the block cache percentage from 24 to 40%, which would lead to a higher
>> probability of a future random read to hit an already cached block.
>>
>>
>> -- Lars
>>
>>
>>
>> ----- Original Message -----
>> From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
>> To: dev@hbase.apache.org <javascript:;>; lars hofhansl <larsh@apache.org<javascript:;>
>> >
>> Cc:
>> Sent: Friday, June 28, 2013 1:18 PM
>> Subject: Re: 30% random performance in 0.95+
>>
>> I have the script done to run over a list of "svn releases", so if
>> required, just give me a bunch of them or a range and I can restart.
>> Just keep me posted.
>>
>> JM
>>
>> 2013/6/28 lars hofhansl <larsh@apache.org <javascript:;>>:
>> > I did a few more test (on my laptop, which is not quite representative),
>> and found only a 2-3% improvement from HBASE-8001+HBASE-8012 in the end.
>> > I'll look through the issues that you identified.
>> >
>> > -- Lars
>> >
>> >
>> >
>> > ----- Original Message -----
>> > From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
>> > To: dev@hbase.apache.org <javascript:;>
>> > Cc:
>> > Sent: Friday, June 28, 2013 12:51 PM
>> > Subject: Re: 30% random performance in 0.95+
>> >
>> > Sorry folks,
>> >
>> > I'm a bit late to run the tests... 0.94.8 and 0.94.9 are currently
>> > running, but here is what I have been able to capture so far for 0.95
>> > over the last year:
>> > r1357480 1513196
>> > r1367009 1440244.4
>> > r1375812 1287143.5
>> > r1381671 1287200.2
>> > r1388620 1295262.6
>> > r1394335 1022140.2
>> > r1403898 884171.9
>> > r1410631 804229.9
>> > r1419787 846816.9
>> > r1426557 853535.3
>> > r1433514 873265.1
>> > r1438972 840666.9
>> > r1446106 877432.2
>> > r1452661 883974.8
>> > r1458421 882233.3
>> > r1464267 847000.8
>> > r1478964 877433.5
>> > r1485868 744905.5
>> > r1494869 765105.9
>> >
>> > So seems that there was some improvements between r1367009 and
>> > r1403898 but they are old. Also another major improvement between
>> > r1478964 and r1485868...
>> >
>> > Let me know if you want me to dig further and I will be very happy to do
>> so.
>> >
>> > JM
>> >
>> > 2013/6/28 Stack <stack@duboce.net <javascript:;>>:
>> >> On Fri, Jun 28, 2013 at 10:53 AM, lars hofhansl <larsh@apache.org<javascript:;>>
>> wrote:
>> >>
>> >>> I partially tracked this down to HBASE-8001 and HBASE-8012 by looking
>> at
>> >>> the call stacks in a profiling session.
>> >>> HBASE-8767 is a backport of both patched to 0.94.
>> >>>
>> >>
>> >> Sounds like nice work by Raymond Liu...
>> >> St.Ack
>> >
>>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: 30% random performance in 0.95+
Posted by lars hofhansl <la...@apache.org>.
Yep.
Now the question is: Make these changes to 0.94 as well? Or just document these better.
-- Lars
----- Original Message -----
From: Andrew Purtell <ap...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <la...@apache.org>
Cc:
Sent: Friday, June 28, 2013 2:08 PM
Subject: Re: 30% random performance in 0.95+
I've been thinking about how to periodically search through some of our
parameter space to see what changes to defaults are better all the way
around. Probably will so something based on Bigtop.
On Friday, June 28, 2013, lars hofhansl wrote:
> And indeed just this makes a tremendous difference. Unpatched 0.94 with
> 40% block cache configured is actually faster than 0.95 with the same block
> cache size.
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: lars hofhansl <larsh@apache.org <javascript:;>>
> To: "dev@hbase.apache.org <javascript:;>" <dev@hbase.apache.org<javascript:;>
> >
> Cc:
> Sent: Friday, June 28, 2013 1:34 PM
> Subject: Re: 30% random performance in 0.95+
>
> Thanks JM,
>
> HBASE-8450 (r1485562) is interesting. It increases (among other things)
> the block cache percentage from 24 to 40%, which would lead to a higher
> probability of a future random read to hit an already cached block.
>
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
> To: dev@hbase.apache.org <javascript:;>; lars hofhansl <larsh@apache.org<javascript:;>
> >
> Cc:
> Sent: Friday, June 28, 2013 1:18 PM
> Subject: Re: 30% random performance in 0.95+
>
> I have the script done to run over a list of "svn releases", so if
> required, just give me a bunch of them or a range and I can restart.
> Just keep me posted.
>
> JM
>
> 2013/6/28 lars hofhansl <larsh@apache.org <javascript:;>>:
> > I did a few more test (on my laptop, which is not quite representative),
> and found only a 2-3% improvement from HBASE-8001+HBASE-8012 in the end.
> > I'll look through the issues that you identified.
> >
> > -- Lars
> >
> >
> >
> > ----- Original Message -----
> > From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
> > To: dev@hbase.apache.org <javascript:;>
> > Cc:
> > Sent: Friday, June 28, 2013 12:51 PM
> > Subject: Re: 30% random performance in 0.95+
> >
> > Sorry folks,
> >
> > I'm a bit late to run the tests... 0.94.8 and 0.94.9 are currently
> > running, but here is what I have been able to capture so far for 0.95
> > over the last year:
> > r1357480 1513196
> > r1367009 1440244.4
> > r1375812 1287143.5
> > r1381671 1287200.2
> > r1388620 1295262.6
> > r1394335 1022140.2
> > r1403898 884171.9
> > r1410631 804229.9
> > r1419787 846816.9
> > r1426557 853535.3
> > r1433514 873265.1
> > r1438972 840666.9
> > r1446106 877432.2
> > r1452661 883974.8
> > r1458421 882233.3
> > r1464267 847000.8
> > r1478964 877433.5
> > r1485868 744905.5
> > r1494869 765105.9
> >
> > So seems that there was some improvements between r1367009 and
> > r1403898 but they are old. Also another major improvement between
> > r1478964 and r1485868...
> >
> > Let me know if you want me to dig further and I will be very happy to do
> so.
> >
> > JM
> >
> > 2013/6/28 Stack <stack@duboce.net <javascript:;>>:
> >> On Fri, Jun 28, 2013 at 10:53 AM, lars hofhansl <larsh@apache.org<javascript:;>>
> wrote:
> >>
> >>> I partially tracked this down to HBASE-8001 and HBASE-8012 by looking
> at
> >>> the call stacks in a profiling session.
> >>> HBASE-8767 is a backport of both patched to 0.94.
> >>>
> >>
> >> Sounds like nice work by Raymond Liu...
> >> St.Ack
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: 30% random performance in 0.95+
Posted by Andrew Purtell <ap...@apache.org>.
I've been thinking about how to periodically search through some of our
parameter space to see what changes to defaults are better all the way
around. Probably will so something based on Bigtop.
On Friday, June 28, 2013, lars hofhansl wrote:
> And indeed just this makes a tremendous difference. Unpatched 0.94 with
> 40% block cache configured is actually faster than 0.95 with the same block
> cache size.
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: lars hofhansl <larsh@apache.org <javascript:;>>
> To: "dev@hbase.apache.org <javascript:;>" <dev@hbase.apache.org<javascript:;>
> >
> Cc:
> Sent: Friday, June 28, 2013 1:34 PM
> Subject: Re: 30% random performance in 0.95+
>
> Thanks JM,
>
> HBASE-8450 (r1485562) is interesting. It increases (among other things)
> the block cache percentage from 24 to 40%, which would lead to a higher
> probability of a future random read to hit an already cached block.
>
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
> To: dev@hbase.apache.org <javascript:;>; lars hofhansl <larsh@apache.org<javascript:;>
> >
> Cc:
> Sent: Friday, June 28, 2013 1:18 PM
> Subject: Re: 30% random performance in 0.95+
>
> I have the script done to run over a list of "svn releases", so if
> required, just give me a bunch of them or a range and I can restart.
> Just keep me posted.
>
> JM
>
> 2013/6/28 lars hofhansl <larsh@apache.org <javascript:;>>:
> > I did a few more test (on my laptop, which is not quite representative),
> and found only a 2-3% improvement from HBASE-8001+HBASE-8012 in the end.
> > I'll look through the issues that you identified.
> >
> > -- Lars
> >
> >
> >
> > ----- Original Message -----
> > From: Jean-Marc Spaggiari <jean-marc@spaggiari.org <javascript:;>>
> > To: dev@hbase.apache.org <javascript:;>
> > Cc:
> > Sent: Friday, June 28, 2013 12:51 PM
> > Subject: Re: 30% random performance in 0.95+
> >
> > Sorry folks,
> >
> > I'm a bit late to run the tests... 0.94.8 and 0.94.9 are currently
> > running, but here is what I have been able to capture so far for 0.95
> > over the last year:
> > r1357480 1513196
> > r1367009 1440244.4
> > r1375812 1287143.5
> > r1381671 1287200.2
> > r1388620 1295262.6
> > r1394335 1022140.2
> > r1403898 884171.9
> > r1410631 804229.9
> > r1419787 846816.9
> > r1426557 853535.3
> > r1433514 873265.1
> > r1438972 840666.9
> > r1446106 877432.2
> > r1452661 883974.8
> > r1458421 882233.3
> > r1464267 847000.8
> > r1478964 877433.5
> > r1485868 744905.5
> > r1494869 765105.9
> >
> > So seems that there was some improvements between r1367009 and
> > r1403898 but they are old. Also another major improvement between
> > r1478964 and r1485868...
> >
> > Let me know if you want me to dig further and I will be very happy to do
> so.
> >
> > JM
> >
> > 2013/6/28 Stack <stack@duboce.net <javascript:;>>:
> >> On Fri, Jun 28, 2013 at 10:53 AM, lars hofhansl <larsh@apache.org<javascript:;>>
> wrote:
> >>
> >>> I partially tracked this down to HBASE-8001 and HBASE-8012 by looking
> at
> >>> the call stacks in a profiling session.
> >>> HBASE-8767 is a backport of both patched to 0.94.
> >>>
> >>
> >> Sounds like nice work by Raymond Liu...
> >> St.Ack
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: 30% random performance in 0.95+
Posted by lars hofhansl <la...@apache.org>.
And indeed just this makes a tremendous difference. Unpatched 0.94 with 40% block cache configured is actually faster than 0.95 with the same block cache size.
-- Lars
----- Original Message -----
From: lars hofhansl <la...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>
Cc:
Sent: Friday, June 28, 2013 1:34 PM
Subject: Re: 30% random performance in 0.95+
Thanks JM,
HBASE-8450 (r1485562) is interesting. It increases (among other things) the block cache percentage from 24 to 40%, which would lead to a higher probability of a future random read to hit an already cached block.
-- Lars
----- Original Message -----
From: Jean-Marc Spaggiari <je...@spaggiari.org>
To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
Cc:
Sent: Friday, June 28, 2013 1:18 PM
Subject: Re: 30% random performance in 0.95+
I have the script done to run over a list of "svn releases", so if
required, just give me a bunch of them or a range and I can restart.
Just keep me posted.
JM
2013/6/28 lars hofhansl <la...@apache.org>:
> I did a few more test (on my laptop, which is not quite representative), and found only a 2-3% improvement from HBASE-8001+HBASE-8012 in the end.
> I'll look through the issues that you identified.
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: Jean-Marc Spaggiari <je...@spaggiari.org>
> To: dev@hbase.apache.org
> Cc:
> Sent: Friday, June 28, 2013 12:51 PM
> Subject: Re: 30% random performance in 0.95+
>
> Sorry folks,
>
> I'm a bit late to run the tests... 0.94.8 and 0.94.9 are currently
> running, but here is what I have been able to capture so far for 0.95
> over the last year:
> r1357480 1513196
> r1367009 1440244.4
> r1375812 1287143.5
> r1381671 1287200.2
> r1388620 1295262.6
> r1394335 1022140.2
> r1403898 884171.9
> r1410631 804229.9
> r1419787 846816.9
> r1426557 853535.3
> r1433514 873265.1
> r1438972 840666.9
> r1446106 877432.2
> r1452661 883974.8
> r1458421 882233.3
> r1464267 847000.8
> r1478964 877433.5
> r1485868 744905.5
> r1494869 765105.9
>
> So seems that there was some improvements between r1367009 and
> r1403898 but they are old. Also another major improvement between
> r1478964 and r1485868...
>
> Let me know if you want me to dig further and I will be very happy to do so.
>
> JM
>
> 2013/6/28 Stack <st...@duboce.net>:
>> On Fri, Jun 28, 2013 at 10:53 AM, lars hofhansl <la...@apache.org> wrote:
>>
>>> I partially tracked this down to HBASE-8001 and HBASE-8012 by looking at
>>> the call stacks in a profiling session.
>>> HBASE-8767 is a backport of both patched to 0.94.
>>>
>>
>> Sounds like nice work by Raymond Liu...
>> St.Ack
>
Re: 30% random performance in 0.95+
Posted by lars hofhansl <la...@apache.org>.
Thanks JM,
HBASE-8450 (r1485562) is interesting. It increases (among other things) the block cache percentage from 24 to 40%, which would lead to a higher probability of a future random read to hit an already cached block.
-- Lars
----- Original Message -----
From: Jean-Marc Spaggiari <je...@spaggiari.org>
To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
Cc:
Sent: Friday, June 28, 2013 1:18 PM
Subject: Re: 30% random performance in 0.95+
I have the script done to run over a list of "svn releases", so if
required, just give me a bunch of them or a range and I can restart.
Just keep me posted.
JM
2013/6/28 lars hofhansl <la...@apache.org>:
> I did a few more test (on my laptop, which is not quite representative), and found only a 2-3% improvement from HBASE-8001+HBASE-8012 in the end.
> I'll look through the issues that you identified.
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: Jean-Marc Spaggiari <je...@spaggiari.org>
> To: dev@hbase.apache.org
> Cc:
> Sent: Friday, June 28, 2013 12:51 PM
> Subject: Re: 30% random performance in 0.95+
>
> Sorry folks,
>
> I'm a bit late to run the tests... 0.94.8 and 0.94.9 are currently
> running, but here is what I have been able to capture so far for 0.95
> over the last year:
> r1357480 1513196
> r1367009 1440244.4
> r1375812 1287143.5
> r1381671 1287200.2
> r1388620 1295262.6
> r1394335 1022140.2
> r1403898 884171.9
> r1410631 804229.9
> r1419787 846816.9
> r1426557 853535.3
> r1433514 873265.1
> r1438972 840666.9
> r1446106 877432.2
> r1452661 883974.8
> r1458421 882233.3
> r1464267 847000.8
> r1478964 877433.5
> r1485868 744905.5
> r1494869 765105.9
>
> So seems that there was some improvements between r1367009 and
> r1403898 but they are old. Also another major improvement between
> r1478964 and r1485868...
>
> Let me know if you want me to dig further and I will be very happy to do so.
>
> JM
>
> 2013/6/28 Stack <st...@duboce.net>:
>> On Fri, Jun 28, 2013 at 10:53 AM, lars hofhansl <la...@apache.org> wrote:
>>
>>> I partially tracked this down to HBASE-8001 and HBASE-8012 by looking at
>>> the call stacks in a profiling session.
>>> HBASE-8767 is a backport of both patched to 0.94.
>>>
>>
>> Sounds like nice work by Raymond Liu...
>> St.Ack
>
Re: 30% random performance in 0.95+
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
I have the script done to run over a list of "svn releases", so if
required, just give me a bunch of them or a range and I can restart.
Just keep me posted.
JM
2013/6/28 lars hofhansl <la...@apache.org>:
> I did a few more test (on my laptop, which is not quite representative), and found only a 2-3% improvement from HBASE-8001+HBASE-8012 in the end.
> I'll look through the issues that you identified.
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: Jean-Marc Spaggiari <je...@spaggiari.org>
> To: dev@hbase.apache.org
> Cc:
> Sent: Friday, June 28, 2013 12:51 PM
> Subject: Re: 30% random performance in 0.95+
>
> Sorry folks,
>
> I'm a bit late to run the tests... 0.94.8 and 0.94.9 are currently
> running, but here is what I have been able to capture so far for 0.95
> over the last year:
> r1357480 1513196
> r1367009 1440244.4
> r1375812 1287143.5
> r1381671 1287200.2
> r1388620 1295262.6
> r1394335 1022140.2
> r1403898 884171.9
> r1410631 804229.9
> r1419787 846816.9
> r1426557 853535.3
> r1433514 873265.1
> r1438972 840666.9
> r1446106 877432.2
> r1452661 883974.8
> r1458421 882233.3
> r1464267 847000.8
> r1478964 877433.5
> r1485868 744905.5
> r1494869 765105.9
>
> So seems that there was some improvements between r1367009 and
> r1403898 but they are old. Also another major improvement between
> r1478964 and r1485868...
>
> Let me know if you want me to dig further and I will be very happy to do so.
>
> JM
>
> 2013/6/28 Stack <st...@duboce.net>:
>> On Fri, Jun 28, 2013 at 10:53 AM, lars hofhansl <la...@apache.org> wrote:
>>
>>> I partially tracked this down to HBASE-8001 and HBASE-8012 by looking at
>>> the call stacks in a profiling session.
>>> HBASE-8767 is a backport of both patched to 0.94.
>>>
>>
>> Sounds like nice work by Raymond Liu...
>> St.Ack
>
Re: 30% random performance in 0.95+
Posted by lars hofhansl <la...@apache.org>.
I did a few more test (on my laptop, which is not quite representative), and found only a 2-3% improvement from HBASE-8001+HBASE-8012 in the end.
I'll look through the issues that you identified.
-- Lars
----- Original Message -----
From: Jean-Marc Spaggiari <je...@spaggiari.org>
To: dev@hbase.apache.org
Cc:
Sent: Friday, June 28, 2013 12:51 PM
Subject: Re: 30% random performance in 0.95+
Sorry folks,
I'm a bit late to run the tests... 0.94.8 and 0.94.9 are currently
running, but here is what I have been able to capture so far for 0.95
over the last year:
r1357480 1513196
r1367009 1440244.4
r1375812 1287143.5
r1381671 1287200.2
r1388620 1295262.6
r1394335 1022140.2
r1403898 884171.9
r1410631 804229.9
r1419787 846816.9
r1426557 853535.3
r1433514 873265.1
r1438972 840666.9
r1446106 877432.2
r1452661 883974.8
r1458421 882233.3
r1464267 847000.8
r1478964 877433.5
r1485868 744905.5
r1494869 765105.9
So seems that there was some improvements between r1367009 and
r1403898 but they are old. Also another major improvement between
r1478964 and r1485868...
Let me know if you want me to dig further and I will be very happy to do so.
JM
2013/6/28 Stack <st...@duboce.net>:
> On Fri, Jun 28, 2013 at 10:53 AM, lars hofhansl <la...@apache.org> wrote:
>
>> I partially tracked this down to HBASE-8001 and HBASE-8012 by looking at
>> the call stacks in a profiling session.
>> HBASE-8767 is a backport of both patched to 0.94.
>>
>
> Sounds like nice work by Raymond Liu...
> St.Ack
Re: 30% random performance in 0.95+
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Sorry folks,
I'm a bit late to run the tests... 0.94.8 and 0.94.9 are currently
running, but here is what I have been able to capture so far for 0.95
over the last year:
r1357480 1513196
r1367009 1440244.4
r1375812 1287143.5
r1381671 1287200.2
r1388620 1295262.6
r1394335 1022140.2
r1403898 884171.9
r1410631 804229.9
r1419787 846816.9
r1426557 853535.3
r1433514 873265.1
r1438972 840666.9
r1446106 877432.2
r1452661 883974.8
r1458421 882233.3
r1464267 847000.8
r1478964 877433.5
r1485868 744905.5
r1494869 765105.9
So seems that there was some improvements between r1367009 and
r1403898 but they are old. Also another major improvement between
r1478964 and r1485868...
Let me know if you want me to dig further and I will be very happy to do so.
JM
2013/6/28 Stack <st...@duboce.net>:
> On Fri, Jun 28, 2013 at 10:53 AM, lars hofhansl <la...@apache.org> wrote:
>
>> I partially tracked this down to HBASE-8001 and HBASE-8012 by looking at
>> the call stacks in a profiling session.
>> HBASE-8767 is a backport of both patched to 0.94.
>>
>
> Sounds like nice work by Raymond Liu...
> St.Ack
Re: 30% random performance in 0.95+
Posted by Stack <st...@duboce.net>.
On Fri, Jun 28, 2013 at 10:53 AM, lars hofhansl <la...@apache.org> wrote:
> I partially tracked this down to HBASE-8001 and HBASE-8012 by looking at
> the call stacks in a profiling session.
> HBASE-8767 is a backport of both patched to 0.94.
>
Sounds like nice work by Raymond Liu...
St.Ack
Re: 30% random performance in 0.95+
Posted by lars hofhansl <la...@apache.org>.
I partially tracked this down to HBASE-8001 and HBASE-8012 by looking at the call stacks in a profiling session.
HBASE-8767 is a backport of both patched to 0.94.
-- Lars
________________________________
From: lars hofhansl <la...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>
Sent: Sunday, June 23, 2013 4:41 AM
Subject: Re: 30% random performance in 0.95+
Thanks JM!
________________________________
From: Jean-Marc Spaggiari <je...@spaggiari.org>
To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
Sent: Saturday, June 22, 2013 7:21 AM
Subject: Re: 30% random performance in 0.95+
Hey, a dichotomic search.
(http://en.wikipedia.org/wiki/Dichotomic_search) I thought about that,
but it's a bit more difficult to script.
So, I reduced it to test only 20 builds over the last year. Every 78
commits. Then we will tackle down from there. Should have the results
in about 48h (it's about half a day for 5 runs.)
JM
2013/6/22 lars hofhansl <la...@apache.org>:
> Maybe do a "binary search" instead?
> I.e. run it at the beginning and end of the past year, then in the middle, then in that middle, etc.
> Assuming we're looking for a single major improvement, that should work and take much less time.
>
> Or maybe do every 100 commits, and after we narrowed the range I can look at the patches in that range.
>
>
> -- Lars
>
>
>
> ________________________________
> From: Jean-Marc Spaggiari <je...@spaggiari.org>
> To: dev@hbase.apache.org
> Sent: Saturday, June 22, 2013 6:19 AM
> Subject: Re: 30% random performance in 0.95+
>
>
> Tests are ready to run. I scripted something which is going to run
> tests over the last year for 0.95 and run the read tests for every 30
> commits, which is about 50 tests over the year.
>
> It will take about 5 days to run. So I'm first running the tests for
> HBASE-8755 with 10 threads and will run this one after.
>
> JM
>
> 2013/6/21 Jean-Marc Spaggiari <je...@spaggiari.org>:
>> Yep, I agree. If we found where the improvement is coming from, having
>> it on 0.94 might be awesome. No more reasons for me to migrate to 0.95
>> ;)
>>
>> 2013/6/21 Andrew Purtell <ap...@apache.org>:
>>> At least I would be curious if any of that could be bottled up and
>>> sprinkled on 0.94.
>>>
>>>
>>> On Fri, Jun 21, 2013 at 6:25 PM, Jean-Marc Spaggiari <
>>> jean-marc@spaggiari.org> wrote:
>>>
>>>> Should we really worry about that? ;) If it will have been a -30%, it
>>>> will have been an issue ;)
>>>>
>>>> If you want I can run the RandomRead test for the last 12 months,
>>>> running a test for every 5 or 10 commits? so we will see when this
>>>> happened?
>>>>
>>>> JM
>>>>
>>>> 2013/6/21 lars hofhansl <la...@apache.org>:
>>>> > Is anybody ware of what set of changes would be responsible for giving
>>>> 0.95+ a 30% boost in random read performance over 0.94?
>>>> >
>>>> > Thanks.
>>>> >
>>>> > -- Lars
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>> (via Tom White)
Re: 30% random performance in 0.95+
Posted by lars hofhansl <la...@apache.org>.
Thanks JM!
________________________________
From: Jean-Marc Spaggiari <je...@spaggiari.org>
To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
Sent: Saturday, June 22, 2013 7:21 AM
Subject: Re: 30% random performance in 0.95+
Hey, a dichotomic search.
(http://en.wikipedia.org/wiki/Dichotomic_search) I thought about that,
but it's a bit more difficult to script.
So, I reduced it to test only 20 builds over the last year. Every 78
commits. Then we will tackle down from there. Should have the results
in about 48h (it's about half a day for 5 runs.)
JM
2013/6/22 lars hofhansl <la...@apache.org>:
> Maybe do a "binary search" instead?
> I.e. run it at the beginning and end of the past year, then in the middle, then in that middle, etc.
> Assuming we're looking for a single major improvement, that should work and take much less time.
>
> Or maybe do every 100 commits, and after we narrowed the range I can look at the patches in that range.
>
>
> -- Lars
>
>
>
> ________________________________
> From: Jean-Marc Spaggiari <je...@spaggiari.org>
> To: dev@hbase.apache.org
> Sent: Saturday, June 22, 2013 6:19 AM
> Subject: Re: 30% random performance in 0.95+
>
>
> Tests are ready to run. I scripted something which is going to run
> tests over the last year for 0.95 and run the read tests for every 30
> commits, which is about 50 tests over the year.
>
> It will take about 5 days to run. So I'm first running the tests for
> HBASE-8755 with 10 threads and will run this one after.
>
> JM
>
> 2013/6/21 Jean-Marc Spaggiari <je...@spaggiari.org>:
>> Yep, I agree. If we found where the improvement is coming from, having
>> it on 0.94 might be awesome. No more reasons for me to migrate to 0.95
>> ;)
>>
>> 2013/6/21 Andrew Purtell <ap...@apache.org>:
>>> At least I would be curious if any of that could be bottled up and
>>> sprinkled on 0.94.
>>>
>>>
>>> On Fri, Jun 21, 2013 at 6:25 PM, Jean-Marc Spaggiari <
>>> jean-marc@spaggiari.org> wrote:
>>>
>>>> Should we really worry about that? ;) If it will have been a -30%, it
>>>> will have been an issue ;)
>>>>
>>>> If you want I can run the RandomRead test for the last 12 months,
>>>> running a test for every 5 or 10 commits? so we will see when this
>>>> happened?
>>>>
>>>> JM
>>>>
>>>> 2013/6/21 lars hofhansl <la...@apache.org>:
>>>> > Is anybody ware of what set of changes would be responsible for giving
>>>> 0.95+ a 30% boost in random read performance over 0.94?
>>>> >
>>>> > Thanks.
>>>> >
>>>> > -- Lars
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>> (via Tom White)
Re: 30% random performance in 0.95+
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hey, a dichotomic search.
(http://en.wikipedia.org/wiki/Dichotomic_search) I thought about that,
but it's a bit more difficult to script.
So, I reduced it to test only 20 builds over the last year. Every 78
commits. Then we will tackle down from there. Should have the results
in about 48h (it's about half a day for 5 runs.)
JM
2013/6/22 lars hofhansl <la...@apache.org>:
> Maybe do a "binary search" instead?
> I.e. run it at the beginning and end of the past year, then in the middle, then in that middle, etc.
> Assuming we're looking for a single major improvement, that should work and take much less time.
>
> Or maybe do every 100 commits, and after we narrowed the range I can look at the patches in that range.
>
>
> -- Lars
>
>
>
> ________________________________
> From: Jean-Marc Spaggiari <je...@spaggiari.org>
> To: dev@hbase.apache.org
> Sent: Saturday, June 22, 2013 6:19 AM
> Subject: Re: 30% random performance in 0.95+
>
>
> Tests are ready to run. I scripted something which is going to run
> tests over the last year for 0.95 and run the read tests for every 30
> commits, which is about 50 tests over the year.
>
> It will take about 5 days to run. So I'm first running the tests for
> HBASE-8755 with 10 threads and will run this one after.
>
> JM
>
> 2013/6/21 Jean-Marc Spaggiari <je...@spaggiari.org>:
>> Yep, I agree. If we found where the improvement is coming from, having
>> it on 0.94 might be awesome. No more reasons for me to migrate to 0.95
>> ;)
>>
>> 2013/6/21 Andrew Purtell <ap...@apache.org>:
>>> At least I would be curious if any of that could be bottled up and
>>> sprinkled on 0.94.
>>>
>>>
>>> On Fri, Jun 21, 2013 at 6:25 PM, Jean-Marc Spaggiari <
>>> jean-marc@spaggiari.org> wrote:
>>>
>>>> Should we really worry about that? ;) If it will have been a -30%, it
>>>> will have been an issue ;)
>>>>
>>>> If you want I can run the RandomRead test for the last 12 months,
>>>> running a test for every 5 or 10 commits? so we will see when this
>>>> happened?
>>>>
>>>> JM
>>>>
>>>> 2013/6/21 lars hofhansl <la...@apache.org>:
>>>> > Is anybody ware of what set of changes would be responsible for giving
>>>> 0.95+ a 30% boost in random read performance over 0.94?
>>>> >
>>>> > Thanks.
>>>> >
>>>> > -- Lars
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>> (via Tom White)
Re: 30% random performance in 0.95+
Posted by lars hofhansl <la...@apache.org>.
Maybe do a "binary search" instead?
I.e. run it at the beginning and end of the past year, then in the middle, then in that middle, etc.
Assuming we're looking for a single major improvement, that should work and take much less time.
Or maybe do every 100 commits, and after we narrowed the range I can look at the patches in that range.
-- Lars
________________________________
From: Jean-Marc Spaggiari <je...@spaggiari.org>
To: dev@hbase.apache.org
Sent: Saturday, June 22, 2013 6:19 AM
Subject: Re: 30% random performance in 0.95+
Tests are ready to run. I scripted something which is going to run
tests over the last year for 0.95 and run the read tests for every 30
commits, which is about 50 tests over the year.
It will take about 5 days to run. So I'm first running the tests for
HBASE-8755 with 10 threads and will run this one after.
JM
2013/6/21 Jean-Marc Spaggiari <je...@spaggiari.org>:
> Yep, I agree. If we found where the improvement is coming from, having
> it on 0.94 might be awesome. No more reasons for me to migrate to 0.95
> ;)
>
> 2013/6/21 Andrew Purtell <ap...@apache.org>:
>> At least I would be curious if any of that could be bottled up and
>> sprinkled on 0.94.
>>
>>
>> On Fri, Jun 21, 2013 at 6:25 PM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org> wrote:
>>
>>> Should we really worry about that? ;) If it will have been a -30%, it
>>> will have been an issue ;)
>>>
>>> If you want I can run the RandomRead test for the last 12 months,
>>> running a test for every 5 or 10 commits? so we will see when this
>>> happened?
>>>
>>> JM
>>>
>>> 2013/6/21 lars hofhansl <la...@apache.org>:
>>> > Is anybody ware of what set of changes would be responsible for giving
>>> 0.95+ a 30% boost in random read performance over 0.94?
>>> >
>>> > Thanks.
>>> >
>>> > -- Lars
>>> >
>>>
>>
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
Re: 30% random performance in 0.95+
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Tests are ready to run. I scripted something which is going to run
tests over the last year for 0.95 and run the read tests for every 30
commits, which is about 50 tests over the year.
It will take about 5 days to run. So I'm first running the tests for
HBASE-8755 with 10 threads and will run this one after.
JM
2013/6/21 Jean-Marc Spaggiari <je...@spaggiari.org>:
> Yep, I agree. If we found where the improvement is coming from, having
> it on 0.94 might be awesome. No more reasons for me to migrate to 0.95
> ;)
>
> 2013/6/21 Andrew Purtell <ap...@apache.org>:
>> At least I would be curious if any of that could be bottled up and
>> sprinkled on 0.94.
>>
>>
>> On Fri, Jun 21, 2013 at 6:25 PM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org> wrote:
>>
>>> Should we really worry about that? ;) If it will have been a -30%, it
>>> will have been an issue ;)
>>>
>>> If you want I can run the RandomRead test for the last 12 months,
>>> running a test for every 5 or 10 commits? so we will see when this
>>> happened?
>>>
>>> JM
>>>
>>> 2013/6/21 lars hofhansl <la...@apache.org>:
>>> > Is anybody ware of what set of changes would be responsible for giving
>>> 0.95+ a 30% boost in random read performance over 0.94?
>>> >
>>> > Thanks.
>>> >
>>> > -- Lars
>>> >
>>>
>>
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
Re: 30% random performance in 0.95+
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Yep, I agree. If we found where the improvement is coming from, having
it on 0.94 might be awesome. No more reasons for me to migrate to 0.95
;)
2013/6/21 Andrew Purtell <ap...@apache.org>:
> At least I would be curious if any of that could be bottled up and
> sprinkled on 0.94.
>
>
> On Fri, Jun 21, 2013 at 6:25 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> Should we really worry about that? ;) If it will have been a -30%, it
>> will have been an issue ;)
>>
>> If you want I can run the RandomRead test for the last 12 months,
>> running a test for every 5 or 10 commits? so we will see when this
>> happened?
>>
>> JM
>>
>> 2013/6/21 lars hofhansl <la...@apache.org>:
>> > Is anybody ware of what set of changes would be responsible for giving
>> 0.95+ a 30% boost in random read performance over 0.94?
>> >
>> > Thanks.
>> >
>> > -- Lars
>> >
>>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: 30% random performance in 0.95+
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Yep, I will start that tonight or tomorrow morning when 8755 will be
done. Since it will take at least 5 days I will most probably order a
new server in the meantime to not block the other performances
investigations ;)
So don't expect results before wednesday....
Another option is to run tests every 50 commits to reduce the number
of tests and get an idea faster, but the range will be bigger so will
be harder to spot the modification which improved the read.
JM
2013/6/22 lars hofhansl <la...@apache.org>:
> That's what I am thinking.
> JM, if it's not too much work and you can it over night or something I wouldn't say no :)
>
>
> -- Lars
>
>
>
> ________________________________
> From: Andrew Purtell <ap...@apache.org>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> Sent: Friday, June 21, 2013 6:33 PM
> Subject: Re: 30% random performance in 0.95+
>
>
> At least I would be curious if any of that could be bottled up and
> sprinkled on 0.94.
>
>
> On Fri, Jun 21, 2013 at 6:25 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> Should we really worry about that? ;) If it will have been a -30%, it
>> will have been an issue ;)
>>
>> If you want I can run the RandomRead test for the last 12 months,
>> running a test for every 5 or 10 commits? so we will see when this
>> happened?
>>
>> JM
>>
>> 2013/6/21 lars hofhansl <la...@apache.org>:
>> > Is anybody ware of what set of changes would be responsible for giving
>> 0.95+ a 30% boost in random read performance over 0.94?
>> >
>> > Thanks.
>> >
>> > -- Lars
>> >
>>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: 30% random performance in 0.95+
Posted by lars hofhansl <la...@apache.org>.
That's what I am thinking.
JM, if it's not too much work and you can it over night or something I wouldn't say no :)
-- Lars
________________________________
From: Andrew Purtell <ap...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>
Sent: Friday, June 21, 2013 6:33 PM
Subject: Re: 30% random performance in 0.95+
At least I would be curious if any of that could be bottled up and
sprinkled on 0.94.
On Fri, Jun 21, 2013 at 6:25 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Should we really worry about that? ;) If it will have been a -30%, it
> will have been an issue ;)
>
> If you want I can run the RandomRead test for the last 12 months,
> running a test for every 5 or 10 commits? so we will see when this
> happened?
>
> JM
>
> 2013/6/21 lars hofhansl <la...@apache.org>:
> > Is anybody ware of what set of changes would be responsible for giving
> 0.95+ a 30% boost in random read performance over 0.94?
> >
> > Thanks.
> >
> > -- Lars
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: 30% random performance in 0.95+
Posted by Andrew Purtell <ap...@apache.org>.
At least I would be curious if any of that could be bottled up and
sprinkled on 0.94.
On Fri, Jun 21, 2013 at 6:25 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Should we really worry about that? ;) If it will have been a -30%, it
> will have been an issue ;)
>
> If you want I can run the RandomRead test for the last 12 months,
> running a test for every 5 or 10 commits? so we will see when this
> happened?
>
> JM
>
> 2013/6/21 lars hofhansl <la...@apache.org>:
> > Is anybody ware of what set of changes would be responsible for giving
> 0.95+ a 30% boost in random read performance over 0.94?
> >
> > Thanks.
> >
> > -- Lars
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: 30% random performance in 0.95+
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Should we really worry about that? ;) If it will have been a -30%, it
will have been an issue ;)
If you want I can run the RandomRead test for the last 12 months,
running a test for every 5 or 10 commits? so we will see when this
happened?
JM
2013/6/21 lars hofhansl <la...@apache.org>:
> Is anybody ware of what set of changes would be responsible for giving 0.95+ a 30% boost in random read performance over 0.94?
>
> Thanks.
>
> -- Lars
>