You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Eshcar Hillel <es...@oath.com.INVALID> on 2017/11/23 15:35:22 UTC
[DISCUSS] Performance degradation in master (compared to 2-alpha-1)
Happy Thanksgiving all,
In recent benchmarks I ran in HBASE-18294 I discovered major performance degradation of master code w.r.t 2-alpha-1 code.I am running write-only workload (similar to the one reported in HBASE-16417). I am using the same hardware and same configuration settings (specifically, I testes both basic memstore compaction with optimal parameters, and no memsore compaction).While in 2-alpha-1 code I see throughput of ~110Kops for basic compaction and ~80Kops for no compaction, in the master code I get only 60Kops and 55Kops, respectively. *This is almost 50% reduction in performance*.
(1) Did anyone else noticed such degradation?(2) Do we have any systematic automatic/semi-automatic method to track the sources of this performance issue?
Thanks,Eshcar
Re: [DISCUSS] Performance degradation in master (compared to 2-alpha-1)
Posted by Ted Yu <yu...@gmail.com>.
Happy Thanksgiving to you.
Since handling regression isn't the goal of HBASE-18294, mind logging
another JIRA ?
I applied patch v6 from HBASE-18294 on branch-2 and observed minor conflict
(CellUtil.java). Meaning compaction / flush code is mostly the same between
branch-2 and master.
Over HBASE-19338, > 20% boost was reported with patch. However, that is not
enough to bring performance back.
Cheers
On Thu, Nov 23, 2017 at 7:35 AM, Eshcar Hillel <es...@oath.com.invalid>
wrote:
> Happy Thanksgiving all,
> In recent benchmarks I ran in HBASE-18294 I discovered major performance
> degradation of master code w.r.t 2-alpha-1 code.I am running write-only
> workload (similar to the one reported in HBASE-16417). I am using the same
> hardware and same configuration settings (specifically, I testes both basic
> memstore compaction with optimal parameters, and no memsore
> compaction).While in 2-alpha-1 code I see throughput of ~110Kops for basic
> compaction and ~80Kops for no compaction, in the master code I get only
> 60Kops and 55Kops, respectively. *This is almost 50% reduction in
> performance*.
> (1) Did anyone else noticed such degradation?(2) Do we have any systematic
> automatic/semi-automatic method to track the sources of this performance
> issue?
> Thanks,Eshcar
>
>
>
Re: [DISCUSS] Performance degradation in master (compared to 2-alpha-1)
Posted by Vladimir Rodionov <vl...@gmail.com>.
I do not know if this is related, but test suite for backup now runs 57 min
on master. It used to run under 34
-Vlad
On Tue, Nov 28, 2017 at 9:16 AM, Mike Drob <md...@apache.org> wrote:
> Eshcar - do you have time to try the other alpha releases and see where
> exactly we introduced the regressions?
>
> Also, I'm worried that the performance regression may be related to an
> important bug-fix, where before we may have had fast writes but also risked
> incorrect behavior somehow.
>
> Mike
>
> On Tue, Nov 28, 2017 at 2:48 AM, Eshcar Hillel <es...@oath.com.invalid>
> wrote:
>
> > I agree, so will wait till we focus on performance.
> > Just one more update, I also ran the same experiment (write-only) with
> > banch-2 beta-1.Here is a summary of the throughput I see in each
> tag/branch:
> > ------------------------------- | BASIC | NONE |
> > -------------------------------2-alpha-1| 110K | 80K |
> > 2-beta-1 | 81K | 62K |
> > master | 60K | 55K |-------------------------------
> > This means there are multiple sources for the regression.
> >
> > Thanks
> >
> > On Saturday, November 25, 2017, 7:44:01 AM GMT+2, 张铎(Duo Zhang) <
> > palomino219@gmail.com> wrote:
> >
> > I think first we need a release plan on when we will begin to focus on
> the
> > performance issue?
> >
> > I do not think it is a good time to focus on performance issue now as we
> > haven’t stabilized our build yet. The performance regression may come
> back
> > again after some bug fixes and maybe we use a wrong way to increase
> > performance and finally we find that it is just a bug...
> >
> > Of course I do not mean we can not do any performance related issues now,
> > for example, HBASE-19338 is a good catch and can be fixed right now.
> >
> > And also, for AsyncFSWAL and in memory compaction, we need to consider
> the
> > performance right now as they are born for performance, but let’s focus
> on
> > the comparison to other policies, not a previous release so we can find
> the
> > correct things to fix.
> >
> > Of course, if there is a big performance downgrading comparing to the
> > previous release and we find it then we should tell others, just like
> this
> > email. An earlier notification is always welcomed.
> >
> > Thanks.
> >
> > Stack <st...@duboce.net>于2017年11月25日 周六13:22写道：
> >
> > > On Thu, Nov 23, 2017 at 7:35 AM, Eshcar Hillel <eshcar@oath.com.invalid
> >
> > > wrote:
> > >
> > > > Happy Thanksgiving all,
> > > >
> > >
> > > And to you Eshcar.
> > >
> > >
> > >
> > > > In recent benchmarks I ran in HBASE-18294 I discovered major
> > performance
> > > > degradation of master code w.r.t 2-alpha-1 code.I am running
> write-only
> > > > workload (similar to the one reported in HBASE-16417). I am using the
> > > same
> > > > hardware and same configuration settings (specifically, I testes both
> > > basic
> > > > memstore compaction with optimal parameters, and no memsore
> > > > compaction).While in 2-alpha-1 code I see throughput of ~110Kops for
> > > basic
> > > > compaction and ~80Kops for no compaction, in the master code I get
> only
> > > > 60Kops and 55Kops, respectively. *This is almost 50% reduction in
> > > > performance*.
> > > > (1) Did anyone else noticed such degradation?(2) Do we have any
> > > systematic
> > > > automatic/semi-automatic method to track the sources of this
> > performance
> > > > issue?
> > > > Thanks,Eshcar
> > > >
> > >
> > >
> > > On #1, no. I've not done perf compare. I wonder if later alpha versions
> > > include the regression (I'll have to check and see).
> > >
> > > On #2, again no. I intend to do a bit of perf tuning and compare before
> > > release.
> > >
> > > If you don't file an issue, I will do so later for myself as a task to
> > > compare at least to alpha-1.
> > >
> > > Thanks Eshcar,
> > >
> > > St.Ack
> > >
> >
>
Re: [DISCUSS] Performance degradation in master (compared to 2-alpha-1)
Posted by Anoop John <an...@gmail.com>.
Eschar,
In ur tests did u notice any increase in Log size? Specially
LOG.warn("Large batch operation detected (greater than " + rowSizeWarnThreshold
+ ") (HBASE-18023)." + " Requested Number of Rows: " + sum
+ " Client: "
+ RpcServer.getRequestUserName().orElse(null) + "/"
+ RpcServer.getRemoteAddress().orElse(null)
+ " first region in multi=" + firstRegionName);
-Anoop-
On Wed, Nov 29, 2017 at 4:01 PM, Eshcar Hillel <es...@oath.com.invalid> wrote:
> I'm currently using my cluster for other purposes.When it is available I can run the same experiment against alpha-3.
>
>
> On Tuesday, November 28, 2017, 7:17:08 PM GMT+2, Mike Drob <md...@apache.org> wrote:
>
> Eshcar - do you have time to try the other alpha releases and see where
> exactly we introduced the regressions?
>
> Also, I'm worried that the performance regression may be related to an
> important bug-fix, where before we may have had fast writes but also risked
> incorrect behavior somehow.
>
> Mike
>
> On Tue, Nov 28, 2017 at 2:48 AM, Eshcar Hillel <es...@oath.com.invalid>
> wrote:
>
>> I agree, so will wait till we focus on performance.
>> Just one more update, I also ran the same experiment (write-only) with
>> banch-2 beta-1.Here is a summary of the throughput I see in each tag/branch:
>> ------------------------------- | BASIC | NONE |
>> -------------------------------2-alpha-1| 110K | 80K |
>> 2-beta-1 | 81K | 62K |
>> master | 60K | 55K |-------------------------------
>> This means there are multiple sources for the regression.
>>
>> Thanks
>>
>> On Saturday, November 25, 2017, 7:44:01 AM GMT+2, 张铎(Duo Zhang) <
>> palomino219@gmail.com> wrote:
>>
>> I think first we need a release plan on when we will begin to focus on the
>> performance issue?
>>
>> I do not think it is a good time to focus on performance issue now as we
>> haven’t stabilized our build yet. The performance regression may come back
>> again after some bug fixes and maybe we use a wrong way to increase
>> performance and finally we find that it is just a bug...
>>
>> Of course I do not mean we can not do any performance related issues now,
>> for example, HBASE-19338 is a good catch and can be fixed right now.
>>
>> And also, for AsyncFSWAL and in memory compaction, we need to consider the
>> performance right now as they are born for performance, but let’s focus on
>> the comparison to other policies, not a previous release so we can find the
>> correct things to fix.
>>
>> Of course, if there is a big performance downgrading comparing to the
>> previous release and we find it then we should tell others, just like this
>> email. An earlier notification is always welcomed.
>>
>> Thanks.
>>
>> Stack <st...@duboce.net>于2017年11月25日 周六13:22写道：
>>
>> > On Thu, Nov 23, 2017 at 7:35 AM, Eshcar Hillel <es...@oath.com.invalid>
>> > wrote:
>> >
>> > > Happy Thanksgiving all,
>> > >
>> >
>> > And to you Eshcar.
>> >
>> >
>> >
>> > > In recent benchmarks I ran in HBASE-18294 I discovered major
>> performance
>> > > degradation of master code w.r.t 2-alpha-1 code.I am running write-only
>> > > workload (similar to the one reported in HBASE-16417). I am using the
>> > same
>> > > hardware and same configuration settings (specifically, I testes both
>> > basic
>> > > memstore compaction with optimal parameters, and no memsore
>> > > compaction).While in 2-alpha-1 code I see throughput of ~110Kops for
>> > basic
>> > > compaction and ~80Kops for no compaction, in the master code I get only
>> > > 60Kops and 55Kops, respectively. *This is almost 50% reduction in
>> > > performance*.
>> > > (1) Did anyone else noticed such degradation?(2) Do we have any
>> > systematic
>> > > automatic/semi-automatic method to track the sources of this
>> performance
>> > > issue?
>> > > Thanks,Eshcar
>> > >
>> >
>> >
>> > On #1, no. I've not done perf compare. I wonder if later alpha versions
>> > include the regression (I'll have to check and see).
>> >
>> > On #2, again no. I intend to do a bit of perf tuning and compare before
>> > release.
>> >
>> > If you don't file an issue, I will do so later for myself as a task to
>> > compare at least to alpha-1.
>> >
>> > Thanks Eshcar,
>> >
>> > St.Ack
>> >
>>
Re: [DISCUSS] Performance degradation in master (compared to
2-alpha-1)
Posted by Eshcar Hillel <es...@oath.com.INVALID>.
I'm currently using my cluster for other purposes.When it is available I can run the same experiment against alpha-3.
On Tuesday, November 28, 2017, 7:17:08 PM GMT+2, Mike Drob <md...@apache.org> wrote:
Eshcar - do you have time to try the other alpha releases and see where
exactly we introduced the regressions?
Also, I'm worried that the performance regression may be related to an
important bug-fix, where before we may have had fast writes but also risked
incorrect behavior somehow.
Mike
On Tue, Nov 28, 2017 at 2:48 AM, Eshcar Hillel <es...@oath.com.invalid>
wrote:
> I agree, so will wait till we focus on performance.
> Just one more update, I also ran the same experiment (write-only) with
> banch-2 beta-1.Here is a summary of the throughput I see in each tag/branch:
> ------------------------------- | BASIC | NONE |
> -------------------------------2-alpha-1| 110K | 80K |
> 2-beta-1 | 81K | 62K |
> master | 60K | 55K |-------------------------------
> This means there are multiple sources for the regression.
>
> Thanks
>
> On Saturday, November 25, 2017, 7:44:01 AM GMT+2, 张铎(Duo Zhang) <
> palomino219@gmail.com> wrote:
>
> I think first we need a release plan on when we will begin to focus on the
> performance issue?
>
> I do not think it is a good time to focus on performance issue now as we
> haven’t stabilized our build yet. The performance regression may come back
> again after some bug fixes and maybe we use a wrong way to increase
> performance and finally we find that it is just a bug...
>
> Of course I do not mean we can not do any performance related issues now,
> for example, HBASE-19338 is a good catch and can be fixed right now.
>
> And also, for AsyncFSWAL and in memory compaction, we need to consider the
> performance right now as they are born for performance, but let’s focus on
> the comparison to other policies, not a previous release so we can find the
> correct things to fix.
>
> Of course, if there is a big performance downgrading comparing to the
> previous release and we find it then we should tell others, just like this
> email. An earlier notification is always welcomed.
>
> Thanks.
>
> Stack <st...@duboce.net>于2017年11月25日 周六13:22写道：
>
> > On Thu, Nov 23, 2017 at 7:35 AM, Eshcar Hillel <es...@oath.com.invalid>
> > wrote:
> >
> > > Happy Thanksgiving all,
> > >
> >
> > And to you Eshcar.
> >
> >
> >
> > > In recent benchmarks I ran in HBASE-18294 I discovered major
> performance
> > > degradation of master code w.r.t 2-alpha-1 code.I am running write-only
> > > workload (similar to the one reported in HBASE-16417). I am using the
> > same
> > > hardware and same configuration settings (specifically, I testes both
> > basic
> > > memstore compaction with optimal parameters, and no memsore
> > > compaction).While in 2-alpha-1 code I see throughput of ~110Kops for
> > basic
> > > compaction and ~80Kops for no compaction, in the master code I get only
> > > 60Kops and 55Kops, respectively. *This is almost 50% reduction in
> > > performance*.
> > > (1) Did anyone else noticed such degradation?(2) Do we have any
> > systematic
> > > automatic/semi-automatic method to track the sources of this
> performance
> > > issue?
> > > Thanks,Eshcar
> > >
> >
> >
> > On #1, no. I've not done perf compare. I wonder if later alpha versions
> > include the regression (I'll have to check and see).
> >
> > On #2, again no. I intend to do a bit of perf tuning and compare before
> > release.
> >
> > If you don't file an issue, I will do so later for myself as a task to
> > compare at least to alpha-1.
> >
> > Thanks Eshcar,
> >
> > St.Ack
> >
>
Re: [DISCUSS] Performance degradation in master (compared to 2-alpha-1)
Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.
I do not mean this is the ONLY problem, and this may not be the problem in
your case, but I think it is worth to tell people about the possible reason
of performance downgrading right?
Thanks.
2017-11-29 11:07 GMT+08:00 Vladimir Rodionov <vl...@gmail.com>:
> >> CompletableFuture more and more in our code base, but at least before
> >>jdk8u131 there is a performance regression for CompletableFuture.
>
> The performance regression is 2x, so this should be something else
>
> On Tue, Nov 28, 2017 at 6:49 PM, 宾莉金（binlijin） <bi...@gmail.com> wrote:
>
> > HBASE-19338 has committed, do you want to update the master branch and
> test
> > it again?
> >
> > 2017-11-29 10:32 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
> >
> > > And one thing may effect performance is that, now we rely on
> > > CompletableFuture more and more in our code base, but at least before
> > > jdk8u131 there is a performance regression for CompletableFuture. So
> > > consider moving to the newest jdk if you are still on an older version.
> > >
> > > Thanks.
> > >
> > > 2017-11-29 3:35 GMT+08:00 Mike Drob <md...@apache.org>:
> > >
> > > > Ted,
> > > >
> > > > To clarify, I'm talking about a hypothetical bugfix that impacted
> > > > performance, not to imply that I know of a specific such change.
> > > >
> > > > I've seen it often enough before that performance is blazing fast at
> > the
> > > > expense of accuracy and people are surprised when correctness takes
> > > longer.
> > > >
> > > >
> > > > Mike
> > > >
> > > > On Tue, Nov 28, 2017 at 1:01 PM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > Mike:
> > > > > Which JIRA was the important bug-fix ?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Tue, Nov 28, 2017 at 9:16 AM, Mike Drob <md...@apache.org>
> wrote:
> > > > >
> > > > > > Eshcar - do you have time to try the other alpha releases and see
> > > where
> > > > > > exactly we introduced the regressions?
> > > > > >
> > > > > > Also, I'm worried that the performance regression may be related
> to
> > > an
> > > > > > important bug-fix, where before we may have had fast writes but
> > also
> > > > > risked
> > > > > > incorrect behavior somehow.
> > > > > >
> > > > > > Mike
> > > > > >
> > > > > > On Tue, Nov 28, 2017 at 2:48 AM, Eshcar Hillel
> > > <eshcar@oath.com.invalid
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I agree, so will wait till we focus on performance.
> > > > > > > Just one more update, I also ran the same experiment
> (write-only)
> > > > with
> > > > > > > banch-2 beta-1.Here is a summary of the throughput I see in
> each
> > > > > > tag/branch:
> > > > > > > ------------------------------- | BASIC | NONE |
> > > > > > > -------------------------------2-alpha-1| 110K | 80K |
> > > > > > > 2-beta-1 | 81K | 62K |
> > > > > > > master | 60K | 55K |-------------------------------
> > > > > > > This means there are multiple sources for the regression.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Saturday, November 25, 2017, 7:44:01 AM GMT+2, 张铎(Duo
> > > Zhang) <
> > > > > > > palomino219@gmail.com> wrote:
> > > > > > >
> > > > > > > I think first we need a release plan on when we will begin to
> > > focus
> > > > on
> > > > > > the
> > > > > > > performance issue?
> > > > > > >
> > > > > > > I do not think it is a good time to focus on performance issue
> > now
> > > as
> > > > > we
> > > > > > > haven’t stabilized our build yet. The performance regression
> may
> > > come
> > > > > > back
> > > > > > > again after some bug fixes and maybe we use a wrong way to
> > increase
> > > > > > > performance and finally we find that it is just a bug...
> > > > > > >
> > > > > > > Of course I do not mean we can not do any performance related
> > > issues
> > > > > now,
> > > > > > > for example, HBASE-19338 is a good catch and can be fixed right
> > > now.
> > > > > > >
> > > > > > > And also, for AsyncFSWAL and in memory compaction, we need to
> > > > consider
> > > > > > the
> > > > > > > performance right now as they are born for performance, but
> let’s
> > > > focus
> > > > > > on
> > > > > > > the comparison to other policies, not a previous release so we
> > can
> > > > find
> > > > > > the
> > > > > > > correct things to fix.
> > > > > > >
> > > > > > > Of course, if there is a big performance downgrading comparing
> to
> > > the
> > > > > > > previous release and we find it then we should tell others,
> just
> > > like
> > > > > > this
> > > > > > > email. An earlier notification is always welcomed.
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > Stack <st...@duboce.net>于2017年11月25日 周六13:22写道：
> > > > > > >
> > > > > > > > On Thu, Nov 23, 2017 at 7:35 AM, Eshcar Hillel
> > > > > <eshcar@oath.com.invalid
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Happy Thanksgiving all,
> > > > > > > > >
> > > > > > > >
> > > > > > > > And to you Eshcar.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > In recent benchmarks I ran in HBASE-18294 I discovered
> major
> > > > > > > performance
> > > > > > > > > degradation of master code w.r.t 2-alpha-1 code.I am
> running
> > > > > > write-only
> > > > > > > > > workload (similar to the one reported in HBASE-16417). I am
> > > using
> > > > > the
> > > > > > > > same
> > > > > > > > > hardware and same configuration settings (specifically, I
> > > testes
> > > > > both
> > > > > > > > basic
> > > > > > > > > memstore compaction with optimal parameters, and no memsore
> > > > > > > > > compaction).While in 2-alpha-1 code I see throughput of
> > > ~110Kops
> > > > > for
> > > > > > > > basic
> > > > > > > > > compaction and ~80Kops for no compaction, in the master
> code
> > I
> > > > get
> > > > > > only
> > > > > > > > > 60Kops and 55Kops, respectively. *This is almost 50%
> > reduction
> > > in
> > > > > > > > > performance*.
> > > > > > > > > (1) Did anyone else noticed such degradation?(2) Do we have
> > any
> > > > > > > > systematic
> > > > > > > > > automatic/semi-automatic method to track the sources of
> this
> > > > > > > performance
> > > > > > > > > issue?
> > > > > > > > > Thanks,Eshcar
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On #1, no. I've not done perf compare. I wonder if later
> alpha
> > > > > versions
> > > > > > > > include the regression (I'll have to check and see).
> > > > > > > >
> > > > > > > > On #2, again no. I intend to do a bit of perf tuning and
> > compare
> > > > > before
> > > > > > > > release.
> > > > > > > >
> > > > > > > > If you don't file an issue, I will do so later for myself as
> a
> > > task
> > > > > to
> > > > > > > > compare at least to alpha-1.
> > > > > > > >
> > > > > > > > Thanks Eshcar,
> > > > > > > >
> > > > > > > > St.Ack
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > *Best Regards,*
> > lijin bin
> >
>
Re: [DISCUSS] Performance degradation in master (compared to 2-alpha-1)
Posted by Vladimir Rodionov <vl...@gmail.com>.
>> CompletableFuture more and more in our code base, but at least before
>>jdk8u131 there is a performance regression for CompletableFuture.
The performance regression is 2x, so this should be something else
On Tue, Nov 28, 2017 at 6:49 PM, 宾莉金（binlijin） <bi...@gmail.com> wrote:
> HBASE-19338 has committed, do you want to update the master branch and test
> it again?
>
> 2017-11-29 10:32 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
>
> > And one thing may effect performance is that, now we rely on
> > CompletableFuture more and more in our code base, but at least before
> > jdk8u131 there is a performance regression for CompletableFuture. So
> > consider moving to the newest jdk if you are still on an older version.
> >
> > Thanks.
> >
> > 2017-11-29 3:35 GMT+08:00 Mike Drob <md...@apache.org>:
> >
> > > Ted,
> > >
> > > To clarify, I'm talking about a hypothetical bugfix that impacted
> > > performance, not to imply that I know of a specific such change.
> > >
> > > I've seen it often enough before that performance is blazing fast at
> the
> > > expense of accuracy and people are surprised when correctness takes
> > longer.
> > >
> > >
> > > Mike
> > >
> > > On Tue, Nov 28, 2017 at 1:01 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > Mike:
> > > > Which JIRA was the important bug-fix ?
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Nov 28, 2017 at 9:16 AM, Mike Drob <md...@apache.org> wrote:
> > > >
> > > > > Eshcar - do you have time to try the other alpha releases and see
> > where
> > > > > exactly we introduced the regressions?
> > > > >
> > > > > Also, I'm worried that the performance regression may be related to
> > an
> > > > > important bug-fix, where before we may have had fast writes but
> also
> > > > risked
> > > > > incorrect behavior somehow.
> > > > >
> > > > > Mike
> > > > >
> > > > > On Tue, Nov 28, 2017 at 2:48 AM, Eshcar Hillel
> > <eshcar@oath.com.invalid
> > > >
> > > > > wrote:
> > > > >
> > > > > > I agree, so will wait till we focus on performance.
> > > > > > Just one more update, I also ran the same experiment (write-only)
> > > with
> > > > > > banch-2 beta-1.Here is a summary of the throughput I see in each
> > > > > tag/branch:
> > > > > > ------------------------------- | BASIC | NONE |
> > > > > > -------------------------------2-alpha-1| 110K | 80K |
> > > > > > 2-beta-1 | 81K | 62K |
> > > > > > master | 60K | 55K |-------------------------------
> > > > > > This means there are multiple sources for the regression.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Saturday, November 25, 2017, 7:44:01 AM GMT+2, 张铎(Duo
> > Zhang) <
> > > > > > palomino219@gmail.com> wrote:
> > > > > >
> > > > > > I think first we need a release plan on when we will begin to
> > focus
> > > on
> > > > > the
> > > > > > performance issue?
> > > > > >
> > > > > > I do not think it is a good time to focus on performance issue
> now
> > as
> > > > we
> > > > > > haven’t stabilized our build yet. The performance regression may
> > come
> > > > > back
> > > > > > again after some bug fixes and maybe we use a wrong way to
> increase
> > > > > > performance and finally we find that it is just a bug...
> > > > > >
> > > > > > Of course I do not mean we can not do any performance related
> > issues
> > > > now,
> > > > > > for example, HBASE-19338 is a good catch and can be fixed right
> > now.
> > > > > >
> > > > > > And also, for AsyncFSWAL and in memory compaction, we need to
> > > consider
> > > > > the
> > > > > > performance right now as they are born for performance, but let’s
> > > focus
> > > > > on
> > > > > > the comparison to other policies, not a previous release so we
> can
> > > find
> > > > > the
> > > > > > correct things to fix.
> > > > > >
> > > > > > Of course, if there is a big performance downgrading comparing to
> > the
> > > > > > previous release and we find it then we should tell others, just
> > like
> > > > > this
> > > > > > email. An earlier notification is always welcomed.
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > Stack <st...@duboce.net>于2017年11月25日 周六13:22写道：
> > > > > >
> > > > > > > On Thu, Nov 23, 2017 at 7:35 AM, Eshcar Hillel
> > > > <eshcar@oath.com.invalid
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Happy Thanksgiving all,
> > > > > > > >
> > > > > > >
> > > > > > > And to you Eshcar.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > In recent benchmarks I ran in HBASE-18294 I discovered major
> > > > > > performance
> > > > > > > > degradation of master code w.r.t 2-alpha-1 code.I am running
> > > > > write-only
> > > > > > > > workload (similar to the one reported in HBASE-16417). I am
> > using
> > > > the
> > > > > > > same
> > > > > > > > hardware and same configuration settings (specifically, I
> > testes
> > > > both
> > > > > > > basic
> > > > > > > > memstore compaction with optimal parameters, and no memsore
> > > > > > > > compaction).While in 2-alpha-1 code I see throughput of
> > ~110Kops
> > > > for
> > > > > > > basic
> > > > > > > > compaction and ~80Kops for no compaction, in the master code
> I
> > > get
> > > > > only
> > > > > > > > 60Kops and 55Kops, respectively. *This is almost 50%
> reduction
> > in
> > > > > > > > performance*.
> > > > > > > > (1) Did anyone else noticed such degradation?(2) Do we have
> any
> > > > > > > systematic
> > > > > > > > automatic/semi-automatic method to track the sources of this
> > > > > > performance
> > > > > > > > issue?
> > > > > > > > Thanks,Eshcar
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On #1, no. I've not done perf compare. I wonder if later alpha
> > > > versions
> > > > > > > include the regression (I'll have to check and see).
> > > > > > >
> > > > > > > On #2, again no. I intend to do a bit of perf tuning and
> compare
> > > > before
> > > > > > > release.
> > > > > > >
> > > > > > > If you don't file an issue, I will do so later for myself as a
> > task
> > > > to
> > > > > > > compare at least to alpha-1.
> > > > > > >
> > > > > > > Thanks Eshcar,
> > > > > > >
> > > > > > > St.Ack
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> *Best Regards,*
> lijin bin
>
Re: [DISCUSS] Performance degradation in master (compared to 2-alpha-1)
Posted by 宾莉金（binlijin） <bi...@gmail.com>.
HBASE-19338 has committed, do you want to update the master branch and test
it again?
2017-11-29 10:32 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
> And one thing may effect performance is that, now we rely on
> CompletableFuture more and more in our code base, but at least before
> jdk8u131 there is a performance regression for CompletableFuture. So
> consider moving to the newest jdk if you are still on an older version.
>
> Thanks.
>
> 2017-11-29 3:35 GMT+08:00 Mike Drob <md...@apache.org>:
>
> > Ted,
> >
> > To clarify, I'm talking about a hypothetical bugfix that impacted
> > performance, not to imply that I know of a specific such change.
> >
> > I've seen it often enough before that performance is blazing fast at the
> > expense of accuracy and people are surprised when correctness takes
> longer.
> >
> >
> > Mike
> >
> > On Tue, Nov 28, 2017 at 1:01 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Mike:
> > > Which JIRA was the important bug-fix ?
> > >
> > > Thanks
> > >
> > > On Tue, Nov 28, 2017 at 9:16 AM, Mike Drob <md...@apache.org> wrote:
> > >
> > > > Eshcar - do you have time to try the other alpha releases and see
> where
> > > > exactly we introduced the regressions?
> > > >
> > > > Also, I'm worried that the performance regression may be related to
> an
> > > > important bug-fix, where before we may have had fast writes but also
> > > risked
> > > > incorrect behavior somehow.
> > > >
> > > > Mike
> > > >
> > > > On Tue, Nov 28, 2017 at 2:48 AM, Eshcar Hillel
> <eshcar@oath.com.invalid
> > >
> > > > wrote:
> > > >
> > > > > I agree, so will wait till we focus on performance.
> > > > > Just one more update, I also ran the same experiment (write-only)
> > with
> > > > > banch-2 beta-1.Here is a summary of the throughput I see in each
> > > > tag/branch:
> > > > > ------------------------------- | BASIC | NONE |
> > > > > -------------------------------2-alpha-1| 110K | 80K |
> > > > > 2-beta-1 | 81K | 62K |
> > > > > master | 60K | 55K |-------------------------------
> > > > > This means there are multiple sources for the regression.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Saturday, November 25, 2017, 7:44:01 AM GMT+2, 张铎(Duo
> Zhang) <
> > > > > palomino219@gmail.com> wrote:
> > > > >
> > > > > I think first we need a release plan on when we will begin to
> focus
> > on
> > > > the
> > > > > performance issue?
> > > > >
> > > > > I do not think it is a good time to focus on performance issue now
> as
> > > we
> > > > > haven’t stabilized our build yet. The performance regression may
> come
> > > > back
> > > > > again after some bug fixes and maybe we use a wrong way to increase
> > > > > performance and finally we find that it is just a bug...
> > > > >
> > > > > Of course I do not mean we can not do any performance related
> issues
> > > now,
> > > > > for example, HBASE-19338 is a good catch and can be fixed right
> now.
> > > > >
> > > > > And also, for AsyncFSWAL and in memory compaction, we need to
> > consider
> > > > the
> > > > > performance right now as they are born for performance, but let’s
> > focus
> > > > on
> > > > > the comparison to other policies, not a previous release so we can
> > find
> > > > the
> > > > > correct things to fix.
> > > > >
> > > > > Of course, if there is a big performance downgrading comparing to
> the
> > > > > previous release and we find it then we should tell others, just
> like
> > > > this
> > > > > email. An earlier notification is always welcomed.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > Stack <st...@duboce.net>于2017年11月25日 周六13:22写道：
> > > > >
> > > > > > On Thu, Nov 23, 2017 at 7:35 AM, Eshcar Hillel
> > > <eshcar@oath.com.invalid
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Happy Thanksgiving all,
> > > > > > >
> > > > > >
> > > > > > And to you Eshcar.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > In recent benchmarks I ran in HBASE-18294 I discovered major
> > > > > performance
> > > > > > > degradation of master code w.r.t 2-alpha-1 code.I am running
> > > > write-only
> > > > > > > workload (similar to the one reported in HBASE-16417). I am
> using
> > > the
> > > > > > same
> > > > > > > hardware and same configuration settings (specifically, I
> testes
> > > both
> > > > > > basic
> > > > > > > memstore compaction with optimal parameters, and no memsore
> > > > > > > compaction).While in 2-alpha-1 code I see throughput of
> ~110Kops
> > > for
> > > > > > basic
> > > > > > > compaction and ~80Kops for no compaction, in the master code I
> > get
> > > > only
> > > > > > > 60Kops and 55Kops, respectively. *This is almost 50% reduction
> in
> > > > > > > performance*.
> > > > > > > (1) Did anyone else noticed such degradation?(2) Do we have any
> > > > > > systematic
> > > > > > > automatic/semi-automatic method to track the sources of this
> > > > > performance
> > > > > > > issue?
> > > > > > > Thanks,Eshcar
> > > > > > >
> > > > > >
> > > > > >
> > > > > > On #1, no. I've not done perf compare. I wonder if later alpha
> > > versions
> > > > > > include the regression (I'll have to check and see).
> > > > > >
> > > > > > On #2, again no. I intend to do a bit of perf tuning and compare
> > > before
> > > > > > release.
> > > > > >
> > > > > > If you don't file an issue, I will do so later for myself as a
> task
> > > to
> > > > > > compare at least to alpha-1.
> > > > > >
> > > > > > Thanks Eshcar,
> > > > > >
> > > > > > St.Ack
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
*Best Regards,*
lijin bin
Re: [DISCUSS] Performance degradation in master (compared to 2-alpha-1)
Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.
And one thing may effect performance is that, now we rely on
CompletableFuture more and more in our code base, but at least before
jdk8u131 there is a performance regression for CompletableFuture. So
consider moving to the newest jdk if you are still on an older version.
Thanks.
2017-11-29 3:35 GMT+08:00 Mike Drob <md...@apache.org>:
> Ted,
>
> To clarify, I'm talking about a hypothetical bugfix that impacted
> performance, not to imply that I know of a specific such change.
>
> I've seen it often enough before that performance is blazing fast at the
> expense of accuracy and people are surprised when correctness takes longer.
>
>
> Mike
>
> On Tue, Nov 28, 2017 at 1:01 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Mike:
> > Which JIRA was the important bug-fix ?
> >
> > Thanks
> >
> > On Tue, Nov 28, 2017 at 9:16 AM, Mike Drob <md...@apache.org> wrote:
> >
> > > Eshcar - do you have time to try the other alpha releases and see where
> > > exactly we introduced the regressions?
> > >
> > > Also, I'm worried that the performance regression may be related to an
> > > important bug-fix, where before we may have had fast writes but also
> > risked
> > > incorrect behavior somehow.
> > >
> > > Mike
> > >
> > > On Tue, Nov 28, 2017 at 2:48 AM, Eshcar Hillel <eshcar@oath.com.invalid
> >
> > > wrote:
> > >
> > > > I agree, so will wait till we focus on performance.
> > > > Just one more update, I also ran the same experiment (write-only)
> with
> > > > banch-2 beta-1.Here is a summary of the throughput I see in each
> > > tag/branch:
> > > > ------------------------------- | BASIC | NONE |
> > > > -------------------------------2-alpha-1| 110K | 80K |
> > > > 2-beta-1 | 81K | 62K |
> > > > master | 60K | 55K |-------------------------------
> > > > This means there are multiple sources for the regression.
> > > >
> > > > Thanks
> > > >
> > > > On Saturday, November 25, 2017, 7:44:01 AM GMT+2, 张铎(Duo Zhang) <
> > > > palomino219@gmail.com> wrote:
> > > >
> > > > I think first we need a release plan on when we will begin to focus
> on
> > > the
> > > > performance issue?
> > > >
> > > > I do not think it is a good time to focus on performance issue now as
> > we
> > > > haven’t stabilized our build yet. The performance regression may come
> > > back
> > > > again after some bug fixes and maybe we use a wrong way to increase
> > > > performance and finally we find that it is just a bug...
> > > >
> > > > Of course I do not mean we can not do any performance related issues
> > now,
> > > > for example, HBASE-19338 is a good catch and can be fixed right now.
> > > >
> > > > And also, for AsyncFSWAL and in memory compaction, we need to
> consider
> > > the
> > > > performance right now as they are born for performance, but let’s
> focus
> > > on
> > > > the comparison to other policies, not a previous release so we can
> find
> > > the
> > > > correct things to fix.
> > > >
> > > > Of course, if there is a big performance downgrading comparing to the
> > > > previous release and we find it then we should tell others, just like
> > > this
> > > > email. An earlier notification is always welcomed.
> > > >
> > > > Thanks.
> > > >
> > > > Stack <st...@duboce.net>于2017年11月25日 周六13:22写道：
> > > >
> > > > > On Thu, Nov 23, 2017 at 7:35 AM, Eshcar Hillel
> > <eshcar@oath.com.invalid
> > > >
> > > > > wrote:
> > > > >
> > > > > > Happy Thanksgiving all,
> > > > > >
> > > > >
> > > > > And to you Eshcar.
> > > > >
> > > > >
> > > > >
> > > > > > In recent benchmarks I ran in HBASE-18294 I discovered major
> > > > performance
> > > > > > degradation of master code w.r.t 2-alpha-1 code.I am running
> > > write-only
> > > > > > workload (similar to the one reported in HBASE-16417). I am using
> > the
> > > > > same
> > > > > > hardware and same configuration settings (specifically, I testes
> > both
> > > > > basic
> > > > > > memstore compaction with optimal parameters, and no memsore
> > > > > > compaction).While in 2-alpha-1 code I see throughput of ~110Kops
> > for
> > > > > basic
> > > > > > compaction and ~80Kops for no compaction, in the master code I
> get
> > > only
> > > > > > 60Kops and 55Kops, respectively. *This is almost 50% reduction in
> > > > > > performance*.
> > > > > > (1) Did anyone else noticed such degradation?(2) Do we have any
> > > > > systematic
> > > > > > automatic/semi-automatic method to track the sources of this
> > > > performance
> > > > > > issue?
> > > > > > Thanks,Eshcar
> > > > > >
> > > > >
> > > > >
> > > > > On #1, no. I've not done perf compare. I wonder if later alpha
> > versions
> > > > > include the regression (I'll have to check and see).
> > > > >
> > > > > On #2, again no. I intend to do a bit of perf tuning and compare
> > before
> > > > > release.
> > > > >
> > > > > If you don't file an issue, I will do so later for myself as a task
> > to
> > > > > compare at least to alpha-1.
> > > > >
> > > > > Thanks Eshcar,
> > > > >
> > > > > St.Ack
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Performance degradation in master (compared to 2-alpha-1)
Posted by Mike Drob <md...@apache.org>.
Ted,
To clarify, I'm talking about a hypothetical bugfix that impacted
performance, not to imply that I know of a specific such change.
I've seen it often enough before that performance is blazing fast at the
expense of accuracy and people are surprised when correctness takes longer.
Mike
On Tue, Nov 28, 2017 at 1:01 PM, Ted Yu <yu...@gmail.com> wrote:
> Mike:
> Which JIRA was the important bug-fix ?
>
> Thanks
>
> On Tue, Nov 28, 2017 at 9:16 AM, Mike Drob <md...@apache.org> wrote:
>
> > Eshcar - do you have time to try the other alpha releases and see where
> > exactly we introduced the regressions?
> >
> > Also, I'm worried that the performance regression may be related to an
> > important bug-fix, where before we may have had fast writes but also
> risked
> > incorrect behavior somehow.
> >
> > Mike
> >
> > On Tue, Nov 28, 2017 at 2:48 AM, Eshcar Hillel <es...@oath.com.invalid>
> > wrote:
> >
> > > I agree, so will wait till we focus on performance.
> > > Just one more update, I also ran the same experiment (write-only) with
> > > banch-2 beta-1.Here is a summary of the throughput I see in each
> > tag/branch:
> > > ------------------------------- | BASIC | NONE |
> > > -------------------------------2-alpha-1| 110K | 80K |
> > > 2-beta-1 | 81K | 62K |
> > > master | 60K | 55K |-------------------------------
> > > This means there are multiple sources for the regression.
> > >
> > > Thanks
> > >
> > > On Saturday, November 25, 2017, 7:44:01 AM GMT+2, 张铎(Duo Zhang) <
> > > palomino219@gmail.com> wrote:
> > >
> > > I think first we need a release plan on when we will begin to focus on
> > the
> > > performance issue?
> > >
> > > I do not think it is a good time to focus on performance issue now as
> we
> > > haven’t stabilized our build yet. The performance regression may come
> > back
> > > again after some bug fixes and maybe we use a wrong way to increase
> > > performance and finally we find that it is just a bug...
> > >
> > > Of course I do not mean we can not do any performance related issues
> now,
> > > for example, HBASE-19338 is a good catch and can be fixed right now.
> > >
> > > And also, for AsyncFSWAL and in memory compaction, we need to consider
> > the
> > > performance right now as they are born for performance, but let’s focus
> > on
> > > the comparison to other policies, not a previous release so we can find
> > the
> > > correct things to fix.
> > >
> > > Of course, if there is a big performance downgrading comparing to the
> > > previous release and we find it then we should tell others, just like
> > this
> > > email. An earlier notification is always welcomed.
> > >
> > > Thanks.
> > >
> > > Stack <st...@duboce.net>于2017年11月25日 周六13:22写道：
> > >
> > > > On Thu, Nov 23, 2017 at 7:35 AM, Eshcar Hillel
> <eshcar@oath.com.invalid
> > >
> > > > wrote:
> > > >
> > > > > Happy Thanksgiving all,
> > > > >
> > > >
> > > > And to you Eshcar.
> > > >
> > > >
> > > >
> > > > > In recent benchmarks I ran in HBASE-18294 I discovered major
> > > performance
> > > > > degradation of master code w.r.t 2-alpha-1 code.I am running
> > write-only
> > > > > workload (similar to the one reported in HBASE-16417). I am using
> the
> > > > same
> > > > > hardware and same configuration settings (specifically, I testes
> both
> > > > basic
> > > > > memstore compaction with optimal parameters, and no memsore
> > > > > compaction).While in 2-alpha-1 code I see throughput of ~110Kops
> for
> > > > basic
> > > > > compaction and ~80Kops for no compaction, in the master code I get
> > only
> > > > > 60Kops and 55Kops, respectively. *This is almost 50% reduction in
> > > > > performance*.
> > > > > (1) Did anyone else noticed such degradation?(2) Do we have any
> > > > systematic
> > > > > automatic/semi-automatic method to track the sources of this
> > > performance
> > > > > issue?
> > > > > Thanks,Eshcar
> > > > >
> > > >
> > > >
> > > > On #1, no. I've not done perf compare. I wonder if later alpha
> versions
> > > > include the regression (I'll have to check and see).
> > > >
> > > > On #2, again no. I intend to do a bit of perf tuning and compare
> before
> > > > release.
> > > >
> > > > If you don't file an issue, I will do so later for myself as a task
> to
> > > > compare at least to alpha-1.
> > > >
> > > > Thanks Eshcar,
> > > >
> > > > St.Ack
> > > >
> > >
> >
>
Re: [DISCUSS] Performance degradation in master (compared to 2-alpha-1)
Posted by Ted Yu <yu...@gmail.com>.
Mike:
Which JIRA was the important bug-fix ?
Thanks
On Tue, Nov 28, 2017 at 9:16 AM, Mike Drob <md...@apache.org> wrote:
> Eshcar - do you have time to try the other alpha releases and see where
> exactly we introduced the regressions?
>
> Also, I'm worried that the performance regression may be related to an
> important bug-fix, where before we may have had fast writes but also risked
> incorrect behavior somehow.
>
> Mike
>
> On Tue, Nov 28, 2017 at 2:48 AM, Eshcar Hillel <es...@oath.com.invalid>
> wrote:
>
> > I agree, so will wait till we focus on performance.
> > Just one more update, I also ran the same experiment (write-only) with
> > banch-2 beta-1.Here is a summary of the throughput I see in each
> tag/branch:
> > ------------------------------- | BASIC | NONE |
> > -------------------------------2-alpha-1| 110K | 80K |
> > 2-beta-1 | 81K | 62K |
> > master | 60K | 55K |-------------------------------
> > This means there are multiple sources for the regression.
> >
> > Thanks
> >
> > On Saturday, November 25, 2017, 7:44:01 AM GMT+2, 张铎(Duo Zhang) <
> > palomino219@gmail.com> wrote:
> >
> > I think first we need a release plan on when we will begin to focus on
> the
> > performance issue?
> >
> > I do not think it is a good time to focus on performance issue now as we
> > haven’t stabilized our build yet. The performance regression may come
> back
> > again after some bug fixes and maybe we use a wrong way to increase
> > performance and finally we find that it is just a bug...
> >
> > Of course I do not mean we can not do any performance related issues now,
> > for example, HBASE-19338 is a good catch and can be fixed right now.
> >
> > And also, for AsyncFSWAL and in memory compaction, we need to consider
> the
> > performance right now as they are born for performance, but let’s focus
> on
> > the comparison to other policies, not a previous release so we can find
> the
> > correct things to fix.
> >
> > Of course, if there is a big performance downgrading comparing to the
> > previous release and we find it then we should tell others, just like
> this
> > email. An earlier notification is always welcomed.
> >
> > Thanks.
> >
> > Stack <st...@duboce.net>于2017年11月25日 周六13:22写道：
> >
> > > On Thu, Nov 23, 2017 at 7:35 AM, Eshcar Hillel <eshcar@oath.com.invalid
> >
> > > wrote:
> > >
> > > > Happy Thanksgiving all,
> > > >
> > >
> > > And to you Eshcar.
> > >
> > >
> > >
> > > > In recent benchmarks I ran in HBASE-18294 I discovered major
> > performance
> > > > degradation of master code w.r.t 2-alpha-1 code.I am running
> write-only
> > > > workload (similar to the one reported in HBASE-16417). I am using the
> > > same
> > > > hardware and same configuration settings (specifically, I testes both
> > > basic
> > > > memstore compaction with optimal parameters, and no memsore
> > > > compaction).While in 2-alpha-1 code I see throughput of ~110Kops for
> > > basic
> > > > compaction and ~80Kops for no compaction, in the master code I get
> only
> > > > 60Kops and 55Kops, respectively. *This is almost 50% reduction in
> > > > performance*.
> > > > (1) Did anyone else noticed such degradation?(2) Do we have any
> > > systematic
> > > > automatic/semi-automatic method to track the sources of this
> > performance
> > > > issue?
> > > > Thanks,Eshcar
> > > >
> > >
> > >
> > > On #1, no. I've not done perf compare. I wonder if later alpha versions
> > > include the regression (I'll have to check and see).
> > >
> > > On #2, again no. I intend to do a bit of perf tuning and compare before
> > > release.
> > >
> > > If you don't file an issue, I will do so later for myself as a task to
> > > compare at least to alpha-1.
> > >
> > > Thanks Eshcar,
> > >
> > > St.Ack
> > >
> >
>
Re: [DISCUSS] Performance degradation in master (compared to 2-alpha-1)
Posted by Mike Drob <md...@apache.org>.
Eshcar - do you have time to try the other alpha releases and see where
exactly we introduced the regressions?
Also, I'm worried that the performance regression may be related to an
important bug-fix, where before we may have had fast writes but also risked
incorrect behavior somehow.
Mike
On Tue, Nov 28, 2017 at 2:48 AM, Eshcar Hillel <es...@oath.com.invalid>
wrote:
> I agree, so will wait till we focus on performance.
> Just one more update, I also ran the same experiment (write-only) with
> banch-2 beta-1.Here is a summary of the throughput I see in each tag/branch:
> ------------------------------- | BASIC | NONE |
> -------------------------------2-alpha-1| 110K | 80K |
> 2-beta-1 | 81K | 62K |
> master | 60K | 55K |-------------------------------
> This means there are multiple sources for the regression.
>
> Thanks
>
> On Saturday, November 25, 2017, 7:44:01 AM GMT+2, 张铎(Duo Zhang) <
> palomino219@gmail.com> wrote:
>
> I think first we need a release plan on when we will begin to focus on the
> performance issue?
>
> I do not think it is a good time to focus on performance issue now as we
> haven’t stabilized our build yet. The performance regression may come back
> again after some bug fixes and maybe we use a wrong way to increase
> performance and finally we find that it is just a bug...
>
> Of course I do not mean we can not do any performance related issues now,
> for example, HBASE-19338 is a good catch and can be fixed right now.
>
> And also, for AsyncFSWAL and in memory compaction, we need to consider the
> performance right now as they are born for performance, but let’s focus on
> the comparison to other policies, not a previous release so we can find the
> correct things to fix.
>
> Of course, if there is a big performance downgrading comparing to the
> previous release and we find it then we should tell others, just like this
> email. An earlier notification is always welcomed.
>
> Thanks.
>
> Stack <st...@duboce.net>于2017年11月25日 周六13:22写道：
>
> > On Thu, Nov 23, 2017 at 7:35 AM, Eshcar Hillel <es...@oath.com.invalid>
> > wrote:
> >
> > > Happy Thanksgiving all,
> > >
> >
> > And to you Eshcar.
> >
> >
> >
> > > In recent benchmarks I ran in HBASE-18294 I discovered major
> performance
> > > degradation of master code w.r.t 2-alpha-1 code.I am running write-only
> > > workload (similar to the one reported in HBASE-16417). I am using the
> > same
> > > hardware and same configuration settings (specifically, I testes both
> > basic
> > > memstore compaction with optimal parameters, and no memsore
> > > compaction).While in 2-alpha-1 code I see throughput of ~110Kops for
> > basic
> > > compaction and ~80Kops for no compaction, in the master code I get only
> > > 60Kops and 55Kops, respectively. *This is almost 50% reduction in
> > > performance*.
> > > (1) Did anyone else noticed such degradation?(2) Do we have any
> > systematic
> > > automatic/semi-automatic method to track the sources of this
> performance
> > > issue?
> > > Thanks,Eshcar
> > >
> >
> >
> > On #1, no. I've not done perf compare. I wonder if later alpha versions
> > include the regression (I'll have to check and see).
> >
> > On #2, again no. I intend to do a bit of perf tuning and compare before
> > release.
> >
> > If you don't file an issue, I will do so later for myself as a task to
> > compare at least to alpha-1.
> >
> > Thanks Eshcar,
> >
> > St.Ack
> >
>
Re: [DISCUSS] Performance degradation in master (compared to
2-alpha-1)
Posted by Eshcar Hillel <es...@oath.com.INVALID>.
I agree, so will wait till we focus on performance.
Just one more update, I also ran the same experiment (write-only) with banch-2 beta-1.Here is a summary of the throughput I see in each tag/branch:
------------------------------- | BASIC | NONE |
-------------------------------2-alpha-1| 110K | 80K |
2-beta-1 | 81K | 62K |
master | 60K | 55K |-------------------------------
This means there are multiple sources for the regression.
Thanks
On Saturday, November 25, 2017, 7:44:01 AM GMT+2, 张铎(Duo Zhang) <pa...@gmail.com> wrote:
I think first we need a release plan on when we will begin to focus on the
performance issue?
I do not think it is a good time to focus on performance issue now as we
haven’t stabilized our build yet. The performance regression may come back
again after some bug fixes and maybe we use a wrong way to increase
performance and finally we find that it is just a bug...
Of course I do not mean we can not do any performance related issues now,
for example, HBASE-19338 is a good catch and can be fixed right now.
And also, for AsyncFSWAL and in memory compaction, we need to consider the
performance right now as they are born for performance, but let’s focus on
the comparison to other policies, not a previous release so we can find the
correct things to fix.
Of course, if there is a big performance downgrading comparing to the
previous release and we find it then we should tell others, just like this
email. An earlier notification is always welcomed.
Thanks.
Stack <st...@duboce.net>于2017年11月25日 周六13:22写道：
> On Thu, Nov 23, 2017 at 7:35 AM, Eshcar Hillel <es...@oath.com.invalid>
> wrote:
>
> > Happy Thanksgiving all,
> >
>
> And to you Eshcar.
>
>
>
> > In recent benchmarks I ran in HBASE-18294 I discovered major performance
> > degradation of master code w.r.t 2-alpha-1 code.I am running write-only
> > workload (similar to the one reported in HBASE-16417). I am using the
> same
> > hardware and same configuration settings (specifically, I testes both
> basic
> > memstore compaction with optimal parameters, and no memsore
> > compaction).While in 2-alpha-1 code I see throughput of ~110Kops for
> basic
> > compaction and ~80Kops for no compaction, in the master code I get only
> > 60Kops and 55Kops, respectively. *This is almost 50% reduction in
> > performance*.
> > (1) Did anyone else noticed such degradation?(2) Do we have any
> systematic
> > automatic/semi-automatic method to track the sources of this performance
> > issue?
> > Thanks,Eshcar
> >
>
>
> On #1, no. I've not done perf compare. I wonder if later alpha versions
> include the regression (I'll have to check and see).
>
> On #2, again no. I intend to do a bit of perf tuning and compare before
> release.
>
> If you don't file an issue, I will do so later for myself as a task to
> compare at least to alpha-1.
>
> Thanks Eshcar,
>
> St.Ack
>
Re: [DISCUSS] Performance degradation in master (compared to 2-alpha-1)
Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.
I think first we need a release plan on when we will begin to focus on the
performance issue?
I do not think it is a good time to focus on performance issue now as we
haven’t stabilized our build yet. The performance regression may come back
again after some bug fixes and maybe we use a wrong way to increase
performance and finally we find that it is just a bug...
Of course I do not mean we can not do any performance related issues now,
for example, HBASE-19338 is a good catch and can be fixed right now.
And also, for AsyncFSWAL and in memory compaction, we need to consider the
performance right now as they are born for performance, but let’s focus on
the comparison to other policies, not a previous release so we can find the
correct things to fix.
Of course, if there is a big performance downgrading comparing to the
previous release and we find it then we should tell others, just like this
email. An earlier notification is always welcomed.
Thanks.
Stack <st...@duboce.net>于2017年11月25日 周六13:22写道：
> On Thu, Nov 23, 2017 at 7:35 AM, Eshcar Hillel <es...@oath.com.invalid>
> wrote:
>
> > Happy Thanksgiving all,
> >
>
> And to you Eshcar.
>
>
>
> > In recent benchmarks I ran in HBASE-18294 I discovered major performance
> > degradation of master code w.r.t 2-alpha-1 code.I am running write-only
> > workload (similar to the one reported in HBASE-16417). I am using the
> same
> > hardware and same configuration settings (specifically, I testes both
> basic
> > memstore compaction with optimal parameters, and no memsore
> > compaction).While in 2-alpha-1 code I see throughput of ~110Kops for
> basic
> > compaction and ~80Kops for no compaction, in the master code I get only
> > 60Kops and 55Kops, respectively. *This is almost 50% reduction in
> > performance*.
> > (1) Did anyone else noticed such degradation?(2) Do we have any
> systematic
> > automatic/semi-automatic method to track the sources of this performance
> > issue?
> > Thanks,Eshcar
> >
>
>
> On #1, no. I've not done perf compare. I wonder if later alpha versions
> include the regression (I'll have to check and see).
>
> On #2, again no. I intend to do a bit of perf tuning and compare before
> release.
>
> If you don't file an issue, I will do so later for myself as a task to
> compare at least to alpha-1.
>
> Thanks Eshcar,
>
> St.Ack
>
Re: [DISCUSS] Performance degradation in master (compared to 2-alpha-1)
Posted by Stack <st...@duboce.net>.
On Thu, Nov 23, 2017 at 7:35 AM, Eshcar Hillel <es...@oath.com.invalid>
wrote:
> Happy Thanksgiving all,
>
And to you Eshcar.
> In recent benchmarks I ran in HBASE-18294 I discovered major performance
> degradation of master code w.r.t 2-alpha-1 code.I am running write-only
> workload (similar to the one reported in HBASE-16417). I am using the same
> hardware and same configuration settings (specifically, I testes both basic
> memstore compaction with optimal parameters, and no memsore
> compaction).While in 2-alpha-1 code I see throughput of ~110Kops for basic
> compaction and ~80Kops for no compaction, in the master code I get only
> 60Kops and 55Kops, respectively. *This is almost 50% reduction in
> performance*.
> (1) Did anyone else noticed such degradation?(2) Do we have any systematic
> automatic/semi-automatic method to track the sources of this performance
> issue?
> Thanks,Eshcar
>
On #1, no. I've not done perf compare. I wonder if later alpha versions
include the regression (I'll have to check and see).
On #2, again no. I intend to do a bit of perf tuning and compare before
release.
If you don't file an issue, I will do so later for myself as a task to
compare at least to alpha-1.
Thanks Eshcar,
St.Ack