You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Ted Yu <yu...@gmail.com> on 2016/03/03 18:02:51 UTC
[DISCUSS] Criteria for including MOB feature backport in branch-1
Hi,
As requested by Sean Busbey, I am starting a new thread w.r.t. backporting
MOB feature to branch-1.
This is to solicit discussion on the criteria for including MOB feature
backport in branch-1.
I can think of the following:
1. whether there is customer interest
There is.
See Jonathan's response here: http://search-hadoop.com/m/YGbbDSqxD1PYXK62
2. whether unit test stability can be maintained in branch-1
Inclusion of the backport should keep unit tests (both existing and new) in
branch-1 green.
Preliminary test runs showed that MOB / snapshot related tests consistently
pass.
https://issues.apache.org/jira/browse/HBASE-15370?focusedCommentId=15176094&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15176094
Comments are welcome
<https://issues.apache.org/jira/browse/HBASE-15370#>
Re: [DISCUSS] Criteria for including MOB feature backport in
branch-1
Posted by Devaraj Das <dd...@hortonworks.com>.
Thanks for chiming in Jincheng. Really appreciated. But from this thread, it's apparent that people don't have enough enthusiasm to see this backported in branch-1. So, maybe what would be really nice to instead do is to make sure that MOB is performant / reliable / releasable in the HBASE-2.0 release. As suggested in this thread, let's run YCSB / ITBLL and other such tests to make sure we have enough confidence on this before HBASE-2.0 goes out. Of course, Unit Tests is another area in the master branch that MOB is still flaky in....
________________________________________
From: Du, Jingcheng <ji...@intel.com>
Sent: Thursday, March 03, 2016 9:55 PM
To: dev@hbase.apache.org
Subject: RE: [DISCUSS] Criteria for including MOB feature backport in branch-1
Sorry for the late chime in and thank you all for the discussion.
I agree with Jon's suggestion, to make a new branch for MOB in branch-1.x, and backport it when it is definitely ready.
I can start to work on that branch after the distributed mob compaction is finished.
I think the distributed mob compaction is not a blocker. The old mob compaction can be disabled by configuration, this would not impact the read/write performance of mob files. Users can count on the TTL cleaner to remove the expired mob files which is very light weighted.
And I will keep eyes on the UT results, and get ready to fix the coming up failures. Thanks.
Regards,
Jingcheng
-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, March 4, 2016 9:53 AM
To: dev@hbase.apache.org
Subject: Re: [DISCUSS] Criteria for including MOB feature backport in branch-1
Good questions, Enis.
If my bandwidth permits, I am planning to collect performance statistics using ycsb against cluster with and without MOB feature enabled.
Cheers
On Thu, Mar 3, 2016 at 3:49 PM, Enis Söztutar <en...@gmail.com> wrote:
> Regardless of the backport, did we do the same regression analysis for
> the master branch merge at the time of the merge? Like make sure that
> the latencies and stability of non-mob is affected or not. Sorry, I
> was not following the merge vote closely.
>
> The reason I am asking is that although we can allow some flexibility
> in master, we still want to keep master releasable and prevent a
> 1-year effort to re-stabilize master when we want to release 2.0. If
> we have design / stability / impact questions, when and how they will
> be addressed for a 2.0 release? Let's say that we would like to
> release 2.0 by summer time as Matteo was quoting, how do we make sure
> that the feature as is, is stable and supportable?
>
> Enis
>
> On Thu, Mar 3, 2016 at 3:36 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > Just to be clear, I did not actually oppose a backport of MOB.
> > Although
> - I
> > have to say am sympathetic to Elliott's position that there's a
> > project management reason not to. My insistence here is "merely" for
> > this change
> in
> > particular a backport to the branch we are making production
> > releases
> from
> > demands a credible data driven assessment on stability,
> > functionality,
> and
> > performance - both avoidance of regression and affirmative
> > indication of the alleged benefit.
> >
> > On Thu, Mar 3, 2016 at 3:16 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > In the middle of writing response to this thread, I saw subsequent
> > comments
> > > from Andrew, Jon and Elliott.
> > >
> > > In light of opposition from the community w.r.t. backporting MOB
> > feature, I
> > > think it suffices to say that this wouldn't be done now.
> > >
> > > Thanks everyone for chiming in.
> > >
> > > On Thu, Mar 3, 2016 at 2:42 PM, Elliott Clark <ec...@apache.org>
> wrote:
> > >
> > > > I just don't see a why we would back port. We're going to
> > > > release a
> 2.0
> > > > when things are ready. It will be a major feature release. MOB
> > > > is a
> > major
> > > > feature. Without compelling reason backporting to branch-1 seems
> > > > like
> > an
> > > > end run around what sem ver is supposed to mean (not the api
> > guarantees,
> > > > the actual meaning).
> > > >
> > > > Backporting major features is a bad habit that the Hadoop
> > > > community
> > seems
> > > > to have. We shouldn't follow their lead.
> > > >
> > > > On Thu, Mar 3, 2016 at 1:01 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > On Thu, Mar 3, 2016 at 9:02 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi,
> > > > > > As requested by Sean Busbey, I am starting a new thread w.r.t.
> > > > > backporting
> > > > > > MOB feature to branch-1.
> > > > > >
> > > > > This is to solicit discussion on the criteria for including
> > > > > MOB
> > feature
> > > > > > backport in branch-1.
> > > > > >
> > > > > > I can think of the following:
> > > > > > 1. whether there is customer interest
> > > > > >
> > > > > > There is.
> > > > > > See Jonathan's response here:
> > > > > http://search-hadoop.com/m/YGbbDSqxD1PYXK62
> > > > > >
> > > > > >
> > > > > You should probably mention that a note requesting if there
> > > > > was
> > > interest
> > > > > got no response here on the public lists. This would seem to
> > > > > imply
> no
> > > > > interest by the community?
> > > > >
> > > > > Jon's note is pregnant but opaque on the details of these
> 'supported'
> > > > > deploys. He is in a bit of an awkward spot unable to share
> > > > > detail
> on
> > > > > someone else's deploy.
> > > > >
> > > > > Would be good to see more interest than Jon's note as evidence
> > > > > of
> > > > interest
> > > > > I'd say.
> > > > >
> > > > > You know of any? Even if they could be described in outline
> > > > > and
> > > hopefully
> > > > > more than one instance and that MOB makes sense for this deploy.
> And
> > > they
> > > > > need it now in a branch-1?
> > > > >
> > > > > Which version of hbase are we talking of backporting too? 1.3? 1.4?
> > > > >
> > > > >
> > > > >
> > > > > > 2. whether unit test stability can be maintained in branch-1
> > > > > >
> > > > > > Inclusion of the backport should keep unit tests (both
> > > > > > existing
> and
> > > > new)
> > > > > in
> > > > > > branch-1 green.
> > > > > > Preliminary test runs showed that MOB / snapshot related
> > > > > > tests
> > > > > consistently
> > > > > > pass.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HBASE-15370?focusedCommentId=151
> 76094&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tab
> panel#comment-15176094
> > > > >
> > > > >
> > > > > Hopefully we are aiming for more than just 'test stability'.
> > > > > Some
> > large
> > > > > community users have placed big bets on branch-1 being stable
> > > > > at
> > scale;
> > > > > this is the stability we should be precious of -- not just
> > > > > 'test stability'.
> > > > >
> > > > > Besides, I see failures in your listing. MOB is pervasive. Why
> > > > > is
> it
> > > not
> > > > > responsible for the mentioned test failures? (And "... passes
> > > > > on my
> > > local
> > > > > machine... " doesn't cut it; builds.apache.org is our public
> > > > > CI
> > where
> > > > > community comes together on state of branch and master, not
> > > > > some
> devs
> > > > > laptop).
> > > > >
> > > > > Branch-1 builds have been generally stable after large
> > > > > investment
> > > fixing,
> > > > > tuning and pruning tests. Master branch had been rendered
> > > > > stable
> but
> > > > seems
> > > > > to be rotting again though it is the most important of our
> > > > > builds
> > given
> > > > it
> > > > > can catch the bad stuff before the commits make it in. During
> > > > > the
> > > > campaign
> > > > > to stabilize Master, MOB tests failed often. I see MOB
> > > > > failures in
> > > master
> > > > > patch build from time to time still (I've been lax of late but
> > > > > our
> > > > flakies
> > > > > list contains a few mentionds, see HBASE-15012). They go
> unaddressed
> > > > > (though, to be fair, Jingcheng, the original author, addressed
> > > > > a
> few
> > > > > failures early on when asked). Just this morning I note:
> > > > >
> > > > >
> > > >
> > >
> >
> https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Buil
> d/818/testReport/junit/org.apache.hadoop.hbase.mob.mapreduce/TestMobSw
> eepMapper/org_apache_hadoop_hbase_mob_mapreduce_TestMobSweepMapper/
> > > > >
> > > > >
> > > > > (Suggestion: MOB seems to be better in master branch of late
> > > > > --
> > Matteo
> > > > and
> > > > > Heng work I believe (or just my disabling of broke tests) --
> > > > > so a
> > > review
> > > > > and report of master and patch builds looking for MOB failures
> > > > > and
> > > fixing
> > > > > any found would help address the above concern. Another way to
> build
> > > > > confidence in a patched branch-1 would be doing the branch
> suggested
> > up
> > > > in
> > > > > the backport issue and then running builds on apache for a period.
> Or
> > > > > better, copy what was done by Jon et al. to build confidence;
> > > > > run long-running ITBLLs on a cluster w/ MOB set with obnoxious
> > > > > configs
> > and
> > > > post
> > > > > evidence it all holds up).
> > > > >
> > > > >
> > > > > > Comments are welcome
> > > > > >
> > > > > > <https://issues.apache.org/jira/browse/HBASE-15370#>
> > > > > >
> > > > >
> > > > >
> > > > > Is MOB an 'owned' feature. The MOB original author is mostly
> > > > > absent
> > > (for
> > > > > instance, no participation in these conversations and no
> > > > > general
> > > > follow-up
> > > > > on test failures). As has been said before many times,
> > > > > features
> need
> > to
> > > > be
> > > > > owned. Writing the code is just one part of owning a feature.
> > Features
> > > > that
> > > > > are not owned become a burden on the community to maintain. We
> > > > > have
> > > > enough
> > > > > of this latter type of feature already.
> > > > >
> > > > > (Jingcheng did show up in the last day though with
> > > HBASE-15381"Implement
> > > > a
> > > > > distributed MOB compaction by procedure" which looks like a
> > > > > pretty important issue and begs the question, is MOB finished?
> > > > > And if not, shouldn't we wait till its done before we
> > > > > backport?)
> > > > >
> > > > > Finally, MOB backport should be done by someone who is
> > > > > familiar
> with
> > > > MOB. I
> > > > > see no evidence of your expertise in MOB other than an odd
> > > > > review
> nor
> > > > even
> > > > > that you've run it in other than a unit test mode. Not to
> > > > > mention
> > that
> > > > the
> > > > > way you go about the backport, dumping an 800k blob of
> > > > > unattributed
> > > code
> > > > > into the issue for review in HBASE-15370, does not bode well
> > > > > for
> our
> > > > > continued stability.
> > > > >
> > > > > St.Ack
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein (via Tom White)
> >
>
RE: [DISCUSS] Criteria for including MOB feature backport in
branch-1
Posted by "Du, Jingcheng" <ji...@intel.com>.
Sorry for the late chime in and thank you all for the discussion.
I agree with Jon's suggestion, to make a new branch for MOB in branch-1.x, and backport it when it is definitely ready.
I can start to work on that branch after the distributed mob compaction is finished.
I think the distributed mob compaction is not a blocker. The old mob compaction can be disabled by configuration, this would not impact the read/write performance of mob files. Users can count on the TTL cleaner to remove the expired mob files which is very light weighted.
And I will keep eyes on the UT results, and get ready to fix the coming up failures. Thanks.
Regards,
Jingcheng
-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, March 4, 2016 9:53 AM
To: dev@hbase.apache.org
Subject: Re: [DISCUSS] Criteria for including MOB feature backport in branch-1
Good questions, Enis.
If my bandwidth permits, I am planning to collect performance statistics using ycsb against cluster with and without MOB feature enabled.
Cheers
On Thu, Mar 3, 2016 at 3:49 PM, Enis Söztutar <en...@gmail.com> wrote:
> Regardless of the backport, did we do the same regression analysis for
> the master branch merge at the time of the merge? Like make sure that
> the latencies and stability of non-mob is affected or not. Sorry, I
> was not following the merge vote closely.
>
> The reason I am asking is that although we can allow some flexibility
> in master, we still want to keep master releasable and prevent a
> 1-year effort to re-stabilize master when we want to release 2.0. If
> we have design / stability / impact questions, when and how they will
> be addressed for a 2.0 release? Let's say that we would like to
> release 2.0 by summer time as Matteo was quoting, how do we make sure
> that the feature as is, is stable and supportable?
>
> Enis
>
> On Thu, Mar 3, 2016 at 3:36 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > Just to be clear, I did not actually oppose a backport of MOB.
> > Although
> - I
> > have to say am sympathetic to Elliott's position that there's a
> > project management reason not to. My insistence here is "merely" for
> > this change
> in
> > particular a backport to the branch we are making production
> > releases
> from
> > demands a credible data driven assessment on stability,
> > functionality,
> and
> > performance - both avoidance of regression and affirmative
> > indication of the alleged benefit.
> >
> > On Thu, Mar 3, 2016 at 3:16 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > In the middle of writing response to this thread, I saw subsequent
> > comments
> > > from Andrew, Jon and Elliott.
> > >
> > > In light of opposition from the community w.r.t. backporting MOB
> > feature, I
> > > think it suffices to say that this wouldn't be done now.
> > >
> > > Thanks everyone for chiming in.
> > >
> > > On Thu, Mar 3, 2016 at 2:42 PM, Elliott Clark <ec...@apache.org>
> wrote:
> > >
> > > > I just don't see a why we would back port. We're going to
> > > > release a
> 2.0
> > > > when things are ready. It will be a major feature release. MOB
> > > > is a
> > major
> > > > feature. Without compelling reason backporting to branch-1 seems
> > > > like
> > an
> > > > end run around what sem ver is supposed to mean (not the api
> > guarantees,
> > > > the actual meaning).
> > > >
> > > > Backporting major features is a bad habit that the Hadoop
> > > > community
> > seems
> > > > to have. We shouldn't follow their lead.
> > > >
> > > > On Thu, Mar 3, 2016 at 1:01 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > On Thu, Mar 3, 2016 at 9:02 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi,
> > > > > > As requested by Sean Busbey, I am starting a new thread w.r.t.
> > > > > backporting
> > > > > > MOB feature to branch-1.
> > > > > >
> > > > > This is to solicit discussion on the criteria for including
> > > > > MOB
> > feature
> > > > > > backport in branch-1.
> > > > > >
> > > > > > I can think of the following:
> > > > > > 1. whether there is customer interest
> > > > > >
> > > > > > There is.
> > > > > > See Jonathan's response here:
> > > > > http://search-hadoop.com/m/YGbbDSqxD1PYXK62
> > > > > >
> > > > > >
> > > > > You should probably mention that a note requesting if there
> > > > > was
> > > interest
> > > > > got no response here on the public lists. This would seem to
> > > > > imply
> no
> > > > > interest by the community?
> > > > >
> > > > > Jon's note is pregnant but opaque on the details of these
> 'supported'
> > > > > deploys. He is in a bit of an awkward spot unable to share
> > > > > detail
> on
> > > > > someone else's deploy.
> > > > >
> > > > > Would be good to see more interest than Jon's note as evidence
> > > > > of
> > > > interest
> > > > > I'd say.
> > > > >
> > > > > You know of any? Even if they could be described in outline
> > > > > and
> > > hopefully
> > > > > more than one instance and that MOB makes sense for this deploy.
> And
> > > they
> > > > > need it now in a branch-1?
> > > > >
> > > > > Which version of hbase are we talking of backporting too? 1.3? 1.4?
> > > > >
> > > > >
> > > > >
> > > > > > 2. whether unit test stability can be maintained in branch-1
> > > > > >
> > > > > > Inclusion of the backport should keep unit tests (both
> > > > > > existing
> and
> > > > new)
> > > > > in
> > > > > > branch-1 green.
> > > > > > Preliminary test runs showed that MOB / snapshot related
> > > > > > tests
> > > > > consistently
> > > > > > pass.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HBASE-15370?focusedCommentId=151
> 76094&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tab
> panel#comment-15176094
> > > > >
> > > > >
> > > > > Hopefully we are aiming for more than just 'test stability'.
> > > > > Some
> > large
> > > > > community users have placed big bets on branch-1 being stable
> > > > > at
> > scale;
> > > > > this is the stability we should be precious of -- not just
> > > > > 'test stability'.
> > > > >
> > > > > Besides, I see failures in your listing. MOB is pervasive. Why
> > > > > is
> it
> > > not
> > > > > responsible for the mentioned test failures? (And "... passes
> > > > > on my
> > > local
> > > > > machine... " doesn't cut it; builds.apache.org is our public
> > > > > CI
> > where
> > > > > community comes together on state of branch and master, not
> > > > > some
> devs
> > > > > laptop).
> > > > >
> > > > > Branch-1 builds have been generally stable after large
> > > > > investment
> > > fixing,
> > > > > tuning and pruning tests. Master branch had been rendered
> > > > > stable
> but
> > > > seems
> > > > > to be rotting again though it is the most important of our
> > > > > builds
> > given
> > > > it
> > > > > can catch the bad stuff before the commits make it in. During
> > > > > the
> > > > campaign
> > > > > to stabilize Master, MOB tests failed often. I see MOB
> > > > > failures in
> > > master
> > > > > patch build from time to time still (I've been lax of late but
> > > > > our
> > > > flakies
> > > > > list contains a few mentionds, see HBASE-15012). They go
> unaddressed
> > > > > (though, to be fair, Jingcheng, the original author, addressed
> > > > > a
> few
> > > > > failures early on when asked). Just this morning I note:
> > > > >
> > > > >
> > > >
> > >
> >
> https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Buil
> d/818/testReport/junit/org.apache.hadoop.hbase.mob.mapreduce/TestMobSw
> eepMapper/org_apache_hadoop_hbase_mob_mapreduce_TestMobSweepMapper/
> > > > >
> > > > >
> > > > > (Suggestion: MOB seems to be better in master branch of late
> > > > > --
> > Matteo
> > > > and
> > > > > Heng work I believe (or just my disabling of broke tests) --
> > > > > so a
> > > review
> > > > > and report of master and patch builds looking for MOB failures
> > > > > and
> > > fixing
> > > > > any found would help address the above concern. Another way to
> build
> > > > > confidence in a patched branch-1 would be doing the branch
> suggested
> > up
> > > > in
> > > > > the backport issue and then running builds on apache for a period.
> Or
> > > > > better, copy what was done by Jon et al. to build confidence;
> > > > > run long-running ITBLLs on a cluster w/ MOB set with obnoxious
> > > > > configs
> > and
> > > > post
> > > > > evidence it all holds up).
> > > > >
> > > > >
> > > > > > Comments are welcome
> > > > > >
> > > > > > <https://issues.apache.org/jira/browse/HBASE-15370#>
> > > > > >
> > > > >
> > > > >
> > > > > Is MOB an 'owned' feature. The MOB original author is mostly
> > > > > absent
> > > (for
> > > > > instance, no participation in these conversations and no
> > > > > general
> > > > follow-up
> > > > > on test failures). As has been said before many times,
> > > > > features
> need
> > to
> > > > be
> > > > > owned. Writing the code is just one part of owning a feature.
> > Features
> > > > that
> > > > > are not owned become a burden on the community to maintain. We
> > > > > have
> > > > enough
> > > > > of this latter type of feature already.
> > > > >
> > > > > (Jingcheng did show up in the last day though with
> > > HBASE-15381"Implement
> > > > a
> > > > > distributed MOB compaction by procedure" which looks like a
> > > > > pretty important issue and begs the question, is MOB finished?
> > > > > And if not, shouldn't we wait till its done before we
> > > > > backport?)
> > > > >
> > > > > Finally, MOB backport should be done by someone who is
> > > > > familiar
> with
> > > > MOB. I
> > > > > see no evidence of your expertise in MOB other than an odd
> > > > > review
> nor
> > > > even
> > > > > that you've run it in other than a unit test mode. Not to
> > > > > mention
> > that
> > > > the
> > > > > way you go about the backport, dumping an 800k blob of
> > > > > unattributed
> > > code
> > > > > into the issue for review in HBASE-15370, does not bode well
> > > > > for
> our
> > > > > continued stability.
> > > > >
> > > > > St.Ack
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein (via Tom White)
> >
>
Re: [DISCUSS] Criteria for including MOB feature backport in branch-1
Posted by Ted Yu <yu...@gmail.com>.
Good questions, Enis.
If my bandwidth permits, I am planning to collect performance statistics
using ycsb against cluster with and without MOB feature enabled.
Cheers
On Thu, Mar 3, 2016 at 3:49 PM, Enis Söztutar <en...@gmail.com> wrote:
> Regardless of the backport, did we do the same regression analysis for the
> master branch merge at the time of the merge? Like make sure that the
> latencies and stability of non-mob is affected or not. Sorry, I was not
> following the merge vote closely.
>
> The reason I am asking is that although we can allow some flexibility in
> master, we still want to keep master releasable and prevent a 1-year effort
> to re-stabilize master when we want to release 2.0. If we have design /
> stability / impact questions, when and how they will be addressed for a 2.0
> release? Let's say that we would like to release 2.0 by summer time as
> Matteo was quoting, how do we make sure that the feature as is, is stable
> and supportable?
>
> Enis
>
> On Thu, Mar 3, 2016 at 3:36 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > Just to be clear, I did not actually oppose a backport of MOB. Although
> - I
> > have to say am sympathetic to Elliott's position that there's a project
> > management reason not to. My insistence here is "merely" for this change
> in
> > particular a backport to the branch we are making production releases
> from
> > demands a credible data driven assessment on stability, functionality,
> and
> > performance - both avoidance of regression and affirmative indication of
> > the alleged benefit.
> >
> > On Thu, Mar 3, 2016 at 3:16 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > In the middle of writing response to this thread, I saw subsequent
> > comments
> > > from Andrew, Jon and Elliott.
> > >
> > > In light of opposition from the community w.r.t. backporting MOB
> > feature, I
> > > think it suffices to say that this wouldn't be done now.
> > >
> > > Thanks everyone for chiming in.
> > >
> > > On Thu, Mar 3, 2016 at 2:42 PM, Elliott Clark <ec...@apache.org>
> wrote:
> > >
> > > > I just don't see a why we would back port. We're going to release a
> 2.0
> > > > when things are ready. It will be a major feature release. MOB is a
> > major
> > > > feature. Without compelling reason backporting to branch-1 seems like
> > an
> > > > end run around what sem ver is supposed to mean (not the api
> > guarantees,
> > > > the actual meaning).
> > > >
> > > > Backporting major features is a bad habit that the Hadoop community
> > seems
> > > > to have. We shouldn't follow their lead.
> > > >
> > > > On Thu, Mar 3, 2016 at 1:01 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > On Thu, Mar 3, 2016 at 9:02 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi,
> > > > > > As requested by Sean Busbey, I am starting a new thread w.r.t.
> > > > > backporting
> > > > > > MOB feature to branch-1.
> > > > > >
> > > > > This is to solicit discussion on the criteria for including MOB
> > feature
> > > > > > backport in branch-1.
> > > > > >
> > > > > > I can think of the following:
> > > > > > 1. whether there is customer interest
> > > > > >
> > > > > > There is.
> > > > > > See Jonathan's response here:
> > > > > http://search-hadoop.com/m/YGbbDSqxD1PYXK62
> > > > > >
> > > > > >
> > > > > You should probably mention that a note requesting if there was
> > > interest
> > > > > got no response here on the public lists. This would seem to imply
> no
> > > > > interest by the community?
> > > > >
> > > > > Jon's note is pregnant but opaque on the details of these
> 'supported'
> > > > > deploys. He is in a bit of an awkward spot unable to share detail
> on
> > > > > someone else's deploy.
> > > > >
> > > > > Would be good to see more interest than Jon's note as evidence of
> > > > interest
> > > > > I'd say.
> > > > >
> > > > > You know of any? Even if they could be described in outline and
> > > hopefully
> > > > > more than one instance and that MOB makes sense for this deploy.
> And
> > > they
> > > > > need it now in a branch-1?
> > > > >
> > > > > Which version of hbase are we talking of backporting too? 1.3? 1.4?
> > > > >
> > > > >
> > > > >
> > > > > > 2. whether unit test stability can be maintained in branch-1
> > > > > >
> > > > > > Inclusion of the backport should keep unit tests (both existing
> and
> > > > new)
> > > > > in
> > > > > > branch-1 green.
> > > > > > Preliminary test runs showed that MOB / snapshot related tests
> > > > > consistently
> > > > > > pass.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HBASE-15370?focusedCommentId=15176094&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15176094
> > > > >
> > > > >
> > > > > Hopefully we are aiming for more than just 'test stability'. Some
> > large
> > > > > community users have placed big bets on branch-1 being stable at
> > scale;
> > > > > this is the stability we should be precious of -- not just 'test
> > > > > stability'.
> > > > >
> > > > > Besides, I see failures in your listing. MOB is pervasive. Why is
> it
> > > not
> > > > > responsible for the mentioned test failures? (And "... passes on my
> > > local
> > > > > machine... " doesn't cut it; builds.apache.org is our public CI
> > where
> > > > > community comes together on state of branch and master, not some
> devs
> > > > > laptop).
> > > > >
> > > > > Branch-1 builds have been generally stable after large investment
> > > fixing,
> > > > > tuning and pruning tests. Master branch had been rendered stable
> but
> > > > seems
> > > > > to be rotting again though it is the most important of our builds
> > given
> > > > it
> > > > > can catch the bad stuff before the commits make it in. During the
> > > > campaign
> > > > > to stabilize Master, MOB tests failed often. I see MOB failures in
> > > master
> > > > > patch build from time to time still (I've been lax of late but our
> > > > flakies
> > > > > list contains a few mentionds, see HBASE-15012). They go
> unaddressed
> > > > > (though, to be fair, Jingcheng, the original author, addressed a
> few
> > > > > failures early on when asked). Just this morning I note:
> > > > >
> > > > >
> > > >
> > >
> >
> https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/818/testReport/junit/org.apache.hadoop.hbase.mob.mapreduce/TestMobSweepMapper/org_apache_hadoop_hbase_mob_mapreduce_TestMobSweepMapper/
> > > > >
> > > > >
> > > > > (Suggestion: MOB seems to be better in master branch of late --
> > Matteo
> > > > and
> > > > > Heng work I believe (or just my disabling of broke tests) -- so a
> > > review
> > > > > and report of master and patch builds looking for MOB failures and
> > > fixing
> > > > > any found would help address the above concern. Another way to
> build
> > > > > confidence in a patched branch-1 would be doing the branch
> suggested
> > up
> > > > in
> > > > > the backport issue and then running builds on apache for a period.
> Or
> > > > > better, copy what was done by Jon et al. to build confidence; run
> > > > > long-running ITBLLs on a cluster w/ MOB set with obnoxious configs
> > and
> > > > post
> > > > > evidence it all holds up).
> > > > >
> > > > >
> > > > > > Comments are welcome
> > > > > >
> > > > > > <https://issues.apache.org/jira/browse/HBASE-15370#>
> > > > > >
> > > > >
> > > > >
> > > > > Is MOB an 'owned' feature. The MOB original author is mostly absent
> > > (for
> > > > > instance, no participation in these conversations and no general
> > > > follow-up
> > > > > on test failures). As has been said before many times, features
> need
> > to
> > > > be
> > > > > owned. Writing the code is just one part of owning a feature.
> > Features
> > > > that
> > > > > are not owned become a burden on the community to maintain. We have
> > > > enough
> > > > > of this latter type of feature already.
> > > > >
> > > > > (Jingcheng did show up in the last day though with
> > > HBASE-15381"Implement
> > > > a
> > > > > distributed MOB compaction by procedure" which looks like a pretty
> > > > > important issue and begs the question, is MOB finished? And if not,
> > > > > shouldn't we wait till its done before we backport?)
> > > > >
> > > > > Finally, MOB backport should be done by someone who is familiar
> with
> > > > MOB. I
> > > > > see no evidence of your expertise in MOB other than an odd review
> nor
> > > > even
> > > > > that you've run it in other than a unit test mode. Not to mention
> > that
> > > > the
> > > > > way you go about the backport, dumping an 800k blob of unattributed
> > > code
> > > > > into the issue for review in HBASE-15370, does not bode well for
> our
> > > > > continued stability.
> > > > >
> > > > > St.Ack
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
Re: [DISCUSS] Criteria for including MOB feature backport in branch-1
Posted by Enis Söztutar <en...@gmail.com>.
Regardless of the backport, did we do the same regression analysis for the
master branch merge at the time of the merge? Like make sure that the
latencies and stability of non-mob is affected or not. Sorry, I was not
following the merge vote closely.
The reason I am asking is that although we can allow some flexibility in
master, we still want to keep master releasable and prevent a 1-year effort
to re-stabilize master when we want to release 2.0. If we have design /
stability / impact questions, when and how they will be addressed for a 2.0
release? Let's say that we would like to release 2.0 by summer time as
Matteo was quoting, how do we make sure that the feature as is, is stable
and supportable?
Enis
On Thu, Mar 3, 2016 at 3:36 PM, Andrew Purtell <ap...@apache.org> wrote:
> Just to be clear, I did not actually oppose a backport of MOB. Although - I
> have to say am sympathetic to Elliott's position that there's a project
> management reason not to. My insistence here is "merely" for this change in
> particular a backport to the branch we are making production releases from
> demands a credible data driven assessment on stability, functionality, and
> performance - both avoidance of regression and affirmative indication of
> the alleged benefit.
>
> On Thu, Mar 3, 2016 at 3:16 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > In the middle of writing response to this thread, I saw subsequent
> comments
> > from Andrew, Jon and Elliott.
> >
> > In light of opposition from the community w.r.t. backporting MOB
> feature, I
> > think it suffices to say that this wouldn't be done now.
> >
> > Thanks everyone for chiming in.
> >
> > On Thu, Mar 3, 2016 at 2:42 PM, Elliott Clark <ec...@apache.org> wrote:
> >
> > > I just don't see a why we would back port. We're going to release a 2.0
> > > when things are ready. It will be a major feature release. MOB is a
> major
> > > feature. Without compelling reason backporting to branch-1 seems like
> an
> > > end run around what sem ver is supposed to mean (not the api
> guarantees,
> > > the actual meaning).
> > >
> > > Backporting major features is a bad habit that the Hadoop community
> seems
> > > to have. We shouldn't follow their lead.
> > >
> > > On Thu, Mar 3, 2016 at 1:01 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > On Thu, Mar 3, 2016 at 9:02 AM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > > As requested by Sean Busbey, I am starting a new thread w.r.t.
> > > > backporting
> > > > > MOB feature to branch-1.
> > > > >
> > > > This is to solicit discussion on the criteria for including MOB
> feature
> > > > > backport in branch-1.
> > > > >
> > > > > I can think of the following:
> > > > > 1. whether there is customer interest
> > > > >
> > > > > There is.
> > > > > See Jonathan's response here:
> > > > http://search-hadoop.com/m/YGbbDSqxD1PYXK62
> > > > >
> > > > >
> > > > You should probably mention that a note requesting if there was
> > interest
> > > > got no response here on the public lists. This would seem to imply no
> > > > interest by the community?
> > > >
> > > > Jon's note is pregnant but opaque on the details of these 'supported'
> > > > deploys. He is in a bit of an awkward spot unable to share detail on
> > > > someone else's deploy.
> > > >
> > > > Would be good to see more interest than Jon's note as evidence of
> > > interest
> > > > I'd say.
> > > >
> > > > You know of any? Even if they could be described in outline and
> > hopefully
> > > > more than one instance and that MOB makes sense for this deploy. And
> > they
> > > > need it now in a branch-1?
> > > >
> > > > Which version of hbase are we talking of backporting too? 1.3? 1.4?
> > > >
> > > >
> > > >
> > > > > 2. whether unit test stability can be maintained in branch-1
> > > > >
> > > > > Inclusion of the backport should keep unit tests (both existing and
> > > new)
> > > > in
> > > > > branch-1 green.
> > > > > Preliminary test runs showed that MOB / snapshot related tests
> > > > consistently
> > > > > pass.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HBASE-15370?focusedCommentId=15176094&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15176094
> > > >
> > > >
> > > > Hopefully we are aiming for more than just 'test stability'. Some
> large
> > > > community users have placed big bets on branch-1 being stable at
> scale;
> > > > this is the stability we should be precious of -- not just 'test
> > > > stability'.
> > > >
> > > > Besides, I see failures in your listing. MOB is pervasive. Why is it
> > not
> > > > responsible for the mentioned test failures? (And "... passes on my
> > local
> > > > machine... " doesn't cut it; builds.apache.org is our public CI
> where
> > > > community comes together on state of branch and master, not some devs
> > > > laptop).
> > > >
> > > > Branch-1 builds have been generally stable after large investment
> > fixing,
> > > > tuning and pruning tests. Master branch had been rendered stable but
> > > seems
> > > > to be rotting again though it is the most important of our builds
> given
> > > it
> > > > can catch the bad stuff before the commits make it in. During the
> > > campaign
> > > > to stabilize Master, MOB tests failed often. I see MOB failures in
> > master
> > > > patch build from time to time still (I've been lax of late but our
> > > flakies
> > > > list contains a few mentionds, see HBASE-15012). They go unaddressed
> > > > (though, to be fair, Jingcheng, the original author, addressed a few
> > > > failures early on when asked). Just this morning I note:
> > > >
> > > >
> > >
> >
> https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/818/testReport/junit/org.apache.hadoop.hbase.mob.mapreduce/TestMobSweepMapper/org_apache_hadoop_hbase_mob_mapreduce_TestMobSweepMapper/
> > > >
> > > >
> > > > (Suggestion: MOB seems to be better in master branch of late --
> Matteo
> > > and
> > > > Heng work I believe (or just my disabling of broke tests) -- so a
> > review
> > > > and report of master and patch builds looking for MOB failures and
> > fixing
> > > > any found would help address the above concern. Another way to build
> > > > confidence in a patched branch-1 would be doing the branch suggested
> up
> > > in
> > > > the backport issue and then running builds on apache for a period. Or
> > > > better, copy what was done by Jon et al. to build confidence; run
> > > > long-running ITBLLs on a cluster w/ MOB set with obnoxious configs
> and
> > > post
> > > > evidence it all holds up).
> > > >
> > > >
> > > > > Comments are welcome
> > > > >
> > > > > <https://issues.apache.org/jira/browse/HBASE-15370#>
> > > > >
> > > >
> > > >
> > > > Is MOB an 'owned' feature. The MOB original author is mostly absent
> > (for
> > > > instance, no participation in these conversations and no general
> > > follow-up
> > > > on test failures). As has been said before many times, features need
> to
> > > be
> > > > owned. Writing the code is just one part of owning a feature.
> Features
> > > that
> > > > are not owned become a burden on the community to maintain. We have
> > > enough
> > > > of this latter type of feature already.
> > > >
> > > > (Jingcheng did show up in the last day though with
> > HBASE-15381"Implement
> > > a
> > > > distributed MOB compaction by procedure" which looks like a pretty
> > > > important issue and begs the question, is MOB finished? And if not,
> > > > shouldn't we wait till its done before we backport?)
> > > >
> > > > Finally, MOB backport should be done by someone who is familiar with
> > > MOB. I
> > > > see no evidence of your expertise in MOB other than an odd review nor
> > > even
> > > > that you've run it in other than a unit test mode. Not to mention
> that
> > > the
> > > > way you go about the backport, dumping an 800k blob of unattributed
> > code
> > > > into the issue for review in HBASE-15370, does not bode well for our
> > > > continued stability.
> > > >
> > > > St.Ack
> > > >
> > >
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: [DISCUSS] Criteria for including MOB feature backport in branch-1
Posted by Andrew Purtell <ap...@apache.org>.
Just to be clear, I did not actually oppose a backport of MOB. Although - I
have to say am sympathetic to Elliott's position that there's a project
management reason not to. My insistence here is "merely" for this change in
particular a backport to the branch we are making production releases from
demands a credible data driven assessment on stability, functionality, and
performance - both avoidance of regression and affirmative indication of
the alleged benefit.
On Thu, Mar 3, 2016 at 3:16 PM, Ted Yu <yu...@gmail.com> wrote:
> In the middle of writing response to this thread, I saw subsequent comments
> from Andrew, Jon and Elliott.
>
> In light of opposition from the community w.r.t. backporting MOB feature, I
> think it suffices to say that this wouldn't be done now.
>
> Thanks everyone for chiming in.
>
> On Thu, Mar 3, 2016 at 2:42 PM, Elliott Clark <ec...@apache.org> wrote:
>
> > I just don't see a why we would back port. We're going to release a 2.0
> > when things are ready. It will be a major feature release. MOB is a major
> > feature. Without compelling reason backporting to branch-1 seems like an
> > end run around what sem ver is supposed to mean (not the api guarantees,
> > the actual meaning).
> >
> > Backporting major features is a bad habit that the Hadoop community seems
> > to have. We shouldn't follow their lead.
> >
> > On Thu, Mar 3, 2016 at 1:01 PM, Stack <st...@duboce.net> wrote:
> >
> > > On Thu, Mar 3, 2016 at 9:02 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > Hi,
> > > > As requested by Sean Busbey, I am starting a new thread w.r.t.
> > > backporting
> > > > MOB feature to branch-1.
> > > >
> > > This is to solicit discussion on the criteria for including MOB feature
> > > > backport in branch-1.
> > > >
> > > > I can think of the following:
> > > > 1. whether there is customer interest
> > > >
> > > > There is.
> > > > See Jonathan's response here:
> > > http://search-hadoop.com/m/YGbbDSqxD1PYXK62
> > > >
> > > >
> > > You should probably mention that a note requesting if there was
> interest
> > > got no response here on the public lists. This would seem to imply no
> > > interest by the community?
> > >
> > > Jon's note is pregnant but opaque on the details of these 'supported'
> > > deploys. He is in a bit of an awkward spot unable to share detail on
> > > someone else's deploy.
> > >
> > > Would be good to see more interest than Jon's note as evidence of
> > interest
> > > I'd say.
> > >
> > > You know of any? Even if they could be described in outline and
> hopefully
> > > more than one instance and that MOB makes sense for this deploy. And
> they
> > > need it now in a branch-1?
> > >
> > > Which version of hbase are we talking of backporting too? 1.3? 1.4?
> > >
> > >
> > >
> > > > 2. whether unit test stability can be maintained in branch-1
> > > >
> > > > Inclusion of the backport should keep unit tests (both existing and
> > new)
> > > in
> > > > branch-1 green.
> > > > Preliminary test runs showed that MOB / snapshot related tests
> > > consistently
> > > > pass.
> > > >
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HBASE-15370?focusedCommentId=15176094&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15176094
> > >
> > >
> > > Hopefully we are aiming for more than just 'test stability'. Some large
> > > community users have placed big bets on branch-1 being stable at scale;
> > > this is the stability we should be precious of -- not just 'test
> > > stability'.
> > >
> > > Besides, I see failures in your listing. MOB is pervasive. Why is it
> not
> > > responsible for the mentioned test failures? (And "... passes on my
> local
> > > machine... " doesn't cut it; builds.apache.org is our public CI where
> > > community comes together on state of branch and master, not some devs
> > > laptop).
> > >
> > > Branch-1 builds have been generally stable after large investment
> fixing,
> > > tuning and pruning tests. Master branch had been rendered stable but
> > seems
> > > to be rotting again though it is the most important of our builds given
> > it
> > > can catch the bad stuff before the commits make it in. During the
> > campaign
> > > to stabilize Master, MOB tests failed often. I see MOB failures in
> master
> > > patch build from time to time still (I've been lax of late but our
> > flakies
> > > list contains a few mentionds, see HBASE-15012). They go unaddressed
> > > (though, to be fair, Jingcheng, the original author, addressed a few
> > > failures early on when asked). Just this morning I note:
> > >
> > >
> >
> https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/818/testReport/junit/org.apache.hadoop.hbase.mob.mapreduce/TestMobSweepMapper/org_apache_hadoop_hbase_mob_mapreduce_TestMobSweepMapper/
> > >
> > >
> > > (Suggestion: MOB seems to be better in master branch of late -- Matteo
> > and
> > > Heng work I believe (or just my disabling of broke tests) -- so a
> review
> > > and report of master and patch builds looking for MOB failures and
> fixing
> > > any found would help address the above concern. Another way to build
> > > confidence in a patched branch-1 would be doing the branch suggested up
> > in
> > > the backport issue and then running builds on apache for a period. Or
> > > better, copy what was done by Jon et al. to build confidence; run
> > > long-running ITBLLs on a cluster w/ MOB set with obnoxious configs and
> > post
> > > evidence it all holds up).
> > >
> > >
> > > > Comments are welcome
> > > >
> > > > <https://issues.apache.org/jira/browse/HBASE-15370#>
> > > >
> > >
> > >
> > > Is MOB an 'owned' feature. The MOB original author is mostly absent
> (for
> > > instance, no participation in these conversations and no general
> > follow-up
> > > on test failures). As has been said before many times, features need to
> > be
> > > owned. Writing the code is just one part of owning a feature. Features
> > that
> > > are not owned become a burden on the community to maintain. We have
> > enough
> > > of this latter type of feature already.
> > >
> > > (Jingcheng did show up in the last day though with
> HBASE-15381"Implement
> > a
> > > distributed MOB compaction by procedure" which looks like a pretty
> > > important issue and begs the question, is MOB finished? And if not,
> > > shouldn't we wait till its done before we backport?)
> > >
> > > Finally, MOB backport should be done by someone who is familiar with
> > MOB. I
> > > see no evidence of your expertise in MOB other than an odd review nor
> > even
> > > that you've run it in other than a unit test mode. Not to mention that
> > the
> > > way you go about the backport, dumping an 800k blob of unattributed
> code
> > > into the issue for review in HBASE-15370, does not bode well for our
> > > continued stability.
> > >
> > > St.Ack
> > >
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [DISCUSS] Criteria for including MOB feature backport in branch-1
Posted by Ted Yu <yu...@gmail.com>.
In the middle of writing response to this thread, I saw subsequent comments
from Andrew, Jon and Elliott.
In light of opposition from the community w.r.t. backporting MOB feature, I
think it suffices to say that this wouldn't be done now.
Thanks everyone for chiming in.
On Thu, Mar 3, 2016 at 2:42 PM, Elliott Clark <ec...@apache.org> wrote:
> I just don't see a why we would back port. We're going to release a 2.0
> when things are ready. It will be a major feature release. MOB is a major
> feature. Without compelling reason backporting to branch-1 seems like an
> end run around what sem ver is supposed to mean (not the api guarantees,
> the actual meaning).
>
> Backporting major features is a bad habit that the Hadoop community seems
> to have. We shouldn't follow their lead.
>
> On Thu, Mar 3, 2016 at 1:01 PM, Stack <st...@duboce.net> wrote:
>
> > On Thu, Mar 3, 2016 at 9:02 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Hi,
> > > As requested by Sean Busbey, I am starting a new thread w.r.t.
> > backporting
> > > MOB feature to branch-1.
> > >
> > This is to solicit discussion on the criteria for including MOB feature
> > > backport in branch-1.
> > >
> > > I can think of the following:
> > > 1. whether there is customer interest
> > >
> > > There is.
> > > See Jonathan's response here:
> > http://search-hadoop.com/m/YGbbDSqxD1PYXK62
> > >
> > >
> > You should probably mention that a note requesting if there was interest
> > got no response here on the public lists. This would seem to imply no
> > interest by the community?
> >
> > Jon's note is pregnant but opaque on the details of these 'supported'
> > deploys. He is in a bit of an awkward spot unable to share detail on
> > someone else's deploy.
> >
> > Would be good to see more interest than Jon's note as evidence of
> interest
> > I'd say.
> >
> > You know of any? Even if they could be described in outline and hopefully
> > more than one instance and that MOB makes sense for this deploy. And they
> > need it now in a branch-1?
> >
> > Which version of hbase are we talking of backporting too? 1.3? 1.4?
> >
> >
> >
> > > 2. whether unit test stability can be maintained in branch-1
> > >
> > > Inclusion of the backport should keep unit tests (both existing and
> new)
> > in
> > > branch-1 green.
> > > Preliminary test runs showed that MOB / snapshot related tests
> > consistently
> > > pass.
> > >
> > >
> > >
> >
> https://issues.apache.org/jira/browse/HBASE-15370?focusedCommentId=15176094&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15176094
> >
> >
> > Hopefully we are aiming for more than just 'test stability'. Some large
> > community users have placed big bets on branch-1 being stable at scale;
> > this is the stability we should be precious of -- not just 'test
> > stability'.
> >
> > Besides, I see failures in your listing. MOB is pervasive. Why is it not
> > responsible for the mentioned test failures? (And "... passes on my local
> > machine... " doesn't cut it; builds.apache.org is our public CI where
> > community comes together on state of branch and master, not some devs
> > laptop).
> >
> > Branch-1 builds have been generally stable after large investment fixing,
> > tuning and pruning tests. Master branch had been rendered stable but
> seems
> > to be rotting again though it is the most important of our builds given
> it
> > can catch the bad stuff before the commits make it in. During the
> campaign
> > to stabilize Master, MOB tests failed often. I see MOB failures in master
> > patch build from time to time still (I've been lax of late but our
> flakies
> > list contains a few mentionds, see HBASE-15012). They go unaddressed
> > (though, to be fair, Jingcheng, the original author, addressed a few
> > failures early on when asked). Just this morning I note:
> >
> >
> https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/818/testReport/junit/org.apache.hadoop.hbase.mob.mapreduce/TestMobSweepMapper/org_apache_hadoop_hbase_mob_mapreduce_TestMobSweepMapper/
> >
> >
> > (Suggestion: MOB seems to be better in master branch of late -- Matteo
> and
> > Heng work I believe (or just my disabling of broke tests) -- so a review
> > and report of master and patch builds looking for MOB failures and fixing
> > any found would help address the above concern. Another way to build
> > confidence in a patched branch-1 would be doing the branch suggested up
> in
> > the backport issue and then running builds on apache for a period. Or
> > better, copy what was done by Jon et al. to build confidence; run
> > long-running ITBLLs on a cluster w/ MOB set with obnoxious configs and
> post
> > evidence it all holds up).
> >
> >
> > > Comments are welcome
> > >
> > > <https://issues.apache.org/jira/browse/HBASE-15370#>
> > >
> >
> >
> > Is MOB an 'owned' feature. The MOB original author is mostly absent (for
> > instance, no participation in these conversations and no general
> follow-up
> > on test failures). As has been said before many times, features need to
> be
> > owned. Writing the code is just one part of owning a feature. Features
> that
> > are not owned become a burden on the community to maintain. We have
> enough
> > of this latter type of feature already.
> >
> > (Jingcheng did show up in the last day though with HBASE-15381"Implement
> a
> > distributed MOB compaction by procedure" which looks like a pretty
> > important issue and begs the question, is MOB finished? And if not,
> > shouldn't we wait till its done before we backport?)
> >
> > Finally, MOB backport should be done by someone who is familiar with
> MOB. I
> > see no evidence of your expertise in MOB other than an odd review nor
> even
> > that you've run it in other than a unit test mode. Not to mention that
> the
> > way you go about the backport, dumping an 800k blob of unattributed code
> > into the issue for review in HBASE-15370, does not bode well for our
> > continued stability.
> >
> > St.Ack
> >
>
Re: [DISCUSS] Criteria for including MOB feature backport in branch-1
Posted by Elliott Clark <ec...@apache.org>.
I just don't see a why we would back port. We're going to release a 2.0
when things are ready. It will be a major feature release. MOB is a major
feature. Without compelling reason backporting to branch-1 seems like an
end run around what sem ver is supposed to mean (not the api guarantees,
the actual meaning).
Backporting major features is a bad habit that the Hadoop community seems
to have. We shouldn't follow their lead.
On Thu, Mar 3, 2016 at 1:01 PM, Stack <st...@duboce.net> wrote:
> On Thu, Mar 3, 2016 at 9:02 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Hi,
> > As requested by Sean Busbey, I am starting a new thread w.r.t.
> backporting
> > MOB feature to branch-1.
> >
> This is to solicit discussion on the criteria for including MOB feature
> > backport in branch-1.
> >
> > I can think of the following:
> > 1. whether there is customer interest
> >
> > There is.
> > See Jonathan's response here:
> http://search-hadoop.com/m/YGbbDSqxD1PYXK62
> >
> >
> You should probably mention that a note requesting if there was interest
> got no response here on the public lists. This would seem to imply no
> interest by the community?
>
> Jon's note is pregnant but opaque on the details of these 'supported'
> deploys. He is in a bit of an awkward spot unable to share detail on
> someone else's deploy.
>
> Would be good to see more interest than Jon's note as evidence of interest
> I'd say.
>
> You know of any? Even if they could be described in outline and hopefully
> more than one instance and that MOB makes sense for this deploy. And they
> need it now in a branch-1?
>
> Which version of hbase are we talking of backporting too? 1.3? 1.4?
>
>
>
> > 2. whether unit test stability can be maintained in branch-1
> >
> > Inclusion of the backport should keep unit tests (both existing and new)
> in
> > branch-1 green.
> > Preliminary test runs showed that MOB / snapshot related tests
> consistently
> > pass.
> >
> >
> >
> https://issues.apache.org/jira/browse/HBASE-15370?focusedCommentId=15176094&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15176094
>
>
> Hopefully we are aiming for more than just 'test stability'. Some large
> community users have placed big bets on branch-1 being stable at scale;
> this is the stability we should be precious of -- not just 'test
> stability'.
>
> Besides, I see failures in your listing. MOB is pervasive. Why is it not
> responsible for the mentioned test failures? (And "... passes on my local
> machine... " doesn't cut it; builds.apache.org is our public CI where
> community comes together on state of branch and master, not some devs
> laptop).
>
> Branch-1 builds have been generally stable after large investment fixing,
> tuning and pruning tests. Master branch had been rendered stable but seems
> to be rotting again though it is the most important of our builds given it
> can catch the bad stuff before the commits make it in. During the campaign
> to stabilize Master, MOB tests failed often. I see MOB failures in master
> patch build from time to time still (I've been lax of late but our flakies
> list contains a few mentionds, see HBASE-15012). They go unaddressed
> (though, to be fair, Jingcheng, the original author, addressed a few
> failures early on when asked). Just this morning I note:
>
> https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/818/testReport/junit/org.apache.hadoop.hbase.mob.mapreduce/TestMobSweepMapper/org_apache_hadoop_hbase_mob_mapreduce_TestMobSweepMapper/
>
>
> (Suggestion: MOB seems to be better in master branch of late -- Matteo and
> Heng work I believe (or just my disabling of broke tests) -- so a review
> and report of master and patch builds looking for MOB failures and fixing
> any found would help address the above concern. Another way to build
> confidence in a patched branch-1 would be doing the branch suggested up in
> the backport issue and then running builds on apache for a period. Or
> better, copy what was done by Jon et al. to build confidence; run
> long-running ITBLLs on a cluster w/ MOB set with obnoxious configs and post
> evidence it all holds up).
>
>
> > Comments are welcome
> >
> > <https://issues.apache.org/jira/browse/HBASE-15370#>
> >
>
>
> Is MOB an 'owned' feature. The MOB original author is mostly absent (for
> instance, no participation in these conversations and no general follow-up
> on test failures). As has been said before many times, features need to be
> owned. Writing the code is just one part of owning a feature. Features that
> are not owned become a burden on the community to maintain. We have enough
> of this latter type of feature already.
>
> (Jingcheng did show up in the last day though with HBASE-15381"Implement a
> distributed MOB compaction by procedure" which looks like a pretty
> important issue and begs the question, is MOB finished? And if not,
> shouldn't we wait till its done before we backport?)
>
> Finally, MOB backport should be done by someone who is familiar with MOB. I
> see no evidence of your expertise in MOB other than an odd review nor even
> that you've run it in other than a unit test mode. Not to mention that the
> way you go about the backport, dumping an 800k blob of unattributed code
> into the issue for review in HBASE-15370, does not bode well for our
> continued stability.
>
> St.Ack
>
Re: [DISCUSS] Criteria for including MOB feature backport in branch-1
Posted by Jonathan Hsieh <jo...@cloudera.com>.
On Thu, Mar 3, 2016 at 1:01 PM, Stack <st...@duboce.net> wrote:
> On Thu, Mar 3, 2016 at 9:02 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Hi,
> > As requested by Sean Busbey, I am starting a new thread w.r.t.
> backporting
> > MOB feature to branch-1.
> >
> This is to solicit discussion on the criteria for including MOB feature
> > backport in branch-1.
> >
> > I can think of the following:
> > 1. whether there is customer interest
> >
> > There is.
> > See Jonathan's response here:
> http://search-hadoop.com/m/YGbbDSqxD1PYXK62
> >
> >
> You should probably mention that a note requesting if there was interest
> got no response here on the public lists. This would seem to imply no
> interest by the community?
>
> Jon's note is pregnant but opaque on the details of these 'supported'
> deploys. He is in a bit of an awkward spot unable to share detail on
> someone else's deploy.
>
> Sorry, what I provided is basically the most I could offer.
Would be good to see more interest than Jon's note as evidence of interest
> I'd say.
>
> You know of any? Even if they could be described in outline and hopefully
> more than one instance and that MOB makes sense for this deploy. And they
> need it now in a branch-1?
>
>
There are some notes from the vote thread staying that Ashutosh from Huawei
(at the time) had tested and patched part of mob -- might be interesting to
see if he has anything to share.
> Which version of hbase are we talking of backporting too? 1.3? 1.4?
>
>
>
> > 2. whether unit test stability can be maintained in branch-1
> >
> > Inclusion of the backport should keep unit tests (both existing and new)
> in
> > branch-1 green.
> > Preliminary test runs showed that MOB / snapshot related tests
> consistently
> > pass.
> >
> >
> >
> https://issues.apache.org/jira/browse/HBASE-15370?focusedCommentId=15176094&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15176094
>
>
> Hopefully we are aiming for more than just 'test stability'. Some large
> community users have placed big bets on branch-1 being stable at scale;
> this is the stability we should be precious of -- not just 'test
> stability'.
>
> Besides, I see failures in your listing. MOB is pervasive. Why is it not
> responsible for the mentioned test failures? (And "... passes on my local
> machine... " doesn't cut it; builds.apache.org is our public CI where
> community comes together on state of branch and master, not some devs
> laptop).
>
I agree with stack here -- unit tests passing is not sufficient.
>
> Branch-1 builds have been generally stable after large investment fixing,
> tuning and pruning tests. Master branch had been rendered stable but seems
> to be rotting again though it is the most important of our builds given it
> can catch the bad stuff before the commits make it in. During the campaign
> to stabilize Master, MOB tests failed often. I see MOB failures in master
> patch build from time to time still (I've been lax of late but our flakies
> list contains a few mentionds, see HBASE-15012). They go unaddressed
> (though, to be fair, Jingcheng, the original author, addressed a few
> failures early on when asked). Just this morning I note:
>
> https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/818/testReport/junit/org.apache.hadoop.hbase.mob.mapreduce/TestMobSweepMapper/org_apache_hadoop_hbase_mob_mapreduce_TestMobSweepMapper/
>
>
> (Suggestion: MOB seems to be better in master branch of late -- Matteo and
> Heng work I believe (or just my disabling of broke tests) -- so a review
> and report of master and patch builds looking for MOB failures and fixing
> any found would help address the above concern. Another way to build
> confidence in a patched branch-1 would be doing the branch suggested up in
> the backport issue and then running builds on apache for a period. Or
> better, copy what was done by Jon et al. to build confidence; run
> long-running ITBLLs on a cluster w/ MOB set with obnoxious configs and post
> evidence it all holds up).
>
Before I would +1 a branch merge, I would like to know that several of the
fault injection tests ran and performed reasonably well. Before I proposed
the hbase-11339 merge to trunk I had tested by adding ITIIngestMob and
options to ITBLL, ITAcidGuarantees and LTT and added a monkey that
triggered mob compaction operation. This is necessary to build up
confidence that the feature works beyond just unit tests.
As noted in the original trunk merge discussion thread, branch-1 is a
stable branch, and the bar for merge should be higher than it was for trunk.
>
>
> > Comments are welcome
> >
> > <https://issues.apache.org/jira/browse/HBASE-15370#>
> >
>
>
> Is MOB an 'owned' feature. The MOB original author is mostly absent (for
> instance, no participation in these conversations and no general follow-up
> on test failures). As has been said before many times, features need to be
> owned. Writing the code is just one part of owning a feature. Features that
> are not owned become a burden on the community to maintain. We have enough
> of this latter type of feature already.
>
> (Jingcheng did show up in the last day though with HBASE-15381"Implement a
> distributed MOB compaction by procedure" which looks like a pretty
> important issue and begs the question, is MOB finished? And if not,
> shouldn't we wait till its done before we backport?)
>
The proc v2 work necessary to do this is likely very similar to the work
necessary to port snapshot to pv2. This isn't done yet and not in branch-1
yet. IMO, because of this, this particular criteria is not a blocker for
me.
>
> Finally, MOB backport should be done by someone who is familiar with MOB. I
> see no evidence of your expertise in MOB other than an odd review nor even
> that you've run it in other than a unit test mode. Not to mention that the
> way you go about the backport, dumping an 800k blob of unattributed code
> into the issue for review in HBASE-15370, does not bode well for our
> continued stability.
>
I don't have the cycles to do the backport, but I will do reviews if the
procedure I outline was followed. I will also review any merge into
branch-1 attempt.
>
> St.Ack
>
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: [DISCUSS] Criteria for including MOB feature backport in branch-1
Posted by Andrew Purtell <an...@gmail.com>.
In addition to Stack's points those of us running HBase in production will need assurance in the form of data collected from a plausible test environment that the MOB backport does not affect function, stability, or performance for anyone who doesn't want it. Furthermore some affirmative demonstration of its benefit in a community reproducible way continues to be overdue on master and in my view mandatory for branch-1.
> On Mar 3, 2016, at 1:01 PM, Stack <st...@duboce.net> wrote:
>
>> On Thu, Mar 3, 2016 at 9:02 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>> Hi,
>> As requested by Sean Busbey, I am starting a new thread w.r.t. backporting
>> MOB feature to branch-1.
> This is to solicit discussion on the criteria for including MOB feature
>> backport in branch-1.
>>
>> I can think of the following:
>> 1. whether there is customer interest
>>
>> There is.
>> See Jonathan's response here: http://search-hadoop.com/m/YGbbDSqxD1PYXK62
> You should probably mention that a note requesting if there was interest
> got no response here on the public lists. This would seem to imply no
> interest by the community?
>
> Jon's note is pregnant but opaque on the details of these 'supported'
> deploys. He is in a bit of an awkward spot unable to share detail on
> someone else's deploy.
>
> Would be good to see more interest than Jon's note as evidence of interest
> I'd say.
>
> You know of any? Even if they could be described in outline and hopefully
> more than one instance and that MOB makes sense for this deploy. And they
> need it now in a branch-1?
>
> Which version of hbase are we talking of backporting too? 1.3? 1.4?
>
>
>
>> 2. whether unit test stability can be maintained in branch-1
>>
>> Inclusion of the backport should keep unit tests (both existing and new) in
>> branch-1 green.
>> Preliminary test runs showed that MOB / snapshot related tests consistently
>> pass.
>>
>>
>> https://issues.apache.org/jira/browse/HBASE-15370?focusedCommentId=15176094&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15176094
>
>
> Hopefully we are aiming for more than just 'test stability'. Some large
> community users have placed big bets on branch-1 being stable at scale;
> this is the stability we should be precious of -- not just 'test stability'.
>
> Besides, I see failures in your listing. MOB is pervasive. Why is it not
> responsible for the mentioned test failures? (And "... passes on my local
> machine... " doesn't cut it; builds.apache.org is our public CI where
> community comes together on state of branch and master, not some devs
> laptop).
>
> Branch-1 builds have been generally stable after large investment fixing,
> tuning and pruning tests. Master branch had been rendered stable but seems
> to be rotting again though it is the most important of our builds given it
> can catch the bad stuff before the commits make it in. During the campaign
> to stabilize Master, MOB tests failed often. I see MOB failures in master
> patch build from time to time still (I've been lax of late but our flakies
> list contains a few mentionds, see HBASE-15012). They go unaddressed
> (though, to be fair, Jingcheng, the original author, addressed a few
> failures early on when asked). Just this morning I note:
> https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/818/testReport/junit/org.apache.hadoop.hbase.mob.mapreduce/TestMobSweepMapper/org_apache_hadoop_hbase_mob_mapreduce_TestMobSweepMapper/
>
>
> (Suggestion: MOB seems to be better in master branch of late -- Matteo and
> Heng work I believe (or just my disabling of broke tests) -- so a review
> and report of master and patch builds looking for MOB failures and fixing
> any found would help address the above concern. Another way to build
> confidence in a patched branch-1 would be doing the branch suggested up in
> the backport issue and then running builds on apache for a period. Or
> better, copy what was done by Jon et al. to build confidence; run
> long-running ITBLLs on a cluster w/ MOB set with obnoxious configs and post
> evidence it all holds up).
>
>
>> Comments are welcome
>>
>> <https://issues.apache.org/jira/browse/HBASE-15370#>
>
>
> Is MOB an 'owned' feature. The MOB original author is mostly absent (for
> instance, no participation in these conversations and no general follow-up
> on test failures). As has been said before many times, features need to be
> owned. Writing the code is just one part of owning a feature. Features that
> are not owned become a burden on the community to maintain. We have enough
> of this latter type of feature already.
>
> (Jingcheng did show up in the last day though with HBASE-15381"Implement a
> distributed MOB compaction by procedure" which looks like a pretty
> important issue and begs the question, is MOB finished? And if not,
> shouldn't we wait till its done before we backport?)
>
> Finally, MOB backport should be done by someone who is familiar with MOB. I
> see no evidence of your expertise in MOB other than an odd review nor even
> that you've run it in other than a unit test mode. Not to mention that the
> way you go about the backport, dumping an 800k blob of unattributed code
> into the issue for review in HBASE-15370, does not bode well for our
> continued stability.
>
> St.Ack
Re: [DISCUSS] Criteria for including MOB feature backport in branch-1
Posted by Stack <st...@duboce.net>.
On Thu, Mar 3, 2016 at 9:02 AM, Ted Yu <yu...@gmail.com> wrote:
> Hi,
> As requested by Sean Busbey, I am starting a new thread w.r.t. backporting
> MOB feature to branch-1.
>
This is to solicit discussion on the criteria for including MOB feature
> backport in branch-1.
>
> I can think of the following:
> 1. whether there is customer interest
>
> There is.
> See Jonathan's response here: http://search-hadoop.com/m/YGbbDSqxD1PYXK62
>
>
You should probably mention that a note requesting if there was interest
got no response here on the public lists. This would seem to imply no
interest by the community?
Jon's note is pregnant but opaque on the details of these 'supported'
deploys. He is in a bit of an awkward spot unable to share detail on
someone else's deploy.
Would be good to see more interest than Jon's note as evidence of interest
I'd say.
You know of any? Even if they could be described in outline and hopefully
more than one instance and that MOB makes sense for this deploy. And they
need it now in a branch-1?
Which version of hbase are we talking of backporting too? 1.3? 1.4?
> 2. whether unit test stability can be maintained in branch-1
>
> Inclusion of the backport should keep unit tests (both existing and new) in
> branch-1 green.
> Preliminary test runs showed that MOB / snapshot related tests consistently
> pass.
>
>
> https://issues.apache.org/jira/browse/HBASE-15370?focusedCommentId=15176094&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15176094
Hopefully we are aiming for more than just 'test stability'. Some large
community users have placed big bets on branch-1 being stable at scale;
this is the stability we should be precious of -- not just 'test stability'.
Besides, I see failures in your listing. MOB is pervasive. Why is it not
responsible for the mentioned test failures? (And "... passes on my local
machine... " doesn't cut it; builds.apache.org is our public CI where
community comes together on state of branch and master, not some devs
laptop).
Branch-1 builds have been generally stable after large investment fixing,
tuning and pruning tests. Master branch had been rendered stable but seems
to be rotting again though it is the most important of our builds given it
can catch the bad stuff before the commits make it in. During the campaign
to stabilize Master, MOB tests failed often. I see MOB failures in master
patch build from time to time still (I've been lax of late but our flakies
list contains a few mentionds, see HBASE-15012). They go unaddressed
(though, to be fair, Jingcheng, the original author, addressed a few
failures early on when asked). Just this morning I note:
https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/818/testReport/junit/org.apache.hadoop.hbase.mob.mapreduce/TestMobSweepMapper/org_apache_hadoop_hbase_mob_mapreduce_TestMobSweepMapper/
(Suggestion: MOB seems to be better in master branch of late -- Matteo and
Heng work I believe (or just my disabling of broke tests) -- so a review
and report of master and patch builds looking for MOB failures and fixing
any found would help address the above concern. Another way to build
confidence in a patched branch-1 would be doing the branch suggested up in
the backport issue and then running builds on apache for a period. Or
better, copy what was done by Jon et al. to build confidence; run
long-running ITBLLs on a cluster w/ MOB set with obnoxious configs and post
evidence it all holds up).
> Comments are welcome
>
> <https://issues.apache.org/jira/browse/HBASE-15370#>
>
Is MOB an 'owned' feature. The MOB original author is mostly absent (for
instance, no participation in these conversations and no general follow-up
on test failures). As has been said before many times, features need to be
owned. Writing the code is just one part of owning a feature. Features that
are not owned become a burden on the community to maintain. We have enough
of this latter type of feature already.
(Jingcheng did show up in the last day though with HBASE-15381"Implement a
distributed MOB compaction by procedure" which looks like a pretty
important issue and begs the question, is MOB finished? And if not,
shouldn't we wait till its done before we backport?)
Finally, MOB backport should be done by someone who is familiar with MOB. I
see no evidence of your expertise in MOB other than an odd review nor even
that you've run it in other than a unit test mode. Not to mention that the
way you go about the backport, dumping an 800k blob of unattributed code
into the issue for review in HBASE-15370, does not bode well for our
continued stability.
St.Ack