Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Status update girls and boys!

hbase-2.0.0-alpha1 went out June 22nd.

alpha2 has been a bit slow to follow (holidays) though there has been
steady progress closing out blockers and criticals by a bunch of you all.
The plan is for a release in the first week or so of August. It should be
fully up on hbase-thirdparty using updated (and relocated) versions of
netty, guava, and protobuf as well as a default deploy that has
master-carrying-no-regions.

alpha3 will follow soon after and will focus on making sure our user-facing
APIs are clean (branch-1 compatible, no illicit removals/mods, and so on)
and that basic upgrade 'works'.

betas start in September?

I've been keeping a rough general state here [1] (please update any section
that is lagging actuality) but for details on what blockers and criticals
remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought 2.0
into better focus. Feel free to review and punt items you think can wait
till 3.0 or 2.1. If you want to pull in more stuff, please ask first.

Thanks,
St.Ack

1.
https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#
2. https://issues.apache.org/jira/projects/HBASE/versions/12327188


On Mon, May 15, 2017 at 4:05 PM, Stack <st...@duboce.net> wrote:

>
> On Mon, May 15, 2017 at 1:46 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
>> HBASE-18037 is a new blocker. I'm currently working on it, will be
>> finished
>> soon I think.
>>
>> I made it a blocker then and added it to our hbase2 release doc [1] list
> as a blocker.
>
> Thanks,
> St.Ack
>
> 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> ktczrlKHK8N4SZzs/edit#
>
>
>> 2017-05-15 14:12 GMT+08:00 Stack <st...@duboce.net>:
>>
>>
>> > A month on. Status.
>> >
>> > I've been working on the HBASE-14614 branch cluster testing. After a
>> load
>> > of fixing, the branch passes smaller test runs (an hour or so of ITBLL
>> up
>> > to 2B rows w/ killing monkeys). When I go larger, to a scale I've not
>> done
>> > in a while, I start to run into other interesting issues -- some of
>> which
>> > are related to AMv2 (I'm fixing), but others are not (100G WALs that
>> take
>> > ten minutes to split makes for interesting cascades when monkeys kill
>> > inside the ten minutes...). I intend to keep on with this larger scale
>> > testing since it is uncovering good stuff (especially when HDFS is dog
>> slow
>> > because of background replications) but my thinking is that I should be
>> > large scale testing branch-2, not just HBASE-14614. I think HBASE-14614,
>> > the new AMv2, is good enough to merge to master these times. Given it is
>> > the last blocker, once in, I'll cut the hbase2 branch.
>> >
>> > I'll start up a 'Merge HBASE-14614' DISCUSSION thread in the next day
>> or so
>> > (I need to fix some unit tests...).
>> >
>> > The AMv2 doc is still a work in progress but should give a gist on
>> where we
>> > are currently[1].  There is a bunch of todo still but seems tractable;
>> e.g.
>> > rolling upgrade, finish doc., and we don't have an HBCK since it needs
>> to
>> > be recast in light of how stuff now works but a redo on HBCK is
>> premature
>> > given we don't know failure types as yet (we just fix the problems as
>> they
>> > come up).
>> >
>> > St.Ack
>> > 1.
>> > https://docs.google.com/document/d/1eVKa7FHdeoJ1-
>> > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
>> >
>> >
>> >
>> > On Thu, Apr 13, 2017 at 1:43 PM, Stack <st...@duboce.net> wrote:
>> >
>> > > Some status:
>> > >
>> > > AMv2 (HBASE-14614) is near to passing all tests caveat my disabling of
>> > all
>> > > to-do w/ fsck (fsck needs revamp) and tests that expect that they can
>> > move
>> > > hbase;meta off master (AMv2 enforces this constraint; it is supposed
>> to
>> > be
>> > > enforced on AMv1 but meta-on-master is incompletely realized in AMv1
>> and
>> > > AMv2). A few other tests have been disabled for various reasons. See
>> [1]
>> > > for full list.
>> > >
>> > > There is a hefty list of TODOs still (Again see the messy doc [1]) but
>> > the
>> > > only 'blocker', IMO, is community confidence in AMv2. Currently,
>> cluster
>> > > tests with chaos fail (new form of 'stuck' regions). Takes time
>> > > investigating.
>> > >
>> > > Will keep you all posted.
>> > > St.Ack
>> > >
>> > >
>> > >
>> > > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1-
>> > > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod
>> > >
>> > >
>> > >
>> > > On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell <ap...@apache.org>
>> > > wrote:
>> > >
>> > >> +1 on branching (yay!)
>> > >>
>> > >> I have EC2 resources for running ITBLL etc.
>> > >>
>> > >>
>> > >> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> wrote:
>> > >>
>> > >> > Some notes on progress toward hbase2.
>> > >> >
>> > >> > Given that stability and performance are NOT emergent behaviors but
>> > >> rather
>> > >> > projects unto themselves, my thought is that we commit all that
>> we've
>> > >> > agreed as core for hbase2 (see [1]), branch, and then work on
>> > >> stabilizing
>> > >> > and perf rather than do stabilize, commit, and then branch. What
>> this
>> > >> means
>> > >> > in practice is that for features like Inmemory Compaction, we
>> commit
>> > it
>> > >> > defaulted 'on' ("BASIC" mode) which is what we want in hbase2.
>> Should
>> > it
>> > >> > prove problematic under test, we disable it before release.
>> > >> >
>> > >> > Are folks good w/ this mode? I ask because, in a few issues there
>> are
>> > >> > requests for proof that a master feature is 'stable' before commit.
>> > >> This is
>> > >> > normally a healthy request only in master's case, it is hard to
>> > >> demonstrate
>> > >> > stability given its current state.
>> > >> >
>> > >> > Other outstanding issues such as decisions about whether master
>> hosts
>> > >> > system tables only (by default), I'm thinking, we can work out post
>> > >> branch
>> > >> > in alpha/betas before release.
>> > >> >
>> > >> > The awkward item is the long-pole Assignment Manager. This is an
>> > >> > all-or-nothing affair. Here we are switching in a new Master core.
>> > >> While I
>> > >> > think it fine that AMv2 is incomplete come branch time, those of us
>> > >> working
>> > >> > on the new AM still need to demonstrate to you all that it
>> basically
>> > >> > viable.
>> > >> >
>> > >> > The point-of-no-return is commit of the patch in HBASE-14614.
>> > >> HBASE-14614
>> > >> > (AMv2) is coming close to passing all unit tests. We'll spend some
>> > time
>> > >> > running it on a cluster to make sure it fundamentally sound and
>> will
>> > >> report
>> > >> > back on our experience. There has been an ask for some dev doc and
>> > >> > low-levels on how it works (in progress). Let satisfaction of these
>> > >> > requests be blockers on commit. We'll put the HBASE-14614 commit up
>> > for
>> > >> a
>> > >> > vote on dev list given its import.
>> > >> >
>> > >> > Branch will happen after HBASE-14614 goes in (or its rejection)
>> with
>> > our
>> > >> > first alpha soon after. Its looking like a week or two at least
>> given
>> > >> how
>> > >> > things have been going up to this.
>> > >> >
>> > >> > I intend to start in on hbase2 stability/perf projects after we
>> > branch.
>> > >> >
>> > >> > Interested in any thoughts you all might have on the above (Would
>> also
>> > >> > appreciate updates on state in [1] if you are a feature owner).
>> > >> >
>> > >> > Thanks,
>> > >> > St.Ack
>> > >> >
>> > >> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
>> > >> > z9iEu_ktczrlKHK8N4SZzs/edit#
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <el...@apache.org>
>> > wrote:
>> > >> >
>> > >> > >
>> > >> > > Stack wrote:
>> > >> > >
>> > >> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<el...@apache.org>
>> > >> wrote:
>> > >> > >>
>> > >> > >> Thanks for pulling in the FS Quotas work, Stack. I'm trying to
>> > cross
>> > >> the
>> > >> > >>> last T's and dot the last I's.
>> > >> > >>>
>> > >> > >>> The biggest thing I know I need to do still is to write a new
>> > >> chapter
>> > >> > to
>> > >> > >>> the book. After that, I'd start entertaining larger
>> > >> reviews/discussions
>> > >> > >>> to
>> > >> > >>> merge the feature into master. Anyone with free time (giggles)
>> is
>> > >> more
>> > >> > >>> than
>> > >> > >>> welcome to start perusing :)
>> > >> > >>>
>> > >> > >>>
>> > >> > >>> Out of interest, this could come in after 2.0 Josh? Any 2.0
>> > specific
>> > >> > >> needs
>> > >> > >> to make this work?
>> > >> > >>
>> > >> > >> Meantime, updated the 2.0 doc 1.
>> > >> > >>
>> > >> > >> Thanks Josh,
>> > >> > >> St.Ack
>> > >> > >>
>> > >> > >> 1.
>> > >> > >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> > >> > >> Eu_ktczrlKHK8N4SZzs/edit#
>> > >> > >>
>> > >> > >>
>> > >> > > Nope, no need to block 2.0 on this one (given the other, related
>> > >> > chatter).
>> > >> > > Would be nice to get it in, but I completely understand if it
>> slips
>> > :)
>> > >> > >
>> > >> > > Thanks for updating the doc for me!
>> > >> > >
>> > >> >
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Best regards,
>> > >>
>> > >>    - Andy
>> > >>
>> > >> If you are given a choice, you believe you have acted freely. -
>> Raymond
>> > >> Teller (via Peter Watts)
>> > >>
>> > >
>> > >
>> >
>>
>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Thu, Nov 16, 2017 at 8:08 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:

> HBASE-18946 Stochastic load balancer assigns replica regions to the same RS
>
> Am on this today and probably will put up another patch.
>
> Regards
> Ram
>
> Thanks Ram,
St.Ack



> On Fri, Nov 17, 2017 at 5:18 AM, Stack <st...@duboce.net> wrote:
>
> > hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
> > remaining features and apply final polish to API. There will be a beta-2
> > but it is about upgrade/rolling-upgrade and bug fixes ONLY).
> >
> > Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1
> list
> > this morning. See here [1].
> >
> > There are about ~12 issues in progress with most of these about to land.
> > There are 37 TODO. Many of these are tests we need to run, some are
> related
> > to the backup/restore, but a good few are meaty w/o assignees.
> >
> > The awkward outstanding ones as I see it are the below:
> >
> > HBASE-18946 Stochastic load balancer assigns replica regions to the same
> RS
> > HBASE-17204 Make L2 off heap cache default ON
> > HBASE-19112 Suspect methods on Cell to be deprecated
> > HBASE-19147 All branch-2 unit tests pass
> >
> > We need to make progress on the above or punt on them (can't punt on the
> > last one though).
> >
> > Any ideas on what configs we should update in hbase2? Dump ideas into:
> > HBASE-19148 Edit of default configuration
> >
> > Still hoping for an early December beta-1 RC. beta-2 hopefully will be
> > close behind.
> >
> > Comments? Thoughts?
> >
> > Thanks all,
> > S
> >
> > 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
> >
> > On Tue, Oct 31, 2017 at 2:56 PM, Stack <st...@duboce.net> wrote:
> >
> > >
> > >
> > > On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:
> > >
> > >> Hoping to keep momentum going from our Stack working on alpha4, I
> tried
> > to
> > >> take a stab at triaging some of the open beta-1 issues.
> > >>
> > >> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done
> > sooner
> > >> then I'm happy to see it pulled back in. Trying to balance optimism
> with
> > >> realism here, and knowing that documentation unfortunately often gets
> > >> pushed to the back-burner.
> > >>
> > >> Also, I poked some folks on unassigned issues that they've filed for
> > >> beta-1, especially in the last few days. If issues don't have an owner
> > >> they
> > >> are unlikely to get worked. I chatted with stack and he agreed to take
> > on
> > >> some of the tasks, but there's a lot of surface area to cover.
> > >>
> > >> If you you're working on issues that are critical for beta-1, please
> > mark
> > >> them as such. Then the rest of the community will know to help
> > prioritize
> > >> feedback and reviews there.
> > >>
> > >> Do we have a general theme for the betas like we did with the alphas?
> > >>
> > >> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
> > >> well? Continue to work on tests throughout?
> > >>
> > >>
> > > Thanks Mike for helping to kick off the beta-1 train.
> > >
> > > Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what
> we
> > > are going to ship (beta-2 is rolling upgrade and any minor items turned
> > up
> > > in testing/burn-in).
> > >
> > > Thanks,
> > > S
> > >
> > >
> > >
> > >> Mike
> > >>
> > >> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org>
> wrote:
> > >>
> > >> > +1 go from my POV.
> > >> >
> > >> >
> > >> > On 10/31/17 10:07 AM, Stack wrote:
> > >> >
> > >> >> I want to push an alpha-4 today. A few items didn't make it
> > >> (HBASE-19092).
> > >> >> They need more time. We'll pull them in for beta-1. CP API is
> > basically
> > >> >> done. There may be some changes for beta-1 but hopefully only
> changes
> > >> >> informed by experience trying to port an existing Coprocessor to
> > >> hbase2.
> > >> >>
> > >> >> Shout if there is anything that needs to make alpha-4.
> > >> >>
> > >> >> Thanks,
> > >> >> St.Ack
> > >> >>
> > >> >>
> > >> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
> > >> wrote:
> > >> >>
> > >> >> Yup, that was going to be my plan, Mike!
> > >> >>>
> > >> >>> Making a pass now, and will check back later tonight again. I see
> > >> others
> > >> >>> have already done some work today on this front.
> > >> >>>
> > >> >>>
> > >> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
> > >> >>>
> > >> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know
> how
> > >> to do
> > >> >>>> it directly on the jenkins job, so you don't have to bother with
> > JIRA
> > >> >>>> uploads)
> > >> >>>>
> > >> >>>> If you're busy, then I can make time tomorrow or Sunday to kick
> off
> > >> >>>> jobs,
> > >> >>>> but I want to make sure we're not duplicating effort and jenkins
> > >> cycles.
> > >> >>>>
> > >> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
> > >> wrote:
> > >> >>>>
> > >> >>>> My turn to bump ;)
> > >> >>>>
> > >> >>>>>
> > >> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that
> > >> remain
> > >> >>>>> needing some more work. The rest are just awaiting a good QA
> run.
> > >> >>>>>
> > >> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over
> > the
> > >> >>>>> weekend, bump them along as necessary, and get them committed.
> > >> Would be
> > >> >>>>> great to be able get a vote up on Monday.
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> On 10/24/17 6:03 PM, Stack wrote:
> > >> >>>>>
> > >> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back
> and
> > >> forth
> > >> >>>>>
> > >> >>>>>> what remains to be done. Surfacing our thoughts here so you all
> > >> clued
> > >> >>>>>> in....Or if you think otherwise, please speak up.
> > >> >>>>>>
> > >> >>>>>> We have ~13 issues to land:
> > >> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> 12341594
> > >> About
> > >> >>>>>> two
> > >> >>>>>> are meta-issues that are about process which leaves 11.
> > >> >>>>>>
> > >> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
> > >> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because
> some
> > >> >>>>>> changed
> > >> >>>>>> API/semantic that we need to get out earlier rather than later.
> > >> >>>>>>
> > >> >>>>>> Once the above is merged, HBASE-13346, change of Filter method
> > >> names
> > >> >>>>>> to
> > >> >>>>>> mention Cell instead of KeyValue can land.
> > >> >>>>>>
> > >> >>>>>> HBASE-199048 needs a review (Anoop will probably do it),
> removing
> > >> >>>>>> IA.Private objects as params to MasterObserver... Hopefully
> this
> > >> goes
> > >> >>>>>> in
> > >> >>>>>> soon.
> > >> >>>>>>
> > >> >>>>>> Duo is hard at work on trackers for flush and compaction for
> CPs
> > >> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for
> > >> Tephra)?
> > >> >>>>>>
> > >> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate
> > >> after
> > >> >>>>>> Duo
> > >> >>>>>> is
> > >> >>>>>> done w/ his work above.
> > >> >>>>>>
> > >> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity
> > after
> > >> >>>>>> all
> > >> >>>>>> the
> > >> >>>>>> purges allowing CPs do direct calls against Regions in same
> Host.
> > >> >>>>>>
> > >> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> > >> >>>>>>
> > >> >>>>>> Another day or two?
> > >> >>>>>>
> > >> >>>>>> St.Ack
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net>
> wrote:
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <
> elserj@apache.org>
> > >> >>>>>> wrote:
> > >> >>>>>>
> > >> >>>>>>>
> > >> >>>>>>> +1
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>> I was trying to work on helping out on the outstanding
> alpha-4
> > >> stuff
> > >> >>>>>>>> last
> > >> >>>>>>>> week -- will be continuing to try to do the same this week.
> > >> >>>>>>>>
> > >> >>>>>>>> If you need any help, Stack, or if others need reviews where
> I
> > >> >>>>>>>> haven't
> > >> >>>>>>>> noticed on my own: feel free to @mention me.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are
> > being
> > >> >>>>>>>>
> > >> >>>>>>>> actively
> > >> >>>>>>> worked on. If you get a moment, reviews by you (or anyone
> else)
> > >> helps
> > >> >>>>>>> move
> > >> >>>>>>> the process along.
> > >> >>>>>>>
> > >> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
> > >> >>>>>>> improvements.
> > >> >>>>>>> Then HBASE-13346 can go in.
> > >> >>>>>>>
> > >> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks like
> > >> that
> > >> >>>>>>> will
> > >> >>>>>>> be addressed by other alpha-4s about to land.
> > >> >>>>>>>
> > >> >>>>>>> St.Ack
> > >> >>>>>>> TODOs: https://issues.apache.org/jira
> > >> /projects/HBASE/versions/12341
> > >> >>>>>>> 594
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>> (Reviving this thread)
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that
> > has
> > >> >>>>>>>>> the
> > >> >>>>>>>>> refactor of the Coprocessor API shutting down access to
> > >> internals
> > >> >>>>>>>>> marked
> > >> >>>>>>>>> InterfaceAudience.Private.
> > >> >>>>>>>>>
> > >> >>>>>>>>> The outstanding list is here:
> > >> >>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> > 12341594
> > >> >>>>>>>>>
> > >> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
> > >> >>>>>>>>>
> > >> >>>>>>>>> If issue, talk out loud on this thread. If you need a review
> > to
> > >> >>>>>>>>> land
> > >> >>>>>>>>> an
> > >> >>>>>>>>> item, shout on the issue and here; we'll help you out.
> > >> >>>>>>>>>
> > >> >>>>>>>>> As is, items are coming along nicely I'd say. We need to
> merge
> > >> the
> > >> >>>>>>>>> filter
> > >> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> > >> >>>>>>>>>
> > >> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and
> > help
> > >> >>>>>>>>> them
> > >> >>>>>>>>> test
> > >> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence
> > our
> > >> >>>>>>>>> downstreamers know what to expect (or we discover what we
> > missed
> > >> >>>>>>>>> BEFORE
> > >> >>>>>>>>> we
> > >> >>>>>>>>> beta-1).
> > >> >>>>>>>>>
> > >> >>>>>>>>> Thanks for time,
> > >> >>>>>>>>> S
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net>
> > wrote:
> > >> >>>>>>>>>
> > >> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That
> > >> should
> > >> >>>>>>>>> be
> > >> >>>>>>>>>
> > >> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to
> be
> > >> done.
> > >> >>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled
> for
> > >> >>>>>>>>>> alpha3
> > >> >>>>>>>>>> but
> > >> >>>>>>>>>> it is plain that more time is needed (in spite of valiant
> > >> effort
> > >> >>>>>>>>>> so
> > >> >>>>>>>>>> far
> > >> >>>>>>>>>> by
> > >> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a
> 2.0.0-alpha-4
> > >> whose
> > >> >>>>>>>>>> theme is
> > >> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by
> > the
> > >> >>>>>>>>>> following
> > >> >>>>>>>>>> week.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> We should then be ready for beta (beta == no new features,
> no
> > >> API
> > >> >>>>>>>>>> changes,
> > >> >>>>>>>>>> just fixes).
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Thanks,
> > >> >>>>>>>>>> St.Ack
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
> > >> wrote:
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please
> > vote
> > >> on
> > >> >>>>>>>>>> it.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I
> hope
> > to
> > >> >>>>>>>>>>> get a
> > >> >>>>>>>>>>> release out in the next week or so.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If
> folks
> > >> are
> > >> >>>>>>>>>>> interested in helping out, below are the items I think we
> > need
> > >> >>>>>>>>>>> done
> > >> >>>>>>>>>>> for
> > >> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API
> > possibly
> > >> >>>>>>>>>>> altering
> > >> >>>>>>>>>>> items, and are absent those JIRAs that are making active
> > >> >>>>>>>>>>> progress,
> > >> >>>>>>>>>>> i.e. the
> > >> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed
> that
> > >> >>>>>>>>>>> needs
> > >> >>>>>>>>>>> doing is
> > >> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between
> > branch-1
> > >> >>>>>>>>>>> and
> > >> >>>>>>>>>>> branch-2
> > >> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches
> > >> only
> > >> >>>>>>>>>>> do
> > >> >>>>>>>>>>> it
> > >> >>>>>>>>>>> between branch-1 and branch-2.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing
> Public
> > /
> > >> >>>>>>>>>>> Private
> > >> >>>>>>>>>>> interfaces
> > >> >>>>>>>>>>> This one is almost done. It could do with a finish,
> > attention
> > >> to
> > >> >>>>>>>>>>> the
> > >> >>>>>>>>>>> items in last comment, and then our codebase could do with
> > >> >>>>>>>>>>> another
> > >> >>>>>>>>>>> sweep
> > >> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in
> > since
> > >> >>>>>>>>>>> the
> > >> >>>>>>>>>>> pass
> > >> >>>>>>>>>>> that was the basis of this issue.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
> > >> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
> > >> forced
> > >> >>>>>>>>>>> by
> > >> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> > >> >>>>>>>>>>> Service'
> > >> >>>>>>>>>>> but if
> > >> >>>>>>>>>>> anyone else is interested, be my guest.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
> > >> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked
> on.
> > >> Needs
> > >> >>>>>>>>>>> finishing.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods
> in
> > >> Admin
> > >> >>>>>>>>>>> and
> > >> >>>>>>>>>>> cleanup
> > >> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
> > >> >>>>>>>>>>> AsyncClient
> > >> >>>>>>>>>>> and
> > >> >>>>>>>>>>> make sure symmetry.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> > >> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but
> > >> needs to
> > >> >>>>>>>>>>> be
> > >> >>>>>>>>>>> done.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would
> be
> > >> good
> > >> >>>>>>>>>>> to
> > >> >>>>>>>>>>> try
> > >> >>>>>>>>>>> it and figure what needs fixup (as said above, according
> to
> > >> trial
> > >> >>>>>>>>>>> by
> > >> >>>>>>>>>>> Sean,
> > >> >>>>>>>>>>> we might not be too bad here):
> > >> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state
> talking
> > to
> > >> >>>>>>>>>>> 2.0
> > >> >>>>>>>>>>> cluster
> > >> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
> > >> Master
> > >> >>>>>>>>>>> and
> > >> >>>>>>>>>>> 1.x
> > >> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-17442 Move most of the replication related classes
> > to
> > >> >>>>>>>>>>> hbase-server package
> > >> >>>>>>>>>>> The above would be good to do generally but it may make
> for
> > >> >>>>>>>>>>> ripples
> > >> >>>>>>>>>>> in
> > >> >>>>>>>>>>> API so would be good to do now.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> > >> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid
> adding
> > >> two
> > >> >>>>>>>>>>> new
> > >> >>>>>>>>>>> types
> > >> >>>>>>>>>>> to our API, two types that are nought but curtailed,
> > read-only
> > >> >>>>>>>>>>> views
> > >> >>>>>>>>>>> on
> > >> >>>>>>>>>>> internals. Input if you have time appreciated.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate
> > to
> > >> a
> > >> >>>>>>>>>>> hbase2
> > >> >>>>>>>>>>> cluster; verify
> > >> >>>>>>>>>>> Esteban is looking at this one
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in
> core
> > >> >>>>>>>>>>> * HBASE-17143 Scan improvement
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
> > >> >>>>>>>>>>> implementations
> > >> >>>>>>>>>>> marked audience Private and returns implementations rather
> > >> than
> > >> >>>>>>>>>>> Interfaces.
> > >> >>>>>>>>>>> In a few locations, we allow returning an alternate
> > >> >>>>>>>>>>> implementation
> > >> >>>>>>>>>>> altogether which is probably something we don't want a CP
> > >> doing.
> > >> >>>>>>>>>>> To
> > >> >>>>>>>>>>> that
> > >> >>>>>>>>>>> end, the following issues started by Duo and Anoop need to
> > be
> > >> >>>>>>>>>>> taken
> > >> >>>>>>>>>>> to
> > >> >>>>>>>>>>> the
> > >> >>>>>>>>>>> finish line; ideally they'd have an owner:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
> > >> release <=
> > >> >>>>>>>>>>> The
> > >> >>>>>>>>>>> umbrella issue.
> > >> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for
> CP
> > >> >>>>>>>>>>> expose
> > >> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
> > >> MasterObserver
> > >> >>>>>>>>>>> and
> > >> >>>>>>>>>>> RegionServerObserver
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Nice-to-haves:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
> > >> remove
> > >> >>>>>>>>>>> unused
> > >> >>>>>>>>>>> TimeRange constructors
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
> > >> >>>>>>>>>>> references
> > >> >>>>>>>>>>> existing in the code
> > >> >>>>>>>>>>> This is the end of an old long-running project moving up
> on
> > to
> > >> >>>>>>>>>>> Cell
> > >> >>>>>>>>>>> Interface. We think it is done but for a few little items
> > >> >>>>>>>>>>> (deprecate
> > >> >>>>>>>>>>> KV
> > >> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is
> > >> indeterminate;
> > >> >>>>>>>>>>> needs
> > >> >>>>>>>>>>> fixing
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
> > >> >>>>>>>>>>> * HBASE-14997
> > >> >>>>>>>>>>> Move compareOp and Comparators out of filter to client
> > package
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> What about:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
> > >> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> If interested in any of the above and you need a legup,
> just
> > >> ask
> > >> >>>>>>>>>>> in
> > >> >>>>>>>>>>> the
> > >> >>>>>>>>>>> issue and I'll be by....
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Thanks,
> > >> >>>>>>>>>>> St.Ack
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <stack@duboce.net
> >
> > >> >>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Heads-up:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release
> Candidate.
> > >> >>>>>>>>>>>> Theme
> > >> >>>>>>>>>>>> is
> > >> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
> > >> (guava,
> > >> >>>>>>>>>>>> netty,
> > >> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
> > >> >>>>>>>>>>>> master-carries-no-regions
> > >> >>>>>>>>>>>> by default.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?).
> Its
> > >> theme
> > >> >>>>>>>>>>>> will
> > >> >>>>>>>>>>>> be
> > >> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we
> are
> > >> not
> > >> >>>>>>>>>>>> looking too
> > >> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have
> hbase-1
> > >> >>>>>>>>>>>> client
> > >> >>>>>>>>>>>> running
> > >> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor
> Interface
> > >> >>>>>>>>>>>> revamp
> > >> >>>>>>>>>>>> should be
> > >> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
> > >> >>>>>>>>>>>> Implementations, and
> > >> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
> > >> >>>>>>>>>>>> InterfaceAudience).
> > >> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API;
> > e.g.
> > >> >>>>>>>>>>>> guava
> > >> >>>>>>>>>>>> 0.12
> > >> >>>>>>>>>>>> Service showing through in our replication API and
> > protobufs
> > >> in
> > >> >>>>>>>>>>>> Admin
> > >> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of
> > outreach
> > >> to
> > >> >>>>>>>>>>>> make
> > >> >>>>>>>>>>>> sure
> > >> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> Beta1 in mid-September?
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
> > >> >>>>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> > 12327
> > >> 188
> > >> >>>>>>>>>>>> Edit
> > >> >>>>>>>>>>>> as
> > >> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not
> get
> > >> any
> > >> >>>>>>>>>>>> attention
> > >> >>>>>>>>>>>> in next month or so.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll
> leave
> > >> them
> > >> >>>>>>>>>>>> as
> > >> >>>>>>>>>>>> is
> > >> >>>>>>>>>>>> another while but I'll boot them soon.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> While I have your attention:
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving
> > >> hbase
> > >> >>>>>>>>>>>> thrift
> > >> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy
> > >> enough
> > >> >>>>>>>>>>>> if
> > >> >>>>>>>>>>>> folks
> > >> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> > >> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
> > >> first
> > >> >>>>>>>>>>>> (0.98?).
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> St.Ack
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
> > >> wrote:
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
> > >> elserj@apache.org>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
> > >> >>>>>>>>>>>>>> elserj@apache.org>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> ...
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would
> > make
> > >> the
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> feature a
> > >> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I
> > >> have
> > >> >>>>>>>>>>>>>>>> no
> > >> >>>>>>>>>>>>>>>> reason
> > >> >>>>>>>>>>>>>>>> to be
> > >> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature
> in
> > >> terms
> > >> >>>>>>>>>>>>>>>> of
> > >> >>>>>>>>>>>>>>>> how
> > >> >>>>>>>>>>>>>>>> much it
> > >> >>>>>>>>>>>>>>>> is covered via testing.
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so,
> +1
> > >> Josh.
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See
> > >> 'Space
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> Quota' in
> > >> >>>>>>>>>>>>>>> here:
> > >> >>>>>>>>>>>>>>> https://docs.google.com/docume
> > >> nt/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> > >> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > >> >>>>>>>>>>>>>>> Does this little section need an update Josh?
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> Thanks,
> > >> >>>>>>>>>>>>>>> S
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage
> > and
> > >> >>>>>>>>>>>>>>> some
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> docs
> > >> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my
> > plate
> > >> if
> > >> >>>>>>>>>>>>>> you're good
> > >> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is against
> > it
> > >> :)
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
> > >> before
> > >> >>>>>>>>>>>>>> we
> > >> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll
> > double
> > >> >>>>>>>>>>>>>> check
> > >> >>>>>>>>>>>>>> and update
> > >> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> Thanks Josh,
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> S
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>
> > >> >>>>>>
> > >> >>>>
> > >> >>
> > >>
> > >
> > >
> >
>

Re: Moving 2.0 forward

[ramkrishna vasudevan <ra...@gmail.com>]

HBASE-18946 Stochastic load balancer assigns replica regions to the same RS

Am on this today and probably will put up another patch.

Regards
Ram

On Fri, Nov 17, 2017 at 5:18 AM, Stack <st...@duboce.net> wrote:

> hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
> remaining features and apply final polish to API. There will be a beta-2
> but it is about upgrade/rolling-upgrade and bug fixes ONLY).
>
> Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1 list
> this morning. See here [1].
>
> There are about ~12 issues in progress with most of these about to land.
> There are 37 TODO. Many of these are tests we need to run, some are related
> to the backup/restore, but a good few are meaty w/o assignees.
>
> The awkward outstanding ones as I see it are the below:
>
> HBASE-18946 Stochastic load balancer assigns replica regions to the same RS
> HBASE-17204 Make L2 off heap cache default ON
> HBASE-19112 Suspect methods on Cell to be deprecated
> HBASE-19147 All branch-2 unit tests pass
>
> We need to make progress on the above or punt on them (can't punt on the
> last one though).
>
> Any ideas on what configs we should update in hbase2? Dump ideas into:
> HBASE-19148 Edit of default configuration
>
> Still hoping for an early December beta-1 RC. beta-2 hopefully will be
> close behind.
>
> Comments? Thoughts?
>
> Thanks all,
> S
>
> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>
> On Tue, Oct 31, 2017 at 2:56 PM, Stack <st...@duboce.net> wrote:
>
> >
> >
> > On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:
> >
> >> Hoping to keep momentum going from our Stack working on alpha4, I tried
> to
> >> take a stab at triaging some of the open beta-1 issues.
> >>
> >> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done
> sooner
> >> then I'm happy to see it pulled back in. Trying to balance optimism with
> >> realism here, and knowing that documentation unfortunately often gets
> >> pushed to the back-burner.
> >>
> >> Also, I poked some folks on unassigned issues that they've filed for
> >> beta-1, especially in the last few days. If issues don't have an owner
> >> they
> >> are unlikely to get worked. I chatted with stack and he agreed to take
> on
> >> some of the tasks, but there's a lot of surface area to cover.
> >>
> >> If you you're working on issues that are critical for beta-1, please
> mark
> >> them as such. Then the rest of the community will know to help
> prioritize
> >> feedback and reviews there.
> >>
> >> Do we have a general theme for the betas like we did with the alphas?
> >>
> >> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
> >> well? Continue to work on tests throughout?
> >>
> >>
> > Thanks Mike for helping to kick off the beta-1 train.
> >
> > Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what we
> > are going to ship (beta-2 is rolling upgrade and any minor items turned
> up
> > in testing/burn-in).
> >
> > Thanks,
> > S
> >
> >
> >
> >> Mike
> >>
> >> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org> wrote:
> >>
> >> > +1 go from my POV.
> >> >
> >> >
> >> > On 10/31/17 10:07 AM, Stack wrote:
> >> >
> >> >> I want to push an alpha-4 today. A few items didn't make it
> >> (HBASE-19092).
> >> >> They need more time. We'll pull them in for beta-1. CP API is
> basically
> >> >> done. There may be some changes for beta-1 but hopefully only changes
> >> >> informed by experience trying to port an existing Coprocessor to
> >> hbase2.
> >> >>
> >> >> Shout if there is anything that needs to make alpha-4.
> >> >>
> >> >> Thanks,
> >> >> St.Ack
> >> >>
> >> >>
> >> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
> >> wrote:
> >> >>
> >> >> Yup, that was going to be my plan, Mike!
> >> >>>
> >> >>> Making a pass now, and will check back later tonight again. I see
> >> others
> >> >>> have already done some work today on this front.
> >> >>>
> >> >>>
> >> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
> >> >>>
> >> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how
> >> to do
> >> >>>> it directly on the jenkins job, so you don't have to bother with
> JIRA
> >> >>>> uploads)
> >> >>>>
> >> >>>> If you're busy, then I can make time tomorrow or Sunday to kick off
> >> >>>> jobs,
> >> >>>> but I want to make sure we're not duplicating effort and jenkins
> >> cycles.
> >> >>>>
> >> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
> >> wrote:
> >> >>>>
> >> >>>> My turn to bump ;)
> >> >>>>
> >> >>>>>
> >> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that
> >> remain
> >> >>>>> needing some more work. The rest are just awaiting a good QA run.
> >> >>>>>
> >> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over
> the
> >> >>>>> weekend, bump them along as necessary, and get them committed.
> >> Would be
> >> >>>>> great to be able get a vote up on Monday.
> >> >>>>>
> >> >>>>>
> >> >>>>> On 10/24/17 6:03 PM, Stack wrote:
> >> >>>>>
> >> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back and
> >> forth
> >> >>>>>
> >> >>>>>> what remains to be done. Surfacing our thoughts here so you all
> >> clued
> >> >>>>>> in....Or if you think otherwise, please speak up.
> >> >>>>>>
> >> >>>>>> We have ~13 issues to land:
> >> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >> About
> >> >>>>>> two
> >> >>>>>> are meta-issues that are about process which leaves 11.
> >> >>>>>>
> >> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
> >> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
> >> >>>>>> changed
> >> >>>>>> API/semantic that we need to get out earlier rather than later.
> >> >>>>>>
> >> >>>>>> Once the above is merged, HBASE-13346, change of Filter method
> >> names
> >> >>>>>> to
> >> >>>>>> mention Cell instead of KeyValue can land.
> >> >>>>>>
> >> >>>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
> >> >>>>>> IA.Private objects as params to MasterObserver... Hopefully this
> >> goes
> >> >>>>>> in
> >> >>>>>> soon.
> >> >>>>>>
> >> >>>>>> Duo is hard at work on trackers for flush and compaction for CPs
> >> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for
> >> Tephra)?
> >> >>>>>>
> >> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate
> >> after
> >> >>>>>> Duo
> >> >>>>>> is
> >> >>>>>> done w/ his work above.
> >> >>>>>>
> >> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity
> after
> >> >>>>>> all
> >> >>>>>> the
> >> >>>>>> purges allowing CPs do direct calls against Regions in same Host.
> >> >>>>>>
> >> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> >> >>>>>>
> >> >>>>>> Another day or two?
> >> >>>>>>
> >> >>>>>> St.Ack
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
> >> >>>>>> wrote:
> >> >>>>>>
> >> >>>>>>>
> >> >>>>>>> +1
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>> I was trying to work on helping out on the outstanding alpha-4
> >> stuff
> >> >>>>>>>> last
> >> >>>>>>>> week -- will be continuing to try to do the same this week.
> >> >>>>>>>>
> >> >>>>>>>> If you need any help, Stack, or if others need reviews where I
> >> >>>>>>>> haven't
> >> >>>>>>>> noticed on my own: feel free to @mention me.
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are
> being
> >> >>>>>>>>
> >> >>>>>>>> actively
> >> >>>>>>> worked on. If you get a moment, reviews by you (or anyone else)
> >> helps
> >> >>>>>>> move
> >> >>>>>>> the process along.
> >> >>>>>>>
> >> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
> >> >>>>>>> improvements.
> >> >>>>>>> Then HBASE-13346 can go in.
> >> >>>>>>>
> >> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks like
> >> that
> >> >>>>>>> will
> >> >>>>>>> be addressed by other alpha-4s about to land.
> >> >>>>>>>
> >> >>>>>>> St.Ack
> >> >>>>>>> TODOs: https://issues.apache.org/jira
> >> /projects/HBASE/versions/12341
> >> >>>>>>> 594
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>> (Reviving this thread)
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that
> has
> >> >>>>>>>>> the
> >> >>>>>>>>> refactor of the Coprocessor API shutting down access to
> >> internals
> >> >>>>>>>>> marked
> >> >>>>>>>>> InterfaceAudience.Private.
> >> >>>>>>>>>
> >> >>>>>>>>> The outstanding list is here:
> >> >>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> 12341594
> >> >>>>>>>>>
> >> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
> >> >>>>>>>>>
> >> >>>>>>>>> If issue, talk out loud on this thread. If you need a review
> to
> >> >>>>>>>>> land
> >> >>>>>>>>> an
> >> >>>>>>>>> item, shout on the issue and here; we'll help you out.
> >> >>>>>>>>>
> >> >>>>>>>>> As is, items are coming along nicely I'd say. We need to merge
> >> the
> >> >>>>>>>>> filter
> >> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> >> >>>>>>>>>
> >> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and
> help
> >> >>>>>>>>> them
> >> >>>>>>>>> test
> >> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence
> our
> >> >>>>>>>>> downstreamers know what to expect (or we discover what we
> missed
> >> >>>>>>>>> BEFORE
> >> >>>>>>>>> we
> >> >>>>>>>>> beta-1).
> >> >>>>>>>>>
> >> >>>>>>>>> Thanks for time,
> >> >>>>>>>>> S
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net>
> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That
> >> should
> >> >>>>>>>>> be
> >> >>>>>>>>>
> >> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to be
> >> done.
> >> >>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for
> >> >>>>>>>>>> alpha3
> >> >>>>>>>>>> but
> >> >>>>>>>>>> it is plain that more time is needed (in spite of valiant
> >> effort
> >> >>>>>>>>>> so
> >> >>>>>>>>>> far
> >> >>>>>>>>>> by
> >> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
> >> whose
> >> >>>>>>>>>> theme is
> >> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by
> the
> >> >>>>>>>>>> following
> >> >>>>>>>>>> week.
> >> >>>>>>>>>>
> >> >>>>>>>>>> We should then be ready for beta (beta == no new features, no
> >> API
> >> >>>>>>>>>> changes,
> >> >>>>>>>>>> just fixes).
> >> >>>>>>>>>>
> >> >>>>>>>>>> Thanks,
> >> >>>>>>>>>> St.Ack
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
> >> wrote:
> >> >>>>>>>>>>
> >> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please
> vote
> >> on
> >> >>>>>>>>>> it.
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope
> to
> >> >>>>>>>>>>> get a
> >> >>>>>>>>>>> release out in the next week or so.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks
> >> are
> >> >>>>>>>>>>> interested in helping out, below are the items I think we
> need
> >> >>>>>>>>>>> done
> >> >>>>>>>>>>> for
> >> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API
> possibly
> >> >>>>>>>>>>> altering
> >> >>>>>>>>>>> items, and are absent those JIRAs that are making active
> >> >>>>>>>>>>> progress,
> >> >>>>>>>>>>> i.e. the
> >> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
> >> >>>>>>>>>>> needs
> >> >>>>>>>>>>> doing is
> >> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between
> branch-1
> >> >>>>>>>>>>> and
> >> >>>>>>>>>>> branch-2
> >> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches
> >> only
> >> >>>>>>>>>>> do
> >> >>>>>>>>>>> it
> >> >>>>>>>>>>> between branch-1 and branch-2.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public
> /
> >> >>>>>>>>>>> Private
> >> >>>>>>>>>>> interfaces
> >> >>>>>>>>>>> This one is almost done. It could do with a finish,
> attention
> >> to
> >> >>>>>>>>>>> the
> >> >>>>>>>>>>> items in last comment, and then our codebase could do with
> >> >>>>>>>>>>> another
> >> >>>>>>>>>>> sweep
> >> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in
> since
> >> >>>>>>>>>>> the
> >> >>>>>>>>>>> pass
> >> >>>>>>>>>>> that was the basis of this issue.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
> >> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
> >> forced
> >> >>>>>>>>>>> by
> >> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> >> >>>>>>>>>>> Service'
> >> >>>>>>>>>>> but if
> >> >>>>>>>>>>> anyone else is interested, be my guest.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
> >> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked on.
> >> Needs
> >> >>>>>>>>>>> finishing.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
> >> Admin
> >> >>>>>>>>>>> and
> >> >>>>>>>>>>> cleanup
> >> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
> >> >>>>>>>>>>> AsyncClient
> >> >>>>>>>>>>> and
> >> >>>>>>>>>>> make sure symmetry.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> >> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but
> >> needs to
> >> >>>>>>>>>>> be
> >> >>>>>>>>>>> done.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would be
> >> good
> >> >>>>>>>>>>> to
> >> >>>>>>>>>>> try
> >> >>>>>>>>>>> it and figure what needs fixup (as said above, according to
> >> trial
> >> >>>>>>>>>>> by
> >> >>>>>>>>>>> Sean,
> >> >>>>>>>>>>> we might not be too bad here):
> >> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking
> to
> >> >>>>>>>>>>> 2.0
> >> >>>>>>>>>>> cluster
> >> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
> >> Master
> >> >>>>>>>>>>> and
> >> >>>>>>>>>>> 1.x
> >> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-17442 Move most of the replication related classes
> to
> >> >>>>>>>>>>> hbase-server package
> >> >>>>>>>>>>> The above would be good to do generally but it may make for
> >> >>>>>>>>>>> ripples
> >> >>>>>>>>>>> in
> >> >>>>>>>>>>> API so would be good to do now.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> >> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid adding
> >> two
> >> >>>>>>>>>>> new
> >> >>>>>>>>>>> types
> >> >>>>>>>>>>> to our API, two types that are nought but curtailed,
> read-only
> >> >>>>>>>>>>> views
> >> >>>>>>>>>>> on
> >> >>>>>>>>>>> internals. Input if you have time appreciated.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate
> to
> >> a
> >> >>>>>>>>>>> hbase2
> >> >>>>>>>>>>> cluster; verify
> >> >>>>>>>>>>> Esteban is looking at this one
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> >> >>>>>>>>>>> * HBASE-17143 Scan improvement
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
> >> >>>>>>>>>>> implementations
> >> >>>>>>>>>>> marked audience Private and returns implementations rather
> >> than
> >> >>>>>>>>>>> Interfaces.
> >> >>>>>>>>>>> In a few locations, we allow returning an alternate
> >> >>>>>>>>>>> implementation
> >> >>>>>>>>>>> altogether which is probably something we don't want a CP
> >> doing.
> >> >>>>>>>>>>> To
> >> >>>>>>>>>>> that
> >> >>>>>>>>>>> end, the following issues started by Duo and Anoop need to
> be
> >> >>>>>>>>>>> taken
> >> >>>>>>>>>>> to
> >> >>>>>>>>>>> the
> >> >>>>>>>>>>> finish line; ideally they'd have an owner:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
> >> release <=
> >> >>>>>>>>>>> The
> >> >>>>>>>>>>> umbrella issue.
> >> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
> >> >>>>>>>>>>> expose
> >> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
> >> MasterObserver
> >> >>>>>>>>>>> and
> >> >>>>>>>>>>> RegionServerObserver
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Nice-to-haves:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
> >> remove
> >> >>>>>>>>>>> unused
> >> >>>>>>>>>>> TimeRange constructors
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
> >> >>>>>>>>>>> references
> >> >>>>>>>>>>> existing in the code
> >> >>>>>>>>>>> This is the end of an old long-running project moving up on
> to
> >> >>>>>>>>>>> Cell
> >> >>>>>>>>>>> Interface. We think it is done but for a few little items
> >> >>>>>>>>>>> (deprecate
> >> >>>>>>>>>>> KV
> >> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is
> >> indeterminate;
> >> >>>>>>>>>>> needs
> >> >>>>>>>>>>> fixing
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
> >> >>>>>>>>>>> * HBASE-14997
> >> >>>>>>>>>>> Move compareOp and Comparators out of filter to client
> package
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> What about:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
> >> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> If interested in any of the above and you need a legup, just
> >> ask
> >> >>>>>>>>>>> in
> >> >>>>>>>>>>> the
> >> >>>>>>>>>>> issue and I'll be by....
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Thanks,
> >> >>>>>>>>>>> St.Ack
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
> >> >>>>>>>>>>> wrote:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Heads-up:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
> >> >>>>>>>>>>>> Theme
> >> >>>>>>>>>>>> is
> >> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
> >> (guava,
> >> >>>>>>>>>>>> netty,
> >> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
> >> >>>>>>>>>>>> master-carries-no-regions
> >> >>>>>>>>>>>> by default.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
> >> theme
> >> >>>>>>>>>>>> will
> >> >>>>>>>>>>>> be
> >> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we are
> >> not
> >> >>>>>>>>>>>> looking too
> >> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1
> >> >>>>>>>>>>>> client
> >> >>>>>>>>>>>> running
> >> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface
> >> >>>>>>>>>>>> revamp
> >> >>>>>>>>>>>> should be
> >> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
> >> >>>>>>>>>>>> Implementations, and
> >> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
> >> >>>>>>>>>>>> InterfaceAudience).
> >> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API;
> e.g.
> >> >>>>>>>>>>>> guava
> >> >>>>>>>>>>>> 0.12
> >> >>>>>>>>>>>> Service showing through in our replication API and
> protobufs
> >> in
> >> >>>>>>>>>>>> Admin
> >> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of
> outreach
> >> to
> >> >>>>>>>>>>>> make
> >> >>>>>>>>>>>> sure
> >> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Beta1 in mid-September?
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
> >> >>>>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> 12327
> >> 188
> >> >>>>>>>>>>>> Edit
> >> >>>>>>>>>>>> as
> >> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get
> >> any
> >> >>>>>>>>>>>> attention
> >> >>>>>>>>>>>> in next month or so.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave
> >> them
> >> >>>>>>>>>>>> as
> >> >>>>>>>>>>>> is
> >> >>>>>>>>>>>> another while but I'll boot them soon.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> While I have your attention:
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving
> >> hbase
> >> >>>>>>>>>>>> thrift
> >> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy
> >> enough
> >> >>>>>>>>>>>> if
> >> >>>>>>>>>>>> folks
> >> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> >> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
> >> first
> >> >>>>>>>>>>>> (0.98?).
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> St.Ack
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
> >> wrote:
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
> >> elserj@apache.org>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
> >> >>>>>>>>>>>>>> elserj@apache.org>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> ...
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would
> make
> >> the
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> feature a
> >> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I
> >> have
> >> >>>>>>>>>>>>>>>> no
> >> >>>>>>>>>>>>>>>> reason
> >> >>>>>>>>>>>>>>>> to be
> >> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in
> >> terms
> >> >>>>>>>>>>>>>>>> of
> >> >>>>>>>>>>>>>>>> how
> >> >>>>>>>>>>>>>>>> much it
> >> >>>>>>>>>>>>>>>> is covered via testing.
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
> >> Josh.
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See
> >> 'Space
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Quota' in
> >> >>>>>>>>>>>>>>> here:
> >> >>>>>>>>>>>>>>> https://docs.google.com/docume
> >> nt/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> >> >>>>>>>>>>>>>>> Does this little section need an update Josh?
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Thanks,
> >> >>>>>>>>>>>>>>> S
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage
> and
> >> >>>>>>>>>>>>>>> some
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> docs
> >> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my
> plate
> >> if
> >> >>>>>>>>>>>>>> you're good
> >> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is against
> it
> >> :)
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
> >> before
> >> >>>>>>>>>>>>>> we
> >> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll
> double
> >> >>>>>>>>>>>>>> check
> >> >>>>>>>>>>>>>> and update
> >> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> Thanks Josh,
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> S
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>
> >> >>>>>>
> >> >>>>
> >> >>
> >>
> >
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

In another thread in this list I flag the coming 2.0.0-beta-2. Look for an
RC in the next day or so.

After 2.0.0-beta-2 makes it out, I'll cut branch-2.0. Soon after will come
our first hbase-2.0.0 release candidate. Expect this within a week or so
after beta-2 after remaining blockers are addressed (At this stage,
blockers are mostly documentation). Scream if there is something that has
to make 2.0.0 that I may have missed, but at this point, only critical
bug-fixes or compatibility breakages will be entertained.

I just did a pass over the issues filed against hbase-2.0.0. I did a pretty
brutal edit, sparse on commentary mostly because the count of issues
against 2.0.0 was high. There were a fair amount filed against '2.0.0' just
because 2.0.0 used to be thought of as the end of the rainbow, an event
that was way off on the horizon that would never arrive[1]. I unscheduled
the bulk of these. Reschedule against 2.0.0 if I got it wrong.  A good
bunch were great ideas almost- or half-done or never started (HBASE-14055
RegionServer Refactor, HBASE-15531 Favored Nodes, Andy's security
improvements, etc.). These were also unscheduled. Others that were really
old without recent updates I resolved. A few had been realized via other
channels.

Here is the current list [2]. Help on blockers would be much appreciated.

Your 2.0.0RM

P.S.No scale testing, perf compare, compatibility, or exercise of new
features (at least to my knowledge) has been done. We could do with some
exercise along any of these dimensions if any of you have a few spare
cycles. Thanks.

1. hbase-2.0.0 still has this smell to it, but we do our best.
2.
https://issues.apache.org/jira/projects/HBASE/versions/12327188#release-report-tab-body


On Wed, Dec 27, 2017 at 9:54 AM, Stack <st...@duboce.net> wrote:

> Heads-up, I'm working on cutting a beta-1. Shout if there is something you
> think needs to make it in (The Chia-Ping Tsai cleanups look good). I'm
> mainly waiting on the hbase-thirdparty changes to go in. The rest can wait
> it.
>
> Thanks,
> S
>
>
> On Wed, Dec 13, 2017 at 11:28 PM, Stack <st...@duboce.net> wrote:
>
>> Update. See below.
>>
>> On Tue, Dec 5, 2017 at 10:43 AM, Stack <st...@duboce.net> wrote:
>>
>>> Update on hbase-2.0.0-beta-1. I'm going to put up a beta-1 RC before
>>> Christmas day.
>>>
>>>
>> Still aiming for an hbase2 beta-1 showing up in your christmas stocking.
>>
>>
>>> Here is the list [1]. We're doing pretty good. We've fixed a load since
>>> the last update including some nice cleanups  two weeks ago (e.g. undoing
>>> client dependency on curator/zk watching making the dependence read-only)
>>> but there are still a few hairy ones hanging out there.
>>>
>>>  * HBASE-18946 Stochastic load balancer assigns replica regions to the
>>> same RS
>>>    This one is proving a little tough. Ram has been banging his head on
>>> it. We'll be able to clear a bunch of test failures once this is done.
>>>
>>>
>> This should be going in tomorrow.
>>
>>
>>
>>>  * HBASE-19112 Suspect methods on Cell to be deprecated
>>>    Good discussion up on the JIRA. We need to be crystal clear around
>>> Cell and derivatives API and Audience when we ship 2.0.
>>>
>>>
>> Ram, Chia-Ping Tsai, and Anoop are doing a nice job here and it should be
>> done in next day or so.
>>
>>
>>
>>>  * HBASE-17204 Make L2 off heap cache default ON
>>>    We need to try this at least so offheaping can be default.
>>>
>>>
>> Chatted with Anoop and Ram. Because it is so late in the day and because
>> it critical that the user have a good experience out of the box -- which
>> requires time and testing -- we are going to punt this from 2.0. Will look
>> at it for 2.1 (or 3.0 if soon). 2.0 will still be carrying all of the boys
>> offheap read/write path goodness.
>>
>>
>>
>>> Making asyncfswal default is also ongoing, HBASE-16890
>>> <https://issues.apache.org/jira/browse/HBASE-16890>, making good
>>> progress.
>>>
>>>
>>
>> Good progress here. In perf testing asyncfswal makes us generally faster.
>> Duo making good progress on last few tests that are in the way of our
>> making this default for hbase2.
>>
>> A load of goodness has been landing these last few weeks. Keep up the
>> great work.
>>
>> Your hbase-2 RM.
>>
>>
>>
>>
>>> I punted HBASE-19147 "All branch-2 unit tests pass" out to beta-2. Our
>>> unit test story has gotten better and a few of us are actively working on
>>> flakies [2] but we may make a beta-1 w/o shutting down all failures.
>>>
>>> Speak up if you need help on an issue or if you think we are missing
>>> items from our list.
>>>
>>> Thanks for all the hard work,
>>>
>>> The hbase-2.0.0 RM.
>>>
>>>
>>> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>>> 2. https://builds.apache.org/job/HBASE-Find-Flaky-Tests/last
>>> SuccessfulBuild/artifact/dashboard.html
>>>
>>> On Thu, Nov 16, 2017 at 3:48 PM, Stack <st...@duboce.net> wrote:
>>>
>>>> hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
>>>> remaining features and apply final polish to API. There will be a beta-2
>>>> but it is about upgrade/rolling-upgrade and bug fixes ONLY).
>>>>
>>>> Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1
>>>> list this morning. See here [1].
>>>>
>>>> There are about ~12 issues in progress with most of these about to
>>>> land. There are 37 TODO. Many of these are tests we need to run, some are
>>>> related to the backup/restore, but a good few are meaty w/o assignees.
>>>>
>>>> The awkward outstanding ones as I see it are the below:
>>>>
>>>> HBASE-18946 Stochastic load balancer assigns replica regions to the
>>>> same RS
>>>> HBASE-17204 Make L2 off heap cache default ON
>>>> HBASE-19112 Suspect methods on Cell to be deprecated
>>>> HBASE-19147 All branch-2 unit tests pass
>>>>
>>>> We need to make progress on the above or punt on them (can't punt on
>>>> the last one though).
>>>>
>>>> Any ideas on what configs we should update in hbase2? Dump ideas into:
>>>> HBASE-19148 Edit of default configuration
>>>>
>>>> Still hoping for an early December beta-1 RC. beta-2 hopefully will be
>>>> close behind.
>>>>
>>>> Comments? Thoughts?
>>>>
>>>> Thanks all,
>>>> S
>>>>
>>>> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>>>>
>>>> On Tue, Oct 31, 2017 at 2:56 PM, Stack <st...@duboce.net> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:
>>>>>
>>>>>> Hoping to keep momentum going from our Stack working on alpha4, I
>>>>>> tried to
>>>>>> take a stab at triaging some of the open beta-1 issues.
>>>>>>
>>>>>> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done
>>>>>> sooner
>>>>>> then I'm happy to see it pulled back in. Trying to balance optimism
>>>>>> with
>>>>>> realism here, and knowing that documentation unfortunately often gets
>>>>>> pushed to the back-burner.
>>>>>>
>>>>>> Also, I poked some folks on unassigned issues that they've filed for
>>>>>> beta-1, especially in the last few days. If issues don't have an
>>>>>> owner they
>>>>>> are unlikely to get worked. I chatted with stack and he agreed to
>>>>>> take on
>>>>>> some of the tasks, but there's a lot of surface area to cover.
>>>>>>
>>>>>> If you you're working on issues that are critical for beta-1, please
>>>>>> mark
>>>>>> them as such. Then the rest of the community will know to help
>>>>>> prioritize
>>>>>> feedback and reviews there.
>>>>>>
>>>>>> Do we have a general theme for the betas like we did with the alphas?
>>>>>>
>>>>>> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
>>>>>> well? Continue to work on tests throughout?
>>>>>>
>>>>>>
>>>>> Thanks Mike for helping to kick off the beta-1 train.
>>>>>
>>>>> Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what
>>>>> we are going to ship (beta-2 is rolling upgrade and any minor items turned
>>>>> up in testing/burn-in).
>>>>>
>>>>> Thanks,
>>>>> S
>>>>>
>>>>>
>>>>>
>>>>>> Mike
>>>>>>
>>>>>> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> > +1 go from my POV.
>>>>>> >
>>>>>> >
>>>>>> > On 10/31/17 10:07 AM, Stack wrote:
>>>>>> >
>>>>>> >> I want to push an alpha-4 today. A few items didn't make it
>>>>>> (HBASE-19092).
>>>>>> >> They need more time. We'll pull them in for beta-1. CP API is
>>>>>> basically
>>>>>> >> done. There may be some changes for beta-1 but hopefully only
>>>>>> changes
>>>>>> >> informed by experience trying to port an existing Coprocessor to
>>>>>> hbase2.
>>>>>> >>
>>>>>> >> Shout if there is anything that needs to make alpha-4.
>>>>>> >>
>>>>>> >> Thanks,
>>>>>> >> St.Ack
>>>>>> >>
>>>>>> >>
>>>>>> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> Yup, that was going to be my plan, Mike!
>>>>>> >>>
>>>>>> >>> Making a pass now, and will check back later tonight again. I see
>>>>>> others
>>>>>> >>> have already done some work today on this front.
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
>>>>>> >>>
>>>>>> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know
>>>>>> how to do
>>>>>> >>>> it directly on the jenkins job, so you don't have to bother with
>>>>>> JIRA
>>>>>> >>>> uploads)
>>>>>> >>>>
>>>>>> >>>> If you're busy, then I can make time tomorrow or Sunday to kick
>>>>>> off
>>>>>> >>>> jobs,
>>>>>> >>>> but I want to make sure we're not duplicating effort and jenkins
>>>>>> cycles.
>>>>>> >>>>
>>>>>> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
>>>>>> wrote:
>>>>>> >>>>
>>>>>> >>>> My turn to bump ;)
>>>>>> >>>>
>>>>>> >>>>>
>>>>>> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues
>>>>>> that remain
>>>>>> >>>>> needing some more work. The rest are just awaiting a good QA
>>>>>> run.
>>>>>> >>>>>
>>>>>> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over
>>>>>> the
>>>>>> >>>>> weekend, bump them along as necessary, and get them committed.
>>>>>> Would be
>>>>>> >>>>> great to be able get a vote up on Monday.
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>> On 10/24/17 6:03 PM, Stack wrote:
>>>>>> >>>>>
>>>>>> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back
>>>>>> and forth
>>>>>> >>>>>
>>>>>> >>>>>> what remains to be done. Surfacing our thoughts here so you
>>>>>> all clued
>>>>>> >>>>>> in....Or if you think otherwise, please speak up.
>>>>>> >>>>>>
>>>>>> >>>>>> We have ~13 issues to land:
>>>>>> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341
>>>>>> 594 About
>>>>>> >>>>>> two
>>>>>> >>>>>> are meta-issues that are about process which leaves 11.
>>>>>> >>>>>>
>>>>>> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>>>>> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because
>>>>>> some
>>>>>> >>>>>> changed
>>>>>> >>>>>> API/semantic that we need to get out earlier rather than later.
>>>>>> >>>>>>
>>>>>> >>>>>> Once the above is merged, HBASE-13346, change of Filter method
>>>>>> names
>>>>>> >>>>>> to
>>>>>> >>>>>> mention Cell instead of KeyValue can land.
>>>>>> >>>>>>
>>>>>> >>>>>> HBASE-199048 needs a review (Anoop will probably do it),
>>>>>> removing
>>>>>> >>>>>> IA.Private objects as params to MasterObserver... Hopefully
>>>>>> this goes
>>>>>> >>>>>> in
>>>>>> >>>>>> soon.
>>>>>> >>>>>>
>>>>>> >>>>>> Duo is hard at work on trackers for flush and compaction for
>>>>>> CPs
>>>>>> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for
>>>>>> Tephra)?
>>>>>> >>>>>>
>>>>>> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate
>>>>>> after
>>>>>> >>>>>> Duo
>>>>>> >>>>>> is
>>>>>> >>>>>> done w/ his work above.
>>>>>> >>>>>>
>>>>>> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity
>>>>>> after
>>>>>> >>>>>> all
>>>>>> >>>>>> the
>>>>>> >>>>>> purges allowing CPs do direct calls against Regions in same
>>>>>> Host.
>>>>>> >>>>>>
>>>>>> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>>>>> >>>>>>
>>>>>> >>>>>> Another day or two?
>>>>>> >>>>>>
>>>>>> >>>>>> St.Ack
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net>
>>>>>> wrote:
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <
>>>>>> elserj@apache.org>
>>>>>> >>>>>> wrote:
>>>>>> >>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>> +1
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>> I was trying to work on helping out on the outstanding
>>>>>> alpha-4 stuff
>>>>>> >>>>>>>> last
>>>>>> >>>>>>>> week -- will be continuing to try to do the same this week.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> If you need any help, Stack, or if others need reviews where
>>>>>> I
>>>>>> >>>>>>>> haven't
>>>>>> >>>>>>>> noticed on my own: feel free to @mention me.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are
>>>>>> being
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> actively
>>>>>> >>>>>>> worked on. If you get a moment, reviews by you (or anyone
>>>>>> else) helps
>>>>>> >>>>>>> move
>>>>>> >>>>>>> the process along.
>>>>>> >>>>>>>
>>>>>> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
>>>>>> >>>>>>> improvements.
>>>>>> >>>>>>> Then HBASE-13346 can go in.
>>>>>> >>>>>>>
>>>>>> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks
>>>>>> like that
>>>>>> >>>>>>> will
>>>>>> >>>>>>> be addressed by other alpha-4s about to land.
>>>>>> >>>>>>>
>>>>>> >>>>>>> St.Ack
>>>>>> >>>>>>> TODOs: https://issues.apache.org/jira
>>>>>> /projects/HBASE/versions/12341
>>>>>> >>>>>>> 594
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>> (Reviving this thread)
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release
>>>>>> that has
>>>>>> >>>>>>>>> the
>>>>>> >>>>>>>>> refactor of the Coprocessor API shutting down access to
>>>>>> internals
>>>>>> >>>>>>>>> marked
>>>>>> >>>>>>>>> InterfaceAudience.Private.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> The outstanding list is here:
>>>>>> >>>>>>>>> https://issues.apache.org/jira
>>>>>> /projects/HBASE/versions/12341594
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> If issue, talk out loud on this thread. If you need a
>>>>>> review to
>>>>>> >>>>>>>>> land
>>>>>> >>>>>>>>> an
>>>>>> >>>>>>>>> item, shout on the issue and here; we'll help you out.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> As is, items are coming along nicely I'd say. We need to
>>>>>> merge the
>>>>>> >>>>>>>>> filter
>>>>>> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and
>>>>>> help
>>>>>> >>>>>>>>> them
>>>>>> >>>>>>>>> test
>>>>>> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have
>>>>>> confidence our
>>>>>> >>>>>>>>> downstreamers know what to expect (or we discover what we
>>>>>> missed
>>>>>> >>>>>>>>> BEFORE
>>>>>> >>>>>>>>> we
>>>>>> >>>>>>>>> beta-1).
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Thanks for time,
>>>>>> >>>>>>>>> S
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net>
>>>>>> wrote:
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night.
>>>>>> That should
>>>>>> >>>>>>>>> be
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to
>>>>>> be done.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled
>>>>>> for
>>>>>> >>>>>>>>>> alpha3
>>>>>> >>>>>>>>>> but
>>>>>> >>>>>>>>>> it is plain that more time is needed (in spite of valiant
>>>>>> effort
>>>>>> >>>>>>>>>> so
>>>>>> >>>>>>>>>> far
>>>>>> >>>>>>>>>> by
>>>>>> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a
>>>>>> 2.0.0-alpha-4 whose
>>>>>> >>>>>>>>>> theme is
>>>>>> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by
>>>>>> the
>>>>>> >>>>>>>>>> following
>>>>>> >>>>>>>>>> week.
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> We should then be ready for beta (beta == no new features,
>>>>>> no API
>>>>>> >>>>>>>>>> changes,
>>>>>> >>>>>>>>>> just fixes).
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> Thanks,
>>>>>> >>>>>>>>>> St.Ack
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
>>>>>> wrote:
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please
>>>>>> vote on
>>>>>> >>>>>>>>>> it.
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I
>>>>>> hope to
>>>>>> >>>>>>>>>>> get a
>>>>>> >>>>>>>>>>> release out in the next week or so.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If
>>>>>> folks are
>>>>>> >>>>>>>>>>> interested in helping out, below are the items I think we
>>>>>> need
>>>>>> >>>>>>>>>>> done
>>>>>> >>>>>>>>>>> for
>>>>>> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API
>>>>>> possibly
>>>>>> >>>>>>>>>>> altering
>>>>>> >>>>>>>>>>> items, and are absent those JIRAs that are making active
>>>>>> >>>>>>>>>>> progress,
>>>>>> >>>>>>>>>>> i.e. the
>>>>>> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed
>>>>>> that
>>>>>> >>>>>>>>>>> needs
>>>>>> >>>>>>>>>>> doing is
>>>>>> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between
>>>>>> branch-1
>>>>>> >>>>>>>>>>> and
>>>>>> >>>>>>>>>>> branch-2
>>>>>> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4
>>>>>> branches only
>>>>>> >>>>>>>>>>> do
>>>>>> >>>>>>>>>>> it
>>>>>> >>>>>>>>>>> between branch-1 and branch-2.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing
>>>>>> Public /
>>>>>> >>>>>>>>>>> Private
>>>>>> >>>>>>>>>>> interfaces
>>>>>> >>>>>>>>>>> This one is almost done. It could do with a finish,
>>>>>> attention to
>>>>>> >>>>>>>>>>> the
>>>>>> >>>>>>>>>>> items in last comment, and then our codebase could do with
>>>>>> >>>>>>>>>>> another
>>>>>> >>>>>>>>>>> sweep
>>>>>> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in
>>>>>> since
>>>>>> >>>>>>>>>>> the
>>>>>> >>>>>>>>>>> pass
>>>>>> >>>>>>>>>>> that was the basis of this issue.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
>>>>>> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
>>>>>> forced
>>>>>> >>>>>>>>>>> by
>>>>>> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
>>>>>> >>>>>>>>>>> Service'
>>>>>> >>>>>>>>>>> but if
>>>>>> >>>>>>>>>>> anyone else is interested, be my guest.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>>> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked
>>>>>> on. Needs
>>>>>> >>>>>>>>>>> finishing.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods
>>>>>> in Admin
>>>>>> >>>>>>>>>>> and
>>>>>> >>>>>>>>>>> cleanup
>>>>>> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>>>>>> >>>>>>>>>>> AsyncClient
>>>>>> >>>>>>>>>>> and
>>>>>> >>>>>>>>>>> make sure symmetry.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>>> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but
>>>>>> needs to
>>>>>> >>>>>>>>>>> be
>>>>>> >>>>>>>>>>> done.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would
>>>>>> be good
>>>>>> >>>>>>>>>>> to
>>>>>> >>>>>>>>>>> try
>>>>>> >>>>>>>>>>> it and figure what needs fixup (as said above, according
>>>>>> to trial
>>>>>> >>>>>>>>>>> by
>>>>>> >>>>>>>>>>> Sean,
>>>>>> >>>>>>>>>>> we might not be too bad here):
>>>>>> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state
>>>>>> talking to
>>>>>> >>>>>>>>>>> 2.0
>>>>>> >>>>>>>>>>> cluster
>>>>>> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
>>>>>> Master
>>>>>> >>>>>>>>>>> and
>>>>>> >>>>>>>>>>> 1.x
>>>>>> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-17442 Move most of the replication related
>>>>>> classes to
>>>>>> >>>>>>>>>>> hbase-server package
>>>>>> >>>>>>>>>>> The above would be good to do generally but it may make
>>>>>> for
>>>>>> >>>>>>>>>>> ripples
>>>>>> >>>>>>>>>>> in
>>>>>> >>>>>>>>>>> API so would be good to do now.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>>> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid
>>>>>> adding two
>>>>>> >>>>>>>>>>> new
>>>>>> >>>>>>>>>>> types
>>>>>> >>>>>>>>>>> to our API, two types that are nought but curtailed,
>>>>>> read-only
>>>>>> >>>>>>>>>>> views
>>>>>> >>>>>>>>>>> on
>>>>>> >>>>>>>>>>> internals. Input if you have time appreciated.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to
>>>>>> replicate to a
>>>>>> >>>>>>>>>>> hbase2
>>>>>> >>>>>>>>>>> cluster; verify
>>>>>> >>>>>>>>>>> Esteban is looking at this one
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in
>>>>>> core
>>>>>> >>>>>>>>>>> * HBASE-17143 Scan improvement
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>>> >>>>>>>>>>> implementations
>>>>>> >>>>>>>>>>> marked audience Private and returns implementations
>>>>>> rather than
>>>>>> >>>>>>>>>>> Interfaces.
>>>>>> >>>>>>>>>>> In a few locations, we allow returning an alternate
>>>>>> >>>>>>>>>>> implementation
>>>>>> >>>>>>>>>>> altogether which is probably something we don't want a CP
>>>>>> doing.
>>>>>> >>>>>>>>>>> To
>>>>>> >>>>>>>>>>> that
>>>>>> >>>>>>>>>>> end, the following issues started by Duo and Anoop need
>>>>>> to be
>>>>>> >>>>>>>>>>> taken
>>>>>> >>>>>>>>>>> to
>>>>>> >>>>>>>>>>> the
>>>>>> >>>>>>>>>>> finish line; ideally they'd have an owner:
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
>>>>>> release <=
>>>>>> >>>>>>>>>>> The
>>>>>> >>>>>>>>>>> umbrella issue.
>>>>>> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for
>>>>>> CP
>>>>>> >>>>>>>>>>> expose
>>>>>> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
>>>>>> MasterObserver
>>>>>> >>>>>>>>>>> and
>>>>>> >>>>>>>>>>> RegionServerObserver
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Nice-to-haves:
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
>>>>>> remove
>>>>>> >>>>>>>>>>> unused
>>>>>> >>>>>>>>>>> TimeRange constructors
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
>>>>>> >>>>>>>>>>> references
>>>>>> >>>>>>>>>>> existing in the code
>>>>>> >>>>>>>>>>> This is the end of an old long-running project moving up
>>>>>> on to
>>>>>> >>>>>>>>>>> Cell
>>>>>> >>>>>>>>>>> Interface. We think it is done but for a few little items
>>>>>> >>>>>>>>>>> (deprecate
>>>>>> >>>>>>>>>>> KV
>>>>>> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is
>>>>>> indeterminate;
>>>>>> >>>>>>>>>>> needs
>>>>>> >>>>>>>>>>> fixing
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>>> >>>>>>>>>>> * HBASE-14997
>>>>>> >>>>>>>>>>> Move compareOp and Comparators out of filter to client
>>>>>> package
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> What about:
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>>> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> If interested in any of the above and you need a legup,
>>>>>> just ask
>>>>>> >>>>>>>>>>> in
>>>>>> >>>>>>>>>>> the
>>>>>> >>>>>>>>>>> issue and I'll be by....
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Thanks,
>>>>>> >>>>>>>>>>> St.Ack
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <stack@duboce.net
>>>>>> >
>>>>>> >>>>>>>>>>> wrote:
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Heads-up:
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release
>>>>>> Candidate.
>>>>>> >>>>>>>>>>>> Theme
>>>>>> >>>>>>>>>>>> is
>>>>>> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
>>>>>> (guava,
>>>>>> >>>>>>>>>>>> netty,
>>>>>> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
>>>>>> >>>>>>>>>>>> master-carries-no-regions
>>>>>> >>>>>>>>>>>> by default.
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?).
>>>>>> Its theme
>>>>>> >>>>>>>>>>>> will
>>>>>> >>>>>>>>>>>> be
>>>>>> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we
>>>>>> are not
>>>>>> >>>>>>>>>>>> looking too
>>>>>> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have
>>>>>> hbase-1
>>>>>> >>>>>>>>>>>> client
>>>>>> >>>>>>>>>>>> running
>>>>>> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor
>>>>>> Interface
>>>>>> >>>>>>>>>>>> revamp
>>>>>> >>>>>>>>>>>> should be
>>>>>> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>> >>>>>>>>>>>> Implementations, and
>>>>>> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>> >>>>>>>>>>>> InterfaceAudience).
>>>>>> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API;
>>>>>> e.g.
>>>>>> >>>>>>>>>>>> guava
>>>>>> >>>>>>>>>>>> 0.12
>>>>>> >>>>>>>>>>>> Service showing through in our replication API and
>>>>>> protobufs in
>>>>>> >>>>>>>>>>>> Admin
>>>>>> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of
>>>>>> outreach to
>>>>>> >>>>>>>>>>>> make
>>>>>> >>>>>>>>>>>> sure
>>>>>> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> Beta1 in mid-September?
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>> >>>>>>>>>>>> https://issues.apache.org/jira
>>>>>> /projects/HBASE/versions/12327188
>>>>>> >>>>>>>>>>>> Edit
>>>>>> >>>>>>>>>>>> as
>>>>>> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not
>>>>>> get any
>>>>>> >>>>>>>>>>>> attention
>>>>>> >>>>>>>>>>>> in next month or so.
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll
>>>>>> leave them
>>>>>> >>>>>>>>>>>> as
>>>>>> >>>>>>>>>>>> is
>>>>>> >>>>>>>>>>>> another while but I'll boot them soon.
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> While I have your attention:
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3.
>>>>>> Moving hbase
>>>>>> >>>>>>>>>>>> thrift
>>>>>> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is
>>>>>> easy enough
>>>>>> >>>>>>>>>>>> if
>>>>>> >>>>>>>>>>>> folks
>>>>>> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to
>>>>>> 1.0 first
>>>>>> >>>>>>>>>>>> (0.98?).
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> St.Ack
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
>>>>>> wrote:
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
>>>>>> elserj@apache.org>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> wrote:
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
>>>>>> >>>>>>>>>>>>>> elserj@apache.org>
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>> wrote:
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>> ...
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would
>>>>>> make the
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>> feature a
>>>>>> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous
>>>>>> (I have
>>>>>> >>>>>>>>>>>>>>>> no
>>>>>> >>>>>>>>>>>>>>>> reason
>>>>>> >>>>>>>>>>>>>>>> to be
>>>>>> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature
>>>>>> in terms
>>>>>> >>>>>>>>>>>>>>>> of
>>>>>> >>>>>>>>>>>>>>>> how
>>>>>> >>>>>>>>>>>>>>>> much it
>>>>>> >>>>>>>>>>>>>>>> is covered via testing.
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so,
>>>>>> +1 Josh.
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too.
>>>>>> See 'Space
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>> Quota' in
>>>>>> >>>>>>>>>>>>>>> here:
>>>>>> >>>>>>>>>>>>>>> https://docs.google.com/docume
>>>>>> nt/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>> >>>>>>>>>>>>>>> Does this little section need an update Josh?
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>> Thanks,
>>>>>> >>>>>>>>>>>>>>> S
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test
>>>>>> coverage and
>>>>>> >>>>>>>>>>>>>>> some
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>> docs
>>>>>> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my
>>>>>> plate if
>>>>>> >>>>>>>>>>>>>> you're good
>>>>>> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is
>>>>>> against it :)
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
>>>>>> before
>>>>>> >>>>>>>>>>>>>> we
>>>>>> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll
>>>>>> double
>>>>>> >>>>>>>>>>>>>> check
>>>>>> >>>>>>>>>>>>>> and update
>>>>>> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>> Thanks Josh,
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>> S
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>
>>>>>> >>>>
>>>>>> >>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Heads-up, I'm working on cutting a beta-1. Shout if there is something you
think needs to make it in (The Chia-Ping Tsai cleanups look good). I'm
mainly waiting on the hbase-thirdparty changes to go in. The rest can wait
it.

Thanks,
S


On Wed, Dec 13, 2017 at 11:28 PM, Stack <st...@duboce.net> wrote:

> Update. See below.
>
> On Tue, Dec 5, 2017 at 10:43 AM, Stack <st...@duboce.net> wrote:
>
>> Update on hbase-2.0.0-beta-1. I'm going to put up a beta-1 RC before
>> Christmas day.
>>
>>
> Still aiming for an hbase2 beta-1 showing up in your christmas stocking.
>
>
>> Here is the list [1]. We're doing pretty good. We've fixed a load since
>> the last update including some nice cleanups  two weeks ago (e.g. undoing
>> client dependency on curator/zk watching making the dependence read-only)
>> but there are still a few hairy ones hanging out there.
>>
>>  * HBASE-18946 Stochastic load balancer assigns replica regions to the
>> same RS
>>    This one is proving a little tough. Ram has been banging his head on
>> it. We'll be able to clear a bunch of test failures once this is done.
>>
>>
> This should be going in tomorrow.
>
>
>
>>  * HBASE-19112 Suspect methods on Cell to be deprecated
>>    Good discussion up on the JIRA. We need to be crystal clear around
>> Cell and derivatives API and Audience when we ship 2.0.
>>
>>
> Ram, Chia-Ping Tsai, and Anoop are doing a nice job here and it should be
> done in next day or so.
>
>
>
>>  * HBASE-17204 Make L2 off heap cache default ON
>>    We need to try this at least so offheaping can be default.
>>
>>
> Chatted with Anoop and Ram. Because it is so late in the day and because
> it critical that the user have a good experience out of the box -- which
> requires time and testing -- we are going to punt this from 2.0. Will look
> at it for 2.1 (or 3.0 if soon). 2.0 will still be carrying all of the boys
> offheap read/write path goodness.
>
>
>
>> Making asyncfswal default is also ongoing, HBASE-16890
>> <https://issues.apache.org/jira/browse/HBASE-16890>, making good
>> progress.
>>
>>
>
> Good progress here. In perf testing asyncfswal makes us generally faster.
> Duo making good progress on last few tests that are in the way of our
> making this default for hbase2.
>
> A load of goodness has been landing these last few weeks. Keep up the
> great work.
>
> Your hbase-2 RM.
>
>
>
>
>> I punted HBASE-19147 "All branch-2 unit tests pass" out to beta-2. Our
>> unit test story has gotten better and a few of us are actively working on
>> flakies [2] but we may make a beta-1 w/o shutting down all failures.
>>
>> Speak up if you need help on an issue or if you think we are missing
>> items from our list.
>>
>> Thanks for all the hard work,
>>
>> The hbase-2.0.0 RM.
>>
>>
>> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>> 2. https://builds.apache.org/job/HBASE-Find-Flaky-Tests/last
>> SuccessfulBuild/artifact/dashboard.html
>>
>> On Thu, Nov 16, 2017 at 3:48 PM, Stack <st...@duboce.net> wrote:
>>
>>> hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
>>> remaining features and apply final polish to API. There will be a beta-2
>>> but it is about upgrade/rolling-upgrade and bug fixes ONLY).
>>>
>>> Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1
>>> list this morning. See here [1].
>>>
>>> There are about ~12 issues in progress with most of these about to land.
>>> There are 37 TODO. Many of these are tests we need to run, some are related
>>> to the backup/restore, but a good few are meaty w/o assignees.
>>>
>>> The awkward outstanding ones as I see it are the below:
>>>
>>> HBASE-18946 Stochastic load balancer assigns replica regions to the same
>>> RS
>>> HBASE-17204 Make L2 off heap cache default ON
>>> HBASE-19112 Suspect methods on Cell to be deprecated
>>> HBASE-19147 All branch-2 unit tests pass
>>>
>>> We need to make progress on the above or punt on them (can't punt on the
>>> last one though).
>>>
>>> Any ideas on what configs we should update in hbase2? Dump ideas into:
>>> HBASE-19148 Edit of default configuration
>>>
>>> Still hoping for an early December beta-1 RC. beta-2 hopefully will be
>>> close behind.
>>>
>>> Comments? Thoughts?
>>>
>>> Thanks all,
>>> S
>>>
>>> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>>>
>>> On Tue, Oct 31, 2017 at 2:56 PM, Stack <st...@duboce.net> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:
>>>>
>>>>> Hoping to keep momentum going from our Stack working on alpha4, I
>>>>> tried to
>>>>> take a stab at triaging some of the open beta-1 issues.
>>>>>
>>>>> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done
>>>>> sooner
>>>>> then I'm happy to see it pulled back in. Trying to balance optimism
>>>>> with
>>>>> realism here, and knowing that documentation unfortunately often gets
>>>>> pushed to the back-burner.
>>>>>
>>>>> Also, I poked some folks on unassigned issues that they've filed for
>>>>> beta-1, especially in the last few days. If issues don't have an owner
>>>>> they
>>>>> are unlikely to get worked. I chatted with stack and he agreed to take
>>>>> on
>>>>> some of the tasks, but there's a lot of surface area to cover.
>>>>>
>>>>> If you you're working on issues that are critical for beta-1, please
>>>>> mark
>>>>> them as such. Then the rest of the community will know to help
>>>>> prioritize
>>>>> feedback and reviews there.
>>>>>
>>>>> Do we have a general theme for the betas like we did with the alphas?
>>>>>
>>>>> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
>>>>> well? Continue to work on tests throughout?
>>>>>
>>>>>
>>>> Thanks Mike for helping to kick off the beta-1 train.
>>>>
>>>> Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what
>>>> we are going to ship (beta-2 is rolling upgrade and any minor items turned
>>>> up in testing/burn-in).
>>>>
>>>> Thanks,
>>>> S
>>>>
>>>>
>>>>
>>>>> Mike
>>>>>
>>>>> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org>
>>>>> wrote:
>>>>>
>>>>> > +1 go from my POV.
>>>>> >
>>>>> >
>>>>> > On 10/31/17 10:07 AM, Stack wrote:
>>>>> >
>>>>> >> I want to push an alpha-4 today. A few items didn't make it
>>>>> (HBASE-19092).
>>>>> >> They need more time. We'll pull them in for beta-1. CP API is
>>>>> basically
>>>>> >> done. There may be some changes for beta-1 but hopefully only
>>>>> changes
>>>>> >> informed by experience trying to port an existing Coprocessor to
>>>>> hbase2.
>>>>> >>
>>>>> >> Shout if there is anything that needs to make alpha-4.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> St.Ack
>>>>> >>
>>>>> >>
>>>>> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
>>>>> wrote:
>>>>> >>
>>>>> >> Yup, that was going to be my plan, Mike!
>>>>> >>>
>>>>> >>> Making a pass now, and will check back later tonight again. I see
>>>>> others
>>>>> >>> have already done some work today on this front.
>>>>> >>>
>>>>> >>>
>>>>> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
>>>>> >>>
>>>>> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know
>>>>> how to do
>>>>> >>>> it directly on the jenkins job, so you don't have to bother with
>>>>> JIRA
>>>>> >>>> uploads)
>>>>> >>>>
>>>>> >>>> If you're busy, then I can make time tomorrow or Sunday to kick
>>>>> off
>>>>> >>>> jobs,
>>>>> >>>> but I want to make sure we're not duplicating effort and jenkins
>>>>> cycles.
>>>>> >>>>
>>>>> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>> My turn to bump ;)
>>>>> >>>>
>>>>> >>>>>
>>>>> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that
>>>>> remain
>>>>> >>>>> needing some more work. The rest are just awaiting a good QA run.
>>>>> >>>>>
>>>>> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over
>>>>> the
>>>>> >>>>> weekend, bump them along as necessary, and get them committed.
>>>>> Would be
>>>>> >>>>> great to be able get a vote up on Monday.
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> On 10/24/17 6:03 PM, Stack wrote:
>>>>> >>>>>
>>>>> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back
>>>>> and forth
>>>>> >>>>>
>>>>> >>>>>> what remains to be done. Surfacing our thoughts here so you all
>>>>> clued
>>>>> >>>>>> in....Or if you think otherwise, please speak up.
>>>>> >>>>>>
>>>>> >>>>>> We have ~13 issues to land:
>>>>> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>> About
>>>>> >>>>>> two
>>>>> >>>>>> are meta-issues that are about process which leaves 11.
>>>>> >>>>>>
>>>>> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>>>> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because
>>>>> some
>>>>> >>>>>> changed
>>>>> >>>>>> API/semantic that we need to get out earlier rather than later.
>>>>> >>>>>>
>>>>> >>>>>> Once the above is merged, HBASE-13346, change of Filter method
>>>>> names
>>>>> >>>>>> to
>>>>> >>>>>> mention Cell instead of KeyValue can land.
>>>>> >>>>>>
>>>>> >>>>>> HBASE-199048 needs a review (Anoop will probably do it),
>>>>> removing
>>>>> >>>>>> IA.Private objects as params to MasterObserver... Hopefully
>>>>> this goes
>>>>> >>>>>> in
>>>>> >>>>>> soon.
>>>>> >>>>>>
>>>>> >>>>>> Duo is hard at work on trackers for flush and compaction for CPs
>>>>> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for
>>>>> Tephra)?
>>>>> >>>>>>
>>>>> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate
>>>>> after
>>>>> >>>>>> Duo
>>>>> >>>>>> is
>>>>> >>>>>> done w/ his work above.
>>>>> >>>>>>
>>>>> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity
>>>>> after
>>>>> >>>>>> all
>>>>> >>>>>> the
>>>>> >>>>>> purges allowing CPs do direct calls against Regions in same
>>>>> Host.
>>>>> >>>>>>
>>>>> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>>>> >>>>>>
>>>>> >>>>>> Another day or two?
>>>>> >>>>>>
>>>>> >>>>>> St.Ack
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net>
>>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <elserj@apache.org
>>>>> >
>>>>> >>>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>> +1
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>> I was trying to work on helping out on the outstanding
>>>>> alpha-4 stuff
>>>>> >>>>>>>> last
>>>>> >>>>>>>> week -- will be continuing to try to do the same this week.
>>>>> >>>>>>>>
>>>>> >>>>>>>> If you need any help, Stack, or if others need reviews where I
>>>>> >>>>>>>> haven't
>>>>> >>>>>>>> noticed on my own: feel free to @mention me.
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are
>>>>> being
>>>>> >>>>>>>>
>>>>> >>>>>>>> actively
>>>>> >>>>>>> worked on. If you get a moment, reviews by you (or anyone
>>>>> else) helps
>>>>> >>>>>>> move
>>>>> >>>>>>> the process along.
>>>>> >>>>>>>
>>>>> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
>>>>> >>>>>>> improvements.
>>>>> >>>>>>> Then HBASE-13346 can go in.
>>>>> >>>>>>>
>>>>> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks like
>>>>> that
>>>>> >>>>>>> will
>>>>> >>>>>>> be addressed by other alpha-4s about to land.
>>>>> >>>>>>>
>>>>> >>>>>>> St.Ack
>>>>> >>>>>>> TODOs: https://issues.apache.org/jira
>>>>> /projects/HBASE/versions/12341
>>>>> >>>>>>> 594
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>> (Reviving this thread)
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that
>>>>> has
>>>>> >>>>>>>>> the
>>>>> >>>>>>>>> refactor of the Coprocessor API shutting down access to
>>>>> internals
>>>>> >>>>>>>>> marked
>>>>> >>>>>>>>> InterfaceAudience.Private.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> The outstanding list is here:
>>>>> >>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341
>>>>> 594
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> If issue, talk out loud on this thread. If you need a review
>>>>> to
>>>>> >>>>>>>>> land
>>>>> >>>>>>>>> an
>>>>> >>>>>>>>> item, shout on the issue and here; we'll help you out.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> As is, items are coming along nicely I'd say. We need to
>>>>> merge the
>>>>> >>>>>>>>> filter
>>>>> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and
>>>>> help
>>>>> >>>>>>>>> them
>>>>> >>>>>>>>> test
>>>>> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence
>>>>> our
>>>>> >>>>>>>>> downstreamers know what to expect (or we discover what we
>>>>> missed
>>>>> >>>>>>>>> BEFORE
>>>>> >>>>>>>>> we
>>>>> >>>>>>>>> beta-1).
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Thanks for time,
>>>>> >>>>>>>>> S
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net>
>>>>> wrote:
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That
>>>>> should
>>>>> >>>>>>>>> be
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to
>>>>> be done.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled
>>>>> for
>>>>> >>>>>>>>>> alpha3
>>>>> >>>>>>>>>> but
>>>>> >>>>>>>>>> it is plain that more time is needed (in spite of valiant
>>>>> effort
>>>>> >>>>>>>>>> so
>>>>> >>>>>>>>>> far
>>>>> >>>>>>>>>> by
>>>>> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a
>>>>> 2.0.0-alpha-4 whose
>>>>> >>>>>>>>>> theme is
>>>>> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by
>>>>> the
>>>>> >>>>>>>>>> following
>>>>> >>>>>>>>>> week.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> We should then be ready for beta (beta == no new features,
>>>>> no API
>>>>> >>>>>>>>>> changes,
>>>>> >>>>>>>>>> just fixes).
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Thanks,
>>>>> >>>>>>>>>> St.Ack
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
>>>>> wrote:
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please
>>>>> vote on
>>>>> >>>>>>>>>> it.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I
>>>>> hope to
>>>>> >>>>>>>>>>> get a
>>>>> >>>>>>>>>>> release out in the next week or so.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If
>>>>> folks are
>>>>> >>>>>>>>>>> interested in helping out, below are the items I think we
>>>>> need
>>>>> >>>>>>>>>>> done
>>>>> >>>>>>>>>>> for
>>>>> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API
>>>>> possibly
>>>>> >>>>>>>>>>> altering
>>>>> >>>>>>>>>>> items, and are absent those JIRAs that are making active
>>>>> >>>>>>>>>>> progress,
>>>>> >>>>>>>>>>> i.e. the
>>>>> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed
>>>>> that
>>>>> >>>>>>>>>>> needs
>>>>> >>>>>>>>>>> doing is
>>>>> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between
>>>>> branch-1
>>>>> >>>>>>>>>>> and
>>>>> >>>>>>>>>>> branch-2
>>>>> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches
>>>>> only
>>>>> >>>>>>>>>>> do
>>>>> >>>>>>>>>>> it
>>>>> >>>>>>>>>>> between branch-1 and branch-2.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing
>>>>> Public /
>>>>> >>>>>>>>>>> Private
>>>>> >>>>>>>>>>> interfaces
>>>>> >>>>>>>>>>> This one is almost done. It could do with a finish,
>>>>> attention to
>>>>> >>>>>>>>>>> the
>>>>> >>>>>>>>>>> items in last comment, and then our codebase could do with
>>>>> >>>>>>>>>>> another
>>>>> >>>>>>>>>>> sweep
>>>>> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in
>>>>> since
>>>>> >>>>>>>>>>> the
>>>>> >>>>>>>>>>> pass
>>>>> >>>>>>>>>>> that was the basis of this issue.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
>>>>> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
>>>>> forced
>>>>> >>>>>>>>>>> by
>>>>> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
>>>>> >>>>>>>>>>> Service'
>>>>> >>>>>>>>>>> but if
>>>>> >>>>>>>>>>> anyone else is interested, be my guest.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked
>>>>> on. Needs
>>>>> >>>>>>>>>>> finishing.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods
>>>>> in Admin
>>>>> >>>>>>>>>>> and
>>>>> >>>>>>>>>>> cleanup
>>>>> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>>>>> >>>>>>>>>>> AsyncClient
>>>>> >>>>>>>>>>> and
>>>>> >>>>>>>>>>> make sure symmetry.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but
>>>>> needs to
>>>>> >>>>>>>>>>> be
>>>>> >>>>>>>>>>> done.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would
>>>>> be good
>>>>> >>>>>>>>>>> to
>>>>> >>>>>>>>>>> try
>>>>> >>>>>>>>>>> it and figure what needs fixup (as said above, according
>>>>> to trial
>>>>> >>>>>>>>>>> by
>>>>> >>>>>>>>>>> Sean,
>>>>> >>>>>>>>>>> we might not be too bad here):
>>>>> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state
>>>>> talking to
>>>>> >>>>>>>>>>> 2.0
>>>>> >>>>>>>>>>> cluster
>>>>> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
>>>>> Master
>>>>> >>>>>>>>>>> and
>>>>> >>>>>>>>>>> 1.x
>>>>> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-17442 Move most of the replication related classes
>>>>> to
>>>>> >>>>>>>>>>> hbase-server package
>>>>> >>>>>>>>>>> The above would be good to do generally but it may make for
>>>>> >>>>>>>>>>> ripples
>>>>> >>>>>>>>>>> in
>>>>> >>>>>>>>>>> API so would be good to do now.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid
>>>>> adding two
>>>>> >>>>>>>>>>> new
>>>>> >>>>>>>>>>> types
>>>>> >>>>>>>>>>> to our API, two types that are nought but curtailed,
>>>>> read-only
>>>>> >>>>>>>>>>> views
>>>>> >>>>>>>>>>> on
>>>>> >>>>>>>>>>> internals. Input if you have time appreciated.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate
>>>>> to a
>>>>> >>>>>>>>>>> hbase2
>>>>> >>>>>>>>>>> cluster; verify
>>>>> >>>>>>>>>>> Esteban is looking at this one
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in
>>>>> core
>>>>> >>>>>>>>>>> * HBASE-17143 Scan improvement
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>> >>>>>>>>>>> implementations
>>>>> >>>>>>>>>>> marked audience Private and returns implementations rather
>>>>> than
>>>>> >>>>>>>>>>> Interfaces.
>>>>> >>>>>>>>>>> In a few locations, we allow returning an alternate
>>>>> >>>>>>>>>>> implementation
>>>>> >>>>>>>>>>> altogether which is probably something we don't want a CP
>>>>> doing.
>>>>> >>>>>>>>>>> To
>>>>> >>>>>>>>>>> that
>>>>> >>>>>>>>>>> end, the following issues started by Duo and Anoop need to
>>>>> be
>>>>> >>>>>>>>>>> taken
>>>>> >>>>>>>>>>> to
>>>>> >>>>>>>>>>> the
>>>>> >>>>>>>>>>> finish line; ideally they'd have an owner:
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
>>>>> release <=
>>>>> >>>>>>>>>>> The
>>>>> >>>>>>>>>>> umbrella issue.
>>>>> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
>>>>> >>>>>>>>>>> expose
>>>>> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
>>>>> MasterObserver
>>>>> >>>>>>>>>>> and
>>>>> >>>>>>>>>>> RegionServerObserver
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Nice-to-haves:
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
>>>>> remove
>>>>> >>>>>>>>>>> unused
>>>>> >>>>>>>>>>> TimeRange constructors
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
>>>>> >>>>>>>>>>> references
>>>>> >>>>>>>>>>> existing in the code
>>>>> >>>>>>>>>>> This is the end of an old long-running project moving up
>>>>> on to
>>>>> >>>>>>>>>>> Cell
>>>>> >>>>>>>>>>> Interface. We think it is done but for a few little items
>>>>> >>>>>>>>>>> (deprecate
>>>>> >>>>>>>>>>> KV
>>>>> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is
>>>>> indeterminate;
>>>>> >>>>>>>>>>> needs
>>>>> >>>>>>>>>>> fixing
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>> >>>>>>>>>>> * HBASE-14997
>>>>> >>>>>>>>>>> Move compareOp and Comparators out of filter to client
>>>>> package
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> What about:
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> If interested in any of the above and you need a legup,
>>>>> just ask
>>>>> >>>>>>>>>>> in
>>>>> >>>>>>>>>>> the
>>>>> >>>>>>>>>>> issue and I'll be by....
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Thanks,
>>>>> >>>>>>>>>>> St.Ack
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
>>>>> >>>>>>>>>>> wrote:
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Heads-up:
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release
>>>>> Candidate.
>>>>> >>>>>>>>>>>> Theme
>>>>> >>>>>>>>>>>> is
>>>>> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
>>>>> (guava,
>>>>> >>>>>>>>>>>> netty,
>>>>> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
>>>>> >>>>>>>>>>>> master-carries-no-regions
>>>>> >>>>>>>>>>>> by default.
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?).
>>>>> Its theme
>>>>> >>>>>>>>>>>> will
>>>>> >>>>>>>>>>>> be
>>>>> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we
>>>>> are not
>>>>> >>>>>>>>>>>> looking too
>>>>> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have
>>>>> hbase-1
>>>>> >>>>>>>>>>>> client
>>>>> >>>>>>>>>>>> running
>>>>> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface
>>>>> >>>>>>>>>>>> revamp
>>>>> >>>>>>>>>>>> should be
>>>>> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>> >>>>>>>>>>>> Implementations, and
>>>>> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>> >>>>>>>>>>>> InterfaceAudience).
>>>>> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API;
>>>>> e.g.
>>>>> >>>>>>>>>>>> guava
>>>>> >>>>>>>>>>>> 0.12
>>>>> >>>>>>>>>>>> Service showing through in our replication API and
>>>>> protobufs in
>>>>> >>>>>>>>>>>> Admin
>>>>> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of
>>>>> outreach to
>>>>> >>>>>>>>>>>> make
>>>>> >>>>>>>>>>>> sure
>>>>> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> Beta1 in mid-September?
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>> >>>>>>>>>>>> https://issues.apache.org/jira
>>>>> /projects/HBASE/versions/12327188
>>>>> >>>>>>>>>>>> Edit
>>>>> >>>>>>>>>>>> as
>>>>> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not
>>>>> get any
>>>>> >>>>>>>>>>>> attention
>>>>> >>>>>>>>>>>> in next month or so.
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll
>>>>> leave them
>>>>> >>>>>>>>>>>> as
>>>>> >>>>>>>>>>>> is
>>>>> >>>>>>>>>>>> another while but I'll boot them soon.
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> While I have your attention:
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving
>>>>> hbase
>>>>> >>>>>>>>>>>> thrift
>>>>> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy
>>>>> enough
>>>>> >>>>>>>>>>>> if
>>>>> >>>>>>>>>>>> folks
>>>>> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
>>>>> first
>>>>> >>>>>>>>>>>> (0.98?).
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> St.Ack
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
>>>>> wrote:
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
>>>>> elserj@apache.org>
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>>> wrote:
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
>>>>> >>>>>>>>>>>>>> elserj@apache.org>
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>> wrote:
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>> ...
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would
>>>>> make the
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>> feature a
>>>>> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I
>>>>> have
>>>>> >>>>>>>>>>>>>>>> no
>>>>> >>>>>>>>>>>>>>>> reason
>>>>> >>>>>>>>>>>>>>>> to be
>>>>> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature
>>>>> in terms
>>>>> >>>>>>>>>>>>>>>> of
>>>>> >>>>>>>>>>>>>>>> how
>>>>> >>>>>>>>>>>>>>>> much it
>>>>> >>>>>>>>>>>>>>>> is covered via testing.
>>>>> >>>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>> >>>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so,
>>>>> +1 Josh.
>>>>> >>>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See
>>>>> 'Space
>>>>> >>>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>> Quota' in
>>>>> >>>>>>>>>>>>>>> here:
>>>>> >>>>>>>>>>>>>>> https://docs.google.com/docume
>>>>> nt/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>> >>>>>>>>>>>>>>> Does this little section need an update Josh?
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>> Thanks,
>>>>> >>>>>>>>>>>>>>> S
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage
>>>>> and
>>>>> >>>>>>>>>>>>>>> some
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>> docs
>>>>> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my
>>>>> plate if
>>>>> >>>>>>>>>>>>>> you're good
>>>>> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is against
>>>>> it :)
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
>>>>> before
>>>>> >>>>>>>>>>>>>> we
>>>>> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll
>>>>> double
>>>>> >>>>>>>>>>>>>> check
>>>>> >>>>>>>>>>>>>> and update
>>>>> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>> Thanks Josh,
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>> S
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>
>>>>> >>>>
>>>>> >>
>>>>>
>>>>
>>>>
>>>
>>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Update. See below.

On Tue, Dec 5, 2017 at 10:43 AM, Stack <st...@duboce.net> wrote:

> Update on hbase-2.0.0-beta-1. I'm going to put up a beta-1 RC before
> Christmas day.
>
>
Still aiming for an hbase2 beta-1 showing up in your christmas stocking.


> Here is the list [1]. We're doing pretty good. We've fixed a load since
> the last update including some nice cleanups  two weeks ago (e.g. undoing
> client dependency on curator/zk watching making the dependence read-only)
> but there are still a few hairy ones hanging out there.
>
>  * HBASE-18946 Stochastic load balancer assigns replica regions to the
> same RS
>    This one is proving a little tough. Ram has been banging his head on
> it. We'll be able to clear a bunch of test failures once this is done.
>
>
This should be going in tomorrow.



>  * HBASE-19112 Suspect methods on Cell to be deprecated
>    Good discussion up on the JIRA. We need to be crystal clear around Cell
> and derivatives API and Audience when we ship 2.0.
>
>
Ram, Chia-Ping Tsai, and Anoop are doing a nice job here and it should be
done in next day or so.



>  * HBASE-17204 Make L2 off heap cache default ON
>    We need to try this at least so offheaping can be default.
>
>
Chatted with Anoop and Ram. Because it is so late in the day and because it
critical that the user have a good experience out of the box -- which
requires time and testing -- we are going to punt this from 2.0. Will look
at it for 2.1 (or 3.0 if soon). 2.0 will still be carrying all of the boys
offheap read/write path goodness.



> Making asyncfswal default is also ongoing, HBASE-16890
> <https://issues.apache.org/jira/browse/HBASE-16890>, making good progress.
>
>

Good progress here. In perf testing asyncfswal makes us generally faster.
Duo making good progress on last few tests that are in the way of our
making this default for hbase2.

A load of goodness has been landing these last few weeks. Keep up the great
work.

Your hbase-2 RM.




> I punted HBASE-19147 "All branch-2 unit tests pass" out to beta-2. Our
> unit test story has gotten better and a few of us are actively working on
> flakies [2] but we may make a beta-1 w/o shutting down all failures.
>
> Speak up if you need help on an issue or if you think we are missing items
> from our list.
>
> Thanks for all the hard work,
>
> The hbase-2.0.0 RM.
>
>
> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
> 2. https://builds.apache.org/job/HBASE-Find-Flaky-Tests/
> lastSuccessfulBuild/artifact/dashboard.html
>
> On Thu, Nov 16, 2017 at 3:48 PM, Stack <st...@duboce.net> wrote:
>
>> hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
>> remaining features and apply final polish to API. There will be a beta-2
>> but it is about upgrade/rolling-upgrade and bug fixes ONLY).
>>
>> Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1
>> list this morning. See here [1].
>>
>> There are about ~12 issues in progress with most of these about to land.
>> There are 37 TODO. Many of these are tests we need to run, some are related
>> to the backup/restore, but a good few are meaty w/o assignees.
>>
>> The awkward outstanding ones as I see it are the below:
>>
>> HBASE-18946 Stochastic load balancer assigns replica regions to the same
>> RS
>> HBASE-17204 Make L2 off heap cache default ON
>> HBASE-19112 Suspect methods on Cell to be deprecated
>> HBASE-19147 All branch-2 unit tests pass
>>
>> We need to make progress on the above or punt on them (can't punt on the
>> last one though).
>>
>> Any ideas on what configs we should update in hbase2? Dump ideas into:
>> HBASE-19148 Edit of default configuration
>>
>> Still hoping for an early December beta-1 RC. beta-2 hopefully will be
>> close behind.
>>
>> Comments? Thoughts?
>>
>> Thanks all,
>> S
>>
>> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>>
>> On Tue, Oct 31, 2017 at 2:56 PM, Stack <st...@duboce.net> wrote:
>>
>>>
>>>
>>> On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:
>>>
>>>> Hoping to keep momentum going from our Stack working on alpha4, I tried
>>>> to
>>>> take a stab at triaging some of the open beta-1 issues.
>>>>
>>>> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done
>>>> sooner
>>>> then I'm happy to see it pulled back in. Trying to balance optimism with
>>>> realism here, and knowing that documentation unfortunately often gets
>>>> pushed to the back-burner.
>>>>
>>>> Also, I poked some folks on unassigned issues that they've filed for
>>>> beta-1, especially in the last few days. If issues don't have an owner
>>>> they
>>>> are unlikely to get worked. I chatted with stack and he agreed to take
>>>> on
>>>> some of the tasks, but there's a lot of surface area to cover.
>>>>
>>>> If you you're working on issues that are critical for beta-1, please
>>>> mark
>>>> them as such. Then the rest of the community will know to help
>>>> prioritize
>>>> feedback and reviews there.
>>>>
>>>> Do we have a general theme for the betas like we did with the alphas?
>>>>
>>>> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
>>>> well? Continue to work on tests throughout?
>>>>
>>>>
>>> Thanks Mike for helping to kick off the beta-1 train.
>>>
>>> Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what
>>> we are going to ship (beta-2 is rolling upgrade and any minor items turned
>>> up in testing/burn-in).
>>>
>>> Thanks,
>>> S
>>>
>>>
>>>
>>>> Mike
>>>>
>>>> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org> wrote:
>>>>
>>>> > +1 go from my POV.
>>>> >
>>>> >
>>>> > On 10/31/17 10:07 AM, Stack wrote:
>>>> >
>>>> >> I want to push an alpha-4 today. A few items didn't make it
>>>> (HBASE-19092).
>>>> >> They need more time. We'll pull them in for beta-1. CP API is
>>>> basically
>>>> >> done. There may be some changes for beta-1 but hopefully only changes
>>>> >> informed by experience trying to port an existing Coprocessor to
>>>> hbase2.
>>>> >>
>>>> >> Shout if there is anything that needs to make alpha-4.
>>>> >>
>>>> >> Thanks,
>>>> >> St.Ack
>>>> >>
>>>> >>
>>>> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
>>>> wrote:
>>>> >>
>>>> >> Yup, that was going to be my plan, Mike!
>>>> >>>
>>>> >>> Making a pass now, and will check back later tonight again. I see
>>>> others
>>>> >>> have already done some work today on this front.
>>>> >>>
>>>> >>>
>>>> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
>>>> >>>
>>>> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how
>>>> to do
>>>> >>>> it directly on the jenkins job, so you don't have to bother with
>>>> JIRA
>>>> >>>> uploads)
>>>> >>>>
>>>> >>>> If you're busy, then I can make time tomorrow or Sunday to kick off
>>>> >>>> jobs,
>>>> >>>> but I want to make sure we're not duplicating effort and jenkins
>>>> cycles.
>>>> >>>>
>>>> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
>>>> wrote:
>>>> >>>>
>>>> >>>> My turn to bump ;)
>>>> >>>>
>>>> >>>>>
>>>> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that
>>>> remain
>>>> >>>>> needing some more work. The rest are just awaiting a good QA run.
>>>> >>>>>
>>>> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over
>>>> the
>>>> >>>>> weekend, bump them along as necessary, and get them committed.
>>>> Would be
>>>> >>>>> great to be able get a vote up on Monday.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On 10/24/17 6:03 PM, Stack wrote:
>>>> >>>>>
>>>> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back and
>>>> forth
>>>> >>>>>
>>>> >>>>>> what remains to be done. Surfacing our thoughts here so you all
>>>> clued
>>>> >>>>>> in....Or if you think otherwise, please speak up.
>>>> >>>>>>
>>>> >>>>>> We have ~13 issues to land:
>>>> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>> About
>>>> >>>>>> two
>>>> >>>>>> are meta-issues that are about process which leaves 11.
>>>> >>>>>>
>>>> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>>> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
>>>> >>>>>> changed
>>>> >>>>>> API/semantic that we need to get out earlier rather than later.
>>>> >>>>>>
>>>> >>>>>> Once the above is merged, HBASE-13346, change of Filter method
>>>> names
>>>> >>>>>> to
>>>> >>>>>> mention Cell instead of KeyValue can land.
>>>> >>>>>>
>>>> >>>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
>>>> >>>>>> IA.Private objects as params to MasterObserver... Hopefully this
>>>> goes
>>>> >>>>>> in
>>>> >>>>>> soon.
>>>> >>>>>>
>>>> >>>>>> Duo is hard at work on trackers for flush and compaction for CPs
>>>> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for
>>>> Tephra)?
>>>> >>>>>>
>>>> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate
>>>> after
>>>> >>>>>> Duo
>>>> >>>>>> is
>>>> >>>>>> done w/ his work above.
>>>> >>>>>>
>>>> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity
>>>> after
>>>> >>>>>> all
>>>> >>>>>> the
>>>> >>>>>> purges allowing CPs do direct calls against Regions in same Host.
>>>> >>>>>>
>>>> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>>> >>>>>>
>>>> >>>>>> Another day or two?
>>>> >>>>>>
>>>> >>>>>> St.Ack
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
>>>> >>>>>> wrote:
>>>> >>>>>>
>>>> >>>>>>>
>>>> >>>>>>> +1
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>> I was trying to work on helping out on the outstanding alpha-4
>>>> stuff
>>>> >>>>>>>> last
>>>> >>>>>>>> week -- will be continuing to try to do the same this week.
>>>> >>>>>>>>
>>>> >>>>>>>> If you need any help, Stack, or if others need reviews where I
>>>> >>>>>>>> haven't
>>>> >>>>>>>> noticed on my own: feel free to @mention me.
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are
>>>> being
>>>> >>>>>>>>
>>>> >>>>>>>> actively
>>>> >>>>>>> worked on. If you get a moment, reviews by you (or anyone else)
>>>> helps
>>>> >>>>>>> move
>>>> >>>>>>> the process along.
>>>> >>>>>>>
>>>> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
>>>> >>>>>>> improvements.
>>>> >>>>>>> Then HBASE-13346 can go in.
>>>> >>>>>>>
>>>> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks like
>>>> that
>>>> >>>>>>> will
>>>> >>>>>>> be addressed by other alpha-4s about to land.
>>>> >>>>>>>
>>>> >>>>>>> St.Ack
>>>> >>>>>>> TODOs: https://issues.apache.org/jira
>>>> /projects/HBASE/versions/12341
>>>> >>>>>>> 594
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>> (Reviving this thread)
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that
>>>> has
>>>> >>>>>>>>> the
>>>> >>>>>>>>> refactor of the Coprocessor API shutting down access to
>>>> internals
>>>> >>>>>>>>> marked
>>>> >>>>>>>>> InterfaceAudience.Private.
>>>> >>>>>>>>>
>>>> >>>>>>>>> The outstanding list is here:
>>>> >>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341
>>>> 594
>>>> >>>>>>>>>
>>>> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>> >>>>>>>>>
>>>> >>>>>>>>> If issue, talk out loud on this thread. If you need a review
>>>> to
>>>> >>>>>>>>> land
>>>> >>>>>>>>> an
>>>> >>>>>>>>> item, shout on the issue and here; we'll help you out.
>>>> >>>>>>>>>
>>>> >>>>>>>>> As is, items are coming along nicely I'd say. We need to
>>>> merge the
>>>> >>>>>>>>> filter
>>>> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>> >>>>>>>>>
>>>> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and
>>>> help
>>>> >>>>>>>>> them
>>>> >>>>>>>>> test
>>>> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence
>>>> our
>>>> >>>>>>>>> downstreamers know what to expect (or we discover what we
>>>> missed
>>>> >>>>>>>>> BEFORE
>>>> >>>>>>>>> we
>>>> >>>>>>>>> beta-1).
>>>> >>>>>>>>>
>>>> >>>>>>>>> Thanks for time,
>>>> >>>>>>>>> S
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net>
>>>> wrote:
>>>> >>>>>>>>>
>>>> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That
>>>> should
>>>> >>>>>>>>> be
>>>> >>>>>>>>>
>>>> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to be
>>>> done.
>>>> >>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for
>>>> >>>>>>>>>> alpha3
>>>> >>>>>>>>>> but
>>>> >>>>>>>>>> it is plain that more time is needed (in spite of valiant
>>>> effort
>>>> >>>>>>>>>> so
>>>> >>>>>>>>>> far
>>>> >>>>>>>>>> by
>>>> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
>>>> whose
>>>> >>>>>>>>>> theme is
>>>> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by
>>>> the
>>>> >>>>>>>>>> following
>>>> >>>>>>>>>> week.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> We should then be ready for beta (beta == no new features,
>>>> no API
>>>> >>>>>>>>>> changes,
>>>> >>>>>>>>>> just fixes).
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Thanks,
>>>> >>>>>>>>>> St.Ack
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
>>>> wrote:
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please
>>>> vote on
>>>> >>>>>>>>>> it.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope
>>>> to
>>>> >>>>>>>>>>> get a
>>>> >>>>>>>>>>> release out in the next week or so.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks
>>>> are
>>>> >>>>>>>>>>> interested in helping out, below are the items I think we
>>>> need
>>>> >>>>>>>>>>> done
>>>> >>>>>>>>>>> for
>>>> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API
>>>> possibly
>>>> >>>>>>>>>>> altering
>>>> >>>>>>>>>>> items, and are absent those JIRAs that are making active
>>>> >>>>>>>>>>> progress,
>>>> >>>>>>>>>>> i.e. the
>>>> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
>>>> >>>>>>>>>>> needs
>>>> >>>>>>>>>>> doing is
>>>> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between
>>>> branch-1
>>>> >>>>>>>>>>> and
>>>> >>>>>>>>>>> branch-2
>>>> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches
>>>> only
>>>> >>>>>>>>>>> do
>>>> >>>>>>>>>>> it
>>>> >>>>>>>>>>> between branch-1 and branch-2.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public
>>>> /
>>>> >>>>>>>>>>> Private
>>>> >>>>>>>>>>> interfaces
>>>> >>>>>>>>>>> This one is almost done. It could do with a finish,
>>>> attention to
>>>> >>>>>>>>>>> the
>>>> >>>>>>>>>>> items in last comment, and then our codebase could do with
>>>> >>>>>>>>>>> another
>>>> >>>>>>>>>>> sweep
>>>> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in
>>>> since
>>>> >>>>>>>>>>> the
>>>> >>>>>>>>>>> pass
>>>> >>>>>>>>>>> that was the basis of this issue.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
>>>> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
>>>> forced
>>>> >>>>>>>>>>> by
>>>> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
>>>> >>>>>>>>>>> Service'
>>>> >>>>>>>>>>> but if
>>>> >>>>>>>>>>> anyone else is interested, be my guest.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked on.
>>>> Needs
>>>> >>>>>>>>>>> finishing.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
>>>> Admin
>>>> >>>>>>>>>>> and
>>>> >>>>>>>>>>> cleanup
>>>> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>>>> >>>>>>>>>>> AsyncClient
>>>> >>>>>>>>>>> and
>>>> >>>>>>>>>>> make sure symmetry.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but
>>>> needs to
>>>> >>>>>>>>>>> be
>>>> >>>>>>>>>>> done.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would
>>>> be good
>>>> >>>>>>>>>>> to
>>>> >>>>>>>>>>> try
>>>> >>>>>>>>>>> it and figure what needs fixup (as said above, according to
>>>> trial
>>>> >>>>>>>>>>> by
>>>> >>>>>>>>>>> Sean,
>>>> >>>>>>>>>>> we might not be too bad here):
>>>> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking
>>>> to
>>>> >>>>>>>>>>> 2.0
>>>> >>>>>>>>>>> cluster
>>>> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
>>>> Master
>>>> >>>>>>>>>>> and
>>>> >>>>>>>>>>> 1.x
>>>> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-17442 Move most of the replication related classes
>>>> to
>>>> >>>>>>>>>>> hbase-server package
>>>> >>>>>>>>>>> The above would be good to do generally but it may make for
>>>> >>>>>>>>>>> ripples
>>>> >>>>>>>>>>> in
>>>> >>>>>>>>>>> API so would be good to do now.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid adding
>>>> two
>>>> >>>>>>>>>>> new
>>>> >>>>>>>>>>> types
>>>> >>>>>>>>>>> to our API, two types that are nought but curtailed,
>>>> read-only
>>>> >>>>>>>>>>> views
>>>> >>>>>>>>>>> on
>>>> >>>>>>>>>>> internals. Input if you have time appreciated.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate
>>>> to a
>>>> >>>>>>>>>>> hbase2
>>>> >>>>>>>>>>> cluster; verify
>>>> >>>>>>>>>>> Esteban is looking at this one
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>> >>>>>>>>>>> * HBASE-17143 Scan improvement
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>> >>>>>>>>>>> implementations
>>>> >>>>>>>>>>> marked audience Private and returns implementations rather
>>>> than
>>>> >>>>>>>>>>> Interfaces.
>>>> >>>>>>>>>>> In a few locations, we allow returning an alternate
>>>> >>>>>>>>>>> implementation
>>>> >>>>>>>>>>> altogether which is probably something we don't want a CP
>>>> doing.
>>>> >>>>>>>>>>> To
>>>> >>>>>>>>>>> that
>>>> >>>>>>>>>>> end, the following issues started by Duo and Anoop need to
>>>> be
>>>> >>>>>>>>>>> taken
>>>> >>>>>>>>>>> to
>>>> >>>>>>>>>>> the
>>>> >>>>>>>>>>> finish line; ideally they'd have an owner:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
>>>> release <=
>>>> >>>>>>>>>>> The
>>>> >>>>>>>>>>> umbrella issue.
>>>> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
>>>> >>>>>>>>>>> expose
>>>> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
>>>> MasterObserver
>>>> >>>>>>>>>>> and
>>>> >>>>>>>>>>> RegionServerObserver
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Nice-to-haves:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
>>>> remove
>>>> >>>>>>>>>>> unused
>>>> >>>>>>>>>>> TimeRange constructors
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
>>>> >>>>>>>>>>> references
>>>> >>>>>>>>>>> existing in the code
>>>> >>>>>>>>>>> This is the end of an old long-running project moving up on
>>>> to
>>>> >>>>>>>>>>> Cell
>>>> >>>>>>>>>>> Interface. We think it is done but for a few little items
>>>> >>>>>>>>>>> (deprecate
>>>> >>>>>>>>>>> KV
>>>> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is
>>>> indeterminate;
>>>> >>>>>>>>>>> needs
>>>> >>>>>>>>>>> fixing
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>> >>>>>>>>>>> * HBASE-14997
>>>> >>>>>>>>>>> Move compareOp and Comparators out of filter to client
>>>> package
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> What about:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> If interested in any of the above and you need a legup,
>>>> just ask
>>>> >>>>>>>>>>> in
>>>> >>>>>>>>>>> the
>>>> >>>>>>>>>>> issue and I'll be by....
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Thanks,
>>>> >>>>>>>>>>> St.Ack
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
>>>> >>>>>>>>>>> wrote:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Heads-up:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
>>>> >>>>>>>>>>>> Theme
>>>> >>>>>>>>>>>> is
>>>> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
>>>> (guava,
>>>> >>>>>>>>>>>> netty,
>>>> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
>>>> >>>>>>>>>>>> master-carries-no-regions
>>>> >>>>>>>>>>>> by default.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
>>>> theme
>>>> >>>>>>>>>>>> will
>>>> >>>>>>>>>>>> be
>>>> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we
>>>> are not
>>>> >>>>>>>>>>>> looking too
>>>> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1
>>>> >>>>>>>>>>>> client
>>>> >>>>>>>>>>>> running
>>>> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface
>>>> >>>>>>>>>>>> revamp
>>>> >>>>>>>>>>>> should be
>>>> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>> >>>>>>>>>>>> Implementations, and
>>>> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>> >>>>>>>>>>>> InterfaceAudience).
>>>> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API;
>>>> e.g.
>>>> >>>>>>>>>>>> guava
>>>> >>>>>>>>>>>> 0.12
>>>> >>>>>>>>>>>> Service showing through in our replication API and
>>>> protobufs in
>>>> >>>>>>>>>>>> Admin
>>>> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of
>>>> outreach to
>>>> >>>>>>>>>>>> make
>>>> >>>>>>>>>>>> sure
>>>> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> Beta1 in mid-September?
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>> >>>>>>>>>>>> https://issues.apache.org/jira
>>>> /projects/HBASE/versions/12327188
>>>> >>>>>>>>>>>> Edit
>>>> >>>>>>>>>>>> as
>>>> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not
>>>> get any
>>>> >>>>>>>>>>>> attention
>>>> >>>>>>>>>>>> in next month or so.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll
>>>> leave them
>>>> >>>>>>>>>>>> as
>>>> >>>>>>>>>>>> is
>>>> >>>>>>>>>>>> another while but I'll boot them soon.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> While I have your attention:
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving
>>>> hbase
>>>> >>>>>>>>>>>> thrift
>>>> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy
>>>> enough
>>>> >>>>>>>>>>>> if
>>>> >>>>>>>>>>>> folks
>>>> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
>>>> first
>>>> >>>>>>>>>>>> (0.98?).
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> St.Ack
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
>>>> wrote:
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
>>>> elserj@apache.org>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>> wrote:
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
>>>> >>>>>>>>>>>>>> elserj@apache.org>
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> wrote:
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>> ...
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would
>>>> make the
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>> feature a
>>>> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I
>>>> have
>>>> >>>>>>>>>>>>>>>> no
>>>> >>>>>>>>>>>>>>>> reason
>>>> >>>>>>>>>>>>>>>> to be
>>>> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in
>>>> terms
>>>> >>>>>>>>>>>>>>>> of
>>>> >>>>>>>>>>>>>>>> how
>>>> >>>>>>>>>>>>>>>> much it
>>>> >>>>>>>>>>>>>>>> is covered via testing.
>>>> >>>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>> >>>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
>>>> Josh.
>>>> >>>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See
>>>> 'Space
>>>> >>>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>> Quota' in
>>>> >>>>>>>>>>>>>>> here:
>>>> >>>>>>>>>>>>>>> https://docs.google.com/docume
>>>> nt/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>> >>>>>>>>>>>>>>> Does this little section need an update Josh?
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>> Thanks,
>>>> >>>>>>>>>>>>>>> S
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage
>>>> and
>>>> >>>>>>>>>>>>>>> some
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>> docs
>>>> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my
>>>> plate if
>>>> >>>>>>>>>>>>>> you're good
>>>> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is against
>>>> it :)
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
>>>> before
>>>> >>>>>>>>>>>>>> we
>>>> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll
>>>> double
>>>> >>>>>>>>>>>>>> check
>>>> >>>>>>>>>>>>>> and update
>>>> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> Thanks Josh,
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> S
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>>
>>>
>>>
>>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Update on hbase-2.0.0-beta-1. I'm going to put up a beta-1 RC before
Christmas day.

Here is the list [1]. We're doing pretty good. We've fixed a load since the
last update including some nice cleanups  two weeks ago (e.g. undoing
client dependency on curator/zk watching making the dependence read-only)
but there are still a few hairy ones hanging out there.

 * HBASE-18946 Stochastic load balancer assigns replica regions to the same
RS
   This one is proving a little tough. Ram has been banging his head on it.
We'll be able to clear a bunch of test failures once this is done.

 * HBASE-19112 Suspect methods on Cell to be deprecated
   Good discussion up on the JIRA. We need to be crystal clear around Cell
and derivatives API and Audience when we ship 2.0.

 * HBASE-17204 Make L2 off heap cache default ON
   We need to try this at least so offheaping can be default.

Making asyncfswal default is also ongoing, HBASE-16890
<https://issues.apache.org/jira/browse/HBASE-16890>, making good progress.

I punted HBASE-19147 "All branch-2 unit tests pass" out to beta-2. Our unit
test story has gotten better and a few of us are actively working on
flakies [2] but we may make a beta-1 w/o shutting down all failures.

Speak up if you need help on an issue or if you think we are missing items
from our list.

Thanks for all the hard work,

The hbase-2.0.0 RM.


1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
2.
https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html

On Thu, Nov 16, 2017 at 3:48 PM, Stack <st...@duboce.net> wrote:

> hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
> remaining features and apply final polish to API. There will be a beta-2
> but it is about upgrade/rolling-upgrade and bug fixes ONLY).
>
> Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1
> list this morning. See here [1].
>
> There are about ~12 issues in progress with most of these about to land.
> There are 37 TODO. Many of these are tests we need to run, some are related
> to the backup/restore, but a good few are meaty w/o assignees.
>
> The awkward outstanding ones as I see it are the below:
>
> HBASE-18946 Stochastic load balancer assigns replica regions to the same RS
> HBASE-17204 Make L2 off heap cache default ON
> HBASE-19112 Suspect methods on Cell to be deprecated
> HBASE-19147 All branch-2 unit tests pass
>
> We need to make progress on the above or punt on them (can't punt on the
> last one though).
>
> Any ideas on what configs we should update in hbase2? Dump ideas into:
> HBASE-19148 Edit of default configuration
>
> Still hoping for an early December beta-1 RC. beta-2 hopefully will be
> close behind.
>
> Comments? Thoughts?
>
> Thanks all,
> S
>
> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>
> On Tue, Oct 31, 2017 at 2:56 PM, Stack <st...@duboce.net> wrote:
>
>>
>>
>> On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:
>>
>>> Hoping to keep momentum going from our Stack working on alpha4, I tried
>>> to
>>> take a stab at triaging some of the open beta-1 issues.
>>>
>>> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done sooner
>>> then I'm happy to see it pulled back in. Trying to balance optimism with
>>> realism here, and knowing that documentation unfortunately often gets
>>> pushed to the back-burner.
>>>
>>> Also, I poked some folks on unassigned issues that they've filed for
>>> beta-1, especially in the last few days. If issues don't have an owner
>>> they
>>> are unlikely to get worked. I chatted with stack and he agreed to take on
>>> some of the tasks, but there's a lot of surface area to cover.
>>>
>>> If you you're working on issues that are critical for beta-1, please mark
>>> them as such. Then the rest of the community will know to help prioritize
>>> feedback and reviews there.
>>>
>>> Do we have a general theme for the betas like we did with the alphas?
>>>
>>> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
>>> well? Continue to work on tests throughout?
>>>
>>>
>> Thanks Mike for helping to kick off the beta-1 train.
>>
>> Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what we
>> are going to ship (beta-2 is rolling upgrade and any minor items turned up
>> in testing/burn-in).
>>
>> Thanks,
>> S
>>
>>
>>
>>> Mike
>>>
>>> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org> wrote:
>>>
>>> > +1 go from my POV.
>>> >
>>> >
>>> > On 10/31/17 10:07 AM, Stack wrote:
>>> >
>>> >> I want to push an alpha-4 today. A few items didn't make it
>>> (HBASE-19092).
>>> >> They need more time. We'll pull them in for beta-1. CP API is
>>> basically
>>> >> done. There may be some changes for beta-1 but hopefully only changes
>>> >> informed by experience trying to port an existing Coprocessor to
>>> hbase2.
>>> >>
>>> >> Shout if there is anything that needs to make alpha-4.
>>> >>
>>> >> Thanks,
>>> >> St.Ack
>>> >>
>>> >>
>>> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
>>> wrote:
>>> >>
>>> >> Yup, that was going to be my plan, Mike!
>>> >>>
>>> >>> Making a pass now, and will check back later tonight again. I see
>>> others
>>> >>> have already done some work today on this front.
>>> >>>
>>> >>>
>>> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
>>> >>>
>>> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how
>>> to do
>>> >>>> it directly on the jenkins job, so you don't have to bother with
>>> JIRA
>>> >>>> uploads)
>>> >>>>
>>> >>>> If you're busy, then I can make time tomorrow or Sunday to kick off
>>> >>>> jobs,
>>> >>>> but I want to make sure we're not duplicating effort and jenkins
>>> cycles.
>>> >>>>
>>> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
>>> wrote:
>>> >>>>
>>> >>>> My turn to bump ;)
>>> >>>>
>>> >>>>>
>>> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that
>>> remain
>>> >>>>> needing some more work. The rest are just awaiting a good QA run.
>>> >>>>>
>>> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over the
>>> >>>>> weekend, bump them along as necessary, and get them committed.
>>> Would be
>>> >>>>> great to be able get a vote up on Monday.
>>> >>>>>
>>> >>>>>
>>> >>>>> On 10/24/17 6:03 PM, Stack wrote:
>>> >>>>>
>>> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back and
>>> forth
>>> >>>>>
>>> >>>>>> what remains to be done. Surfacing our thoughts here so you all
>>> clued
>>> >>>>>> in....Or if you think otherwise, please speak up.
>>> >>>>>>
>>> >>>>>> We have ~13 issues to land:
>>> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>> About
>>> >>>>>> two
>>> >>>>>> are meta-issues that are about process which leaves 11.
>>> >>>>>>
>>> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
>>> >>>>>> changed
>>> >>>>>> API/semantic that we need to get out earlier rather than later.
>>> >>>>>>
>>> >>>>>> Once the above is merged, HBASE-13346, change of Filter method
>>> names
>>> >>>>>> to
>>> >>>>>> mention Cell instead of KeyValue can land.
>>> >>>>>>
>>> >>>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
>>> >>>>>> IA.Private objects as params to MasterObserver... Hopefully this
>>> goes
>>> >>>>>> in
>>> >>>>>> soon.
>>> >>>>>>
>>> >>>>>> Duo is hard at work on trackers for flush and compaction for CPs
>>> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for
>>> Tephra)?
>>> >>>>>>
>>> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate
>>> after
>>> >>>>>> Duo
>>> >>>>>> is
>>> >>>>>> done w/ his work above.
>>> >>>>>>
>>> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity
>>> after
>>> >>>>>> all
>>> >>>>>> the
>>> >>>>>> purges allowing CPs do direct calls against Regions in same Host.
>>> >>>>>>
>>> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>> >>>>>>
>>> >>>>>> Another day or two?
>>> >>>>>>
>>> >>>>>> St.Ack
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>>>
>>> >>>>>>> +1
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>> I was trying to work on helping out on the outstanding alpha-4
>>> stuff
>>> >>>>>>>> last
>>> >>>>>>>> week -- will be continuing to try to do the same this week.
>>> >>>>>>>>
>>> >>>>>>>> If you need any help, Stack, or if others need reviews where I
>>> >>>>>>>> haven't
>>> >>>>>>>> noticed on my own: feel free to @mention me.
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are being
>>> >>>>>>>>
>>> >>>>>>>> actively
>>> >>>>>>> worked on. If you get a moment, reviews by you (or anyone else)
>>> helps
>>> >>>>>>> move
>>> >>>>>>> the process along.
>>> >>>>>>>
>>> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
>>> >>>>>>> improvements.
>>> >>>>>>> Then HBASE-13346 can go in.
>>> >>>>>>>
>>> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks like
>>> that
>>> >>>>>>> will
>>> >>>>>>> be addressed by other alpha-4s about to land.
>>> >>>>>>>
>>> >>>>>>> St.Ack
>>> >>>>>>> TODOs: https://issues.apache.org/jira
>>> /projects/HBASE/versions/12341
>>> >>>>>>> 594
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>> (Reviving this thread)
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that
>>> has
>>> >>>>>>>>> the
>>> >>>>>>>>> refactor of the Coprocessor API shutting down access to
>>> internals
>>> >>>>>>>>> marked
>>> >>>>>>>>> InterfaceAudience.Private.
>>> >>>>>>>>>
>>> >>>>>>>>> The outstanding list is here:
>>> >>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341
>>> 594
>>> >>>>>>>>>
>>> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>> >>>>>>>>>
>>> >>>>>>>>> If issue, talk out loud on this thread. If you need a review to
>>> >>>>>>>>> land
>>> >>>>>>>>> an
>>> >>>>>>>>> item, shout on the issue and here; we'll help you out.
>>> >>>>>>>>>
>>> >>>>>>>>> As is, items are coming along nicely I'd say. We need to merge
>>> the
>>> >>>>>>>>> filter
>>> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>> >>>>>>>>>
>>> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and
>>> help
>>> >>>>>>>>> them
>>> >>>>>>>>> test
>>> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence
>>> our
>>> >>>>>>>>> downstreamers know what to expect (or we discover what we
>>> missed
>>> >>>>>>>>> BEFORE
>>> >>>>>>>>> we
>>> >>>>>>>>> beta-1).
>>> >>>>>>>>>
>>> >>>>>>>>> Thanks for time,
>>> >>>>>>>>> S
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net>
>>> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That
>>> should
>>> >>>>>>>>> be
>>> >>>>>>>>>
>>> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to be
>>> done.
>>> >>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for
>>> >>>>>>>>>> alpha3
>>> >>>>>>>>>> but
>>> >>>>>>>>>> it is plain that more time is needed (in spite of valiant
>>> effort
>>> >>>>>>>>>> so
>>> >>>>>>>>>> far
>>> >>>>>>>>>> by
>>> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
>>> whose
>>> >>>>>>>>>> theme is
>>> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>>> >>>>>>>>>> following
>>> >>>>>>>>>> week.
>>> >>>>>>>>>>
>>> >>>>>>>>>> We should then be ready for beta (beta == no new features, no
>>> API
>>> >>>>>>>>>> changes,
>>> >>>>>>>>>> just fixes).
>>> >>>>>>>>>>
>>> >>>>>>>>>> Thanks,
>>> >>>>>>>>>> St.Ack
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
>>> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please
>>> vote on
>>> >>>>>>>>>> it.
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope
>>> to
>>> >>>>>>>>>>> get a
>>> >>>>>>>>>>> release out in the next week or so.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks
>>> are
>>> >>>>>>>>>>> interested in helping out, below are the items I think we
>>> need
>>> >>>>>>>>>>> done
>>> >>>>>>>>>>> for
>>> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API
>>> possibly
>>> >>>>>>>>>>> altering
>>> >>>>>>>>>>> items, and are absent those JIRAs that are making active
>>> >>>>>>>>>>> progress,
>>> >>>>>>>>>>> i.e. the
>>> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
>>> >>>>>>>>>>> needs
>>> >>>>>>>>>>> doing is
>>> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between
>>> branch-1
>>> >>>>>>>>>>> and
>>> >>>>>>>>>>> branch-2
>>> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches
>>> only
>>> >>>>>>>>>>> do
>>> >>>>>>>>>>> it
>>> >>>>>>>>>>> between branch-1 and branch-2.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public /
>>> >>>>>>>>>>> Private
>>> >>>>>>>>>>> interfaces
>>> >>>>>>>>>>> This one is almost done. It could do with a finish,
>>> attention to
>>> >>>>>>>>>>> the
>>> >>>>>>>>>>> items in last comment, and then our codebase could do with
>>> >>>>>>>>>>> another
>>> >>>>>>>>>>> sweep
>>> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in
>>> since
>>> >>>>>>>>>>> the
>>> >>>>>>>>>>> pass
>>> >>>>>>>>>>> that was the basis of this issue.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
>>> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
>>> forced
>>> >>>>>>>>>>> by
>>> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
>>> >>>>>>>>>>> Service'
>>> >>>>>>>>>>> but if
>>> >>>>>>>>>>> anyone else is interested, be my guest.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked on.
>>> Needs
>>> >>>>>>>>>>> finishing.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
>>> Admin
>>> >>>>>>>>>>> and
>>> >>>>>>>>>>> cleanup
>>> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>>> >>>>>>>>>>> AsyncClient
>>> >>>>>>>>>>> and
>>> >>>>>>>>>>> make sure symmetry.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but
>>> needs to
>>> >>>>>>>>>>> be
>>> >>>>>>>>>>> done.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would be
>>> good
>>> >>>>>>>>>>> to
>>> >>>>>>>>>>> try
>>> >>>>>>>>>>> it and figure what needs fixup (as said above, according to
>>> trial
>>> >>>>>>>>>>> by
>>> >>>>>>>>>>> Sean,
>>> >>>>>>>>>>> we might not be too bad here):
>>> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking
>>> to
>>> >>>>>>>>>>> 2.0
>>> >>>>>>>>>>> cluster
>>> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
>>> Master
>>> >>>>>>>>>>> and
>>> >>>>>>>>>>> 1.x
>>> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-17442 Move most of the replication related classes to
>>> >>>>>>>>>>> hbase-server package
>>> >>>>>>>>>>> The above would be good to do generally but it may make for
>>> >>>>>>>>>>> ripples
>>> >>>>>>>>>>> in
>>> >>>>>>>>>>> API so would be good to do now.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid adding
>>> two
>>> >>>>>>>>>>> new
>>> >>>>>>>>>>> types
>>> >>>>>>>>>>> to our API, two types that are nought but curtailed,
>>> read-only
>>> >>>>>>>>>>> views
>>> >>>>>>>>>>> on
>>> >>>>>>>>>>> internals. Input if you have time appreciated.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate
>>> to a
>>> >>>>>>>>>>> hbase2
>>> >>>>>>>>>>> cluster; verify
>>> >>>>>>>>>>> Esteban is looking at this one
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>> >>>>>>>>>>> * HBASE-17143 Scan improvement
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>> >>>>>>>>>>> implementations
>>> >>>>>>>>>>> marked audience Private and returns implementations rather
>>> than
>>> >>>>>>>>>>> Interfaces.
>>> >>>>>>>>>>> In a few locations, we allow returning an alternate
>>> >>>>>>>>>>> implementation
>>> >>>>>>>>>>> altogether which is probably something we don't want a CP
>>> doing.
>>> >>>>>>>>>>> To
>>> >>>>>>>>>>> that
>>> >>>>>>>>>>> end, the following issues started by Duo and Anoop need to be
>>> >>>>>>>>>>> taken
>>> >>>>>>>>>>> to
>>> >>>>>>>>>>> the
>>> >>>>>>>>>>> finish line; ideally they'd have an owner:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
>>> release <=
>>> >>>>>>>>>>> The
>>> >>>>>>>>>>> umbrella issue.
>>> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
>>> >>>>>>>>>>> expose
>>> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
>>> MasterObserver
>>> >>>>>>>>>>> and
>>> >>>>>>>>>>> RegionServerObserver
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Nice-to-haves:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
>>> remove
>>> >>>>>>>>>>> unused
>>> >>>>>>>>>>> TimeRange constructors
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
>>> >>>>>>>>>>> references
>>> >>>>>>>>>>> existing in the code
>>> >>>>>>>>>>> This is the end of an old long-running project moving up on
>>> to
>>> >>>>>>>>>>> Cell
>>> >>>>>>>>>>> Interface. We think it is done but for a few little items
>>> >>>>>>>>>>> (deprecate
>>> >>>>>>>>>>> KV
>>> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is
>>> indeterminate;
>>> >>>>>>>>>>> needs
>>> >>>>>>>>>>> fixing
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>> >>>>>>>>>>> * HBASE-14997
>>> >>>>>>>>>>> Move compareOp and Comparators out of filter to client
>>> package
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> What about:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> If interested in any of the above and you need a legup, just
>>> ask
>>> >>>>>>>>>>> in
>>> >>>>>>>>>>> the
>>> >>>>>>>>>>> issue and I'll be by....
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Thanks,
>>> >>>>>>>>>>> St.Ack
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
>>> >>>>>>>>>>> wrote:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Heads-up:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
>>> >>>>>>>>>>>> Theme
>>> >>>>>>>>>>>> is
>>> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
>>> (guava,
>>> >>>>>>>>>>>> netty,
>>> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
>>> >>>>>>>>>>>> master-carries-no-regions
>>> >>>>>>>>>>>> by default.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
>>> theme
>>> >>>>>>>>>>>> will
>>> >>>>>>>>>>>> be
>>> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we are
>>> not
>>> >>>>>>>>>>>> looking too
>>> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1
>>> >>>>>>>>>>>> client
>>> >>>>>>>>>>>> running
>>> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface
>>> >>>>>>>>>>>> revamp
>>> >>>>>>>>>>>> should be
>>> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>> >>>>>>>>>>>> Implementations, and
>>> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>> >>>>>>>>>>>> InterfaceAudience).
>>> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API; e.g.
>>> >>>>>>>>>>>> guava
>>> >>>>>>>>>>>> 0.12
>>> >>>>>>>>>>>> Service showing through in our replication API and
>>> protobufs in
>>> >>>>>>>>>>>> Admin
>>> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of
>>> outreach to
>>> >>>>>>>>>>>> make
>>> >>>>>>>>>>>> sure
>>> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> Beta1 in mid-September?
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>> >>>>>>>>>>>> https://issues.apache.org/jira
>>> /projects/HBASE/versions/12327188
>>> >>>>>>>>>>>> Edit
>>> >>>>>>>>>>>> as
>>> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get
>>> any
>>> >>>>>>>>>>>> attention
>>> >>>>>>>>>>>> in next month or so.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave
>>> them
>>> >>>>>>>>>>>> as
>>> >>>>>>>>>>>> is
>>> >>>>>>>>>>>> another while but I'll boot them soon.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> While I have your attention:
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving
>>> hbase
>>> >>>>>>>>>>>> thrift
>>> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy
>>> enough
>>> >>>>>>>>>>>> if
>>> >>>>>>>>>>>> folks
>>> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
>>> first
>>> >>>>>>>>>>>> (0.98?).
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> St.Ack
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
>>> wrote:
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
>>> elserj@apache.org>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
>>> >>>>>>>>>>>>>> elserj@apache.org>
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> ...
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would
>>> make the
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> feature a
>>> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I
>>> have
>>> >>>>>>>>>>>>>>>> no
>>> >>>>>>>>>>>>>>>> reason
>>> >>>>>>>>>>>>>>>> to be
>>> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in
>>> terms
>>> >>>>>>>>>>>>>>>> of
>>> >>>>>>>>>>>>>>>> how
>>> >>>>>>>>>>>>>>>> much it
>>> >>>>>>>>>>>>>>>> is covered via testing.
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
>>> Josh.
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See
>>> 'Space
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Quota' in
>>> >>>>>>>>>>>>>>> here:
>>> >>>>>>>>>>>>>>> https://docs.google.com/docume
>>> nt/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>> >>>>>>>>>>>>>>> Does this little section need an update Josh?
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Thanks,
>>> >>>>>>>>>>>>>>> S
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage
>>> and
>>> >>>>>>>>>>>>>>> some
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> docs
>>> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my
>>> plate if
>>> >>>>>>>>>>>>>> you're good
>>> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is against
>>> it :)
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
>>> before
>>> >>>>>>>>>>>>>> we
>>> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll double
>>> >>>>>>>>>>>>>> check
>>> >>>>>>>>>>>>>> and update
>>> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> Thanks Josh,
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> S
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>>
>>
>>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
remaining features and apply final polish to API. There will be a beta-2
but it is about upgrade/rolling-upgrade and bug fixes ONLY).

Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1 list
this morning. See here [1].

There are about ~12 issues in progress with most of these about to land.
There are 37 TODO. Many of these are tests we need to run, some are related
to the backup/restore, but a good few are meaty w/o assignees.

The awkward outstanding ones as I see it are the below:

HBASE-18946 Stochastic load balancer assigns replica regions to the same RS
HBASE-17204 Make L2 off heap cache default ON
HBASE-19112 Suspect methods on Cell to be deprecated
HBASE-19147 All branch-2 unit tests pass

We need to make progress on the above or punt on them (can't punt on the
last one though).

Any ideas on what configs we should update in hbase2? Dump ideas into:
HBASE-19148 Edit of default configuration

Still hoping for an early December beta-1 RC. beta-2 hopefully will be
close behind.

Comments? Thoughts?

Thanks all,
S

1. https://issues.apache.org/jira/projects/HBASE/versions/12340861

On Tue, Oct 31, 2017 at 2:56 PM, Stack <st...@duboce.net> wrote:

>
>
> On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:
>
>> Hoping to keep momentum going from our Stack working on alpha4, I tried to
>> take a stab at triaging some of the open beta-1 issues.
>>
>> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done sooner
>> then I'm happy to see it pulled back in. Trying to balance optimism with
>> realism here, and knowing that documentation unfortunately often gets
>> pushed to the back-burner.
>>
>> Also, I poked some folks on unassigned issues that they've filed for
>> beta-1, especially in the last few days. If issues don't have an owner
>> they
>> are unlikely to get worked. I chatted with stack and he agreed to take on
>> some of the tasks, but there's a lot of surface area to cover.
>>
>> If you you're working on issues that are critical for beta-1, please mark
>> them as such. Then the rest of the community will know to help prioritize
>> feedback and reviews there.
>>
>> Do we have a general theme for the betas like we did with the alphas?
>>
>> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
>> well? Continue to work on tests throughout?
>>
>>
> Thanks Mike for helping to kick off the beta-1 train.
>
> Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what we
> are going to ship (beta-2 is rolling upgrade and any minor items turned up
> in testing/burn-in).
>
> Thanks,
> S
>
>
>
>> Mike
>>
>> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org> wrote:
>>
>> > +1 go from my POV.
>> >
>> >
>> > On 10/31/17 10:07 AM, Stack wrote:
>> >
>> >> I want to push an alpha-4 today. A few items didn't make it
>> (HBASE-19092).
>> >> They need more time. We'll pull them in for beta-1. CP API is basically
>> >> done. There may be some changes for beta-1 but hopefully only changes
>> >> informed by experience trying to port an existing Coprocessor to
>> hbase2.
>> >>
>> >> Shout if there is anything that needs to make alpha-4.
>> >>
>> >> Thanks,
>> >> St.Ack
>> >>
>> >>
>> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
>> wrote:
>> >>
>> >> Yup, that was going to be my plan, Mike!
>> >>>
>> >>> Making a pass now, and will check back later tonight again. I see
>> others
>> >>> have already done some work today on this front.
>> >>>
>> >>>
>> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
>> >>>
>> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how
>> to do
>> >>>> it directly on the jenkins job, so you don't have to bother with JIRA
>> >>>> uploads)
>> >>>>
>> >>>> If you're busy, then I can make time tomorrow or Sunday to kick off
>> >>>> jobs,
>> >>>> but I want to make sure we're not duplicating effort and jenkins
>> cycles.
>> >>>>
>> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
>> wrote:
>> >>>>
>> >>>> My turn to bump ;)
>> >>>>
>> >>>>>
>> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that
>> remain
>> >>>>> needing some more work. The rest are just awaiting a good QA run.
>> >>>>>
>> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over the
>> >>>>> weekend, bump them along as necessary, and get them committed.
>> Would be
>> >>>>> great to be able get a vote up on Monday.
>> >>>>>
>> >>>>>
>> >>>>> On 10/24/17 6:03 PM, Stack wrote:
>> >>>>>
>> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back and
>> forth
>> >>>>>
>> >>>>>> what remains to be done. Surfacing our thoughts here so you all
>> clued
>> >>>>>> in....Or if you think otherwise, please speak up.
>> >>>>>>
>> >>>>>> We have ~13 issues to land:
>> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>> About
>> >>>>>> two
>> >>>>>> are meta-issues that are about process which leaves 11.
>> >>>>>>
>> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
>> >>>>>> changed
>> >>>>>> API/semantic that we need to get out earlier rather than later.
>> >>>>>>
>> >>>>>> Once the above is merged, HBASE-13346, change of Filter method
>> names
>> >>>>>> to
>> >>>>>> mention Cell instead of KeyValue can land.
>> >>>>>>
>> >>>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
>> >>>>>> IA.Private objects as params to MasterObserver... Hopefully this
>> goes
>> >>>>>> in
>> >>>>>> soon.
>> >>>>>>
>> >>>>>> Duo is hard at work on trackers for flush and compaction for CPs
>> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for
>> Tephra)?
>> >>>>>>
>> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate
>> after
>> >>>>>> Duo
>> >>>>>> is
>> >>>>>> done w/ his work above.
>> >>>>>>
>> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after
>> >>>>>> all
>> >>>>>> the
>> >>>>>> purges allowing CPs do direct calls against Regions in same Host.
>> >>>>>>
>> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>> >>>>>>
>> >>>>>> Another day or two?
>> >>>>>>
>> >>>>>> St.Ack
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>> >>>>>>
>> >>>>>>
>> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>>
>> >>>>>>> +1
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>> I was trying to work on helping out on the outstanding alpha-4
>> stuff
>> >>>>>>>> last
>> >>>>>>>> week -- will be continuing to try to do the same this week.
>> >>>>>>>>
>> >>>>>>>> If you need any help, Stack, or if others need reviews where I
>> >>>>>>>> haven't
>> >>>>>>>> noticed on my own: feel free to @mention me.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are being
>> >>>>>>>>
>> >>>>>>>> actively
>> >>>>>>> worked on. If you get a moment, reviews by you (or anyone else)
>> helps
>> >>>>>>> move
>> >>>>>>> the process along.
>> >>>>>>>
>> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
>> >>>>>>> improvements.
>> >>>>>>> Then HBASE-13346 can go in.
>> >>>>>>>
>> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks like
>> that
>> >>>>>>> will
>> >>>>>>> be addressed by other alpha-4s about to land.
>> >>>>>>>
>> >>>>>>> St.Ack
>> >>>>>>> TODOs: https://issues.apache.org/jira
>> /projects/HBASE/versions/12341
>> >>>>>>> 594
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>> (Reviving this thread)
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has
>> >>>>>>>>> the
>> >>>>>>>>> refactor of the Coprocessor API shutting down access to
>> internals
>> >>>>>>>>> marked
>> >>>>>>>>> InterfaceAudience.Private.
>> >>>>>>>>>
>> >>>>>>>>> The outstanding list is here:
>> >>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>> >>>>>>>>>
>> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>> >>>>>>>>>
>> >>>>>>>>> If issue, talk out loud on this thread. If you need a review to
>> >>>>>>>>> land
>> >>>>>>>>> an
>> >>>>>>>>> item, shout on the issue and here; we'll help you out.
>> >>>>>>>>>
>> >>>>>>>>> As is, items are coming along nicely I'd say. We need to merge
>> the
>> >>>>>>>>> filter
>> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>> >>>>>>>>>
>> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and help
>> >>>>>>>>> them
>> >>>>>>>>> test
>> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>> >>>>>>>>> downstreamers know what to expect (or we discover what we missed
>> >>>>>>>>> BEFORE
>> >>>>>>>>> we
>> >>>>>>>>> beta-1).
>> >>>>>>>>>
>> >>>>>>>>> Thanks for time,
>> >>>>>>>>> S
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>> >>>>>>>>>
>> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That
>> should
>> >>>>>>>>> be
>> >>>>>>>>>
>> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to be
>> done.
>> >>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for
>> >>>>>>>>>> alpha3
>> >>>>>>>>>> but
>> >>>>>>>>>> it is plain that more time is needed (in spite of valiant
>> effort
>> >>>>>>>>>> so
>> >>>>>>>>>> far
>> >>>>>>>>>> by
>> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
>> whose
>> >>>>>>>>>> theme is
>> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>> >>>>>>>>>> following
>> >>>>>>>>>> week.
>> >>>>>>>>>>
>> >>>>>>>>>> We should then be ready for beta (beta == no new features, no
>> API
>> >>>>>>>>>> changes,
>> >>>>>>>>>> just fixes).
>> >>>>>>>>>>
>> >>>>>>>>>> Thanks,
>> >>>>>>>>>> St.Ack
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote
>> on
>> >>>>>>>>>> it.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
>> >>>>>>>>>>> get a
>> >>>>>>>>>>> release out in the next week or so.
>> >>>>>>>>>>>
>> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks
>> are
>> >>>>>>>>>>> interested in helping out, below are the items I think we need
>> >>>>>>>>>>> done
>> >>>>>>>>>>> for
>> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API possibly
>> >>>>>>>>>>> altering
>> >>>>>>>>>>> items, and are absent those JIRAs that are making active
>> >>>>>>>>>>> progress,
>> >>>>>>>>>>> i.e. the
>> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
>> >>>>>>>>>>> needs
>> >>>>>>>>>>> doing is
>> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1
>> >>>>>>>>>>> and
>> >>>>>>>>>>> branch-2
>> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches
>> only
>> >>>>>>>>>>> do
>> >>>>>>>>>>> it
>> >>>>>>>>>>> between branch-1 and branch-2.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public /
>> >>>>>>>>>>> Private
>> >>>>>>>>>>> interfaces
>> >>>>>>>>>>> This one is almost done. It could do with a finish, attention
>> to
>> >>>>>>>>>>> the
>> >>>>>>>>>>> items in last comment, and then our codebase could do with
>> >>>>>>>>>>> another
>> >>>>>>>>>>> sweep
>> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in since
>> >>>>>>>>>>> the
>> >>>>>>>>>>> pass
>> >>>>>>>>>>> that was the basis of this issue.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
>> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
>> forced
>> >>>>>>>>>>> by
>> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
>> >>>>>>>>>>> Service'
>> >>>>>>>>>>> but if
>> >>>>>>>>>>> anyone else is interested, be my guest.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked on.
>> Needs
>> >>>>>>>>>>> finishing.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
>> Admin
>> >>>>>>>>>>> and
>> >>>>>>>>>>> cleanup
>> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>> >>>>>>>>>>> AsyncClient
>> >>>>>>>>>>> and
>> >>>>>>>>>>> make sure symmetry.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but
>> needs to
>> >>>>>>>>>>> be
>> >>>>>>>>>>> done.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would be
>> good
>> >>>>>>>>>>> to
>> >>>>>>>>>>> try
>> >>>>>>>>>>> it and figure what needs fixup (as said above, according to
>> trial
>> >>>>>>>>>>> by
>> >>>>>>>>>>> Sean,
>> >>>>>>>>>>> we might not be too bad here):
>> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking to
>> >>>>>>>>>>> 2.0
>> >>>>>>>>>>> cluster
>> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
>> Master
>> >>>>>>>>>>> and
>> >>>>>>>>>>> 1.x
>> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-17442 Move most of the replication related classes to
>> >>>>>>>>>>> hbase-server package
>> >>>>>>>>>>> The above would be good to do generally but it may make for
>> >>>>>>>>>>> ripples
>> >>>>>>>>>>> in
>> >>>>>>>>>>> API so would be good to do now.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid adding
>> two
>> >>>>>>>>>>> new
>> >>>>>>>>>>> types
>> >>>>>>>>>>> to our API, two types that are nought but curtailed, read-only
>> >>>>>>>>>>> views
>> >>>>>>>>>>> on
>> >>>>>>>>>>> internals. Input if you have time appreciated.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to
>> a
>> >>>>>>>>>>> hbase2
>> >>>>>>>>>>> cluster; verify
>> >>>>>>>>>>> Esteban is looking at this one
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>> >>>>>>>>>>> * HBASE-17143 Scan improvement
>> >>>>>>>>>>>
>> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>> >>>>>>>>>>> implementations
>> >>>>>>>>>>> marked audience Private and returns implementations rather
>> than
>> >>>>>>>>>>> Interfaces.
>> >>>>>>>>>>> In a few locations, we allow returning an alternate
>> >>>>>>>>>>> implementation
>> >>>>>>>>>>> altogether which is probably something we don't want a CP
>> doing.
>> >>>>>>>>>>> To
>> >>>>>>>>>>> that
>> >>>>>>>>>>> end, the following issues started by Duo and Anoop need to be
>> >>>>>>>>>>> taken
>> >>>>>>>>>>> to
>> >>>>>>>>>>> the
>> >>>>>>>>>>> finish line; ideally they'd have an owner:
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
>> release <=
>> >>>>>>>>>>> The
>> >>>>>>>>>>> umbrella issue.
>> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
>> >>>>>>>>>>> expose
>> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
>> MasterObserver
>> >>>>>>>>>>> and
>> >>>>>>>>>>> RegionServerObserver
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Nice-to-haves:
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
>> remove
>> >>>>>>>>>>> unused
>> >>>>>>>>>>> TimeRange constructors
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
>> >>>>>>>>>>> references
>> >>>>>>>>>>> existing in the code
>> >>>>>>>>>>> This is the end of an old long-running project moving up on to
>> >>>>>>>>>>> Cell
>> >>>>>>>>>>> Interface. We think it is done but for a few little items
>> >>>>>>>>>>> (deprecate
>> >>>>>>>>>>> KV
>> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is
>> indeterminate;
>> >>>>>>>>>>> needs
>> >>>>>>>>>>> fixing
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>> >>>>>>>>>>> * HBASE-14997
>> >>>>>>>>>>> Move compareOp and Comparators out of filter to client package
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>> >>>>>>>>>>>
>> >>>>>>>>>>> What about:
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>> >>>>>>>>>>>
>> >>>>>>>>>>> If interested in any of the above and you need a legup, just
>> ask
>> >>>>>>>>>>> in
>> >>>>>>>>>>> the
>> >>>>>>>>>>> issue and I'll be by....
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thanks,
>> >>>>>>>>>>> St.Ack
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Heads-up:
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
>> >>>>>>>>>>>> Theme
>> >>>>>>>>>>>> is
>> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
>> (guava,
>> >>>>>>>>>>>> netty,
>> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
>> >>>>>>>>>>>> master-carries-no-regions
>> >>>>>>>>>>>> by default.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
>> theme
>> >>>>>>>>>>>> will
>> >>>>>>>>>>>> be
>> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we are
>> not
>> >>>>>>>>>>>> looking too
>> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1
>> >>>>>>>>>>>> client
>> >>>>>>>>>>>> running
>> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface
>> >>>>>>>>>>>> revamp
>> >>>>>>>>>>>> should be
>> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>> >>>>>>>>>>>> Implementations, and
>> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>> >>>>>>>>>>>> InterfaceAudience).
>> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API; e.g.
>> >>>>>>>>>>>> guava
>> >>>>>>>>>>>> 0.12
>> >>>>>>>>>>>> Service showing through in our replication API and protobufs
>> in
>> >>>>>>>>>>>> Admin
>> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach
>> to
>> >>>>>>>>>>>> make
>> >>>>>>>>>>>> sure
>> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Beta1 in mid-September?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>> >>>>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327
>> 188
>> >>>>>>>>>>>> Edit
>> >>>>>>>>>>>> as
>> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get
>> any
>> >>>>>>>>>>>> attention
>> >>>>>>>>>>>> in next month or so.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave
>> them
>> >>>>>>>>>>>> as
>> >>>>>>>>>>>> is
>> >>>>>>>>>>>> another while but I'll boot them soon.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> While I have your attention:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving
>> hbase
>> >>>>>>>>>>>> thrift
>> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy
>> enough
>> >>>>>>>>>>>> if
>> >>>>>>>>>>>> folks
>> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
>> first
>> >>>>>>>>>>>> (0.98?).
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> St.Ack
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
>> elserj@apache.org>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
>> >>>>>>>>>>>>>> elserj@apache.org>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> ...
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make
>> the
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> feature a
>> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I
>> have
>> >>>>>>>>>>>>>>>> no
>> >>>>>>>>>>>>>>>> reason
>> >>>>>>>>>>>>>>>> to be
>> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in
>> terms
>> >>>>>>>>>>>>>>>> of
>> >>>>>>>>>>>>>>>> how
>> >>>>>>>>>>>>>>>> much it
>> >>>>>>>>>>>>>>>> is covered via testing.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
>> Josh.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See
>> 'Space
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Quota' in
>> >>>>>>>>>>>>>>> here:
>> >>>>>>>>>>>>>>> https://docs.google.com/docume
>> nt/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>> >>>>>>>>>>>>>>> Does this little section need an update Josh?
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>>>> S
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and
>> >>>>>>>>>>>>>>> some
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> docs
>> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my plate
>> if
>> >>>>>>>>>>>>>> you're good
>> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is against it
>> :)
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
>> before
>> >>>>>>>>>>>>>> we
>> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll double
>> >>>>>>>>>>>>>> check
>> >>>>>>>>>>>>>> and update
>> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thanks Josh,
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> S
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>
>> >>
>>
>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:

> Hoping to keep momentum going from our Stack working on alpha4, I tried to
> take a stab at triaging some of the open beta-1 issues.
>
> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done sooner
> then I'm happy to see it pulled back in. Trying to balance optimism with
> realism here, and knowing that documentation unfortunately often gets
> pushed to the back-burner.
>
> Also, I poked some folks on unassigned issues that they've filed for
> beta-1, especially in the last few days. If issues don't have an owner they
> are unlikely to get worked. I chatted with stack and he agreed to take on
> some of the tasks, but there's a lot of surface area to cover.
>
> If you you're working on issues that are critical for beta-1, please mark
> them as such. Then the rest of the community will know to help prioritize
> feedback and reviews there.
>
> Do we have a general theme for the betas like we did with the alphas?
>
> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
> well? Continue to work on tests throughout?
>
>
Thanks Mike for helping to kick off the beta-1 train.

Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what we
are going to ship (beta-2 is rolling upgrade and any minor items turned up
in testing/burn-in).

Thanks,
S



> Mike
>
> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org> wrote:
>
> > +1 go from my POV.
> >
> >
> > On 10/31/17 10:07 AM, Stack wrote:
> >
> >> I want to push an alpha-4 today. A few items didn't make it
> (HBASE-19092).
> >> They need more time. We'll pull them in for beta-1. CP API is basically
> >> done. There may be some changes for beta-1 but hopefully only changes
> >> informed by experience trying to port an existing Coprocessor to hbase2.
> >>
> >> Shout if there is anything that needs to make alpha-4.
> >>
> >> Thanks,
> >> St.Ack
> >>
> >>
> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
> wrote:
> >>
> >> Yup, that was going to be my plan, Mike!
> >>>
> >>> Making a pass now, and will check back later tonight again. I see
> others
> >>> have already done some work today on this front.
> >>>
> >>>
> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
> >>>
> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how to
> do
> >>>> it directly on the jenkins job, so you don't have to bother with JIRA
> >>>> uploads)
> >>>>
> >>>> If you're busy, then I can make time tomorrow or Sunday to kick off
> >>>> jobs,
> >>>> but I want to make sure we're not duplicating effort and jenkins
> cycles.
> >>>>
> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
> wrote:
> >>>>
> >>>> My turn to bump ;)
> >>>>
> >>>>>
> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that
> remain
> >>>>> needing some more work. The rest are just awaiting a good QA run.
> >>>>>
> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over the
> >>>>> weekend, bump them along as necessary, and get them committed. Would
> be
> >>>>> great to be able get a vote up on Monday.
> >>>>>
> >>>>>
> >>>>> On 10/24/17 6:03 PM, Stack wrote:
> >>>>>
> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back and
> forth
> >>>>>
> >>>>>> what remains to be done. Surfacing our thoughts here so you all
> clued
> >>>>>> in....Or if you think otherwise, please speak up.
> >>>>>>
> >>>>>> We have ~13 issues to land:
> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> About
> >>>>>> two
> >>>>>> are meta-issues that are about process which leaves 11.
> >>>>>>
> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
> >>>>>> changed
> >>>>>> API/semantic that we need to get out earlier rather than later.
> >>>>>>
> >>>>>> Once the above is merged, HBASE-13346, change of Filter method names
> >>>>>> to
> >>>>>> mention Cell instead of KeyValue can land.
> >>>>>>
> >>>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
> >>>>>> IA.Private objects as params to MasterObserver... Hopefully this
> goes
> >>>>>> in
> >>>>>> soon.
> >>>>>>
> >>>>>> Duo is hard at work on trackers for flush and compaction for CPs
> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
> >>>>>>
> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after
> >>>>>> Duo
> >>>>>> is
> >>>>>> done w/ his work above.
> >>>>>>
> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after
> >>>>>> all
> >>>>>> the
> >>>>>> purges allowing CPs do direct calls against Regions in same Host.
> >>>>>>
> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> >>>>>>
> >>>>>> Another day or two?
> >>>>>>
> >>>>>> St.Ack
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>>
> >>>>>>> +1
> >>>>>>>
> >>>>>>>
> >>>>>>>> I was trying to work on helping out on the outstanding alpha-4
> stuff
> >>>>>>>> last
> >>>>>>>> week -- will be continuing to try to do the same this week.
> >>>>>>>>
> >>>>>>>> If you need any help, Stack, or if others need reviews where I
> >>>>>>>> haven't
> >>>>>>>> noticed on my own: feel free to @mention me.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are being
> >>>>>>>>
> >>>>>>>> actively
> >>>>>>> worked on. If you get a moment, reviews by you (or anyone else)
> helps
> >>>>>>> move
> >>>>>>> the process along.
> >>>>>>>
> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
> >>>>>>> improvements.
> >>>>>>> Then HBASE-13346 can go in.
> >>>>>>>
> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks like that
> >>>>>>> will
> >>>>>>> be addressed by other alpha-4s about to land.
> >>>>>>>
> >>>>>>> St.Ack
> >>>>>>> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/
> 12341
> >>>>>>> 594
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>> (Reviving this thread)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has
> >>>>>>>>> the
> >>>>>>>>> refactor of the Coprocessor API shutting down access to internals
> >>>>>>>>> marked
> >>>>>>>>> InterfaceAudience.Private.
> >>>>>>>>>
> >>>>>>>>> The outstanding list is here:
> >>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >>>>>>>>>
> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
> >>>>>>>>>
> >>>>>>>>> If issue, talk out loud on this thread. If you need a review to
> >>>>>>>>> land
> >>>>>>>>> an
> >>>>>>>>> item, shout on the issue and here; we'll help you out.
> >>>>>>>>>
> >>>>>>>>> As is, items are coming along nicely I'd say. We need to merge
> the
> >>>>>>>>> filter
> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> >>>>>>>>>
> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and help
> >>>>>>>>> them
> >>>>>>>>> test
> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
> >>>>>>>>> downstreamers know what to expect (or we discover what we missed
> >>>>>>>>> BEFORE
> >>>>>>>>> we
> >>>>>>>>> beta-1).
> >>>>>>>>>
> >>>>>>>>> Thanks for time,
> >>>>>>>>> S
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> >>>>>>>>>
> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That
> should
> >>>>>>>>> be
> >>>>>>>>>
> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to be
> done.
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for
> >>>>>>>>>> alpha3
> >>>>>>>>>> but
> >>>>>>>>>> it is plain that more time is needed (in spite of valiant effort
> >>>>>>>>>> so
> >>>>>>>>>> far
> >>>>>>>>>> by
> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
> whose
> >>>>>>>>>> theme is
> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> >>>>>>>>>> following
> >>>>>>>>>> week.
> >>>>>>>>>>
> >>>>>>>>>> We should then be ready for beta (beta == no new features, no
> API
> >>>>>>>>>> changes,
> >>>>>>>>>> just fixes).
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> St.Ack
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote
> on
> >>>>>>>>>> it.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
> >>>>>>>>>>> get a
> >>>>>>>>>>> release out in the next week or so.
> >>>>>>>>>>>
> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> >>>>>>>>>>> interested in helping out, below are the items I think we need
> >>>>>>>>>>> done
> >>>>>>>>>>> for
> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API possibly
> >>>>>>>>>>> altering
> >>>>>>>>>>> items, and are absent those JIRAs that are making active
> >>>>>>>>>>> progress,
> >>>>>>>>>>> i.e. the
> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
> >>>>>>>>>>> needs
> >>>>>>>>>>> doing is
> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1
> >>>>>>>>>>> and
> >>>>>>>>>>> branch-2
> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only
> >>>>>>>>>>> do
> >>>>>>>>>>> it
> >>>>>>>>>>> between branch-1 and branch-2.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public /
> >>>>>>>>>>> Private
> >>>>>>>>>>> interfaces
> >>>>>>>>>>> This one is almost done. It could do with a finish, attention
> to
> >>>>>>>>>>> the
> >>>>>>>>>>> items in last comment, and then our codebase could do with
> >>>>>>>>>>> another
> >>>>>>>>>>> sweep
> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in since
> >>>>>>>>>>> the
> >>>>>>>>>>> pass
> >>>>>>>>>>> that was the basis of this issue.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
> forced
> >>>>>>>>>>> by
> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> >>>>>>>>>>> Service'
> >>>>>>>>>>> but if
> >>>>>>>>>>> anyone else is interested, be my guest.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked on.
> Needs
> >>>>>>>>>>> finishing.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
> Admin
> >>>>>>>>>>> and
> >>>>>>>>>>> cleanup
> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
> >>>>>>>>>>> AsyncClient
> >>>>>>>>>>> and
> >>>>>>>>>>> make sure symmetry.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but needs
> to
> >>>>>>>>>>> be
> >>>>>>>>>>> done.
> >>>>>>>>>>>
> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would be
> good
> >>>>>>>>>>> to
> >>>>>>>>>>> try
> >>>>>>>>>>> it and figure what needs fixup (as said above, according to
> trial
> >>>>>>>>>>> by
> >>>>>>>>>>> Sean,
> >>>>>>>>>>> we might not be too bad here):
> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking to
> >>>>>>>>>>> 2.0
> >>>>>>>>>>> cluster
> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
> Master
> >>>>>>>>>>> and
> >>>>>>>>>>> 1.x
> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-17442 Move most of the replication related classes to
> >>>>>>>>>>> hbase-server package
> >>>>>>>>>>> The above would be good to do generally but it may make for
> >>>>>>>>>>> ripples
> >>>>>>>>>>> in
> >>>>>>>>>>> API so would be good to do now.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid adding two
> >>>>>>>>>>> new
> >>>>>>>>>>> types
> >>>>>>>>>>> to our API, two types that are nought but curtailed, read-only
> >>>>>>>>>>> views
> >>>>>>>>>>> on
> >>>>>>>>>>> internals. Input if you have time appreciated.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
> >>>>>>>>>>> hbase2
> >>>>>>>>>>> cluster; verify
> >>>>>>>>>>> Esteban is looking at this one
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> >>>>>>>>>>> * HBASE-17143 Scan improvement
> >>>>>>>>>>>
> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
> >>>>>>>>>>> implementations
> >>>>>>>>>>> marked audience Private and returns implementations rather than
> >>>>>>>>>>> Interfaces.
> >>>>>>>>>>> In a few locations, we allow returning an alternate
> >>>>>>>>>>> implementation
> >>>>>>>>>>> altogether which is probably something we don't want a CP
> doing.
> >>>>>>>>>>> To
> >>>>>>>>>>> that
> >>>>>>>>>>> end, the following issues started by Duo and Anoop need to be
> >>>>>>>>>>> taken
> >>>>>>>>>>> to
> >>>>>>>>>>> the
> >>>>>>>>>>> finish line; ideally they'd have an owner:
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release
> <=
> >>>>>>>>>>> The
> >>>>>>>>>>> umbrella issue.
> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
> >>>>>>>>>>> expose
> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
> MasterObserver
> >>>>>>>>>>> and
> >>>>>>>>>>> RegionServerObserver
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Nice-to-haves:
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
> >>>>>>>>>>> unused
> >>>>>>>>>>> TimeRange constructors
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
> >>>>>>>>>>> references
> >>>>>>>>>>> existing in the code
> >>>>>>>>>>> This is the end of an old long-running project moving up on to
> >>>>>>>>>>> Cell
> >>>>>>>>>>> Interface. We think it is done but for a few little items
> >>>>>>>>>>> (deprecate
> >>>>>>>>>>> KV
> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
> >>>>>>>>>>> needs
> >>>>>>>>>>> fixing
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
> >>>>>>>>>>> * HBASE-14997
> >>>>>>>>>>> Move compareOp and Comparators out of filter to client package
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
> >>>>>>>>>>>
> >>>>>>>>>>> What about:
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
> >>>>>>>>>>>
> >>>>>>>>>>> If interested in any of the above and you need a legup, just
> ask
> >>>>>>>>>>> in
> >>>>>>>>>>> the
> >>>>>>>>>>> issue and I'll be by....
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> St.Ack
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Heads-up:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
> >>>>>>>>>>>> Theme
> >>>>>>>>>>>> is
> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
> (guava,
> >>>>>>>>>>>> netty,
> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
> >>>>>>>>>>>> master-carries-no-regions
> >>>>>>>>>>>> by default.
> >>>>>>>>>>>>
> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
> theme
> >>>>>>>>>>>> will
> >>>>>>>>>>>> be
> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we are
> not
> >>>>>>>>>>>> looking too
> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1
> >>>>>>>>>>>> client
> >>>>>>>>>>>> running
> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface
> >>>>>>>>>>>> revamp
> >>>>>>>>>>>> should be
> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
> >>>>>>>>>>>> Implementations, and
> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
> >>>>>>>>>>>> InterfaceAudience).
> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API; e.g.
> >>>>>>>>>>>> guava
> >>>>>>>>>>>> 0.12
> >>>>>>>>>>>> Service showing through in our replication API and protobufs
> in
> >>>>>>>>>>>> Admin
> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach
> to
> >>>>>>>>>>>> make
> >>>>>>>>>>>> sure
> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Beta1 in mid-September?
> >>>>>>>>>>>>
> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
> >>>>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> 12327188
> >>>>>>>>>>>> Edit
> >>>>>>>>>>>> as
> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get
> any
> >>>>>>>>>>>> attention
> >>>>>>>>>>>> in next month or so.
> >>>>>>>>>>>>
> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave
> them
> >>>>>>>>>>>> as
> >>>>>>>>>>>> is
> >>>>>>>>>>>> another while but I'll boot them soon.
> >>>>>>>>>>>>
> >>>>>>>>>>>> While I have your attention:
> >>>>>>>>>>>>
> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving
> hbase
> >>>>>>>>>>>> thrift
> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy
> enough
> >>>>>>>>>>>> if
> >>>>>>>>>>>> folks
> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
> first
> >>>>>>>>>>>> (0.98?).
> >>>>>>>>>>>>
> >>>>>>>>>>>> St.Ack
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <elserj@apache.org
> >
> >>>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
> >>>>>>>>>>>>>> elserj@apache.org>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make
> the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> feature a
> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I have
> >>>>>>>>>>>>>>>> no
> >>>>>>>>>>>>>>>> reason
> >>>>>>>>>>>>>>>> to be
> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in
> terms
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>> how
> >>>>>>>>>>>>>>>> much it
> >>>>>>>>>>>>>>>> is covered via testing.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
> Josh.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See
> 'Space
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Quota' in
> >>>>>>>>>>>>>>> here:
> >>>>>>>>>>>>>>> https://docs.google.com/document/d/
> 1WCsVlnHjJeKUcl7wHwqb4z9i
> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> >>>>>>>>>>>>>>> Does this little section need an update Josh?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> S
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and
> >>>>>>>>>>>>>>> some
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> docs
> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my plate
> if
> >>>>>>>>>>>>>> you're good
> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is against it
> :)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
> before
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll double
> >>>>>>>>>>>>>> check
> >>>>>>>>>>>>>> and update
> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks Josh,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> S
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
>

Re: Moving 2.0 forward

[Mike Drob <md...@apache.org>]

Hoping to keep momentum going from our Stack working on alpha4, I tried to
take a stab at triaging some of the open beta-1 issues.

I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done sooner
then I'm happy to see it pulled back in. Trying to balance optimism with
realism here, and knowing that documentation unfortunately often gets
pushed to the back-burner.

Also, I poked some folks on unassigned issues that they've filed for
beta-1, especially in the last few days. If issues don't have an owner they
are unlikely to get worked. I chatted with stack and he agreed to take on
some of the tasks, but there's a lot of surface area to cover.

If you you're working on issues that are critical for beta-1, please mark
them as such. Then the rest of the community will know to help prioritize
feedback and reviews there.

Do we have a general theme for the betas like we did with the alphas?

Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
well? Continue to work on tests throughout?

Mike

On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org> wrote:

> +1 go from my POV.
>
>
> On 10/31/17 10:07 AM, Stack wrote:
>
>> I want to push an alpha-4 today. A few items didn't make it (HBASE-19092).
>> They need more time. We'll pull them in for beta-1. CP API is basically
>> done. There may be some changes for beta-1 but hopefully only changes
>> informed by experience trying to port an existing Coprocessor to hbase2.
>>
>> Shout if there is anything that needs to make alpha-4.
>>
>> Thanks,
>> St.Ack
>>
>>
>> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com> wrote:
>>
>> Yup, that was going to be my plan, Mike!
>>>
>>> Making a pass now, and will check back later tonight again. I see others
>>> have already done some work today on this front.
>>>
>>>
>>> On 10/27/17 11:38 PM, Mike Drob wrote:
>>>
>>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how to do
>>>> it directly on the jenkins job, so you don't have to bother with JIRA
>>>> uploads)
>>>>
>>>> If you're busy, then I can make time tomorrow or Sunday to kick off
>>>> jobs,
>>>> but I want to make sure we're not duplicating effort and jenkins cycles.
>>>>
>>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org> wrote:
>>>>
>>>> My turn to bump ;)
>>>>
>>>>>
>>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that remain
>>>>> needing some more work. The rest are just awaiting a good QA run.
>>>>>
>>>>> Unless I hear otherwise, I'll try to keep an eye on things over the
>>>>> weekend, bump them along as necessary, and get them committed. Would be
>>>>> great to be able get a vote up on Monday.
>>>>>
>>>>>
>>>>> On 10/24/17 6:03 PM, Stack wrote:
>>>>>
>>>>> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
>>>>>
>>>>>> what remains to be done. Surfacing our thoughts here so you all clued
>>>>>> in....Or if you think otherwise, please speak up.
>>>>>>
>>>>>> We have ~13 issues to land:
>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About
>>>>>> two
>>>>>> are meta-issues that are about process which leaves 11.
>>>>>>
>>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
>>>>>> changed
>>>>>> API/semantic that we need to get out earlier rather than later.
>>>>>>
>>>>>> Once the above is merged, HBASE-13346, change of Filter method names
>>>>>> to
>>>>>> mention Cell instead of KeyValue can land.
>>>>>>
>>>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
>>>>>> IA.Private objects as params to MasterObserver... Hopefully this goes
>>>>>> in
>>>>>> soon.
>>>>>>
>>>>>> Duo is hard at work on trackers for flush and compaction for CPs
>>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>>>>>>
>>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after
>>>>>> Duo
>>>>>> is
>>>>>> done w/ his work above.
>>>>>>
>>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after
>>>>>> all
>>>>>> the
>>>>>> purges allowing CPs do direct calls against Regions in same Host.
>>>>>>
>>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>>>>>
>>>>>> Another day or two?
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>>
>>>>>>>> I was trying to work on helping out on the outstanding alpha-4 stuff
>>>>>>>> last
>>>>>>>> week -- will be continuing to try to do the same this week.
>>>>>>>>
>>>>>>>> If you need any help, Stack, or if others need reviews where I
>>>>>>>> haven't
>>>>>>>> noticed on my own: feel free to @mention me.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks for the offer Josh. All items seem assigned and are being
>>>>>>>>
>>>>>>>> actively
>>>>>>> worked on. If you get a moment, reviews by you (or anyone else) helps
>>>>>>> move
>>>>>>> the process along.
>>>>>>>
>>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
>>>>>>> improvements.
>>>>>>> Then HBASE-13346 can go in.
>>>>>>>
>>>>>>> You are already helping out on HBASE-18906, thanks. Looks like that
>>>>>>> will
>>>>>>> be addressed by other alpha-4s about to land.
>>>>>>>
>>>>>>> St.Ack
>>>>>>> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341
>>>>>>> 594
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 10/23/17 12:53 PM, Stack wrote:
>>>>>>>
>>>>>>>
>>>>>>>> (Reviving this thread)
>>>>>>>>
>>>>>>>>
>>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has
>>>>>>>>> the
>>>>>>>>> refactor of the Coprocessor API shutting down access to internals
>>>>>>>>> marked
>>>>>>>>> InterfaceAudience.Private.
>>>>>>>>>
>>>>>>>>> The outstanding list is here:
>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>>>>>>
>>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>>>>>>
>>>>>>>>> If issue, talk out loud on this thread. If you need a review to
>>>>>>>>> land
>>>>>>>>> an
>>>>>>>>> item, shout on the issue and here; we'll help you out.
>>>>>>>>>
>>>>>>>>> As is, items are coming along nicely I'd say. We need to merge the
>>>>>>>>> filter
>>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>>>>>>
>>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and help
>>>>>>>>> them
>>>>>>>>> test
>>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>>>>>>>>> downstreamers know what to expect (or we discover what we missed
>>>>>>>>> BEFORE
>>>>>>>>> we
>>>>>>>>> beta-1).
>>>>>>>>>
>>>>>>>>> Thanks for time,
>>>>>>>>> S
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>>>>>>>>
>>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That should
>>>>>>>>> be
>>>>>>>>>
>>>>>>>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for
>>>>>>>>>> alpha3
>>>>>>>>>> but
>>>>>>>>>> it is plain that more time is needed (in spite of valiant effort
>>>>>>>>>> so
>>>>>>>>>> far
>>>>>>>>>> by
>>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>>>>>>>>>> theme is
>>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>>>>>>>>>> following
>>>>>>>>>> week.
>>>>>>>>>>
>>>>>>>>>> We should then be ready for beta (beta == no new features, no API
>>>>>>>>>> changes,
>>>>>>>>>> just fixes).
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> St.Ack
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>>>>>>>>
>>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on
>>>>>>>>>> it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
>>>>>>>>>>> get a
>>>>>>>>>>> release out in the next week or so.
>>>>>>>>>>>
>>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>>>>>>>>> interested in helping out, below are the items I think we need
>>>>>>>>>>> done
>>>>>>>>>>> for
>>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API possibly
>>>>>>>>>>> altering
>>>>>>>>>>> items, and are absent those JIRAs that are making active
>>>>>>>>>>> progress,
>>>>>>>>>>> i.e. the
>>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
>>>>>>>>>>> needs
>>>>>>>>>>> doing is
>>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1
>>>>>>>>>>> and
>>>>>>>>>>> branch-2
>>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only
>>>>>>>>>>> do
>>>>>>>>>>> it
>>>>>>>>>>> between branch-1 and branch-2.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public /
>>>>>>>>>>> Private
>>>>>>>>>>> interfaces
>>>>>>>>>>> This one is almost done. It could do with a finish, attention to
>>>>>>>>>>> the
>>>>>>>>>>> items in last comment, and then our codebase could do with
>>>>>>>>>>> another
>>>>>>>>>>> sweep
>>>>>>>>>>> after the spirit of this issue since a bunch has gone in since
>>>>>>>>>>> the
>>>>>>>>>>> pass
>>>>>>>>>>> that was the basis of this issue.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-10504 Define Replication Interface
>>>>>>>>>>> I was going to take a crack at this as part of the revamp forced
>>>>>>>>>>> by
>>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
>>>>>>>>>>> Service'
>>>>>>>>>>> but if
>>>>>>>>>>> anyone else is interested, be my guest.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>>>>>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>>>>>>>>> finishing.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
>>>>>>>>>>> and
>>>>>>>>>>> cleanup
>>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>>>>>>>>>>> AsyncClient
>>>>>>>>>>> and
>>>>>>>>>>> make sure symmetry.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but needs to
>>>>>>>>>>> be
>>>>>>>>>>> done.
>>>>>>>>>>>
>>>>>>>>>>> Rolling upgrade will have implications for our API. Would be good
>>>>>>>>>>> to
>>>>>>>>>>> try
>>>>>>>>>>> it and figure what needs fixup (as said above, according to trial
>>>>>>>>>>> by
>>>>>>>>>>> Sean,
>>>>>>>>>>> we might not be too bad here):
>>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking to
>>>>>>>>>>> 2.0
>>>>>>>>>>> cluster
>>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master
>>>>>>>>>>> and
>>>>>>>>>>> 1.x
>>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-17442 Move most of the replication related classes to
>>>>>>>>>>> hbase-server package
>>>>>>>>>>> The above would be good to do generally but it may make for
>>>>>>>>>>> ripples
>>>>>>>>>>> in
>>>>>>>>>>> API so would be good to do now.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>>>>>>>> Balazs is working on this. The idea is that we avoid adding two
>>>>>>>>>>> new
>>>>>>>>>>> types
>>>>>>>>>>> to our API, two types that are nought but curtailed, read-only
>>>>>>>>>>> views
>>>>>>>>>>> on
>>>>>>>>>>> internals. Input if you have time appreciated.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
>>>>>>>>>>> hbase2
>>>>>>>>>>> cluster; verify
>>>>>>>>>>> Esteban is looking at this one
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>>>>>>>>> * HBASE-17143 Scan improvement
>>>>>>>>>>>
>>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>>>>>>>> implementations
>>>>>>>>>>> marked audience Private and returns implementations rather than
>>>>>>>>>>> Interfaces.
>>>>>>>>>>> In a few locations, we allow returning an alternate
>>>>>>>>>>> implementation
>>>>>>>>>>> altogether which is probably something we don't want a CP doing.
>>>>>>>>>>> To
>>>>>>>>>>> that
>>>>>>>>>>> end, the following issues started by Duo and Anoop need to be
>>>>>>>>>>> taken
>>>>>>>>>>> to
>>>>>>>>>>> the
>>>>>>>>>>> finish line; ideally they'd have an owner:
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
>>>>>>>>>>> The
>>>>>>>>>>> umbrella issue.
>>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
>>>>>>>>>>> expose
>>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver
>>>>>>>>>>> and
>>>>>>>>>>> RegionServerObserver
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Nice-to-haves:
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
>>>>>>>>>>> unused
>>>>>>>>>>> TimeRange constructors
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
>>>>>>>>>>> references
>>>>>>>>>>> existing in the code
>>>>>>>>>>> This is the end of an old long-running project moving up on to
>>>>>>>>>>> Cell
>>>>>>>>>>> Interface. We think it is done but for a few little items
>>>>>>>>>>> (deprecate
>>>>>>>>>>> KV
>>>>>>>>>>> methods in MR and provide Cell versions instead...)
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
>>>>>>>>>>> needs
>>>>>>>>>>> fixing
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>>>>>>>> * HBASE-14997
>>>>>>>>>>> Move compareOp and Comparators out of filter to client package
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>>>>>>>
>>>>>>>>>>> What about:
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>>>>>>>
>>>>>>>>>>> If interested in any of the above and you need a legup, just ask
>>>>>>>>>>> in
>>>>>>>>>>> the
>>>>>>>>>>> issue and I'll be by....
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> St.Ack
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Heads-up:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
>>>>>>>>>>>> Theme
>>>>>>>>>>>> is
>>>>>>>>>>>> updated dependencies, reliance on relocated popular libs (guava,
>>>>>>>>>>>> netty,
>>>>>>>>>>>> protobuf), purge of checked-in generated src, and
>>>>>>>>>>>> master-carries-no-regions
>>>>>>>>>>>> by default.
>>>>>>>>>>>>
>>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
>>>>>>>>>>>> will
>>>>>>>>>>>> be
>>>>>>>>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>>>>>>>>> looking too
>>>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1
>>>>>>>>>>>> client
>>>>>>>>>>>> running
>>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface
>>>>>>>>>>>> revamp
>>>>>>>>>>>> should be
>>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>>>>>>>> Implementations, and
>>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>>>>>>>> InterfaceAudience).
>>>>>>>>>>>> We'll also have purged thirdparty classes from our API; e.g.
>>>>>>>>>>>> guava
>>>>>>>>>>>> 0.12
>>>>>>>>>>>> Service showing through in our replication API and protobufs in
>>>>>>>>>>>> Admin
>>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach to
>>>>>>>>>>>> make
>>>>>>>>>>>> sure
>>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>>>>>>>>
>>>>>>>>>>>> Beta1 in mid-September?
>>>>>>>>>>>>
>>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>>>>>>>>>>> Edit
>>>>>>>>>>>> as
>>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>>>>>>>>> attention
>>>>>>>>>>>> in next month or so.
>>>>>>>>>>>>
>>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave them
>>>>>>>>>>>> as
>>>>>>>>>>>> is
>>>>>>>>>>>> another while but I'll boot them soon.
>>>>>>>>>>>>
>>>>>>>>>>>> While I have your attention:
>>>>>>>>>>>>
>>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
>>>>>>>>>>>> thrift
>>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy enough
>>>>>>>>>>>> if
>>>>>>>>>>>> folks
>>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>>>>>>>>> (0.98?).
>>>>>>>>>>>>
>>>>>>>>>>>> St.Ack
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
>>>>>>>>>>>>>> elserj@apache.org>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> feature a
>>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I have
>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>> reason
>>>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in terms
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>> much it
>>>>>>>>>>>>>>>> is covered via testing.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See 'Space
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Quota' in
>>>>>>>>>>>>>>> here:
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>>>>>>>>> Does this little section need an update Josh?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> S
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and
>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> docs
>>>>>>>>>>>>>> already included for 17748.  Happy to put that on my plate if
>>>>>>>>>>>>>> you're good
>>>>>>>>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff before
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> merged it in from the original feature branch. I'll double
>>>>>>>>>>>>>> check
>>>>>>>>>>>>>> and update
>>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks Josh,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> S
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>

Re: Moving 2.0 forward

[Josh Elser <el...@apache.org>]

+1 go from my POV.

On 10/31/17 10:07 AM, Stack wrote:
> I want to push an alpha-4 today. A few items didn't make it (HBASE-19092).
> They need more time. We'll pull them in for beta-1. CP API is basically
> done. There may be some changes for beta-1 but hopefully only changes
> informed by experience trying to port an existing Coprocessor to hbase2.
> 
> Shout if there is anything that needs to make alpha-4.
> 
> Thanks,
> St.Ack
> 
> 
> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com> wrote:
> 
>> Yup, that was going to be my plan, Mike!
>>
>> Making a pass now, and will check back later tonight again. I see others
>> have already done some work today on this front.
>>
>>
>> On 10/27/17 11:38 PM, Mike Drob wrote:
>>
>>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how to do
>>> it directly on the jenkins job, so you don't have to bother with JIRA
>>> uploads)
>>>
>>> If you're busy, then I can make time tomorrow or Sunday to kick off jobs,
>>> but I want to make sure we're not duplicating effort and jenkins cycles.
>>>
>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org> wrote:
>>>
>>> My turn to bump ;)
>>>>
>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that remain
>>>> needing some more work. The rest are just awaiting a good QA run.
>>>>
>>>> Unless I hear otherwise, I'll try to keep an eye on things over the
>>>> weekend, bump them along as necessary, and get them committed. Would be
>>>> great to be able get a vote up on Monday.
>>>>
>>>>
>>>> On 10/24/17 6:03 PM, Stack wrote:
>>>>
>>>> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
>>>>> what remains to be done. Surfacing our thoughts here so you all clued
>>>>> in....Or if you think otherwise, please speak up.
>>>>>
>>>>> We have ~13 issues to land:
>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About
>>>>> two
>>>>> are meta-issues that are about process which leaves 11.
>>>>>
>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
>>>>> changed
>>>>> API/semantic that we need to get out earlier rather than later.
>>>>>
>>>>> Once the above is merged, HBASE-13346, change of Filter method names to
>>>>> mention Cell instead of KeyValue can land.
>>>>>
>>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
>>>>> IA.Private objects as params to MasterObserver... Hopefully this goes in
>>>>> soon.
>>>>>
>>>>> Duo is hard at work on trackers for flush and compaction for CPs
>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>>>>>
>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo
>>>>> is
>>>>> done w/ his work above.
>>>>>
>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all
>>>>> the
>>>>> purges allowing CPs do direct calls against Regions in same Host.
>>>>>
>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>>>>
>>>>> Another day or two?
>>>>>
>>>>> St.Ack
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>>>>
>>>>>
>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>>>>>>
>>>>>> +1
>>>>>>
>>>>>>>
>>>>>>> I was trying to work on helping out on the outstanding alpha-4 stuff
>>>>>>> last
>>>>>>> week -- will be continuing to try to do the same this week.
>>>>>>>
>>>>>>> If you need any help, Stack, or if others need reviews where I haven't
>>>>>>> noticed on my own: feel free to @mention me.
>>>>>>>
>>>>>>>
>>>>>>> Thanks for the offer Josh. All items seem assigned and are being
>>>>>>>
>>>>>> actively
>>>>>> worked on. If you get a moment, reviews by you (or anyone else) helps
>>>>>> move
>>>>>> the process along.
>>>>>>
>>>>>> We need to merge in HBASE-18410 branch to pick up Filter improvements.
>>>>>> Then HBASE-13346 can go in.
>>>>>>
>>>>>> You are already helping out on HBASE-18906, thanks. Looks like that
>>>>>> will
>>>>>> be addressed by other alpha-4s about to land.
>>>>>>
>>>>>> St.Ack
>>>>>> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 10/23/17 12:53 PM, Stack wrote:
>>>>>>
>>>>>>>
>>>>>>> (Reviving this thread)
>>>>>>>
>>>>>>>>
>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>>>>>>>> refactor of the Coprocessor API shutting down access to internals
>>>>>>>> marked
>>>>>>>> InterfaceAudience.Private.
>>>>>>>>
>>>>>>>> The outstanding list is here:
>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>>>>>
>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>>>>>
>>>>>>>> If issue, talk out loud on this thread. If you need a review to land
>>>>>>>> an
>>>>>>>> item, shout on the issue and here; we'll help you out.
>>>>>>>>
>>>>>>>> As is, items are coming along nicely I'd say. We need to merge the
>>>>>>>> filter
>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>>>>>
>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and help them
>>>>>>>> test
>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>>>>>>>> downstreamers know what to expect (or we discover what we missed
>>>>>>>> BEFORE
>>>>>>>> we
>>>>>>>> beta-1).
>>>>>>>>
>>>>>>>> Thanks for time,
>>>>>>>> S
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>>>>>>>
>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That should
>>>>>>>> be
>>>>>>>>
>>>>>>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>>>>>>>
>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
>>>>>>>>> but
>>>>>>>>> it is plain that more time is needed (in spite of valiant effort so
>>>>>>>>> far
>>>>>>>>> by
>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>>>>>>>>> theme is
>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>>>>>>>>> following
>>>>>>>>> week.
>>>>>>>>>
>>>>>>>>> We should then be ready for beta (beta == no new features, no API
>>>>>>>>> changes,
>>>>>>>>> just fixes).
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> St.Ack
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>>>>>>>
>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on
>>>>>>>>> it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
>>>>>>>>>> get a
>>>>>>>>>> release out in the next week or so.
>>>>>>>>>>
>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>>>>>>>> interested in helping out, below are the items I think we need done
>>>>>>>>>> for
>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API possibly
>>>>>>>>>> altering
>>>>>>>>>> items, and are absent those JIRAs that are making active progress,
>>>>>>>>>> i.e. the
>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>>>>>>>>>> doing is
>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>>>>>>
>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>>>>>>>>> branch-2
>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do
>>>>>>>>>> it
>>>>>>>>>> between branch-1 and branch-2.
>>>>>>>>>>
>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public /
>>>>>>>>>> Private
>>>>>>>>>> interfaces
>>>>>>>>>> This one is almost done. It could do with a finish, attention to
>>>>>>>>>> the
>>>>>>>>>> items in last comment, and then our codebase could do with another
>>>>>>>>>> sweep
>>>>>>>>>> after the spirit of this issue since a bunch has gone in since the
>>>>>>>>>> pass
>>>>>>>>>> that was the basis of this issue.
>>>>>>>>>>
>>>>>>>>>> * HBASE-10504 Define Replication Interface
>>>>>>>>>> I was going to take a crack at this as part of the revamp forced by
>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
>>>>>>>>>> but if
>>>>>>>>>> anyone else is interested, be my guest.
>>>>>>>>>>
>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>>>>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>>>>>>>> finishing.
>>>>>>>>>>
>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
>>>>>>>>>> and
>>>>>>>>>> cleanup
>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>>>>>>>>>> AsyncClient
>>>>>>>>>> and
>>>>>>>>>> make sure symmetry.
>>>>>>>>>>
>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but needs to
>>>>>>>>>> be
>>>>>>>>>> done.
>>>>>>>>>>
>>>>>>>>>> Rolling upgrade will have implications for our API. Would be good
>>>>>>>>>> to
>>>>>>>>>> try
>>>>>>>>>> it and figure what needs fixup (as said above, according to trial
>>>>>>>>>> by
>>>>>>>>>> Sean,
>>>>>>>>>> we might not be too bad here):
>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>>>>>>>>>> cluster
>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master
>>>>>>>>>> and
>>>>>>>>>> 1.x
>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>>>>>>
>>>>>>>>>> * HBASE-17442 Move most of the replication related classes to
>>>>>>>>>> hbase-server package
>>>>>>>>>> The above would be good to do generally but it may make for ripples
>>>>>>>>>> in
>>>>>>>>>> API so would be good to do now.
>>>>>>>>>>
>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>>>>>>> Balazs is working on this. The idea is that we avoid adding two new
>>>>>>>>>> types
>>>>>>>>>> to our API, two types that are nought but curtailed, read-only
>>>>>>>>>> views
>>>>>>>>>> on
>>>>>>>>>> internals. Input if you have time appreciated.
>>>>>>>>>>
>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
>>>>>>>>>> hbase2
>>>>>>>>>> cluster; verify
>>>>>>>>>> Esteban is looking at this one
>>>>>>>>>>
>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>>>>>>>> * HBASE-17143 Scan improvement
>>>>>>>>>>
>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>>>>>>> implementations
>>>>>>>>>> marked audience Private and returns implementations rather than
>>>>>>>>>> Interfaces.
>>>>>>>>>> In a few locations, we allow returning an alternate implementation
>>>>>>>>>> altogether which is probably something we don't want a CP doing. To
>>>>>>>>>> that
>>>>>>>>>> end, the following issues started by Duo and Anoop need to be taken
>>>>>>>>>> to
>>>>>>>>>> the
>>>>>>>>>> finish line; ideally they'd have an owner:
>>>>>>>>>>
>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
>>>>>>>>>> The
>>>>>>>>>> umbrella issue.
>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver
>>>>>>>>>> and
>>>>>>>>>> RegionServerObserver
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Nice-to-haves:
>>>>>>>>>>
>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
>>>>>>>>>> unused
>>>>>>>>>> TimeRange constructors
>>>>>>>>>>
>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>>>>>>>>> existing in the code
>>>>>>>>>> This is the end of an old long-running project moving up on to Cell
>>>>>>>>>> Interface. We think it is done but for a few little items
>>>>>>>>>> (deprecate
>>>>>>>>>> KV
>>>>>>>>>> methods in MR and provide Cell versions instead...)
>>>>>>>>>>
>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
>>>>>>>>>> needs
>>>>>>>>>> fixing
>>>>>>>>>>
>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>>>>>>
>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>>>>>>> * HBASE-14997
>>>>>>>>>> Move compareOp and Comparators out of filter to client package
>>>>>>>>>>
>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>>>>>>
>>>>>>>>>> What about:
>>>>>>>>>>
>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>>>>>>
>>>>>>>>>> If interested in any of the above and you need a legup, just ask in
>>>>>>>>>> the
>>>>>>>>>> issue and I'll be by....
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> St.Ack
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>>>
>>>>>>>>>> Heads-up:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme
>>>>>>>>>>> is
>>>>>>>>>>> updated dependencies, reliance on relocated popular libs (guava,
>>>>>>>>>>> netty,
>>>>>>>>>>> protobuf), purge of checked-in generated src, and
>>>>>>>>>>> master-carries-no-regions
>>>>>>>>>>> by default.
>>>>>>>>>>>
>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
>>>>>>>>>>> will
>>>>>>>>>>> be
>>>>>>>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>>>>>>>> looking too
>>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>>>>>>>>>>> running
>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>>>>>>>>>>> should be
>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>>>>>>> Implementations, and
>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>>>>>>> InterfaceAudience).
>>>>>>>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
>>>>>>>>>>> 0.12
>>>>>>>>>>> Service showing through in our replication API and protobufs in
>>>>>>>>>>> Admin
>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach to
>>>>>>>>>>> make
>>>>>>>>>>> sure
>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>>>>>>>
>>>>>>>>>>> Beta1 in mid-September?
>>>>>>>>>>>
>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>>>>>>>>>> Edit
>>>>>>>>>>> as
>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>>>>>>>> attention
>>>>>>>>>>> in next month or so.
>>>>>>>>>>>
>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave them
>>>>>>>>>>> as
>>>>>>>>>>> is
>>>>>>>>>>> another while but I'll boot them soon.
>>>>>>>>>>>
>>>>>>>>>>> While I have your attention:
>>>>>>>>>>>
>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
>>>>>>>>>>> thrift
>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy enough
>>>>>>>>>>> if
>>>>>>>>>>> folks
>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>>>>>>>> (0.98?).
>>>>>>>>>>>
>>>>>>>>>>> St.Ack
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>>>>>>>> feature a
>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>>>>>>>>>>>>>>> reason
>>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in terms
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>> much it
>>>>>>>>>>>>>>> is covered via testing.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See 'Space
>>>>>>>>>>>>>> Quota' in
>>>>>>>>>>>>>> here:
>>>>>>>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>>>>>>>> Does this little section need an update Josh?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> S
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
>>>>>>>>>>>>>>
>>>>>>>>>>>>> docs
>>>>>>>>>>>>> already included for 17748.  Happy to put that on my plate if
>>>>>>>>>>>>> you're good
>>>>>>>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>>>>>>>>> merged it in from the original feature branch. I'll double check
>>>>>>>>>>>>> and update
>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Josh,
>>>>>>>>>>>>>
>>>>>>>>>>>> S
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>
> 

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

I want to push an alpha-4 today. A few items didn't make it (HBASE-19092).
They need more time. We'll pull them in for beta-1. CP API is basically
done. There may be some changes for beta-1 but hopefully only changes
informed by experience trying to port an existing Coprocessor to hbase2.

Shout if there is anything that needs to make alpha-4.

Thanks,
St.Ack


On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com> wrote:

> Yup, that was going to be my plan, Mike!
>
> Making a pass now, and will check back later tonight again. I see others
> have already done some work today on this front.
>
>
> On 10/27/17 11:38 PM, Mike Drob wrote:
>
>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how to do
>> it directly on the jenkins job, so you don't have to bother with JIRA
>> uploads)
>>
>> If you're busy, then I can make time tomorrow or Sunday to kick off jobs,
>> but I want to make sure we're not duplicating effort and jenkins cycles.
>>
>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org> wrote:
>>
>> My turn to bump ;)
>>>
>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that remain
>>> needing some more work. The rest are just awaiting a good QA run.
>>>
>>> Unless I hear otherwise, I'll try to keep an eye on things over the
>>> weekend, bump them along as necessary, and get them committed. Would be
>>> great to be able get a vote up on Monday.
>>>
>>>
>>> On 10/24/17 6:03 PM, Stack wrote:
>>>
>>> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
>>>> what remains to be done. Surfacing our thoughts here so you all clued
>>>> in....Or if you think otherwise, please speak up.
>>>>
>>>> We have ~13 issues to land:
>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About
>>>> two
>>>> are meta-issues that are about process which leaves 11.
>>>>
>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
>>>> changed
>>>> API/semantic that we need to get out earlier rather than later.
>>>>
>>>> Once the above is merged, HBASE-13346, change of Filter method names to
>>>> mention Cell instead of KeyValue can land.
>>>>
>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
>>>> IA.Private objects as params to MasterObserver... Hopefully this goes in
>>>> soon.
>>>>
>>>> Duo is hard at work on trackers for flush and compaction for CPs
>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>>>>
>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo
>>>> is
>>>> done w/ his work above.
>>>>
>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all
>>>> the
>>>> purges allowing CPs do direct calls against Regions in same Host.
>>>>
>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>>>
>>>> Another day or two?
>>>>
>>>> St.Ack
>>>>
>>>>
>>>>
>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>>>
>>>>
>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>>>>>
>>>>> +1
>>>>>
>>>>>>
>>>>>> I was trying to work on helping out on the outstanding alpha-4 stuff
>>>>>> last
>>>>>> week -- will be continuing to try to do the same this week.
>>>>>>
>>>>>> If you need any help, Stack, or if others need reviews where I haven't
>>>>>> noticed on my own: feel free to @mention me.
>>>>>>
>>>>>>
>>>>>> Thanks for the offer Josh. All items seem assigned and are being
>>>>>>
>>>>> actively
>>>>> worked on. If you get a moment, reviews by you (or anyone else) helps
>>>>> move
>>>>> the process along.
>>>>>
>>>>> We need to merge in HBASE-18410 branch to pick up Filter improvements.
>>>>> Then HBASE-13346 can go in.
>>>>>
>>>>> You are already helping out on HBASE-18906, thanks. Looks like that
>>>>> will
>>>>> be addressed by other alpha-4s about to land.
>>>>>
>>>>> St.Ack
>>>>> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 10/23/17 12:53 PM, Stack wrote:
>>>>>
>>>>>>
>>>>>> (Reviving this thread)
>>>>>>
>>>>>>>
>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>>>>>>> refactor of the Coprocessor API shutting down access to internals
>>>>>>> marked
>>>>>>> InterfaceAudience.Private.
>>>>>>>
>>>>>>> The outstanding list is here:
>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>>>>
>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>>>>
>>>>>>> If issue, talk out loud on this thread. If you need a review to land
>>>>>>> an
>>>>>>> item, shout on the issue and here; we'll help you out.
>>>>>>>
>>>>>>> As is, items are coming along nicely I'd say. We need to merge the
>>>>>>> filter
>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>>>>
>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and help them
>>>>>>> test
>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>>>>>>> downstreamers know what to expect (or we discover what we missed
>>>>>>> BEFORE
>>>>>>> we
>>>>>>> beta-1).
>>>>>>>
>>>>>>> Thanks for time,
>>>>>>> S
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That should
>>>>>>> be
>>>>>>>
>>>>>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>>>>>>
>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
>>>>>>>> but
>>>>>>>> it is plain that more time is needed (in spite of valiant effort so
>>>>>>>> far
>>>>>>>> by
>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>>>>>>>> theme is
>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>>>>>>>> following
>>>>>>>> week.
>>>>>>>>
>>>>>>>> We should then be ready for beta (beta == no new features, no API
>>>>>>>> changes,
>>>>>>>> just fixes).
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> St.Ack
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>>>>>>
>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on
>>>>>>>> it.
>>>>>>>>
>>>>>>>>
>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
>>>>>>>>> get a
>>>>>>>>> release out in the next week or so.
>>>>>>>>>
>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>>>>>>> interested in helping out, below are the items I think we need done
>>>>>>>>> for
>>>>>>>>> alpha3 (below are at least 'Critical' status, are API possibly
>>>>>>>>> altering
>>>>>>>>> items, and are absent those JIRAs that are making active progress,
>>>>>>>>> i.e. the
>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>>>>>>>>> doing is
>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>>>>>
>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>>>>>>>> branch-2
>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do
>>>>>>>>> it
>>>>>>>>> between branch-1 and branch-2.
>>>>>>>>>
>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public /
>>>>>>>>> Private
>>>>>>>>> interfaces
>>>>>>>>> This one is almost done. It could do with a finish, attention to
>>>>>>>>> the
>>>>>>>>> items in last comment, and then our codebase could do with another
>>>>>>>>> sweep
>>>>>>>>> after the spirit of this issue since a bunch has gone in since the
>>>>>>>>> pass
>>>>>>>>> that was the basis of this issue.
>>>>>>>>>
>>>>>>>>> * HBASE-10504 Define Replication Interface
>>>>>>>>> I was going to take a crack at this as part of the revamp forced by
>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
>>>>>>>>> but if
>>>>>>>>> anyone else is interested, be my guest.
>>>>>>>>>
>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>>>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>>>>>>> finishing.
>>>>>>>>>
>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
>>>>>>>>> and
>>>>>>>>> cleanup
>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>>>>>>>>> AsyncClient
>>>>>>>>> and
>>>>>>>>> make sure symmetry.
>>>>>>>>>
>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but needs to
>>>>>>>>> be
>>>>>>>>> done.
>>>>>>>>>
>>>>>>>>> Rolling upgrade will have implications for our API. Would be good
>>>>>>>>> to
>>>>>>>>> try
>>>>>>>>> it and figure what needs fixup (as said above, according to trial
>>>>>>>>> by
>>>>>>>>> Sean,
>>>>>>>>> we might not be too bad here):
>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>>>>>>>>> cluster
>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master
>>>>>>>>> and
>>>>>>>>> 1.x
>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>>>>>
>>>>>>>>> * HBASE-17442 Move most of the replication related classes to
>>>>>>>>> hbase-server package
>>>>>>>>> The above would be good to do generally but it may make for ripples
>>>>>>>>> in
>>>>>>>>> API so would be good to do now.
>>>>>>>>>
>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>>>>>> Balazs is working on this. The idea is that we avoid adding two new
>>>>>>>>> types
>>>>>>>>> to our API, two types that are nought but curtailed, read-only
>>>>>>>>> views
>>>>>>>>> on
>>>>>>>>> internals. Input if you have time appreciated.
>>>>>>>>>
>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
>>>>>>>>> hbase2
>>>>>>>>> cluster; verify
>>>>>>>>> Esteban is looking at this one
>>>>>>>>>
>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>>>>>>> * HBASE-17143 Scan improvement
>>>>>>>>>
>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>>>>>> implementations
>>>>>>>>> marked audience Private and returns implementations rather than
>>>>>>>>> Interfaces.
>>>>>>>>> In a few locations, we allow returning an alternate implementation
>>>>>>>>> altogether which is probably something we don't want a CP doing. To
>>>>>>>>> that
>>>>>>>>> end, the following issues started by Duo and Anoop need to be taken
>>>>>>>>> to
>>>>>>>>> the
>>>>>>>>> finish line; ideally they'd have an owner:
>>>>>>>>>
>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
>>>>>>>>> The
>>>>>>>>> umbrella issue.
>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver
>>>>>>>>> and
>>>>>>>>> RegionServerObserver
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Nice-to-haves:
>>>>>>>>>
>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
>>>>>>>>> unused
>>>>>>>>> TimeRange constructors
>>>>>>>>>
>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>>>>>>>> existing in the code
>>>>>>>>> This is the end of an old long-running project moving up on to Cell
>>>>>>>>> Interface. We think it is done but for a few little items
>>>>>>>>> (deprecate
>>>>>>>>> KV
>>>>>>>>> methods in MR and provide Cell versions instead...)
>>>>>>>>>
>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
>>>>>>>>> needs
>>>>>>>>> fixing
>>>>>>>>>
>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>>>>>
>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>>>>>> * HBASE-14997
>>>>>>>>> Move compareOp and Comparators out of filter to client package
>>>>>>>>>
>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>>>>>
>>>>>>>>> What about:
>>>>>>>>>
>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>>>>>
>>>>>>>>> If interested in any of the above and you need a legup, just ask in
>>>>>>>>> the
>>>>>>>>> issue and I'll be by....
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> St.Ack
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>>
>>>>>>>>> Heads-up:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme
>>>>>>>>>> is
>>>>>>>>>> updated dependencies, reliance on relocated popular libs (guava,
>>>>>>>>>> netty,
>>>>>>>>>> protobuf), purge of checked-in generated src, and
>>>>>>>>>> master-carries-no-regions
>>>>>>>>>> by default.
>>>>>>>>>>
>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
>>>>>>>>>> will
>>>>>>>>>> be
>>>>>>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>>>>>>> looking too
>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>>>>>>>>>> running
>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>>>>>>>>>> should be
>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>>>>>> Implementations, and
>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>>>>>> InterfaceAudience).
>>>>>>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
>>>>>>>>>> 0.12
>>>>>>>>>> Service showing through in our replication API and protobufs in
>>>>>>>>>> Admin
>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach to
>>>>>>>>>> make
>>>>>>>>>> sure
>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>>>>>>
>>>>>>>>>> Beta1 in mid-September?
>>>>>>>>>>
>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>>>>>>>>> Edit
>>>>>>>>>> as
>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>>>>>>> attention
>>>>>>>>>> in next month or so.
>>>>>>>>>>
>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave them
>>>>>>>>>> as
>>>>>>>>>> is
>>>>>>>>>> another while but I'll boot them soon.
>>>>>>>>>>
>>>>>>>>>> While I have your attention:
>>>>>>>>>>
>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
>>>>>>>>>> thrift
>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy enough
>>>>>>>>>> if
>>>>>>>>>> folks
>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>>>>>>> (0.98?).
>>>>>>>>>>
>>>>>>>>>> St.Ack
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>>>>>>> feature a
>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>>>>>>>>>>>>>> reason
>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in terms
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> how
>>>>>>>>>>>>>> much it
>>>>>>>>>>>>>> is covered via testing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Last think on space quota says that need doc too. See 'Space
>>>>>>>>>>>>> Quota' in
>>>>>>>>>>>>> here:
>>>>>>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>>>>>>> Does this little section need an update Josh?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> S
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
>>>>>>>>>>>>>
>>>>>>>>>>>> docs
>>>>>>>>>>>> already included for 17748.  Happy to put that on my plate if
>>>>>>>>>>>> you're good
>>>>>>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>>>>>>
>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>>>>>>>> merged it in from the original feature branch. I'll double check
>>>>>>>>>>>> and update
>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks Josh,
>>>>>>>>>>>>
>>>>>>>>>>> S
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>

Re: Moving 2.0 forward

[Josh Elser <jo...@gmail.com>]

Yup, that was going to be my plan, Mike!

Making a pass now, and will check back later tonight again. I see others 
have already done some work today on this front.

On 10/27/17 11:38 PM, Mike Drob wrote:
> Josh - Do you want to kick off a bunch of QA runs? (Do you know how to do
> it directly on the jenkins job, so you don't have to bother with JIRA
> uploads)
> 
> If you're busy, then I can make time tomorrow or Sunday to kick off jobs,
> but I want to make sure we're not duplicating effort and jenkins cycles.
> 
> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org> wrote:
> 
>> My turn to bump ;)
>>
>> By my take: HBASE-18770 and HBASE-19092 are the only issues that remain
>> needing some more work. The rest are just awaiting a good QA run.
>>
>> Unless I hear otherwise, I'll try to keep an eye on things over the
>> weekend, bump them along as necessary, and get them committed. Would be
>> great to be able get a vote up on Monday.
>>
>>
>> On 10/24/17 6:03 PM, Stack wrote:
>>
>>> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
>>> what remains to be done. Surfacing our thoughts here so you all clued
>>> in....Or if you think otherwise, please speak up.
>>>
>>> We have ~13 issues to land:
>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About two
>>> are meta-issues that are about process which leaves 11.
>>>
>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some changed
>>> API/semantic that we need to get out earlier rather than later.
>>>
>>> Once the above is merged, HBASE-13346, change of Filter method names to
>>> mention Cell instead of KeyValue can land.
>>>
>>> HBASE-199048 needs a review (Anoop will probably do it), removing
>>> IA.Private objects as params to MasterObserver... Hopefully this goes in
>>> soon.
>>>
>>> Duo is hard at work on trackers for flush and compaction for CPs
>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>>>
>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo
>>> is
>>> done w/ his work above.
>>>
>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all
>>> the
>>> purges allowing CPs do direct calls against Regions in same Host.
>>>
>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>>
>>> Another day or two?
>>>
>>> St.Ack
>>>
>>>
>>>
>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>>
>>>
>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>>>>
>>>> +1
>>>>>
>>>>> I was trying to work on helping out on the outstanding alpha-4 stuff
>>>>> last
>>>>> week -- will be continuing to try to do the same this week.
>>>>>
>>>>> If you need any help, Stack, or if others need reviews where I haven't
>>>>> noticed on my own: feel free to @mention me.
>>>>>
>>>>>
>>>>> Thanks for the offer Josh. All items seem assigned and are being
>>>> actively
>>>> worked on. If you get a moment, reviews by you (or anyone else) helps
>>>> move
>>>> the process along.
>>>>
>>>> We need to merge in HBASE-18410 branch to pick up Filter improvements.
>>>> Then HBASE-13346 can go in.
>>>>
>>>> You are already helping out on HBASE-18906, thanks. Looks like that will
>>>> be addressed by other alpha-4s about to land.
>>>>
>>>> St.Ack
>>>> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 10/23/17 12:53 PM, Stack wrote:
>>>>>
>>>>> (Reviving this thread)
>>>>>>
>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>>>>>> refactor of the Coprocessor API shutting down access to internals
>>>>>> marked
>>>>>> InterfaceAudience.Private.
>>>>>>
>>>>>> The outstanding list is here:
>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>>>
>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>>>
>>>>>> If issue, talk out loud on this thread. If you need a review to land an
>>>>>> item, shout on the issue and here; we'll help you out.
>>>>>>
>>>>>> As is, items are coming along nicely I'd say. We need to merge the
>>>>>> filter
>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>>>
>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and help them
>>>>>> test
>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>>>>>> downstreamers know what to expect (or we discover what we missed BEFORE
>>>>>> we
>>>>>> beta-1).
>>>>>>
>>>>>> Thanks for time,
>>>>>> S
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>>>>>
>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
>>>>>>
>>>>>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>>>>>
>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
>>>>>>> but
>>>>>>> it is plain that more time is needed (in spite of valiant effort so
>>>>>>> far
>>>>>>> by
>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>>>>>>> theme is
>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>>>>>>> following
>>>>>>> week.
>>>>>>>
>>>>>>> We should then be ready for beta (beta == no new features, no API
>>>>>>> changes,
>>>>>>> just fixes).
>>>>>>>
>>>>>>> Thanks,
>>>>>>> St.Ack
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>>>>>>>
>>>>>>>>
>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>>>>>>>> release out in the next week or so.
>>>>>>>>
>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>>>>>> interested in helping out, below are the items I think we need done
>>>>>>>> for
>>>>>>>> alpha3 (below are at least 'Critical' status, are API possibly
>>>>>>>> altering
>>>>>>>> items, and are absent those JIRAs that are making active progress,
>>>>>>>> i.e. the
>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>>>>>>>> doing is
>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>>>>
>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>>>>>>> branch-2
>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
>>>>>>>> between branch-1 and branch-2.
>>>>>>>>
>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public / Private
>>>>>>>> interfaces
>>>>>>>> This one is almost done. It could do with a finish, attention to the
>>>>>>>> items in last comment, and then our codebase could do with another
>>>>>>>> sweep
>>>>>>>> after the spirit of this issue since a bunch has gone in since the
>>>>>>>> pass
>>>>>>>> that was the basis of this issue.
>>>>>>>>
>>>>>>>> * HBASE-10504 Define Replication Interface
>>>>>>>> I was going to take a crack at this as part of the revamp forced by
>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
>>>>>>>> but if
>>>>>>>> anyone else is interested, be my guest.
>>>>>>>>
>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>>>>>> finishing.
>>>>>>>>
>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>>>>>>>> cleanup
>>>>>>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
>>>>>>>> and
>>>>>>>> make sure symmetry.
>>>>>>>>
>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
>>>>>>>> done.
>>>>>>>>
>>>>>>>> Rolling upgrade will have implications for our API. Would be good to
>>>>>>>> try
>>>>>>>> it and figure what needs fixup (as said above, according to trial by
>>>>>>>> Sean,
>>>>>>>> we might not be too bad here):
>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>>>>>>>> cluster
>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
>>>>>>>> 1.x
>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>>>>
>>>>>>>> * HBASE-17442 Move most of the replication related classes to
>>>>>>>> hbase-server package
>>>>>>>> The above would be good to do generally but it may make for ripples
>>>>>>>> in
>>>>>>>> API so would be good to do now.
>>>>>>>>
>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>>>>> Balazs is working on this. The idea is that we avoid adding two new
>>>>>>>> types
>>>>>>>> to our API, two types that are nought but curtailed, read-only views
>>>>>>>> on
>>>>>>>> internals. Input if you have time appreciated.
>>>>>>>>
>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
>>>>>>>> hbase2
>>>>>>>> cluster; verify
>>>>>>>> Esteban is looking at this one
>>>>>>>>
>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>>>>>> * HBASE-17143 Scan improvement
>>>>>>>>
>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>>>>> implementations
>>>>>>>> marked audience Private and returns implementations rather than
>>>>>>>> Interfaces.
>>>>>>>> In a few locations, we allow returning an alternate implementation
>>>>>>>> altogether which is probably something we don't want a CP doing. To
>>>>>>>> that
>>>>>>>> end, the following issues started by Duo and Anoop need to be taken
>>>>>>>> to
>>>>>>>> the
>>>>>>>> finish line; ideally they'd have an owner:
>>>>>>>>
>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
>>>>>>>> umbrella issue.
>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>>>>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>>>>>>>> RegionServerObserver
>>>>>>>>
>>>>>>>>
>>>>>>>> Nice-to-haves:
>>>>>>>>
>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
>>>>>>>> unused
>>>>>>>> TimeRange constructors
>>>>>>>>
>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>>>>>>> existing in the code
>>>>>>>> This is the end of an old long-running project moving up on to Cell
>>>>>>>> Interface. We think it is done but for a few little items (deprecate
>>>>>>>> KV
>>>>>>>> methods in MR and provide Cell versions instead...)
>>>>>>>>
>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
>>>>>>>> fixing
>>>>>>>>
>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>>>>
>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>>>>> * HBASE-14997
>>>>>>>> Move compareOp and Comparators out of filter to client package
>>>>>>>>
>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>>>>
>>>>>>>> What about:
>>>>>>>>
>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>>>>
>>>>>>>> If interested in any of the above and you need a legup, just ask in
>>>>>>>> the
>>>>>>>> issue and I'll be by....
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> St.Ack
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>
>>>>>>>> Heads-up:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme
>>>>>>>>> is
>>>>>>>>> updated dependencies, reliance on relocated popular libs (guava,
>>>>>>>>> netty,
>>>>>>>>> protobuf), purge of checked-in generated src, and
>>>>>>>>> master-carries-no-regions
>>>>>>>>> by default.
>>>>>>>>>
>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
>>>>>>>>> will
>>>>>>>>> be
>>>>>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>>>>>> looking too
>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>>>>>>>>> running
>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>>>>>>>>> should be
>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>>>>> Implementations, and
>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>>>>> InterfaceAudience).
>>>>>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
>>>>>>>>> 0.12
>>>>>>>>> Service showing through in our replication API and protobufs in
>>>>>>>>> Admin
>>>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach to make
>>>>>>>>> sure
>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>>>>>
>>>>>>>>> Beta1 in mid-September?
>>>>>>>>>
>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>>>>>>>> Edit
>>>>>>>>> as
>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>>>>>> attention
>>>>>>>>> in next month or so.
>>>>>>>>>
>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave them as
>>>>>>>>> is
>>>>>>>>> another while but I'll boot them soon.
>>>>>>>>>
>>>>>>>>> While I have your attention:
>>>>>>>>>
>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
>>>>>>>>> thrift
>>>>>>>>> to 0.10.0 will break existing clients. The change is easy enough if
>>>>>>>>> folks
>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>>>>>> (0.98?).
>>>>>>>>>
>>>>>>>>> St.Ack
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> ...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>>>>>> feature a
>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>>>>>>>>>>>>> reason
>>>>>>>>>>>>> to be
>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in terms of
>>>>>>>>>>>>> how
>>>>>>>>>>>>> much it
>>>>>>>>>>>>> is covered via testing.
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Last think on space quota says that need doc too. See 'Space
>>>>>>>>>>>> Quota' in
>>>>>>>>>>>> here:
>>>>>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>>>>>> Does this little section need an update Josh?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> S
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
>>>>>>>>>>> docs
>>>>>>>>>>> already included for 17748.  Happy to put that on my plate if
>>>>>>>>>>> you're good
>>>>>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>>>>>
>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>>>>>>> merged it in from the original feature branch. I'll double check
>>>>>>>>>>> and update
>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks Josh,
>>>>>>>>>> S
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
> 

Re: Moving 2.0 forward

[Mike Drob <md...@apache.org>]

Josh - Do you want to kick off a bunch of QA runs? (Do you know how to do
it directly on the jenkins job, so you don't have to bother with JIRA
uploads)

If you're busy, then I can make time tomorrow or Sunday to kick off jobs,
but I want to make sure we're not duplicating effort and jenkins cycles.

On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org> wrote:

> My turn to bump ;)
>
> By my take: HBASE-18770 and HBASE-19092 are the only issues that remain
> needing some more work. The rest are just awaiting a good QA run.
>
> Unless I hear otherwise, I'll try to keep an eye on things over the
> weekend, bump them along as necessary, and get them committed. Would be
> great to be able get a vote up on Monday.
>
>
> On 10/24/17 6:03 PM, Stack wrote:
>
>> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
>> what remains to be done. Surfacing our thoughts here so you all clued
>> in....Or if you think otherwise, please speak up.
>>
>> We have ~13 issues to land:
>> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About two
>> are meta-issues that are about process which leaves 11.
>>
>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some changed
>> API/semantic that we need to get out earlier rather than later.
>>
>> Once the above is merged, HBASE-13346, change of Filter method names to
>> mention Cell instead of KeyValue can land.
>>
>> HBASE-199048 needs a review (Anoop will probably do it), removing
>> IA.Private objects as params to MasterObserver... Hopefully this goes in
>> soon.
>>
>> Duo is hard at work on trackers for flush and compaction for CPs
>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>>
>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo
>> is
>> done w/ his work above.
>>
>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all
>> the
>> purges allowing CPs do direct calls against Regions in same Host.
>>
>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>
>> Another day or two?
>>
>> St.Ack
>>
>>
>>
>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>
>>
>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>>>
>>> +1
>>>>
>>>> I was trying to work on helping out on the outstanding alpha-4 stuff
>>>> last
>>>> week -- will be continuing to try to do the same this week.
>>>>
>>>> If you need any help, Stack, or if others need reviews where I haven't
>>>> noticed on my own: feel free to @mention me.
>>>>
>>>>
>>>> Thanks for the offer Josh. All items seem assigned and are being
>>> actively
>>> worked on. If you get a moment, reviews by you (or anyone else) helps
>>> move
>>> the process along.
>>>
>>> We need to merge in HBASE-18410 branch to pick up Filter improvements.
>>> Then HBASE-13346 can go in.
>>>
>>> You are already helping out on HBASE-18906, thanks. Looks like that will
>>> be addressed by other alpha-4s about to land.
>>>
>>> St.Ack
>>> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 10/23/17 12:53 PM, Stack wrote:
>>>>
>>>> (Reviving this thread)
>>>>>
>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>>>>> refactor of the Coprocessor API shutting down access to internals
>>>>> marked
>>>>> InterfaceAudience.Private.
>>>>>
>>>>> The outstanding list is here:
>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>>
>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>>
>>>>> If issue, talk out loud on this thread. If you need a review to land an
>>>>> item, shout on the issue and here; we'll help you out.
>>>>>
>>>>> As is, items are coming along nicely I'd say. We need to merge the
>>>>> filter
>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>>
>>>>> Post alpha-4, we'll have to hunt down our downstreamers and help them
>>>>> test
>>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>>>>> downstreamers know what to expect (or we discover what we missed BEFORE
>>>>> we
>>>>> beta-1).
>>>>>
>>>>> Thanks for time,
>>>>> S
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>>>>
>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
>>>>>
>>>>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>>>>
>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
>>>>>> but
>>>>>> it is plain that more time is needed (in spite of valiant effort so
>>>>>> far
>>>>>> by
>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>>>>>> theme is
>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>>>>>> following
>>>>>> week.
>>>>>>
>>>>>> We should then be ready for beta (beta == no new features, no API
>>>>>> changes,
>>>>>> just fixes).
>>>>>>
>>>>>> Thanks,
>>>>>> St.Ack
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>>>>
>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>>>>>>
>>>>>>>
>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>>>>>>> release out in the next week or so.
>>>>>>>
>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>>>>> interested in helping out, below are the items I think we need done
>>>>>>> for
>>>>>>> alpha3 (below are at least 'Critical' status, are API possibly
>>>>>>> altering
>>>>>>> items, and are absent those JIRAs that are making active progress,
>>>>>>> i.e. the
>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>>>>>>> doing is
>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>>>
>>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>>>>>> branch-2
>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
>>>>>>> between branch-1 and branch-2.
>>>>>>>
>>>>>>> * HBASE-10462 Recategorize some of the client facing Public / Private
>>>>>>> interfaces
>>>>>>> This one is almost done. It could do with a finish, attention to the
>>>>>>> items in last comment, and then our codebase could do with another
>>>>>>> sweep
>>>>>>> after the spirit of this issue since a bunch has gone in since the
>>>>>>> pass
>>>>>>> that was the basis of this issue.
>>>>>>>
>>>>>>> * HBASE-10504 Define Replication Interface
>>>>>>> I was going to take a crack at this as part of the revamp forced by
>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
>>>>>>> but if
>>>>>>> anyone else is interested, be my guest.
>>>>>>>
>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>>>>> finishing.
>>>>>>>
>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>>>>>>> cleanup
>>>>>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
>>>>>>> and
>>>>>>> make sure symmetry.
>>>>>>>
>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
>>>>>>> done.
>>>>>>>
>>>>>>> Rolling upgrade will have implications for our API. Would be good to
>>>>>>> try
>>>>>>> it and figure what needs fixup (as said above, according to trial by
>>>>>>> Sean,
>>>>>>> we might not be too bad here):
>>>>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>>>>>>> cluster
>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
>>>>>>> 1.x
>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>>>
>>>>>>> * HBASE-17442 Move most of the replication related classes to
>>>>>>> hbase-server package
>>>>>>> The above would be good to do generally but it may make for ripples
>>>>>>> in
>>>>>>> API so would be good to do now.
>>>>>>>
>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>>>> Balazs is working on this. The idea is that we avoid adding two new
>>>>>>> types
>>>>>>> to our API, two types that are nought but curtailed, read-only views
>>>>>>> on
>>>>>>> internals. Input if you have time appreciated.
>>>>>>>
>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
>>>>>>> hbase2
>>>>>>> cluster; verify
>>>>>>> Esteban is looking at this one
>>>>>>>
>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>>>>> * HBASE-17143 Scan improvement
>>>>>>>
>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>>>> implementations
>>>>>>> marked audience Private and returns implementations rather than
>>>>>>> Interfaces.
>>>>>>> In a few locations, we allow returning an alternate implementation
>>>>>>> altogether which is probably something we don't want a CP doing. To
>>>>>>> that
>>>>>>> end, the following issues started by Duo and Anoop need to be taken
>>>>>>> to
>>>>>>> the
>>>>>>> finish line; ideally they'd have an owner:
>>>>>>>
>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
>>>>>>> umbrella issue.
>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>>>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>>>>>>> RegionServerObserver
>>>>>>>
>>>>>>>
>>>>>>> Nice-to-haves:
>>>>>>>
>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
>>>>>>> unused
>>>>>>> TimeRange constructors
>>>>>>>
>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>>>>>> existing in the code
>>>>>>> This is the end of an old long-running project moving up on to Cell
>>>>>>> Interface. We think it is done but for a few little items (deprecate
>>>>>>> KV
>>>>>>> methods in MR and provide Cell versions instead...)
>>>>>>>
>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
>>>>>>> fixing
>>>>>>>
>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>>>
>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>>>> * HBASE-14997
>>>>>>> Move compareOp and Comparators out of filter to client package
>>>>>>>
>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>>>
>>>>>>> What about:
>>>>>>>
>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>>>
>>>>>>> If interested in any of the above and you need a legup, just ask in
>>>>>>> the
>>>>>>> issue and I'll be by....
>>>>>>>
>>>>>>> Thanks,
>>>>>>> St.Ack
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>> Heads-up:
>>>>>>>
>>>>>>>>
>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme
>>>>>>>> is
>>>>>>>> updated dependencies, reliance on relocated popular libs (guava,
>>>>>>>> netty,
>>>>>>>> protobuf), purge of checked-in generated src, and
>>>>>>>> master-carries-no-regions
>>>>>>>> by default.
>>>>>>>>
>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
>>>>>>>> will
>>>>>>>> be
>>>>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>>>>> looking too
>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>>>>>>>> running
>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>>>>>>>> should be
>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>>>> Implementations, and
>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>>>> InterfaceAudience).
>>>>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
>>>>>>>> 0.12
>>>>>>>> Service showing through in our replication API and protobufs in
>>>>>>>> Admin
>>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach to make
>>>>>>>> sure
>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>>>>
>>>>>>>> Beta1 in mid-September?
>>>>>>>>
>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>>>>>>> Edit
>>>>>>>> as
>>>>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>>>>> attention
>>>>>>>> in next month or so.
>>>>>>>>
>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave them as
>>>>>>>> is
>>>>>>>> another while but I'll boot them soon.
>>>>>>>>
>>>>>>>> While I have your attention:
>>>>>>>>
>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
>>>>>>>> thrift
>>>>>>>> to 0.10.0 will break existing clients. The change is easy enough if
>>>>>>>> folks
>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>>>>> (0.98?).
>>>>>>>>
>>>>>>>> St.Ack
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>>>>
>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> ...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>>>>> feature a
>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>>>>>>>>>>>> reason
>>>>>>>>>>>> to be
>>>>>>>>>>>> nervous though). I am pretty happy with the feature in terms of
>>>>>>>>>>>> how
>>>>>>>>>>>> much it
>>>>>>>>>>>> is covered via testing.
>>>>>>>>>>>>
>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Last think on space quota says that need doc too. See 'Space
>>>>>>>>>>> Quota' in
>>>>>>>>>>> here:
>>>>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>>>>> Does this little section need an update Josh?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> S
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
>>>>>>>>>> docs
>>>>>>>>>> already included for 17748.  Happy to put that on my plate if
>>>>>>>>>> you're good
>>>>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>>>>
>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>>>>>> merged it in from the original feature branch. I'll double check
>>>>>>>>>> and update
>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks Josh,
>>>>>>>>> S
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>>

Re: Moving 2.0 forward

[Josh Elser <el...@apache.org>]

My turn to bump ;)

By my take: HBASE-18770 and HBASE-19092 are the only issues that remain 
needing some more work. The rest are just awaiting a good QA run.

Unless I hear otherwise, I'll try to keep an eye on things over the 
weekend, bump them along as necessary, and get them committed. Would be 
great to be able get a vote up on Monday.

On 10/24/17 6:03 PM, Stack wrote:
> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
> what remains to be done. Surfacing our thoughts here so you all clued
> in....Or if you think otherwise, please speak up.
> 
> We have ~13 issues to land:
> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About two
> are meta-issues that are about process which leaves 11.
> 
> Duo and Zheng Hu are to merge the FilterList fixes improvements
> (HBASE-19057, HBASE-18410 et al.). These are blocker because some changed
> API/semantic that we need to get out earlier rather than later.
> 
> Once the above is merged, HBASE-13346, change of Filter method names to
> mention Cell instead of KeyValue can land.
> 
> HBASE-199048 needs a review (Anoop will probably do it), removing
> IA.Private objects as params to MasterObserver... Hopefully this goes in
> soon.
> 
> Duo is hard at work on trackers for flush and compaction for CPs
> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
> 
> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo is
> done w/ his work above.
> 
> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all the
> purges allowing CPs do direct calls against Regions in same Host.
> 
> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> 
> Another day or two?
> 
> St.Ack
> 
> 
> 
> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
> 
>>
>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>>
>>> +1
>>>
>>> I was trying to work on helping out on the outstanding alpha-4 stuff last
>>> week -- will be continuing to try to do the same this week.
>>>
>>> If you need any help, Stack, or if others need reviews where I haven't
>>> noticed on my own: feel free to @mention me.
>>>
>>>
>> Thanks for the offer Josh. All items seem assigned and are being actively
>> worked on. If you get a moment, reviews by you (or anyone else) helps move
>> the process along.
>>
>> We need to merge in HBASE-18410 branch to pick up Filter improvements.
>> Then HBASE-13346 can go in.
>>
>> You are already helping out on HBASE-18906, thanks. Looks like that will
>> be addressed by other alpha-4s about to land.
>>
>> St.Ack
>> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>> On 10/23/17 12:53 PM, Stack wrote:
>>>
>>>> (Reviving this thread)
>>>>
>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>>>> refactor of the Coprocessor API shutting down access to internals marked
>>>> InterfaceAudience.Private.
>>>>
>>>> The outstanding list is here:
>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>
>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>
>>>> If issue, talk out loud on this thread. If you need a review to land an
>>>> item, shout on the issue and here; we'll help you out.
>>>>
>>>> As is, items are coming along nicely I'd say. We need to merge the filter
>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>
>>>> Post alpha-4, we'll have to hunt down our downstreamers and help them
>>>> test
>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>>>> downstreamers know what to expect (or we discover what we missed BEFORE
>>>> we
>>>> beta-1).
>>>>
>>>> Thanks for time,
>>>> S
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>>>
>>>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
>>>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>>>
>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3 but
>>>>> it is plain that more time is needed (in spite of valiant effort so far
>>>>> by
>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>>>>> theme is
>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the following
>>>>> week.
>>>>>
>>>>> We should then be ready for beta (beta == no new features, no API
>>>>> changes,
>>>>> just fixes).
>>>>>
>>>>> Thanks,
>>>>> St.Ack
>>>>>
>>>>>
>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>>>
>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>>>>>>
>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>>>>>> release out in the next week or so.
>>>>>>
>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>>>> interested in helping out, below are the items I think we need done for
>>>>>> alpha3 (below are at least 'Critical' status, are API possibly altering
>>>>>> items, and are absent those JIRAs that are making active progress,
>>>>>> i.e. the
>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>>>>>> doing is
>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>>
>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>>>>> branch-2
>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
>>>>>> between branch-1 and branch-2.
>>>>>>
>>>>>> * HBASE-10462 Recategorize some of the client facing Public / Private
>>>>>> interfaces
>>>>>> This one is almost done. It could do with a finish, attention to the
>>>>>> items in last comment, and then our codebase could do with another
>>>>>> sweep
>>>>>> after the spirit of this issue since a bunch has gone in since the pass
>>>>>> that was the basis of this issue.
>>>>>>
>>>>>> * HBASE-10504 Define Replication Interface
>>>>>> I was going to take a crack at this as part of the revamp forced by
>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
>>>>>> but if
>>>>>> anyone else is interested, be my guest.
>>>>>>
>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>>>> finishing.
>>>>>>
>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>>>>>> cleanup
>>>>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
>>>>>> and
>>>>>> make sure symmetry.
>>>>>>
>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
>>>>>> done.
>>>>>>
>>>>>> Rolling upgrade will have implications for our API. Would be good to
>>>>>> try
>>>>>> it and figure what needs fixup (as said above, according to trial by
>>>>>> Sean,
>>>>>> we might not be too bad here):
>>>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>>>>>> cluster
>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
>>>>>> 1.x
>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>>
>>>>>> * HBASE-17442 Move most of the replication related classes to
>>>>>> hbase-server package
>>>>>> The above would be good to do generally but it may make for ripples in
>>>>>> API so would be good to do now.
>>>>>>
>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>>> Balazs is working on this. The idea is that we avoid adding two new
>>>>>> types
>>>>>> to our API, two types that are nought but curtailed, read-only views on
>>>>>> internals. Input if you have time appreciated.
>>>>>>
>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
>>>>>> cluster; verify
>>>>>> Esteban is looking at this one
>>>>>>
>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>>>> * HBASE-17143 Scan improvement
>>>>>>
>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>>> implementations
>>>>>> marked audience Private and returns implementations rather than
>>>>>> Interfaces.
>>>>>> In a few locations, we allow returning an alternate implementation
>>>>>> altogether which is probably something we don't want a CP doing. To
>>>>>> that
>>>>>> end, the following issues started by Duo and Anoop need to be taken to
>>>>>> the
>>>>>> finish line; ideally they'd have an owner:
>>>>>>
>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
>>>>>> umbrella issue.
>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>>>>>> RegionServerObserver
>>>>>>
>>>>>>
>>>>>> Nice-to-haves:
>>>>>>
>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
>>>>>> TimeRange constructors
>>>>>>
>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>>>>> existing in the code
>>>>>> This is the end of an old long-running project moving up on to Cell
>>>>>> Interface. We think it is done but for a few little items (deprecate KV
>>>>>> methods in MR and provide Cell versions instead...)
>>>>>>
>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
>>>>>> fixing
>>>>>>
>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>>
>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>>> * HBASE-14997
>>>>>> Move compareOp and Comparators out of filter to client package
>>>>>>
>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>>
>>>>>> What about:
>>>>>>
>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>>
>>>>>> If interested in any of the above and you need a legup, just ask in the
>>>>>> issue and I'll be by....
>>>>>>
>>>>>> Thanks,
>>>>>> St.Ack
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>>>>
>>>>>> Heads-up:
>>>>>>>
>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
>>>>>>> updated dependencies, reliance on relocated popular libs (guava,
>>>>>>> netty,
>>>>>>> protobuf), purge of checked-in generated src, and
>>>>>>> master-carries-no-regions
>>>>>>> by default.
>>>>>>>
>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme will
>>>>>>> be
>>>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>>>> looking too
>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>>>>>>> running
>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>>>>>>> should be
>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>>> Implementations, and
>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>>> InterfaceAudience).
>>>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
>>>>>>> 0.12
>>>>>>> Service showing through in our replication API and protobufs in Admin
>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach to make
>>>>>>> sure
>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>>>
>>>>>>> Beta1 in mid-September?
>>>>>>>
>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit
>>>>>>> as
>>>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>>>> attention
>>>>>>> in next month or so.
>>>>>>>
>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave them as is
>>>>>>> another while but I'll boot them soon.
>>>>>>>
>>>>>>> While I have your attention:
>>>>>>>
>>>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase thrift
>>>>>>> to 0.10.0 will break existing clients. The change is easy enough if
>>>>>>> folks
>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>>>> (0.98?).
>>>>>>>
>>>>>>> St.Ack
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>>>
>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>>>> feature a
>>>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>>>>>>>>>>> reason
>>>>>>>>>>> to be
>>>>>>>>>>> nervous though). I am pretty happy with the feature in terms of
>>>>>>>>>>> how
>>>>>>>>>>> much it
>>>>>>>>>>> is covered via testing.
>>>>>>>>>>>
>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Last think on space quota says that need doc too. See 'Space
>>>>>>>>>> Quota' in
>>>>>>>>>> here:
>>>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>>>> Does this little section need an update Josh?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> S
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>>>>>>>>> already included for 17748.  Happy to put that on my plate if
>>>>>>>>> you're good
>>>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>>>
>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>>>>> merged it in from the original feature branch. I'll double check
>>>>>>>>> and update
>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Thanks Josh,
>>>>>>>> S
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
> 

Re: Moving 2.0 forward

[Guanghao Zhang <zg...@gmail.com>]

>
> For public API we have rules on how to keep compatible so I think it is
> less hurt for users, beta1 is fine.
>
Got it. Thanks for you explanation.

When you think you'd be done w/ the sync of Admin and AsyncAdmin
> Interfaces? Are the changes to Admin or to AsyncAdmin?

There are two issues (HBASE-19009 and HBASE-18950) need more review.  After
resolve these issues, I will upload a new patch for HBASE-18911. Thanks.

2017-10-27 0:14 GMT+08:00 Stack <st...@duboce.net>:

> On Thu, Oct 26, 2017 at 2:27 AM, Guanghao Zhang <zg...@gmail.com>
> wrote:
>
> > I saw beta == no new features, no API changes, just fixes. And I am
> working
> > on HBASE-18805 to unify Admin and AsyncAdmin methods. The fix version was
> > 2.0-beta-1. But I thought this will introduce API change(deprecate some
> API
> > and introduce new one). So should I change the fix versions to
> 2.0-alpha-4
> > and finish it before we release 2.0-alpha-4?
> >
> > Issue HBASE-18978 (Align the methods in Table and AsyncTable) may have
> same
> > problem. Thanks.
> >
> >
> Yeah, we were supposed to be done w/ public API changes when we shipped
> alpha-3 (smile).
>
> For Admin Interface, semantic versioning applies as per Duo comment. I see
> some polish still being done on the Admin/Table APIs (removal of deprecated
> methods, alignment of Admin and AsyncAdmin APIs). Hard to say no to this
> clean up on our most public facing Interfaces.
>
> When you think you'd be done w/ the sync of Admin and AsyncAdmin
> Interfaces? Are the changes to Admin or to AsyncAdmin?
>
> Thank you Guanghao Zhang,
>
> St.Ack
>
>
> > 2017-10-26 9:51 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
> >
> > > OK, skimmed,  we are in trouble! The in memory compaction just use the
> > same
> > > constructor with normal compaction to construct a StoreScanner, and use
> > it
> > > to do compaction...
> > >
> > > We have to provide several preXXX and postXXX for it, at least we
> should
> > > allow user reset TTL and max versions, and also do filtering on the
> > > scanner.
> > >
> > > Thanks.
> > >
> > >
> > >
> > > 2017-10-26 9:41 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
> > >
> > > > When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
> > > > HBASE-19033, I found that there maybe a hole in our CP hools. It is
> the
> > > in
> > > > memory compaction.
> > > >
> > > > As Anoop suggested in HBASE-19001, we still need to give user the
> > ability
> > > > to extend the max versions and TTL config, so I plan to add back the
> > > hooks
> > > > above to let CP users can change the max versions and TTL of a
> ScanInfo
> > > > object. But I'm not sure whether in memory compaction will also
> discard
> > > > expired cells, if so then we are in trouble...
> > > >
> > > > Thanks.
> > > >
> > > > 2017-10-25 6:03 GMT+08:00 Stack <st...@duboce.net>:
> > > >
> > > >> Chatting with my coworker Mr. Mike Drob, we were batting back and
> > forth
> > > >> what remains to be done. Surfacing our thoughts here so you all
> clued
> > > >> in....Or if you think otherwise, please speak up.
> > > >>
> > > >> We have ~13 issues to land:
> > > >> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> About
> > > two
> > > >> are meta-issues that are about process which leaves 11.
> > > >>
> > > >> Duo and Zheng Hu are to merge the FilterList fixes improvements
> > > >> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
> > > changed
> > > >> API/semantic that we need to get out earlier rather than later.
> > > >>
> > > >> Once the above is merged, HBASE-13346, change of Filter method names
> > to
> > > >> mention Cell instead of KeyValue can land.
> > > >>
> > > >> HBASE-199048 needs a review (Anoop will probably do it), removing
> > > >> IA.Private objects as params to MasterObserver... Hopefully this
> goes
> > in
> > > >> soon.
> > > >>
> > > >> Duo is hard at work on trackers for flush and compaction for CPs
> > > >> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
> > > >>
> > > >> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after
> > Duo
> > > >> is
> > > >> done w/ his work above.
> > > >>
> > > >> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after
> > all
> > > >> the
> > > >> purges allowing CPs do direct calls against Regions in same Host.
> > > >>
> > > >> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> > > >>
> > > >> Another day or two?
> > > >>
> > > >> St.Ack
> > > >>
> > > >>
> > > >>
> > > >> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
> > > >>
> > > >> >
> > > >> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
> > > wrote:
> > > >> >
> > > >> >> +1
> > > >> >>
> > > >> >> I was trying to work on helping out on the outstanding alpha-4
> > stuff
> > > >> last
> > > >> >> week -- will be continuing to try to do the same this week.
> > > >> >>
> > > >> >> If you need any help, Stack, or if others need reviews where I
> > > haven't
> > > >> >> noticed on my own: feel free to @mention me.
> > > >> >>
> > > >> >>
> > > >> > Thanks for the offer Josh. All items seem assigned and are being
> > > >> actively
> > > >> > worked on. If you get a moment, reviews by you (or anyone else)
> > helps
> > > >> move
> > > >> > the process along.
> > > >> >
> > > >> > We need to merge in HBASE-18410 branch to pick up Filter
> > improvements.
> > > >> > Then HBASE-13346 can go in.
> > > >> >
> > > >> > You are already helping out on HBASE-18906, thanks. Looks like
> that
> > > will
> > > >> > be addressed by other alpha-4s about to land.
> > > >> >
> > > >> > St.Ack
> > > >> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/
> > > 12341594
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >> On 10/23/17 12:53 PM, Stack wrote:
> > > >> >>
> > > >> >>> (Reviving this thread)
> > > >> >>>
> > > >> >>> Lets push out alpha-4 this week. Alpha-4 is the release that has
> > the
> > > >> >>> refactor of the Coprocessor API shutting down access to
> internals
> > > >> marked
> > > >> >>> InterfaceAudience.Private.
> > > >> >>>
> > > >> >>> The outstanding list is here:
> > > >> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > > >> >>>
> > > >> >>> Please push in anything marked alpha-4 that belongs to you.
> > > >> >>>
> > > >> >>> If issue, talk out loud on this thread. If you need a review to
> > land
> > > >> an
> > > >> >>> item, shout on the issue and here; we'll help you out.
> > > >> >>>
> > > >> >>> As is, items are coming along nicely I'd say. We need to merge
> the
> > > >> filter
> > > >> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> > > >> >>>
> > > >> >>> Post alpha-4, we'll have to hunt down our downstreamers and help
> > > them
> > > >> >>> test
> > > >> >>> on top of alpha-4 so rolling into beta-1, we have confidence our
> > > >> >>> downstreamers know what to expect (or we discover what we missed
> > > >> BEFORE
> > > >> >>> we
> > > >> >>> beta-1).
> > > >> >>>
> > > >> >>> Thanks for time,
> > > >> >>> S
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> > > >> >>>
> > > >> >>> I'll put up an alpha3 RC Monday, probably Monday night. That
> > should
> > > be
> > > >> >>>> time, if we all sprint, for the public-facing API fixes to be
> > done.
> > > >> >>>>
> > > >> >>>> I had a bunch of Coprocessor refactor and fixup scheduled for
> > > alpha3
> > > >> but
> > > >> >>>> it is plain that more time is needed (in spite of valiant
> effort
> > so
> > > >> far
> > > >> >>>> by
> > > >> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
> > whose
> > > >> >>>> theme is
> > > >> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> > > >> following
> > > >> >>>> week.
> > > >> >>>>
> > > >> >>>> We should then be ready for beta (beta == no new features, no
> API
> > > >> >>>> changes,
> > > >> >>>> just fixes).
> > > >> >>>>
> > > >> >>>> Thanks,
> > > >> >>>> St.Ack
> > > >> >>>>
> > > >> >>>>
> > > >> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
> > wrote:
> > > >> >>>>
> > > >> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote
> on
> > > it.
> > > >> >>>>>
> > > >> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope
> to
> > > get
> > > >> a
> > > >> >>>>> release out in the next week or so.
> > > >> >>>>>
> > > >> >>>>> I did a weeding of 2.0.0 issues over the last day. If folks
> are
> > > >> >>>>> interested in helping out, below are the items I think we need
> > > done
> > > >> for
> > > >> >>>>> alpha3 (below are at least 'Critical' status, are API possibly
> > > >> altering
> > > >> >>>>> items, and are absent those JIRAs that are making active
> > progress,
> > > >> >>>>> i.e. the
> > > >> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
> > needs
> > > >> >>>>> doing is
> > > >> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
> > > >> >>>>>
> > > >> >>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1
> > and
> > > >> >>>>> branch-2
> > > >> >>>>> This is to do what Andrew did between 1.3 and 1.4 branches
> only
> > do
> > > >> it
> > > >> >>>>> between branch-1 and branch-2.
> > > >> >>>>>
> > > >> >>>>> * HBASE-10462 Recategorize some of the client facing Public /
> > > >> Private
> > > >> >>>>> interfaces
> > > >> >>>>> This one is almost done. It could do with a finish, attention
> to
> > > the
> > > >> >>>>> items in last comment, and then our codebase could do with
> > another
> > > >> >>>>> sweep
> > > >> >>>>> after the spirit of this issue since a bunch has gone in since
> > the
> > > >> pass
> > > >> >>>>> that was the basis of this issue.
> > > >> >>>>>
> > > >> >>>>> * HBASE-10504 Define Replication Interface
> > > >> >>>>> I was going to take a crack at this as part of the revamp
> forced
> > > by
> > > >> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> > > Service'
> > > >> >>>>> but if
> > > >> >>>>> anyone else is interested, be my guest.
> > > >> >>>>>
> > > >> >>>>> * HBASE-14996 Some more API cleanup for 2.0
> > > >> >>>>> Has a bunch of subtasks, some of which are being worked on.
> > Needs
> > > >> >>>>> finishing.
> > > >> >>>>>
> > > >> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
> > Admin
> > > >> and
> > > >> >>>>> cleanup
> > > >> >>>>> Needs a pass. Small issue I think. Could also look at new
> > > >> AsyncClient
> > > >> >>>>> and
> > > >> >>>>> make sure symmetry.
> > > >> >>>>>
> > > >> >>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> > > >> >>>>> Predicated on result of an ongoing DISCUSSION thread but needs
> > to
> > > be
> > > >> >>>>> done.
> > > >> >>>>>
> > > >> >>>>> Rolling upgrade will have implications for our API. Would be
> > good
> > > to
> > > >> >>>>> try
> > > >> >>>>> it and figure what needs fixup (as said above, according to
> > trial
> > > by
> > > >> >>>>> Sean,
> > > >> >>>>> we might not be too bad here):
> > > >> >>>>> * HBASE-16060 1.x clients cannot access table state talking to
> > 2.0
> > > >> >>>>> cluster
> > > >> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
> Master
> > > and
> > > >> >>>>> 1.x
> > > >> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> > > >> >>>>>
> > > >> >>>>> * HBASE-17442 Move most of the replication related classes to
> > > >> >>>>> hbase-server package
> > > >> >>>>> The above would be good to do generally but it may make for
> > > ripples
> > > >> in
> > > >> >>>>> API so would be good to do now.
> > > >> >>>>>
> > > >> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> > > >> >>>>> Balazs is working on this. The idea is that we avoid adding
> two
> > > new
> > > >> >>>>> types
> > > >> >>>>> to our API, two types that are nought but curtailed, read-only
> > > >> views on
> > > >> >>>>> internals. Input if you have time appreciated.
> > > >> >>>>>
> > > >> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to
> a
> > > >> hbase2
> > > >> >>>>> cluster; verify
> > > >> >>>>> Esteban is looking at this one
> > > >> >>>>>
> > > >> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> > > >> >>>>> * HBASE-17143 Scan improvement
> > > >> >>>>>
> > > >> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
> > > >> >>>>> implementations
> > > >> >>>>> marked audience Private and returns implementations rather
> than
> > > >> >>>>> Interfaces.
> > > >> >>>>> In a few locations, we allow returning an alternate
> > implementation
> > > >> >>>>> altogether which is probably something we don't want a CP
> doing.
> > > To
> > > >> >>>>> that
> > > >> >>>>> end, the following issues started by Duo and Anoop need to be
> > > taken
> > > >> to
> > > >> >>>>> the
> > > >> >>>>> finish line; ideally they'd have an owner:
> > > >> >>>>>
> > > >> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release
> > <=
> > > >> The
> > > >> >>>>> umbrella issue.
> > > >> >>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
> > expose
> > > >> >>>>> * HBASE-16769 Deprecate/remove PB references from
> MasterObserver
> > > and
> > > >> >>>>> RegionServerObserver
> > > >> >>>>>
> > > >> >>>>>
> > > >> >>>>> Nice-to-haves:
> > > >> >>>>>
> > > >> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
> remove
> > > >> unused
> > > >> >>>>> TimeRange constructors
> > > >> >>>>>
> > > >> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
> > references
> > > >> >>>>> existing in the code
> > > >> >>>>> This is the end of an old long-running project moving up on to
> > > Cell
> > > >> >>>>> Interface. We think it is done but for a few little items
> > > >> (deprecate KV
> > > >> >>>>> methods in MR and provide Cell versions instead...)
> > > >> >>>>>
> > > >> >>>>> * HBASE-13271 Table#puts(List<Put>) operation is
> indeterminate;
> > > >> needs
> > > >> >>>>> fixing
> > > >> >>>>>
> > > >> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
> > > >> >>>>>
> > > >> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
> > > >> >>>>> * HBASE-14997
> > > >> >>>>> Move compareOp and Comparators out of filter to client package
> > > >> >>>>>
> > > >> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
> > > >> >>>>>
> > > >> >>>>> What about:
> > > >> >>>>>
> > > >> >>>>> * HBASE-18601 Remove Htrace 3.2
> > > >> >>>>> As has been noted, the HTrace API is our 'trace' API.
> > > >> >>>>>
> > > >> >>>>> If interested in any of the above and you need a legup, just
> ask
> > > in
> > > >> the
> > > >> >>>>> issue and I'll be by....
> > > >> >>>>>
> > > >> >>>>> Thanks,
> > > >> >>>>> St.Ack
> > > >> >>>>>
> > > >> >>>>>
> > > >> >>>>>
> > > >> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
> > wrote:
> > > >> >>>>>
> > > >> >>>>> Heads-up:
> > > >> >>>>>>
> > > >> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
> > > Theme
> > > >> is
> > > >> >>>>>> updated dependencies, reliance on relocated popular libs
> > (guava,
> > > >> >>>>>> netty,
> > > >> >>>>>> protobuf), purge of checked-in generated src, and
> > > >> >>>>>> master-carries-no-regions
> > > >> >>>>>> by default.
> > > >> >>>>>>
> > > >> >>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
> > theme
> > > >> will
> > > >> >>>>>> be
> > > >> >>>>>> settling the APIs and compatibility (At first blush, we are
> not
> > > >> >>>>>> looking too
> > > >> >>>>>> bad; our Sean ran some tests over weekend that have hbase-1
> > > client
> > > >> >>>>>> running
> > > >> >>>>>> against an hbase-2 cluster....). The Coprocessor Interface
> > revamp
> > > >> >>>>>> should be
> > > >> >>>>>> done by alpha3 (i.e. returning Interfaces rather than
> > > >> >>>>>> Implementations, and
> > > >> >>>>>> our shutdown of CPs accessing classes in hbase marked
> > > >> >>>>>> InterfaceAudience).
> > > >> >>>>>> We'll also have purged thirdparty classes from our API; e.g.
> > > guava
> > > >> >>>>>> 0.12
> > > >> >>>>>> Service showing through in our replication API and protobufs
> in
> > > >> Admin
> > > >> >>>>>> Interface. On alpha3, we will have to do a bunch of outreach
> to
> > > >> make
> > > >> >>>>>> sure
> > > >> >>>>>> our downstreamers are up on what is coming down the pipe.
> > > >> >>>>>>
> > > >> >>>>>> Beta1 in mid-September?
> > > >> >>>>>>
> > > >> >>>>>> I encourage you to check out the items marked for hbase2:
> > > >> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> > 12327188
> > > >> Edit
> > > >> >>>>>> as
> > > >> >>>>>> you see appropriate. Punt if you know the JIRA will not get
> any
> > > >> >>>>>> attention
> > > >> >>>>>> in next month or so.
> > > >> >>>>>>
> > > >> >>>>>> A bunch of issues marked blocker are unassigned. I'll leave
> > them
> > > >> as is
> > > >> >>>>>> another while but I'll boot them soon.
> > > >> >>>>>>
> > > >> >>>>>> While I have your attention:
> > > >> >>>>>>
> > > >> >>>>>> + I think we should leave thrift version at 0.9.3. Moving
> hbase
> > > >> thrift
> > > >> >>>>>> to 0.10.0 will break existing clients. The change is easy
> > enough
> > > if
> > > >> >>>>>> folks
> > > >> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> > > >> >>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
> first
> > > >> >>>>>> (0.98?).
> > > >> >>>>>>
> > > >> >>>>>> St.Ack
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
> > wrote:
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
> elserj@apache.org
> > >
> > > >> >>>>>>> wrote:
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> > > >> >>>>>>>>
> > > >> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
> > elserj@apache.org
> > > >
> > > >> >>>>>>>>> wrote:
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> ...
> > > >> >>>>>>>>>
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> I like the idea of this also hitting 2.0 as it would make
> > the
> > > >> >>>>>>>>>> feature a
> > > >> >>>>>>>>>> bit more "real", but am obviously a little nervous (I
> have
> > no
> > > >> >>>>>>>>>> reason
> > > >> >>>>>>>>>> to be
> > > >> >>>>>>>>>> nervous though). I am pretty happy with the feature in
> > terms
> > > of
> > > >> >>>>>>>>>> how
> > > >> >>>>>>>>>> much it
> > > >> >>>>>>>>>> is covered via testing.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
> > Josh.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> Last think on space quota says that need doc too. See
> 'Space
> > > >> >>>>>>>>> Quota' in
> > > >> >>>>>>>>> here:
> > > >> >>>>>>>>> https://docs.google.com/document/d/
> > 1WCsVlnHjJeKUcl7wHwqb4z9i
> > > >> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > > >> >>>>>>>>> Does this little section need an update Josh?
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> Thanks,
> > > >> >>>>>>>>> S
> > > >> >>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and
> > some
> > > >> docs
> > > >> >>>>>>>> already included for 17748.  Happy to put that on my plate
> if
> > > >> >>>>>>>> you're good
> > > >> >>>>>>>> with it. I can reasonably assume that no one is against it
> :)
> > > >> >>>>>>>>
> > > >> >>>>>>>> I think I had knocked out docs for the "phase 1" stuff
> before
> > > we
> > > >> >>>>>>>> merged it in from the original feature branch. I'll double
> > > check
> > > >> >>>>>>>> and update
> > > >> >>>>>>>> the gdoc. Perhaps this was just a timing thing.
> > > >> >>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>>>> Thanks Josh,
> > > >> >>>>>>> S
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Thu, Oct 26, 2017 at 2:27 AM, Guanghao Zhang <zg...@gmail.com> wrote:

> I saw beta == no new features, no API changes, just fixes. And I am working
> on HBASE-18805 to unify Admin and AsyncAdmin methods. The fix version was
> 2.0-beta-1. But I thought this will introduce API change(deprecate some API
> and introduce new one). So should I change the fix versions to 2.0-alpha-4
> and finish it before we release 2.0-alpha-4?
>
> Issue HBASE-18978 (Align the methods in Table and AsyncTable) may have same
> problem. Thanks.
>
>
Yeah, we were supposed to be done w/ public API changes when we shipped
alpha-3 (smile).

For Admin Interface, semantic versioning applies as per Duo comment. I see
some polish still being done on the Admin/Table APIs (removal of deprecated
methods, alignment of Admin and AsyncAdmin APIs). Hard to say no to this
clean up on our most public facing Interfaces.

When you think you'd be done w/ the sync of Admin and AsyncAdmin
Interfaces? Are the changes to Admin or to AsyncAdmin?

Thank you Guanghao Zhang,

St.Ack


> 2017-10-26 9:51 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
>
> > OK, skimmed,  we are in trouble! The in memory compaction just use the
> same
> > constructor with normal compaction to construct a StoreScanner, and use
> it
> > to do compaction...
> >
> > We have to provide several preXXX and postXXX for it, at least we should
> > allow user reset TTL and max versions, and also do filtering on the
> > scanner.
> >
> > Thanks.
> >
> >
> >
> > 2017-10-26 9:41 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
> >
> > > When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
> > > HBASE-19033, I found that there maybe a hole in our CP hools. It is the
> > in
> > > memory compaction.
> > >
> > > As Anoop suggested in HBASE-19001, we still need to give user the
> ability
> > > to extend the max versions and TTL config, so I plan to add back the
> > hooks
> > > above to let CP users can change the max versions and TTL of a ScanInfo
> > > object. But I'm not sure whether in memory compaction will also discard
> > > expired cells, if so then we are in trouble...
> > >
> > > Thanks.
> > >
> > > 2017-10-25 6:03 GMT+08:00 Stack <st...@duboce.net>:
> > >
> > >> Chatting with my coworker Mr. Mike Drob, we were batting back and
> forth
> > >> what remains to be done. Surfacing our thoughts here so you all clued
> > >> in....Or if you think otherwise, please speak up.
> > >>
> > >> We have ~13 issues to land:
> > >> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About
> > two
> > >> are meta-issues that are about process which leaves 11.
> > >>
> > >> Duo and Zheng Hu are to merge the FilterList fixes improvements
> > >> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
> > changed
> > >> API/semantic that we need to get out earlier rather than later.
> > >>
> > >> Once the above is merged, HBASE-13346, change of Filter method names
> to
> > >> mention Cell instead of KeyValue can land.
> > >>
> > >> HBASE-199048 needs a review (Anoop will probably do it), removing
> > >> IA.Private objects as params to MasterObserver... Hopefully this goes
> in
> > >> soon.
> > >>
> > >> Duo is hard at work on trackers for flush and compaction for CPs
> > >> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
> > >>
> > >> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after
> Duo
> > >> is
> > >> done w/ his work above.
> > >>
> > >> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after
> all
> > >> the
> > >> purges allowing CPs do direct calls against Regions in same Host.
> > >>
> > >> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> > >>
> > >> Another day or two?
> > >>
> > >> St.Ack
> > >>
> > >>
> > >>
> > >> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
> > >>
> > >> >
> > >> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
> > wrote:
> > >> >
> > >> >> +1
> > >> >>
> > >> >> I was trying to work on helping out on the outstanding alpha-4
> stuff
> > >> last
> > >> >> week -- will be continuing to try to do the same this week.
> > >> >>
> > >> >> If you need any help, Stack, or if others need reviews where I
> > haven't
> > >> >> noticed on my own: feel free to @mention me.
> > >> >>
> > >> >>
> > >> > Thanks for the offer Josh. All items seem assigned and are being
> > >> actively
> > >> > worked on. If you get a moment, reviews by you (or anyone else)
> helps
> > >> move
> > >> > the process along.
> > >> >
> > >> > We need to merge in HBASE-18410 branch to pick up Filter
> improvements.
> > >> > Then HBASE-13346 can go in.
> > >> >
> > >> > You are already helping out on HBASE-18906, thanks. Looks like that
> > will
> > >> > be addressed by other alpha-4s about to land.
> > >> >
> > >> > St.Ack
> > >> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/
> > 12341594
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >> On 10/23/17 12:53 PM, Stack wrote:
> > >> >>
> > >> >>> (Reviving this thread)
> > >> >>>
> > >> >>> Lets push out alpha-4 this week. Alpha-4 is the release that has
> the
> > >> >>> refactor of the Coprocessor API shutting down access to internals
> > >> marked
> > >> >>> InterfaceAudience.Private.
> > >> >>>
> > >> >>> The outstanding list is here:
> > >> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > >> >>>
> > >> >>> Please push in anything marked alpha-4 that belongs to you.
> > >> >>>
> > >> >>> If issue, talk out loud on this thread. If you need a review to
> land
> > >> an
> > >> >>> item, shout on the issue and here; we'll help you out.
> > >> >>>
> > >> >>> As is, items are coming along nicely I'd say. We need to merge the
> > >> filter
> > >> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> > >> >>>
> > >> >>> Post alpha-4, we'll have to hunt down our downstreamers and help
> > them
> > >> >>> test
> > >> >>> on top of alpha-4 so rolling into beta-1, we have confidence our
> > >> >>> downstreamers know what to expect (or we discover what we missed
> > >> BEFORE
> > >> >>> we
> > >> >>> beta-1).
> > >> >>>
> > >> >>> Thanks for time,
> > >> >>> S
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> > >> >>>
> > >> >>> I'll put up an alpha3 RC Monday, probably Monday night. That
> should
> > be
> > >> >>>> time, if we all sprint, for the public-facing API fixes to be
> done.
> > >> >>>>
> > >> >>>> I had a bunch of Coprocessor refactor and fixup scheduled for
> > alpha3
> > >> but
> > >> >>>> it is plain that more time is needed (in spite of valiant effort
> so
> > >> far
> > >> >>>> by
> > >> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
> whose
> > >> >>>> theme is
> > >> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> > >> following
> > >> >>>> week.
> > >> >>>>
> > >> >>>> We should then be ready for beta (beta == no new features, no API
> > >> >>>> changes,
> > >> >>>> just fixes).
> > >> >>>>
> > >> >>>> Thanks,
> > >> >>>> St.Ack
> > >> >>>>
> > >> >>>>
> > >> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
> wrote:
> > >> >>>>
> > >> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on
> > it.
> > >> >>>>>
> > >> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
> > get
> > >> a
> > >> >>>>> release out in the next week or so.
> > >> >>>>>
> > >> >>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> > >> >>>>> interested in helping out, below are the items I think we need
> > done
> > >> for
> > >> >>>>> alpha3 (below are at least 'Critical' status, are API possibly
> > >> altering
> > >> >>>>> items, and are absent those JIRAs that are making active
> progress,
> > >> >>>>> i.e. the
> > >> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
> needs
> > >> >>>>> doing is
> > >> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
> > >> >>>>>
> > >> >>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1
> and
> > >> >>>>> branch-2
> > >> >>>>> This is to do what Andrew did between 1.3 and 1.4 branches only
> do
> > >> it
> > >> >>>>> between branch-1 and branch-2.
> > >> >>>>>
> > >> >>>>> * HBASE-10462 Recategorize some of the client facing Public /
> > >> Private
> > >> >>>>> interfaces
> > >> >>>>> This one is almost done. It could do with a finish, attention to
> > the
> > >> >>>>> items in last comment, and then our codebase could do with
> another
> > >> >>>>> sweep
> > >> >>>>> after the spirit of this issue since a bunch has gone in since
> the
> > >> pass
> > >> >>>>> that was the basis of this issue.
> > >> >>>>>
> > >> >>>>> * HBASE-10504 Define Replication Interface
> > >> >>>>> I was going to take a crack at this as part of the revamp forced
> > by
> > >> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> > Service'
> > >> >>>>> but if
> > >> >>>>> anyone else is interested, be my guest.
> > >> >>>>>
> > >> >>>>> * HBASE-14996 Some more API cleanup for 2.0
> > >> >>>>> Has a bunch of subtasks, some of which are being worked on.
> Needs
> > >> >>>>> finishing.
> > >> >>>>>
> > >> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
> Admin
> > >> and
> > >> >>>>> cleanup
> > >> >>>>> Needs a pass. Small issue I think. Could also look at new
> > >> AsyncClient
> > >> >>>>> and
> > >> >>>>> make sure symmetry.
> > >> >>>>>
> > >> >>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> > >> >>>>> Predicated on result of an ongoing DISCUSSION thread but needs
> to
> > be
> > >> >>>>> done.
> > >> >>>>>
> > >> >>>>> Rolling upgrade will have implications for our API. Would be
> good
> > to
> > >> >>>>> try
> > >> >>>>> it and figure what needs fixup (as said above, according to
> trial
> > by
> > >> >>>>> Sean,
> > >> >>>>> we might not be too bad here):
> > >> >>>>> * HBASE-16060 1.x clients cannot access table state talking to
> 2.0
> > >> >>>>> cluster
> > >> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master
> > and
> > >> >>>>> 1.x
> > >> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> > >> >>>>>
> > >> >>>>> * HBASE-17442 Move most of the replication related classes to
> > >> >>>>> hbase-server package
> > >> >>>>> The above would be good to do generally but it may make for
> > ripples
> > >> in
> > >> >>>>> API so would be good to do now.
> > >> >>>>>
> > >> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> > >> >>>>> Balazs is working on this. The idea is that we avoid adding two
> > new
> > >> >>>>> types
> > >> >>>>> to our API, two types that are nought but curtailed, read-only
> > >> views on
> > >> >>>>> internals. Input if you have time appreciated.
> > >> >>>>>
> > >> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
> > >> hbase2
> > >> >>>>> cluster; verify
> > >> >>>>> Esteban is looking at this one
> > >> >>>>>
> > >> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> > >> >>>>> * HBASE-17143 Scan improvement
> > >> >>>>>
> > >> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
> > >> >>>>> implementations
> > >> >>>>> marked audience Private and returns implementations rather than
> > >> >>>>> Interfaces.
> > >> >>>>> In a few locations, we allow returning an alternate
> implementation
> > >> >>>>> altogether which is probably something we don't want a CP doing.
> > To
> > >> >>>>> that
> > >> >>>>> end, the following issues started by Duo and Anoop need to be
> > taken
> > >> to
> > >> >>>>> the
> > >> >>>>> finish line; ideally they'd have an owner:
> > >> >>>>>
> > >> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release
> <=
> > >> The
> > >> >>>>> umbrella issue.
> > >> >>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
> expose
> > >> >>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver
> > and
> > >> >>>>> RegionServerObserver
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> Nice-to-haves:
> > >> >>>>>
> > >> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
> > >> unused
> > >> >>>>> TimeRange constructors
> > >> >>>>>
> > >> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
> references
> > >> >>>>> existing in the code
> > >> >>>>> This is the end of an old long-running project moving up on to
> > Cell
> > >> >>>>> Interface. We think it is done but for a few little items
> > >> (deprecate KV
> > >> >>>>> methods in MR and provide Cell versions instead...)
> > >> >>>>>
> > >> >>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
> > >> needs
> > >> >>>>> fixing
> > >> >>>>>
> > >> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
> > >> >>>>>
> > >> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
> > >> >>>>> * HBASE-14997
> > >> >>>>> Move compareOp and Comparators out of filter to client package
> > >> >>>>>
> > >> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
> > >> >>>>>
> > >> >>>>> What about:
> > >> >>>>>
> > >> >>>>> * HBASE-18601 Remove Htrace 3.2
> > >> >>>>> As has been noted, the HTrace API is our 'trace' API.
> > >> >>>>>
> > >> >>>>> If interested in any of the above and you need a legup, just ask
> > in
> > >> the
> > >> >>>>> issue and I'll be by....
> > >> >>>>>
> > >> >>>>> Thanks,
> > >> >>>>> St.Ack
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
> wrote:
> > >> >>>>>
> > >> >>>>> Heads-up:
> > >> >>>>>>
> > >> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
> > Theme
> > >> is
> > >> >>>>>> updated dependencies, reliance on relocated popular libs
> (guava,
> > >> >>>>>> netty,
> > >> >>>>>> protobuf), purge of checked-in generated src, and
> > >> >>>>>> master-carries-no-regions
> > >> >>>>>> by default.
> > >> >>>>>>
> > >> >>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
> theme
> > >> will
> > >> >>>>>> be
> > >> >>>>>> settling the APIs and compatibility (At first blush, we are not
> > >> >>>>>> looking too
> > >> >>>>>> bad; our Sean ran some tests over weekend that have hbase-1
> > client
> > >> >>>>>> running
> > >> >>>>>> against an hbase-2 cluster....). The Coprocessor Interface
> revamp
> > >> >>>>>> should be
> > >> >>>>>> done by alpha3 (i.e. returning Interfaces rather than
> > >> >>>>>> Implementations, and
> > >> >>>>>> our shutdown of CPs accessing classes in hbase marked
> > >> >>>>>> InterfaceAudience).
> > >> >>>>>> We'll also have purged thirdparty classes from our API; e.g.
> > guava
> > >> >>>>>> 0.12
> > >> >>>>>> Service showing through in our replication API and protobufs in
> > >> Admin
> > >> >>>>>> Interface. On alpha3, we will have to do a bunch of outreach to
> > >> make
> > >> >>>>>> sure
> > >> >>>>>> our downstreamers are up on what is coming down the pipe.
> > >> >>>>>>
> > >> >>>>>> Beta1 in mid-September?
> > >> >>>>>>
> > >> >>>>>> I encourage you to check out the items marked for hbase2:
> > >> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> 12327188
> > >> Edit
> > >> >>>>>> as
> > >> >>>>>> you see appropriate. Punt if you know the JIRA will not get any
> > >> >>>>>> attention
> > >> >>>>>> in next month or so.
> > >> >>>>>>
> > >> >>>>>> A bunch of issues marked blocker are unassigned. I'll leave
> them
> > >> as is
> > >> >>>>>> another while but I'll boot them soon.
> > >> >>>>>>
> > >> >>>>>> While I have your attention:
> > >> >>>>>>
> > >> >>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
> > >> thrift
> > >> >>>>>> to 0.10.0 will break existing clients. The change is easy
> enough
> > if
> > >> >>>>>> folks
> > >> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> > >> >>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> > >> >>>>>> (0.98?).
> > >> >>>>>>
> > >> >>>>>> St.Ack
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
> wrote:
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>>
> > >> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <elserj@apache.org
> >
> > >> >>>>>>> wrote:
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> > >> >>>>>>>>
> > >> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
> elserj@apache.org
> > >
> > >> >>>>>>>>> wrote:
> > >> >>>>>>>>>
> > >> >>>>>>>>> ...
> > >> >>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> I like the idea of this also hitting 2.0 as it would make
> the
> > >> >>>>>>>>>> feature a
> > >> >>>>>>>>>> bit more "real", but am obviously a little nervous (I have
> no
> > >> >>>>>>>>>> reason
> > >> >>>>>>>>>> to be
> > >> >>>>>>>>>> nervous though). I am pretty happy with the feature in
> terms
> > of
> > >> >>>>>>>>>> how
> > >> >>>>>>>>>> much it
> > >> >>>>>>>>>> is covered via testing.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
> Josh.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>> Last think on space quota says that need doc too. See 'Space
> > >> >>>>>>>>> Quota' in
> > >> >>>>>>>>> here:
> > >> >>>>>>>>> https://docs.google.com/document/d/
> 1WCsVlnHjJeKUcl7wHwqb4z9i
> > >> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > >> >>>>>>>>> Does this little section need an update Josh?
> > >> >>>>>>>>>
> > >> >>>>>>>>> Thanks,
> > >> >>>>>>>>> S
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and
> some
> > >> docs
> > >> >>>>>>>> already included for 17748.  Happy to put that on my plate if
> > >> >>>>>>>> you're good
> > >> >>>>>>>> with it. I can reasonably assume that no one is against it :)
> > >> >>>>>>>>
> > >> >>>>>>>> I think I had knocked out docs for the "phase 1" stuff before
> > we
> > >> >>>>>>>> merged it in from the original feature branch. I'll double
> > check
> > >> >>>>>>>> and update
> > >> >>>>>>>> the gdoc. Perhaps this was just a timing thing.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>> Thanks Josh,
> > >> >>>>>>> S
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >
> > >>
> > >
> > >
> >
>

Re: Moving 2.0 forward

["张铎 (Duo Zhang)" <pa...@gmail.com>]

Alpha4 aims to freeze the CP related API, not the public API. We break
everything for CP so we need to push out a stable(I hope) version to let CP
users learn how to implement their project on top of the new APIs.

For public API we have rules on how to keep compatible so I think it is
less hurt for users, beta1 is fine.

2017-10-26 17:27 GMT+08:00 Guanghao Zhang <zg...@gmail.com>:

> I saw beta == no new features, no API changes, just fixes. And I am working
> on HBASE-18805 to unify Admin and AsyncAdmin methods. The fix version was
> 2.0-beta-1. But I thought this will introduce API change(deprecate some API
> and introduce new one). So should I change the fix versions to 2.0-alpha-4
> and finish it before we release 2.0-alpha-4?
>
> Issue HBASE-18978 (Align the methods in Table and AsyncTable) may have same
> problem. Thanks.
>
> 2017-10-26 9:51 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
>
> > OK, skimmed,  we are in trouble! The in memory compaction just use the
> same
> > constructor with normal compaction to construct a StoreScanner, and use
> it
> > to do compaction...
> >
> > We have to provide several preXXX and postXXX for it, at least we should
> > allow user reset TTL and max versions, and also do filtering on the
> > scanner.
> >
> > Thanks.
> >
> >
> >
> > 2017-10-26 9:41 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
> >
> > > When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
> > > HBASE-19033, I found that there maybe a hole in our CP hools. It is the
> > in
> > > memory compaction.
> > >
> > > As Anoop suggested in HBASE-19001, we still need to give user the
> ability
> > > to extend the max versions and TTL config, so I plan to add back the
> > hooks
> > > above to let CP users can change the max versions and TTL of a ScanInfo
> > > object. But I'm not sure whether in memory compaction will also discard
> > > expired cells, if so then we are in trouble...
> > >
> > > Thanks.
> > >
> > > 2017-10-25 6:03 GMT+08:00 Stack <st...@duboce.net>:
> > >
> > >> Chatting with my coworker Mr. Mike Drob, we were batting back and
> forth
> > >> what remains to be done. Surfacing our thoughts here so you all clued
> > >> in....Or if you think otherwise, please speak up.
> > >>
> > >> We have ~13 issues to land:
> > >> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About
> > two
> > >> are meta-issues that are about process which leaves 11.
> > >>
> > >> Duo and Zheng Hu are to merge the FilterList fixes improvements
> > >> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
> > changed
> > >> API/semantic that we need to get out earlier rather than later.
> > >>
> > >> Once the above is merged, HBASE-13346, change of Filter method names
> to
> > >> mention Cell instead of KeyValue can land.
> > >>
> > >> HBASE-199048 needs a review (Anoop will probably do it), removing
> > >> IA.Private objects as params to MasterObserver... Hopefully this goes
> in
> > >> soon.
> > >>
> > >> Duo is hard at work on trackers for flush and compaction for CPs
> > >> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
> > >>
> > >> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after
> Duo
> > >> is
> > >> done w/ his work above.
> > >>
> > >> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after
> all
> > >> the
> > >> purges allowing CPs do direct calls against Regions in same Host.
> > >>
> > >> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> > >>
> > >> Another day or two?
> > >>
> > >> St.Ack
> > >>
> > >>
> > >>
> > >> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
> > >>
> > >> >
> > >> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
> > wrote:
> > >> >
> > >> >> +1
> > >> >>
> > >> >> I was trying to work on helping out on the outstanding alpha-4
> stuff
> > >> last
> > >> >> week -- will be continuing to try to do the same this week.
> > >> >>
> > >> >> If you need any help, Stack, or if others need reviews where I
> > haven't
> > >> >> noticed on my own: feel free to @mention me.
> > >> >>
> > >> >>
> > >> > Thanks for the offer Josh. All items seem assigned and are being
> > >> actively
> > >> > worked on. If you get a moment, reviews by you (or anyone else)
> helps
> > >> move
> > >> > the process along.
> > >> >
> > >> > We need to merge in HBASE-18410 branch to pick up Filter
> improvements.
> > >> > Then HBASE-13346 can go in.
> > >> >
> > >> > You are already helping out on HBASE-18906, thanks. Looks like that
> > will
> > >> > be addressed by other alpha-4s about to land.
> > >> >
> > >> > St.Ack
> > >> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/
> > 12341594
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >> On 10/23/17 12:53 PM, Stack wrote:
> > >> >>
> > >> >>> (Reviving this thread)
> > >> >>>
> > >> >>> Lets push out alpha-4 this week. Alpha-4 is the release that has
> the
> > >> >>> refactor of the Coprocessor API shutting down access to internals
> > >> marked
> > >> >>> InterfaceAudience.Private.
> > >> >>>
> > >> >>> The outstanding list is here:
> > >> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > >> >>>
> > >> >>> Please push in anything marked alpha-4 that belongs to you.
> > >> >>>
> > >> >>> If issue, talk out loud on this thread. If you need a review to
> land
> > >> an
> > >> >>> item, shout on the issue and here; we'll help you out.
> > >> >>>
> > >> >>> As is, items are coming along nicely I'd say. We need to merge the
> > >> filter
> > >> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> > >> >>>
> > >> >>> Post alpha-4, we'll have to hunt down our downstreamers and help
> > them
> > >> >>> test
> > >> >>> on top of alpha-4 so rolling into beta-1, we have confidence our
> > >> >>> downstreamers know what to expect (or we discover what we missed
> > >> BEFORE
> > >> >>> we
> > >> >>> beta-1).
> > >> >>>
> > >> >>> Thanks for time,
> > >> >>> S
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> > >> >>>
> > >> >>> I'll put up an alpha3 RC Monday, probably Monday night. That
> should
> > be
> > >> >>>> time, if we all sprint, for the public-facing API fixes to be
> done.
> > >> >>>>
> > >> >>>> I had a bunch of Coprocessor refactor and fixup scheduled for
> > alpha3
> > >> but
> > >> >>>> it is plain that more time is needed (in spite of valiant effort
> so
> > >> far
> > >> >>>> by
> > >> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
> whose
> > >> >>>> theme is
> > >> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> > >> following
> > >> >>>> week.
> > >> >>>>
> > >> >>>> We should then be ready for beta (beta == no new features, no API
> > >> >>>> changes,
> > >> >>>> just fixes).
> > >> >>>>
> > >> >>>> Thanks,
> > >> >>>> St.Ack
> > >> >>>>
> > >> >>>>
> > >> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
> wrote:
> > >> >>>>
> > >> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on
> > it.
> > >> >>>>>
> > >> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
> > get
> > >> a
> > >> >>>>> release out in the next week or so.
> > >> >>>>>
> > >> >>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> > >> >>>>> interested in helping out, below are the items I think we need
> > done
> > >> for
> > >> >>>>> alpha3 (below are at least 'Critical' status, are API possibly
> > >> altering
> > >> >>>>> items, and are absent those JIRAs that are making active
> progress,
> > >> >>>>> i.e. the
> > >> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
> needs
> > >> >>>>> doing is
> > >> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
> > >> >>>>>
> > >> >>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1
> and
> > >> >>>>> branch-2
> > >> >>>>> This is to do what Andrew did between 1.3 and 1.4 branches only
> do
> > >> it
> > >> >>>>> between branch-1 and branch-2.
> > >> >>>>>
> > >> >>>>> * HBASE-10462 Recategorize some of the client facing Public /
> > >> Private
> > >> >>>>> interfaces
> > >> >>>>> This one is almost done. It could do with a finish, attention to
> > the
> > >> >>>>> items in last comment, and then our codebase could do with
> another
> > >> >>>>> sweep
> > >> >>>>> after the spirit of this issue since a bunch has gone in since
> the
> > >> pass
> > >> >>>>> that was the basis of this issue.
> > >> >>>>>
> > >> >>>>> * HBASE-10504 Define Replication Interface
> > >> >>>>> I was going to take a crack at this as part of the revamp forced
> > by
> > >> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> > Service'
> > >> >>>>> but if
> > >> >>>>> anyone else is interested, be my guest.
> > >> >>>>>
> > >> >>>>> * HBASE-14996 Some more API cleanup for 2.0
> > >> >>>>> Has a bunch of subtasks, some of which are being worked on.
> Needs
> > >> >>>>> finishing.
> > >> >>>>>
> > >> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
> Admin
> > >> and
> > >> >>>>> cleanup
> > >> >>>>> Needs a pass. Small issue I think. Could also look at new
> > >> AsyncClient
> > >> >>>>> and
> > >> >>>>> make sure symmetry.
> > >> >>>>>
> > >> >>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> > >> >>>>> Predicated on result of an ongoing DISCUSSION thread but needs
> to
> > be
> > >> >>>>> done.
> > >> >>>>>
> > >> >>>>> Rolling upgrade will have implications for our API. Would be
> good
> > to
> > >> >>>>> try
> > >> >>>>> it and figure what needs fixup (as said above, according to
> trial
> > by
> > >> >>>>> Sean,
> > >> >>>>> we might not be too bad here):
> > >> >>>>> * HBASE-16060 1.x clients cannot access table state talking to
> 2.0
> > >> >>>>> cluster
> > >> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master
> > and
> > >> >>>>> 1.x
> > >> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> > >> >>>>>
> > >> >>>>> * HBASE-17442 Move most of the replication related classes to
> > >> >>>>> hbase-server package
> > >> >>>>> The above would be good to do generally but it may make for
> > ripples
> > >> in
> > >> >>>>> API so would be good to do now.
> > >> >>>>>
> > >> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> > >> >>>>> Balazs is working on this. The idea is that we avoid adding two
> > new
> > >> >>>>> types
> > >> >>>>> to our API, two types that are nought but curtailed, read-only
> > >> views on
> > >> >>>>> internals. Input if you have time appreciated.
> > >> >>>>>
> > >> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
> > >> hbase2
> > >> >>>>> cluster; verify
> > >> >>>>> Esteban is looking at this one
> > >> >>>>>
> > >> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> > >> >>>>> * HBASE-17143 Scan improvement
> > >> >>>>>
> > >> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
> > >> >>>>> implementations
> > >> >>>>> marked audience Private and returns implementations rather than
> > >> >>>>> Interfaces.
> > >> >>>>> In a few locations, we allow returning an alternate
> implementation
> > >> >>>>> altogether which is probably something we don't want a CP doing.
> > To
> > >> >>>>> that
> > >> >>>>> end, the following issues started by Duo and Anoop need to be
> > taken
> > >> to
> > >> >>>>> the
> > >> >>>>> finish line; ideally they'd have an owner:
> > >> >>>>>
> > >> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release
> <=
> > >> The
> > >> >>>>> umbrella issue.
> > >> >>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
> expose
> > >> >>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver
> > and
> > >> >>>>> RegionServerObserver
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> Nice-to-haves:
> > >> >>>>>
> > >> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
> > >> unused
> > >> >>>>> TimeRange constructors
> > >> >>>>>
> > >> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
> references
> > >> >>>>> existing in the code
> > >> >>>>> This is the end of an old long-running project moving up on to
> > Cell
> > >> >>>>> Interface. We think it is done but for a few little items
> > >> (deprecate KV
> > >> >>>>> methods in MR and provide Cell versions instead...)
> > >> >>>>>
> > >> >>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
> > >> needs
> > >> >>>>> fixing
> > >> >>>>>
> > >> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
> > >> >>>>>
> > >> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
> > >> >>>>> * HBASE-14997
> > >> >>>>> Move compareOp and Comparators out of filter to client package
> > >> >>>>>
> > >> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
> > >> >>>>>
> > >> >>>>> What about:
> > >> >>>>>
> > >> >>>>> * HBASE-18601 Remove Htrace 3.2
> > >> >>>>> As has been noted, the HTrace API is our 'trace' API.
> > >> >>>>>
> > >> >>>>> If interested in any of the above and you need a legup, just ask
> > in
> > >> the
> > >> >>>>> issue and I'll be by....
> > >> >>>>>
> > >> >>>>> Thanks,
> > >> >>>>> St.Ack
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
> wrote:
> > >> >>>>>
> > >> >>>>> Heads-up:
> > >> >>>>>>
> > >> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
> > Theme
> > >> is
> > >> >>>>>> updated dependencies, reliance on relocated popular libs
> (guava,
> > >> >>>>>> netty,
> > >> >>>>>> protobuf), purge of checked-in generated src, and
> > >> >>>>>> master-carries-no-regions
> > >> >>>>>> by default.
> > >> >>>>>>
> > >> >>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
> theme
> > >> will
> > >> >>>>>> be
> > >> >>>>>> settling the APIs and compatibility (At first blush, we are not
> > >> >>>>>> looking too
> > >> >>>>>> bad; our Sean ran some tests over weekend that have hbase-1
> > client
> > >> >>>>>> running
> > >> >>>>>> against an hbase-2 cluster....). The Coprocessor Interface
> revamp
> > >> >>>>>> should be
> > >> >>>>>> done by alpha3 (i.e. returning Interfaces rather than
> > >> >>>>>> Implementations, and
> > >> >>>>>> our shutdown of CPs accessing classes in hbase marked
> > >> >>>>>> InterfaceAudience).
> > >> >>>>>> We'll also have purged thirdparty classes from our API; e.g.
> > guava
> > >> >>>>>> 0.12
> > >> >>>>>> Service showing through in our replication API and protobufs in
> > >> Admin
> > >> >>>>>> Interface. On alpha3, we will have to do a bunch of outreach to
> > >> make
> > >> >>>>>> sure
> > >> >>>>>> our downstreamers are up on what is coming down the pipe.
> > >> >>>>>>
> > >> >>>>>> Beta1 in mid-September?
> > >> >>>>>>
> > >> >>>>>> I encourage you to check out the items marked for hbase2:
> > >> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> 12327188
> > >> Edit
> > >> >>>>>> as
> > >> >>>>>> you see appropriate. Punt if you know the JIRA will not get any
> > >> >>>>>> attention
> > >> >>>>>> in next month or so.
> > >> >>>>>>
> > >> >>>>>> A bunch of issues marked blocker are unassigned. I'll leave
> them
> > >> as is
> > >> >>>>>> another while but I'll boot them soon.
> > >> >>>>>>
> > >> >>>>>> While I have your attention:
> > >> >>>>>>
> > >> >>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
> > >> thrift
> > >> >>>>>> to 0.10.0 will break existing clients. The change is easy
> enough
> > if
> > >> >>>>>> folks
> > >> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> > >> >>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> > >> >>>>>> (0.98?).
> > >> >>>>>>
> > >> >>>>>> St.Ack
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
> wrote:
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>>
> > >> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <elserj@apache.org
> >
> > >> >>>>>>> wrote:
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> > >> >>>>>>>>
> > >> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
> elserj@apache.org
> > >
> > >> >>>>>>>>> wrote:
> > >> >>>>>>>>>
> > >> >>>>>>>>> ...
> > >> >>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> I like the idea of this also hitting 2.0 as it would make
> the
> > >> >>>>>>>>>> feature a
> > >> >>>>>>>>>> bit more "real", but am obviously a little nervous (I have
> no
> > >> >>>>>>>>>> reason
> > >> >>>>>>>>>> to be
> > >> >>>>>>>>>> nervous though). I am pretty happy with the feature in
> terms
> > of
> > >> >>>>>>>>>> how
> > >> >>>>>>>>>> much it
> > >> >>>>>>>>>> is covered via testing.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
> Josh.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>> Last think on space quota says that need doc too. See 'Space
> > >> >>>>>>>>> Quota' in
> > >> >>>>>>>>> here:
> > >> >>>>>>>>> https://docs.google.com/document/d/
> 1WCsVlnHjJeKUcl7wHwqb4z9i
> > >> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > >> >>>>>>>>> Does this little section need an update Josh?
> > >> >>>>>>>>>
> > >> >>>>>>>>> Thanks,
> > >> >>>>>>>>> S
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and
> some
> > >> docs
> > >> >>>>>>>> already included for 17748.  Happy to put that on my plate if
> > >> >>>>>>>> you're good
> > >> >>>>>>>> with it. I can reasonably assume that no one is against it :)
> > >> >>>>>>>>
> > >> >>>>>>>> I think I had knocked out docs for the "phase 1" stuff before
> > we
> > >> >>>>>>>> merged it in from the original feature branch. I'll double
> > check
> > >> >>>>>>>> and update
> > >> >>>>>>>> the gdoc. Perhaps this was just a timing thing.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>> Thanks Josh,
> > >> >>>>>>> S
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >
> > >>
> > >
> > >
> >
>

Re: Moving 2.0 forward

[Guanghao Zhang <zg...@gmail.com>]

I saw beta == no new features, no API changes, just fixes. And I am working
on HBASE-18805 to unify Admin and AsyncAdmin methods. The fix version was
2.0-beta-1. But I thought this will introduce API change(deprecate some API
and introduce new one). So should I change the fix versions to 2.0-alpha-4
and finish it before we release 2.0-alpha-4?

Issue HBASE-18978 (Align the methods in Table and AsyncTable) may have same
problem. Thanks.

2017-10-26 9:51 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:

> OK, skimmed,  we are in trouble! The in memory compaction just use the same
> constructor with normal compaction to construct a StoreScanner, and use it
> to do compaction...
>
> We have to provide several preXXX and postXXX for it, at least we should
> allow user reset TTL and max versions, and also do filtering on the
> scanner.
>
> Thanks.
>
>
>
> 2017-10-26 9:41 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
>
> > When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
> > HBASE-19033, I found that there maybe a hole in our CP hools. It is the
> in
> > memory compaction.
> >
> > As Anoop suggested in HBASE-19001, we still need to give user the ability
> > to extend the max versions and TTL config, so I plan to add back the
> hooks
> > above to let CP users can change the max versions and TTL of a ScanInfo
> > object. But I'm not sure whether in memory compaction will also discard
> > expired cells, if so then we are in trouble...
> >
> > Thanks.
> >
> > 2017-10-25 6:03 GMT+08:00 Stack <st...@duboce.net>:
> >
> >> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
> >> what remains to be done. Surfacing our thoughts here so you all clued
> >> in....Or if you think otherwise, please speak up.
> >>
> >> We have ~13 issues to land:
> >> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About
> two
> >> are meta-issues that are about process which leaves 11.
> >>
> >> Duo and Zheng Hu are to merge the FilterList fixes improvements
> >> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
> changed
> >> API/semantic that we need to get out earlier rather than later.
> >>
> >> Once the above is merged, HBASE-13346, change of Filter method names to
> >> mention Cell instead of KeyValue can land.
> >>
> >> HBASE-199048 needs a review (Anoop will probably do it), removing
> >> IA.Private objects as params to MasterObserver... Hopefully this goes in
> >> soon.
> >>
> >> Duo is hard at work on trackers for flush and compaction for CPs
> >> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
> >>
> >> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo
> >> is
> >> done w/ his work above.
> >>
> >> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all
> >> the
> >> purges allowing CPs do direct calls against Regions in same Host.
> >>
> >> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> >>
> >> Another day or two?
> >>
> >> St.Ack
> >>
> >>
> >>
> >> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
> >>
> >> >
> >> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
> wrote:
> >> >
> >> >> +1
> >> >>
> >> >> I was trying to work on helping out on the outstanding alpha-4 stuff
> >> last
> >> >> week -- will be continuing to try to do the same this week.
> >> >>
> >> >> If you need any help, Stack, or if others need reviews where I
> haven't
> >> >> noticed on my own: feel free to @mention me.
> >> >>
> >> >>
> >> > Thanks for the offer Josh. All items seem assigned and are being
> >> actively
> >> > worked on. If you get a moment, reviews by you (or anyone else) helps
> >> move
> >> > the process along.
> >> >
> >> > We need to merge in HBASE-18410 branch to pick up Filter improvements.
> >> > Then HBASE-13346 can go in.
> >> >
> >> > You are already helping out on HBASE-18906, thanks. Looks like that
> will
> >> > be addressed by other alpha-4s about to land.
> >> >
> >> > St.Ack
> >> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/
> 12341594
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >> On 10/23/17 12:53 PM, Stack wrote:
> >> >>
> >> >>> (Reviving this thread)
> >> >>>
> >> >>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
> >> >>> refactor of the Coprocessor API shutting down access to internals
> >> marked
> >> >>> InterfaceAudience.Private.
> >> >>>
> >> >>> The outstanding list is here:
> >> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >> >>>
> >> >>> Please push in anything marked alpha-4 that belongs to you.
> >> >>>
> >> >>> If issue, talk out loud on this thread. If you need a review to land
> >> an
> >> >>> item, shout on the issue and here; we'll help you out.
> >> >>>
> >> >>> As is, items are coming along nicely I'd say. We need to merge the
> >> filter
> >> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> >> >>>
> >> >>> Post alpha-4, we'll have to hunt down our downstreamers and help
> them
> >> >>> test
> >> >>> on top of alpha-4 so rolling into beta-1, we have confidence our
> >> >>> downstreamers know what to expect (or we discover what we missed
> >> BEFORE
> >> >>> we
> >> >>> beta-1).
> >> >>>
> >> >>> Thanks for time,
> >> >>> S
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> >> >>>
> >> >>> I'll put up an alpha3 RC Monday, probably Monday night. That should
> be
> >> >>>> time, if we all sprint, for the public-facing API fixes to be done.
> >> >>>>
> >> >>>> I had a bunch of Coprocessor refactor and fixup scheduled for
> alpha3
> >> but
> >> >>>> it is plain that more time is needed (in spite of valiant effort so
> >> far
> >> >>>> by
> >> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
> >> >>>> theme is
> >> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> >> following
> >> >>>> week.
> >> >>>>
> >> >>>> We should then be ready for beta (beta == no new features, no API
> >> >>>> changes,
> >> >>>> just fixes).
> >> >>>>
> >> >>>> Thanks,
> >> >>>> St.Ack
> >> >>>>
> >> >>>>
> >> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
> >> >>>>
> >> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on
> it.
> >> >>>>>
> >> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
> get
> >> a
> >> >>>>> release out in the next week or so.
> >> >>>>>
> >> >>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> >> >>>>> interested in helping out, below are the items I think we need
> done
> >> for
> >> >>>>> alpha3 (below are at least 'Critical' status, are API possibly
> >> altering
> >> >>>>> items, and are absent those JIRAs that are making active progress,
> >> >>>>> i.e. the
> >> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
> >> >>>>> doing is
> >> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
> >> >>>>>
> >> >>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
> >> >>>>> branch-2
> >> >>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do
> >> it
> >> >>>>> between branch-1 and branch-2.
> >> >>>>>
> >> >>>>> * HBASE-10462 Recategorize some of the client facing Public /
> >> Private
> >> >>>>> interfaces
> >> >>>>> This one is almost done. It could do with a finish, attention to
> the
> >> >>>>> items in last comment, and then our codebase could do with another
> >> >>>>> sweep
> >> >>>>> after the spirit of this issue since a bunch has gone in since the
> >> pass
> >> >>>>> that was the basis of this issue.
> >> >>>>>
> >> >>>>> * HBASE-10504 Define Replication Interface
> >> >>>>> I was going to take a crack at this as part of the revamp forced
> by
> >> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> Service'
> >> >>>>> but if
> >> >>>>> anyone else is interested, be my guest.
> >> >>>>>
> >> >>>>> * HBASE-14996 Some more API cleanup for 2.0
> >> >>>>> Has a bunch of subtasks, some of which are being worked on. Needs
> >> >>>>> finishing.
> >> >>>>>
> >> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
> >> and
> >> >>>>> cleanup
> >> >>>>> Needs a pass. Small issue I think. Could also look at new
> >> AsyncClient
> >> >>>>> and
> >> >>>>> make sure symmetry.
> >> >>>>>
> >> >>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> >> >>>>> Predicated on result of an ongoing DISCUSSION thread but needs to
> be
> >> >>>>> done.
> >> >>>>>
> >> >>>>> Rolling upgrade will have implications for our API. Would be good
> to
> >> >>>>> try
> >> >>>>> it and figure what needs fixup (as said above, according to trial
> by
> >> >>>>> Sean,
> >> >>>>> we might not be too bad here):
> >> >>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
> >> >>>>> cluster
> >> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master
> and
> >> >>>>> 1.x
> >> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> >> >>>>>
> >> >>>>> * HBASE-17442 Move most of the replication related classes to
> >> >>>>> hbase-server package
> >> >>>>> The above would be good to do generally but it may make for
> ripples
> >> in
> >> >>>>> API so would be good to do now.
> >> >>>>>
> >> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> >> >>>>> Balazs is working on this. The idea is that we avoid adding two
> new
> >> >>>>> types
> >> >>>>> to our API, two types that are nought but curtailed, read-only
> >> views on
> >> >>>>> internals. Input if you have time appreciated.
> >> >>>>>
> >> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
> >> hbase2
> >> >>>>> cluster; verify
> >> >>>>> Esteban is looking at this one
> >> >>>>>
> >> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> >> >>>>> * HBASE-17143 Scan improvement
> >> >>>>>
> >> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
> >> >>>>> implementations
> >> >>>>> marked audience Private and returns implementations rather than
> >> >>>>> Interfaces.
> >> >>>>> In a few locations, we allow returning an alternate implementation
> >> >>>>> altogether which is probably something we don't want a CP doing.
> To
> >> >>>>> that
> >> >>>>> end, the following issues started by Duo and Anoop need to be
> taken
> >> to
> >> >>>>> the
> >> >>>>> finish line; ideally they'd have an owner:
> >> >>>>>
> >> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
> >> The
> >> >>>>> umbrella issue.
> >> >>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> >> >>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver
> and
> >> >>>>> RegionServerObserver
> >> >>>>>
> >> >>>>>
> >> >>>>> Nice-to-haves:
> >> >>>>>
> >> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
> >> unused
> >> >>>>> TimeRange constructors
> >> >>>>>
> >> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
> >> >>>>> existing in the code
> >> >>>>> This is the end of an old long-running project moving up on to
> Cell
> >> >>>>> Interface. We think it is done but for a few little items
> >> (deprecate KV
> >> >>>>> methods in MR and provide Cell versions instead...)
> >> >>>>>
> >> >>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
> >> needs
> >> >>>>> fixing
> >> >>>>>
> >> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
> >> >>>>>
> >> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
> >> >>>>> * HBASE-14997
> >> >>>>> Move compareOp and Comparators out of filter to client package
> >> >>>>>
> >> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
> >> >>>>>
> >> >>>>> What about:
> >> >>>>>
> >> >>>>> * HBASE-18601 Remove Htrace 3.2
> >> >>>>> As has been noted, the HTrace API is our 'trace' API.
> >> >>>>>
> >> >>>>> If interested in any of the above and you need a legup, just ask
> in
> >> the
> >> >>>>> issue and I'll be by....
> >> >>>>>
> >> >>>>> Thanks,
> >> >>>>> St.Ack
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
> >> >>>>>
> >> >>>>> Heads-up:
> >> >>>>>>
> >> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
> Theme
> >> is
> >> >>>>>> updated dependencies, reliance on relocated popular libs (guava,
> >> >>>>>> netty,
> >> >>>>>> protobuf), purge of checked-in generated src, and
> >> >>>>>> master-carries-no-regions
> >> >>>>>> by default.
> >> >>>>>>
> >> >>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
> >> will
> >> >>>>>> be
> >> >>>>>> settling the APIs and compatibility (At first blush, we are not
> >> >>>>>> looking too
> >> >>>>>> bad; our Sean ran some tests over weekend that have hbase-1
> client
> >> >>>>>> running
> >> >>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
> >> >>>>>> should be
> >> >>>>>> done by alpha3 (i.e. returning Interfaces rather than
> >> >>>>>> Implementations, and
> >> >>>>>> our shutdown of CPs accessing classes in hbase marked
> >> >>>>>> InterfaceAudience).
> >> >>>>>> We'll also have purged thirdparty classes from our API; e.g.
> guava
> >> >>>>>> 0.12
> >> >>>>>> Service showing through in our replication API and protobufs in
> >> Admin
> >> >>>>>> Interface. On alpha3, we will have to do a bunch of outreach to
> >> make
> >> >>>>>> sure
> >> >>>>>> our downstreamers are up on what is coming down the pipe.
> >> >>>>>>
> >> >>>>>> Beta1 in mid-September?
> >> >>>>>>
> >> >>>>>> I encourage you to check out the items marked for hbase2:
> >> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
> >> Edit
> >> >>>>>> as
> >> >>>>>> you see appropriate. Punt if you know the JIRA will not get any
> >> >>>>>> attention
> >> >>>>>> in next month or so.
> >> >>>>>>
> >> >>>>>> A bunch of issues marked blocker are unassigned. I'll leave them
> >> as is
> >> >>>>>> another while but I'll boot them soon.
> >> >>>>>>
> >> >>>>>> While I have your attention:
> >> >>>>>>
> >> >>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
> >> thrift
> >> >>>>>> to 0.10.0 will break existing clients. The change is easy enough
> if
> >> >>>>>> folks
> >> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> >> >>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> >> >>>>>> (0.98?).
> >> >>>>>>
> >> >>>>>> St.Ack
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>>
> >> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
> >> >>>>>>> wrote:
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> >> >>>>>>>>
> >> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<elserj@apache.org
> >
> >> >>>>>>>>> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>> ...
> >> >>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
> >> >>>>>>>>>> feature a
> >> >>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
> >> >>>>>>>>>> reason
> >> >>>>>>>>>> to be
> >> >>>>>>>>>> nervous though). I am pretty happy with the feature in terms
> of
> >> >>>>>>>>>> how
> >> >>>>>>>>>> much it
> >> >>>>>>>>>> is covered via testing.
> >> >>>>>>>>>>
> >> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> >> >>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>> Last think on space quota says that need doc too. See 'Space
> >> >>>>>>>>> Quota' in
> >> >>>>>>>>> here:
> >> >>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> >> >>>>>>>>> Does this little section need an update Josh?
> >> >>>>>>>>>
> >> >>>>>>>>> Thanks,
> >> >>>>>>>>> S
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
> >> docs
> >> >>>>>>>> already included for 17748.  Happy to put that on my plate if
> >> >>>>>>>> you're good
> >> >>>>>>>> with it. I can reasonably assume that no one is against it :)
> >> >>>>>>>>
> >> >>>>>>>> I think I had knocked out docs for the "phase 1" stuff before
> we
> >> >>>>>>>> merged it in from the original feature branch. I'll double
> check
> >> >>>>>>>> and update
> >> >>>>>>>> the gdoc. Perhaps this was just a timing thing.
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>> Thanks Josh,
> >> >>>>>>> S
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >
> >>
> >
> >
>

Re: Moving 2.0 forward

["张铎 (Duo Zhang)" <pa...@gmail.com>]

OK, skimmed,  we are in trouble! The in memory compaction just use the same
constructor with normal compaction to construct a StoreScanner, and use it
to do compaction...

We have to provide several preXXX and postXXX for it, at least we should
allow user reset TTL and max versions, and also do filtering on the scanner.

Thanks.



2017-10-26 9:41 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:

> When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
> HBASE-19033, I found that there maybe a hole in our CP hools. It is the in
> memory compaction.
>
> As Anoop suggested in HBASE-19001, we still need to give user the ability
> to extend the max versions and TTL config, so I plan to add back the hooks
> above to let CP users can change the max versions and TTL of a ScanInfo
> object. But I'm not sure whether in memory compaction will also discard
> expired cells, if so then we are in trouble...
>
> Thanks.
>
> 2017-10-25 6:03 GMT+08:00 Stack <st...@duboce.net>:
>
>> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
>> what remains to be done. Surfacing our thoughts here so you all clued
>> in....Or if you think otherwise, please speak up.
>>
>> We have ~13 issues to land:
>> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About two
>> are meta-issues that are about process which leaves 11.
>>
>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some changed
>> API/semantic that we need to get out earlier rather than later.
>>
>> Once the above is merged, HBASE-13346, change of Filter method names to
>> mention Cell instead of KeyValue can land.
>>
>> HBASE-199048 needs a review (Anoop will probably do it), removing
>> IA.Private objects as params to MasterObserver... Hopefully this goes in
>> soon.
>>
>> Duo is hard at work on trackers for flush and compaction for CPs
>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>>
>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo
>> is
>> done w/ his work above.
>>
>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all
>> the
>> purges allowing CPs do direct calls against Regions in same Host.
>>
>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>
>> Another day or two?
>>
>> St.Ack
>>
>>
>>
>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>
>> >
>> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>> >
>> >> +1
>> >>
>> >> I was trying to work on helping out on the outstanding alpha-4 stuff
>> last
>> >> week -- will be continuing to try to do the same this week.
>> >>
>> >> If you need any help, Stack, or if others need reviews where I haven't
>> >> noticed on my own: feel free to @mention me.
>> >>
>> >>
>> > Thanks for the offer Josh. All items seem assigned and are being
>> actively
>> > worked on. If you get a moment, reviews by you (or anyone else) helps
>> move
>> > the process along.
>> >
>> > We need to merge in HBASE-18410 branch to pick up Filter improvements.
>> > Then HBASE-13346 can go in.
>> >
>> > You are already helping out on HBASE-18906, thanks. Looks like that will
>> > be addressed by other alpha-4s about to land.
>> >
>> > St.Ack
>> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >> On 10/23/17 12:53 PM, Stack wrote:
>> >>
>> >>> (Reviving this thread)
>> >>>
>> >>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>> >>> refactor of the Coprocessor API shutting down access to internals
>> marked
>> >>> InterfaceAudience.Private.
>> >>>
>> >>> The outstanding list is here:
>> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>> >>>
>> >>> Please push in anything marked alpha-4 that belongs to you.
>> >>>
>> >>> If issue, talk out loud on this thread. If you need a review to land
>> an
>> >>> item, shout on the issue and here; we'll help you out.
>> >>>
>> >>> As is, items are coming along nicely I'd say. We need to merge the
>> filter
>> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>> >>>
>> >>> Post alpha-4, we'll have to hunt down our downstreamers and help them
>> >>> test
>> >>> on top of alpha-4 so rolling into beta-1, we have confidence our
>> >>> downstreamers know what to expect (or we discover what we missed
>> BEFORE
>> >>> we
>> >>> beta-1).
>> >>>
>> >>> Thanks for time,
>> >>> S
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>> >>>
>> >>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
>> >>>> time, if we all sprint, for the public-facing API fixes to be done.
>> >>>>
>> >>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
>> but
>> >>>> it is plain that more time is needed (in spite of valiant effort so
>> far
>> >>>> by
>> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>> >>>> theme is
>> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>> following
>> >>>> week.
>> >>>>
>> >>>> We should then be ready for beta (beta == no new features, no API
>> >>>> changes,
>> >>>> just fixes).
>> >>>>
>> >>>> Thanks,
>> >>>> St.Ack
>> >>>>
>> >>>>
>> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>> >>>>
>> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>> >>>>>
>> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get
>> a
>> >>>>> release out in the next week or so.
>> >>>>>
>> >>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>> >>>>> interested in helping out, below are the items I think we need done
>> for
>> >>>>> alpha3 (below are at least 'Critical' status, are API possibly
>> altering
>> >>>>> items, and are absent those JIRAs that are making active progress,
>> >>>>> i.e. the
>> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>> >>>>> doing is
>> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
>> >>>>>
>> >>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>> >>>>> branch-2
>> >>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do
>> it
>> >>>>> between branch-1 and branch-2.
>> >>>>>
>> >>>>> * HBASE-10462 Recategorize some of the client facing Public /
>> Private
>> >>>>> interfaces
>> >>>>> This one is almost done. It could do with a finish, attention to the
>> >>>>> items in last comment, and then our codebase could do with another
>> >>>>> sweep
>> >>>>> after the spirit of this issue since a bunch has gone in since the
>> pass
>> >>>>> that was the basis of this issue.
>> >>>>>
>> >>>>> * HBASE-10504 Define Replication Interface
>> >>>>> I was going to take a crack at this as part of the revamp forced by
>> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
>> >>>>> but if
>> >>>>> anyone else is interested, be my guest.
>> >>>>>
>> >>>>> * HBASE-14996 Some more API cleanup for 2.0
>> >>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>> >>>>> finishing.
>> >>>>>
>> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
>> and
>> >>>>> cleanup
>> >>>>> Needs a pass. Small issue I think. Could also look at new
>> AsyncClient
>> >>>>> and
>> >>>>> make sure symmetry.
>> >>>>>
>> >>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>> >>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
>> >>>>> done.
>> >>>>>
>> >>>>> Rolling upgrade will have implications for our API. Would be good to
>> >>>>> try
>> >>>>> it and figure what needs fixup (as said above, according to trial by
>> >>>>> Sean,
>> >>>>> we might not be too bad here):
>> >>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>> >>>>> cluster
>> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
>> >>>>> 1.x
>> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>> >>>>>
>> >>>>> * HBASE-17442 Move most of the replication related classes to
>> >>>>> hbase-server package
>> >>>>> The above would be good to do generally but it may make for ripples
>> in
>> >>>>> API so would be good to do now.
>> >>>>>
>> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>> >>>>> Balazs is working on this. The idea is that we avoid adding two new
>> >>>>> types
>> >>>>> to our API, two types that are nought but curtailed, read-only
>> views on
>> >>>>> internals. Input if you have time appreciated.
>> >>>>>
>> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
>> hbase2
>> >>>>> cluster; verify
>> >>>>> Esteban is looking at this one
>> >>>>>
>> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>> >>>>> * HBASE-17143 Scan improvement
>> >>>>>
>> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
>> >>>>> implementations
>> >>>>> marked audience Private and returns implementations rather than
>> >>>>> Interfaces.
>> >>>>> In a few locations, we allow returning an alternate implementation
>> >>>>> altogether which is probably something we don't want a CP doing. To
>> >>>>> that
>> >>>>> end, the following issues started by Duo and Anoop need to be taken
>> to
>> >>>>> the
>> >>>>> finish line; ideally they'd have an owner:
>> >>>>>
>> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
>> The
>> >>>>> umbrella issue.
>> >>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>> >>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>> >>>>> RegionServerObserver
>> >>>>>
>> >>>>>
>> >>>>> Nice-to-haves:
>> >>>>>
>> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
>> unused
>> >>>>> TimeRange constructors
>> >>>>>
>> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>> >>>>> existing in the code
>> >>>>> This is the end of an old long-running project moving up on to Cell
>> >>>>> Interface. We think it is done but for a few little items
>> (deprecate KV
>> >>>>> methods in MR and provide Cell versions instead...)
>> >>>>>
>> >>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
>> needs
>> >>>>> fixing
>> >>>>>
>> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
>> >>>>>
>> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
>> >>>>> * HBASE-14997
>> >>>>> Move compareOp and Comparators out of filter to client package
>> >>>>>
>> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
>> >>>>>
>> >>>>> What about:
>> >>>>>
>> >>>>> * HBASE-18601 Remove Htrace 3.2
>> >>>>> As has been noted, the HTrace API is our 'trace' API.
>> >>>>>
>> >>>>> If interested in any of the above and you need a legup, just ask in
>> the
>> >>>>> issue and I'll be by....
>> >>>>>
>> >>>>> Thanks,
>> >>>>> St.Ack
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>> >>>>>
>> >>>>> Heads-up:
>> >>>>>>
>> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme
>> is
>> >>>>>> updated dependencies, reliance on relocated popular libs (guava,
>> >>>>>> netty,
>> >>>>>> protobuf), purge of checked-in generated src, and
>> >>>>>> master-carries-no-regions
>> >>>>>> by default.
>> >>>>>>
>> >>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
>> will
>> >>>>>> be
>> >>>>>> settling the APIs and compatibility (At first blush, we are not
>> >>>>>> looking too
>> >>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>> >>>>>> running
>> >>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>> >>>>>> should be
>> >>>>>> done by alpha3 (i.e. returning Interfaces rather than
>> >>>>>> Implementations, and
>> >>>>>> our shutdown of CPs accessing classes in hbase marked
>> >>>>>> InterfaceAudience).
>> >>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
>> >>>>>> 0.12
>> >>>>>> Service showing through in our replication API and protobufs in
>> Admin
>> >>>>>> Interface. On alpha3, we will have to do a bunch of outreach to
>> make
>> >>>>>> sure
>> >>>>>> our downstreamers are up on what is coming down the pipe.
>> >>>>>>
>> >>>>>> Beta1 in mid-September?
>> >>>>>>
>> >>>>>> I encourage you to check out the items marked for hbase2:
>> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
>> Edit
>> >>>>>> as
>> >>>>>> you see appropriate. Punt if you know the JIRA will not get any
>> >>>>>> attention
>> >>>>>> in next month or so.
>> >>>>>>
>> >>>>>> A bunch of issues marked blocker are unassigned. I'll leave them
>> as is
>> >>>>>> another while but I'll boot them soon.
>> >>>>>>
>> >>>>>> While I have your attention:
>> >>>>>>
>> >>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
>> thrift
>> >>>>>> to 0.10.0 will break existing clients. The change is easy enough if
>> >>>>>> folks
>> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>> >>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>> >>>>>> (0.98?).
>> >>>>>>
>> >>>>>> St.Ack
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>> >>>>>>
>> >>>>>>
>> >>>>>>>
>> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>> >>>>>>>>
>> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>> >>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>> ...
>> >>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>> >>>>>>>>>> feature a
>> >>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>> >>>>>>>>>> reason
>> >>>>>>>>>> to be
>> >>>>>>>>>> nervous though). I am pretty happy with the feature in terms of
>> >>>>>>>>>> how
>> >>>>>>>>>> much it
>> >>>>>>>>>> is covered via testing.
>> >>>>>>>>>>
>> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> Last think on space quota says that need doc too. See 'Space
>> >>>>>>>>> Quota' in
>> >>>>>>>>> here:
>> >>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>> >>>>>>>>> Does this little section need an update Josh?
>> >>>>>>>>>
>> >>>>>>>>> Thanks,
>> >>>>>>>>> S
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
>> docs
>> >>>>>>>> already included for 17748.  Happy to put that on my plate if
>> >>>>>>>> you're good
>> >>>>>>>> with it. I can reasonably assume that no one is against it :)
>> >>>>>>>>
>> >>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>> >>>>>>>> merged it in from the original feature branch. I'll double check
>> >>>>>>>> and update
>> >>>>>>>> the gdoc. Perhaps this was just a timing thing.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>> Thanks Josh,
>> >>>>>>> S
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >
>>
>
>

Re: Moving 2.0 forward

["张铎 (Duo Zhang)" <pa...@gmail.com>]

When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
HBASE-19033, I found that there maybe a hole in our CP hools. It is the in
memory compaction.

As Anoop suggested in HBASE-19001, we still need to give user the ability
to extend the max versions and TTL config, so I plan to add back the hooks
above to let CP users can change the max versions and TTL of a ScanInfo
object. But I'm not sure whether in memory compaction will also discard
expired cells, if so then we are in trouble...

Thanks.

2017-10-25 6:03 GMT+08:00 Stack <st...@duboce.net>:

> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
> what remains to be done. Surfacing our thoughts here so you all clued
> in....Or if you think otherwise, please speak up.
>
> We have ~13 issues to land:
> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About two
> are meta-issues that are about process which leaves 11.
>
> Duo and Zheng Hu are to merge the FilterList fixes improvements
> (HBASE-19057, HBASE-18410 et al.). These are blocker because some changed
> API/semantic that we need to get out earlier rather than later.
>
> Once the above is merged, HBASE-13346, change of Filter method names to
> mention Cell instead of KeyValue can land.
>
> HBASE-199048 needs a review (Anoop will probably do it), removing
> IA.Private objects as params to MasterObserver... Hopefully this goes in
> soon.
>
> Duo is hard at work on trackers for flush and compaction for CPs
> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>
> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo is
> done w/ his work above.
>
> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all the
> purges allowing CPs do direct calls against Regions in same Host.
>
> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>
> Another day or two?
>
> St.Ack
>
>
>
> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>
> >
> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
> >
> >> +1
> >>
> >> I was trying to work on helping out on the outstanding alpha-4 stuff
> last
> >> week -- will be continuing to try to do the same this week.
> >>
> >> If you need any help, Stack, or if others need reviews where I haven't
> >> noticed on my own: feel free to @mention me.
> >>
> >>
> > Thanks for the offer Josh. All items seem assigned and are being actively
> > worked on. If you get a moment, reviews by you (or anyone else) helps
> move
> > the process along.
> >
> > We need to merge in HBASE-18410 branch to pick up Filter improvements.
> > Then HBASE-13346 can go in.
> >
> > You are already helping out on HBASE-18906, thanks. Looks like that will
> > be addressed by other alpha-4s about to land.
> >
> > St.Ack
> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >> On 10/23/17 12:53 PM, Stack wrote:
> >>
> >>> (Reviving this thread)
> >>>
> >>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
> >>> refactor of the Coprocessor API shutting down access to internals
> marked
> >>> InterfaceAudience.Private.
> >>>
> >>> The outstanding list is here:
> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >>>
> >>> Please push in anything marked alpha-4 that belongs to you.
> >>>
> >>> If issue, talk out loud on this thread. If you need a review to land an
> >>> item, shout on the issue and here; we'll help you out.
> >>>
> >>> As is, items are coming along nicely I'd say. We need to merge the
> filter
> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> >>>
> >>> Post alpha-4, we'll have to hunt down our downstreamers and help them
> >>> test
> >>> on top of alpha-4 so rolling into beta-1, we have confidence our
> >>> downstreamers know what to expect (or we discover what we missed BEFORE
> >>> we
> >>> beta-1).
> >>>
> >>> Thanks for time,
> >>> S
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> >>>
> >>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
> >>>> time, if we all sprint, for the public-facing API fixes to be done.
> >>>>
> >>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
> but
> >>>> it is plain that more time is needed (in spite of valiant effort so
> far
> >>>> by
> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
> >>>> theme is
> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> following
> >>>> week.
> >>>>
> >>>> We should then be ready for beta (beta == no new features, no API
> >>>> changes,
> >>>> just fixes).
> >>>>
> >>>> Thanks,
> >>>> St.Ack
> >>>>
> >>>>
> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
> >>>>
> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
> >>>>>
> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
> >>>>> release out in the next week or so.
> >>>>>
> >>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> >>>>> interested in helping out, below are the items I think we need done
> for
> >>>>> alpha3 (below are at least 'Critical' status, are API possibly
> altering
> >>>>> items, and are absent those JIRAs that are making active progress,
> >>>>> i.e. the
> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
> >>>>> doing is
> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
> >>>>>
> >>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
> >>>>> branch-2
> >>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
> >>>>> between branch-1 and branch-2.
> >>>>>
> >>>>> * HBASE-10462 Recategorize some of the client facing Public / Private
> >>>>> interfaces
> >>>>> This one is almost done. It could do with a finish, attention to the
> >>>>> items in last comment, and then our codebase could do with another
> >>>>> sweep
> >>>>> after the spirit of this issue since a bunch has gone in since the
> pass
> >>>>> that was the basis of this issue.
> >>>>>
> >>>>> * HBASE-10504 Define Replication Interface
> >>>>> I was going to take a crack at this as part of the revamp forced by
> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
> >>>>> but if
> >>>>> anyone else is interested, be my guest.
> >>>>>
> >>>>> * HBASE-14996 Some more API cleanup for 2.0
> >>>>> Has a bunch of subtasks, some of which are being worked on. Needs
> >>>>> finishing.
> >>>>>
> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
> >>>>> cleanup
> >>>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
> >>>>> and
> >>>>> make sure symmetry.
> >>>>>
> >>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> >>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
> >>>>> done.
> >>>>>
> >>>>> Rolling upgrade will have implications for our API. Would be good to
> >>>>> try
> >>>>> it and figure what needs fixup (as said above, according to trial by
> >>>>> Sean,
> >>>>> we might not be too bad here):
> >>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
> >>>>> cluster
> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
> >>>>> 1.x
> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> >>>>>
> >>>>> * HBASE-17442 Move most of the replication related classes to
> >>>>> hbase-server package
> >>>>> The above would be good to do generally but it may make for ripples
> in
> >>>>> API so would be good to do now.
> >>>>>
> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> >>>>> Balazs is working on this. The idea is that we avoid adding two new
> >>>>> types
> >>>>> to our API, two types that are nought but curtailed, read-only views
> on
> >>>>> internals. Input if you have time appreciated.
> >>>>>
> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
> hbase2
> >>>>> cluster; verify
> >>>>> Esteban is looking at this one
> >>>>>
> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> >>>>> * HBASE-17143 Scan improvement
> >>>>>
> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
> >>>>> implementations
> >>>>> marked audience Private and returns implementations rather than
> >>>>> Interfaces.
> >>>>> In a few locations, we allow returning an alternate implementation
> >>>>> altogether which is probably something we don't want a CP doing. To
> >>>>> that
> >>>>> end, the following issues started by Duo and Anoop need to be taken
> to
> >>>>> the
> >>>>> finish line; ideally they'd have an owner:
> >>>>>
> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
> >>>>> umbrella issue.
> >>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> >>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
> >>>>> RegionServerObserver
> >>>>>
> >>>>>
> >>>>> Nice-to-haves:
> >>>>>
> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
> unused
> >>>>> TimeRange constructors
> >>>>>
> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
> >>>>> existing in the code
> >>>>> This is the end of an old long-running project moving up on to Cell
> >>>>> Interface. We think it is done but for a few little items (deprecate
> KV
> >>>>> methods in MR and provide Cell versions instead...)
> >>>>>
> >>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
> >>>>> fixing
> >>>>>
> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
> >>>>>
> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
> >>>>> * HBASE-14997
> >>>>> Move compareOp and Comparators out of filter to client package
> >>>>>
> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
> >>>>>
> >>>>> What about:
> >>>>>
> >>>>> * HBASE-18601 Remove Htrace 3.2
> >>>>> As has been noted, the HTrace API is our 'trace' API.
> >>>>>
> >>>>> If interested in any of the above and you need a legup, just ask in
> the
> >>>>> issue and I'll be by....
> >>>>>
> >>>>> Thanks,
> >>>>> St.Ack
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
> >>>>>
> >>>>> Heads-up:
> >>>>>>
> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme
> is
> >>>>>> updated dependencies, reliance on relocated popular libs (guava,
> >>>>>> netty,
> >>>>>> protobuf), purge of checked-in generated src, and
> >>>>>> master-carries-no-regions
> >>>>>> by default.
> >>>>>>
> >>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
> will
> >>>>>> be
> >>>>>> settling the APIs and compatibility (At first blush, we are not
> >>>>>> looking too
> >>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
> >>>>>> running
> >>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
> >>>>>> should be
> >>>>>> done by alpha3 (i.e. returning Interfaces rather than
> >>>>>> Implementations, and
> >>>>>> our shutdown of CPs accessing classes in hbase marked
> >>>>>> InterfaceAudience).
> >>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
> >>>>>> 0.12
> >>>>>> Service showing through in our replication API and protobufs in
> Admin
> >>>>>> Interface. On alpha3, we will have to do a bunch of outreach to make
> >>>>>> sure
> >>>>>> our downstreamers are up on what is coming down the pipe.
> >>>>>>
> >>>>>> Beta1 in mid-September?
> >>>>>>
> >>>>>> I encourage you to check out the items marked for hbase2:
> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
> Edit
> >>>>>> as
> >>>>>> you see appropriate. Punt if you know the JIRA will not get any
> >>>>>> attention
> >>>>>> in next month or so.
> >>>>>>
> >>>>>> A bunch of issues marked blocker are unassigned. I'll leave them as
> is
> >>>>>> another while but I'll boot them soon.
> >>>>>>
> >>>>>> While I have your attention:
> >>>>>>
> >>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
> thrift
> >>>>>> to 0.10.0 will break existing clients. The change is easy enough if
> >>>>>> folks
> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> >>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> >>>>>> (0.98?).
> >>>>>>
> >>>>>> St.Ack
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> >>>>>>>>
> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> ...
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
> >>>>>>>>>> feature a
> >>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
> >>>>>>>>>> reason
> >>>>>>>>>> to be
> >>>>>>>>>> nervous though). I am pretty happy with the feature in terms of
> >>>>>>>>>> how
> >>>>>>>>>> much it
> >>>>>>>>>> is covered via testing.
> >>>>>>>>>>
> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Last think on space quota says that need doc too. See 'Space
> >>>>>>>>> Quota' in
> >>>>>>>>> here:
> >>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> >>>>>>>>> Does this little section need an update Josh?
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> S
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
> docs
> >>>>>>>> already included for 17748.  Happy to put that on my plate if
> >>>>>>>> you're good
> >>>>>>>> with it. I can reasonably assume that no one is against it :)
> >>>>>>>>
> >>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
> >>>>>>>> merged it in from the original feature branch. I'll double check
> >>>>>>>> and update
> >>>>>>>> the gdoc. Perhaps this was just a timing thing.
> >>>>>>>>
> >>>>>>>>
> >>>>>>> Thanks Josh,
> >>>>>>> S
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Chatting with my coworker Mr. Mike Drob, we were batting back and forth
what remains to be done. Surfacing our thoughts here so you all clued
in....Or if you think otherwise, please speak up.

We have ~13 issues to land:
https://issues.apache.org/jira/projects/HBASE/versions/12341594 About two
are meta-issues that are about process which leaves 11.

Duo and Zheng Hu are to merge the FilterList fixes improvements
(HBASE-19057, HBASE-18410 et al.). These are blocker because some changed
API/semantic that we need to get out earlier rather than later.

Once the above is merged, HBASE-13346, change of Filter method names to
mention Cell instead of KeyValue can land.

HBASE-199048 needs a review (Anoop will probably do it), removing
IA.Private objects as params to MasterObserver... Hopefully this goes in
soon.

Duo is hard at work on trackers for flush and compaction for CPs
(HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?

I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo is
done w/ his work above.

I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all the
purges allowing CPs do direct calls against Regions in same Host.

Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.

Another day or two?

St.Ack



On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:

>
> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>
>> +1
>>
>> I was trying to work on helping out on the outstanding alpha-4 stuff last
>> week -- will be continuing to try to do the same this week.
>>
>> If you need any help, Stack, or if others need reviews where I haven't
>> noticed on my own: feel free to @mention me.
>>
>>
> Thanks for the offer Josh. All items seem assigned and are being actively
> worked on. If you get a moment, reviews by you (or anyone else) helps move
> the process along.
>
> We need to merge in HBASE-18410 branch to pick up Filter improvements.
> Then HBASE-13346 can go in.
>
> You are already helping out on HBASE-18906, thanks. Looks like that will
> be addressed by other alpha-4s about to land.
>
> St.Ack
> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>
>
>
>
>
>
>
>
>
>> On 10/23/17 12:53 PM, Stack wrote:
>>
>>> (Reviving this thread)
>>>
>>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>>> refactor of the Coprocessor API shutting down access to internals marked
>>> InterfaceAudience.Private.
>>>
>>> The outstanding list is here:
>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>
>>> Please push in anything marked alpha-4 that belongs to you.
>>>
>>> If issue, talk out loud on this thread. If you need a review to land an
>>> item, shout on the issue and here; we'll help you out.
>>>
>>> As is, items are coming along nicely I'd say. We need to merge the filter
>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>
>>> Post alpha-4, we'll have to hunt down our downstreamers and help them
>>> test
>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>>> downstreamers know what to expect (or we discover what we missed BEFORE
>>> we
>>> beta-1).
>>>
>>> Thanks for time,
>>> S
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>>
>>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
>>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>>
>>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3 but
>>>> it is plain that more time is needed (in spite of valiant effort so far
>>>> by
>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>>>> theme is
>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the following
>>>> week.
>>>>
>>>> We should then be ready for beta (beta == no new features, no API
>>>> changes,
>>>> just fixes).
>>>>
>>>> Thanks,
>>>> St.Ack
>>>>
>>>>
>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>>
>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>>>>>
>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>>>>> release out in the next week or so.
>>>>>
>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>>> interested in helping out, below are the items I think we need done for
>>>>> alpha3 (below are at least 'Critical' status, are API possibly altering
>>>>> items, and are absent those JIRAs that are making active progress,
>>>>> i.e. the
>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>>>>> doing is
>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>
>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>>>> branch-2
>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
>>>>> between branch-1 and branch-2.
>>>>>
>>>>> * HBASE-10462 Recategorize some of the client facing Public / Private
>>>>> interfaces
>>>>> This one is almost done. It could do with a finish, attention to the
>>>>> items in last comment, and then our codebase could do with another
>>>>> sweep
>>>>> after the spirit of this issue since a bunch has gone in since the pass
>>>>> that was the basis of this issue.
>>>>>
>>>>> * HBASE-10504 Define Replication Interface
>>>>> I was going to take a crack at this as part of the revamp forced by
>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
>>>>> but if
>>>>> anyone else is interested, be my guest.
>>>>>
>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>>> finishing.
>>>>>
>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>>>>> cleanup
>>>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
>>>>> and
>>>>> make sure symmetry.
>>>>>
>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
>>>>> done.
>>>>>
>>>>> Rolling upgrade will have implications for our API. Would be good to
>>>>> try
>>>>> it and figure what needs fixup (as said above, according to trial by
>>>>> Sean,
>>>>> we might not be too bad here):
>>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>>>>> cluster
>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
>>>>> 1.x
>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>
>>>>> * HBASE-17442 Move most of the replication related classes to
>>>>> hbase-server package
>>>>> The above would be good to do generally but it may make for ripples in
>>>>> API so would be good to do now.
>>>>>
>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>> Balazs is working on this. The idea is that we avoid adding two new
>>>>> types
>>>>> to our API, two types that are nought but curtailed, read-only views on
>>>>> internals. Input if you have time appreciated.
>>>>>
>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
>>>>> cluster; verify
>>>>> Esteban is looking at this one
>>>>>
>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>>> * HBASE-17143 Scan improvement
>>>>>
>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>> implementations
>>>>> marked audience Private and returns implementations rather than
>>>>> Interfaces.
>>>>> In a few locations, we allow returning an alternate implementation
>>>>> altogether which is probably something we don't want a CP doing. To
>>>>> that
>>>>> end, the following issues started by Duo and Anoop need to be taken to
>>>>> the
>>>>> finish line; ideally they'd have an owner:
>>>>>
>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
>>>>> umbrella issue.
>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>>>>> RegionServerObserver
>>>>>
>>>>>
>>>>> Nice-to-haves:
>>>>>
>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
>>>>> TimeRange constructors
>>>>>
>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>>>> existing in the code
>>>>> This is the end of an old long-running project moving up on to Cell
>>>>> Interface. We think it is done but for a few little items (deprecate KV
>>>>> methods in MR and provide Cell versions instead...)
>>>>>
>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
>>>>> fixing
>>>>>
>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>
>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>> * HBASE-14997
>>>>> Move compareOp and Comparators out of filter to client package
>>>>>
>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>
>>>>> What about:
>>>>>
>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>
>>>>> If interested in any of the above and you need a legup, just ask in the
>>>>> issue and I'll be by....
>>>>>
>>>>> Thanks,
>>>>> St.Ack
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>>>
>>>>> Heads-up:
>>>>>>
>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
>>>>>> updated dependencies, reliance on relocated popular libs (guava,
>>>>>> netty,
>>>>>> protobuf), purge of checked-in generated src, and
>>>>>> master-carries-no-regions
>>>>>> by default.
>>>>>>
>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme will
>>>>>> be
>>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>>> looking too
>>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>>>>>> running
>>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>>>>>> should be
>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>> Implementations, and
>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>> InterfaceAudience).
>>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
>>>>>> 0.12
>>>>>> Service showing through in our replication API and protobufs in Admin
>>>>>> Interface. On alpha3, we will have to do a bunch of outreach to make
>>>>>> sure
>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>>
>>>>>> Beta1 in mid-September?
>>>>>>
>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit
>>>>>> as
>>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>>> attention
>>>>>> in next month or so.
>>>>>>
>>>>>> A bunch of issues marked blocker are unassigned. I'll leave them as is
>>>>>> another while but I'll boot them soon.
>>>>>>
>>>>>> While I have your attention:
>>>>>>
>>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase thrift
>>>>>> to 0.10.0 will break existing clients. The change is easy enough if
>>>>>> folks
>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>>> (0.98?).
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>>
>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>>> feature a
>>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>>>>>>>>>> reason
>>>>>>>>>> to be
>>>>>>>>>> nervous though). I am pretty happy with the feature in terms of
>>>>>>>>>> how
>>>>>>>>>> much it
>>>>>>>>>> is covered via testing.
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Last think on space quota says that need doc too. See 'Space
>>>>>>>>> Quota' in
>>>>>>>>> here:
>>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>>> Does this little section need an update Josh?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> S
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>>>>>>>> already included for 17748.  Happy to put that on my plate if
>>>>>>>> you're good
>>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>>
>>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>>>> merged it in from the original feature branch. I'll double check
>>>>>>>> and update
>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>>
>>>>>>>>
>>>>>>> Thanks Josh,
>>>>>>> S
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>

Re: Moving 2.0 forward

[ramkrishna vasudevan <ra...@gmail.com>]

Thanks Stack. Started working on it. Will post a patch ASAP.

On Tue, Oct 24, 2017 at 11:38 AM, Stack <st...@duboce.net> wrote:

> That'd be a good one to get in Ram.
> S
>
> On Mon, Oct 23, 2017 at 9:42 PM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
> > Hi Stack
> >
> > Do you want HBASE-18995 before the alpha-4 (REmoving exposed internal
> APIs
> > from CellUtil)? Because you had mentioned no more API changes. If so I
> will
> > start making changes and put up a patch ASAP.
> >
> > Regards
> > Ram
> >
> > On Tue, Oct 24, 2017 at 3:22 AM, Stack <st...@duboce.net> wrote:
> >
> > > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
> wrote:
> > >
> > > > +1
> > > >
> > > > I was trying to work on helping out on the outstanding alpha-4 stuff
> > last
> > > > week -- will be continuing to try to do the same this week.
> > > >
> > > > If you need any help, Stack, or if others need reviews where I
> haven't
> > > > noticed on my own: feel free to @mention me.
> > > >
> > > >
> > > Thanks for the offer Josh. All items seem assigned and are being
> actively
> > > worked on. If you get a moment, reviews by you (or anyone else) helps
> > move
> > > the process along.
> > >
> > > We need to merge in HBASE-18410 branch to pick up Filter improvements.
> > Then
> > > HBASE-13346 can go in.
> > >
> > > You are already helping out on HBASE-18906, thanks. Looks like that
> will
> > be
> > > addressed by other alpha-4s about to land.
> > >
> > > St.Ack
> > > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > > On 10/23/17 12:53 PM, Stack wrote:
> > > >
> > > >> (Reviving this thread)
> > > >>
> > > >> Lets push out alpha-4 this week. Alpha-4 is the release that has the
> > > >> refactor of the Coprocessor API shutting down access to internals
> > marked
> > > >> InterfaceAudience.Private.
> > > >>
> > > >> The outstanding list is here:
> > > >> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > > >>
> > > >> Please push in anything marked alpha-4 that belongs to you.
> > > >>
> > > >> If issue, talk out loud on this thread. If you need a review to land
> > an
> > > >> item, shout on the issue and here; we'll help you out.
> > > >>
> > > >> As is, items are coming along nicely I'd say. We need to merge the
> > > filter
> > > >> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> > > >>
> > > >> Post alpha-4, we'll have to hunt down our downstreamers and help
> them
> > > test
> > > >> on top of alpha-4 so rolling into beta-1, we have confidence our
> > > >> downstreamers know what to expect (or we discover what we missed
> > BEFORE
> > > we
> > > >> beta-1).
> > > >>
> > > >> Thanks for time,
> > > >> S
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> > > >>
> > > >> I'll put up an alpha3 RC Monday, probably Monday night. That should
> be
> > > >>> time, if we all sprint, for the public-facing API fixes to be done.
> > > >>>
> > > >>> I had a bunch of Coprocessor refactor and fixup scheduled for
> alpha3
> > > but
> > > >>> it is plain that more time is needed (in spite of valiant effort so
> > far
> > > >>> by
> > > >>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
> > > theme
> > > >>> is
> > > >>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> > > following
> > > >>> week.
> > > >>>
> > > >>> We should then be ready for beta (beta == no new features, no API
> > > >>> changes,
> > > >>> just fixes).
> > > >>>
> > > >>> Thanks,
> > > >>> St.Ack
> > > >>>
> > > >>>
> > > >>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
> > > >>>
> > > >>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on
> it.
> > > >>>>
> > > >>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
> get
> > a
> > > >>>> release out in the next week or so.
> > > >>>>
> > > >>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> > > >>>> interested in helping out, below are the items I think we need
> done
> > > for
> > > >>>> alpha3 (below are at least 'Critical' status, are API possibly
> > > altering
> > > >>>> items, and are absent those JIRAs that are making active progress,
> > > i.e.
> > > >>>> the
> > > >>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
> > > >>>> doing is
> > > >>>> what Andrew did comparing 1.3. and 1.4 APIs
> > > >>>>
> > > >>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
> > > >>>> branch-2
> > > >>>> This is to do what Andrew did between 1.3 and 1.4 branches only do
> > it
> > > >>>> between branch-1 and branch-2.
> > > >>>>
> > > >>>> * HBASE-10462 Recategorize some of the client facing Public /
> > Private
> > > >>>> interfaces
> > > >>>> This one is almost done. It could do with a finish, attention to
> the
> > > >>>> items in last comment, and then our codebase could do with another
> > > sweep
> > > >>>> after the spirit of this issue since a bunch has gone in since the
> > > pass
> > > >>>> that was the basis of this issue.
> > > >>>>
> > > >>>> * HBASE-10504 Define Replication Interface
> > > >>>> I was going to take a crack at this as part of the revamp forced
> by
> > > >>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> Service'
> > > but
> > > >>>> if
> > > >>>> anyone else is interested, be my guest.
> > > >>>>
> > > >>>> * HBASE-14996 Some more API cleanup for 2.0
> > > >>>> Has a bunch of subtasks, some of which are being worked on. Needs
> > > >>>> finishing.
> > > >>>>
> > > >>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
> > and
> > > >>>> cleanup
> > > >>>> Needs a pass. Small issue I think. Could also look at new
> > AsyncClient
> > > >>>> and
> > > >>>> make sure symmetry.
> > > >>>>
> > > >>>> * HBASE-15607 Remove PB references from Admin for 2.0
> > > >>>> Predicated on result of an ongoing DISCUSSION thread but needs to
> be
> > > >>>> done.
> > > >>>>
> > > >>>> Rolling upgrade will have implications for our API. Would be good
> to
> > > try
> > > >>>> it and figure what needs fixup (as said above, according to trial
> by
> > > >>>> Sean,
> > > >>>> we might not be too bad here):
> > > >>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
> > > >>>> cluster
> > > >>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master
> and
> > > 1.x
> > > >>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> > > >>>>
> > > >>>> * HBASE-17442 Move most of the replication related classes to
> > > >>>> hbase-server package
> > > >>>> The above would be good to do generally but it may make for
> ripples
> > in
> > > >>>> API so would be good to do now.
> > > >>>>
> > > >>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> > > >>>> Balazs is working on this. The idea is that we avoid adding two
> new
> > > >>>> types
> > > >>>> to our API, two types that are nought but curtailed, read-only
> views
> > > on
> > > >>>> internals. Input if you have time appreciated.
> > > >>>>
> > > >>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
> > hbase2
> > > >>>> cluster; verify
> > > >>>> Esteban is looking at this one
> > > >>>>
> > > >>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> > > >>>> * HBASE-17143 Scan improvement
> > > >>>>
> > > >>>> Our Coprocessor Interface needs a tough edit. It exposes
> > > implementations
> > > >>>> marked audience Private and returns implementations rather than
> > > >>>> Interfaces.
> > > >>>> In a few locations, we allow returning an alternate implementation
> > > >>>> altogether which is probably something we don't want a CP doing.
> To
> > > that
> > > >>>> end, the following issues started by Duo and Anoop need to be
> taken
> > to
> > > >>>> the
> > > >>>> finish line; ideally they'd have an owner:
> > > >>>>
> > > >>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
> > The
> > > >>>> umbrella issue.
> > > >>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> > > >>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver
> and
> > > >>>> RegionServerObserver
> > > >>>>
> > > >>>>
> > > >>>> Nice-to-haves:
> > > >>>>
> > > >>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
> > unused
> > > >>>> TimeRange constructors
> > > >>>>
> > > >>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
> > > >>>> existing in the code
> > > >>>> This is the end of an old long-running project moving up on to
> Cell
> > > >>>> Interface. We think it is done but for a few little items
> (deprecate
> > > KV
> > > >>>> methods in MR and provide Cell versions instead...)
> > > >>>>
> > > >>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
> > needs
> > > >>>> fixing
> > > >>>>
> > > >>>> * HBASE-13346 Clean up Filter package for post 1.0
> > > >>>>
> > > >>>> * HBASE-14255 Simplify Cell creation post 1.0
> > > >>>> * HBASE-14997
> > > >>>> Move compareOp and Comparators out of filter to client package
> > > >>>>
> > > >>>> * HBASE-13740 Stop using Hadoop private interfaces
> > > >>>>
> > > >>>> What about:
> > > >>>>
> > > >>>> * HBASE-18601 Remove Htrace 3.2
> > > >>>> As has been noted, the HTrace API is our 'trace' API.
> > > >>>>
> > > >>>> If interested in any of the above and you need a legup, just ask
> in
> > > the
> > > >>>> issue and I'll be by....
> > > >>>>
> > > >>>> Thanks,
> > > >>>> St.Ack
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
> > > >>>>
> > > >>>> Heads-up:
> > > >>>>>
> > > >>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
> Theme
> > is
> > > >>>>> updated dependencies, reliance on relocated popular libs (guava,
> > > netty,
> > > >>>>> protobuf), purge of checked-in generated src, and
> > > >>>>> master-carries-no-regions
> > > >>>>> by default.
> > > >>>>>
> > > >>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
> > will
> > > >>>>> be
> > > >>>>> settling the APIs and compatibility (At first blush, we are not
> > > >>>>> looking too
> > > >>>>> bad; our Sean ran some tests over weekend that have hbase-1
> client
> > > >>>>> running
> > > >>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
> > > >>>>> should be
> > > >>>>> done by alpha3 (i.e. returning Interfaces rather than
> > > Implementations,
> > > >>>>> and
> > > >>>>> our shutdown of CPs accessing classes in hbase marked
> > > >>>>> InterfaceAudience).
> > > >>>>> We'll also have purged thirdparty classes from our API; e.g.
> guava
> > > 0.12
> > > >>>>> Service showing through in our replication API and protobufs in
> > Admin
> > > >>>>> Interface. On alpha3, we will have to do a bunch of outreach to
> > make
> > > >>>>> sure
> > > >>>>> our downstreamers are up on what is coming down the pipe.
> > > >>>>>
> > > >>>>> Beta1 in mid-September?
> > > >>>>>
> > > >>>>> I encourage you to check out the items marked for hbase2:
> > > >>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
> > Edit
> > > >>>>> as
> > > >>>>> you see appropriate. Punt if you know the JIRA will not get any
> > > >>>>> attention
> > > >>>>> in next month or so.
> > > >>>>>
> > > >>>>> A bunch of issues marked blocker are unassigned. I'll leave them
> as
> > > is
> > > >>>>> another while but I'll boot them soon.
> > > >>>>>
> > > >>>>> While I have your attention:
> > > >>>>>
> > > >>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
> > > thrift
> > > >>>>> to 0.10.0 will break existing clients. The change is easy enough
> if
> > > >>>>> folks
> > > >>>>> need to upgrade their hbase thrift. See HBASE-18591.
> > > >>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> > > >>>>> (0.98?).
> > > >>>>>
> > > >>>>> St.Ack
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> > > >>>>>
> > > >>>>>
> > > >>>>>>
> > > >>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
> > > wrote:
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> > > >>>>>>>
> > > >>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<elserj@apache.org
> >
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> ...
> > > >>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
> > > >>>>>>>>> feature a
> > > >>>>>>>>> bit more "real", but am obviously a little nervous (I have no
> > > >>>>>>>>> reason
> > > >>>>>>>>> to be
> > > >>>>>>>>> nervous though). I am pretty happy with the feature in terms
> of
> > > how
> > > >>>>>>>>> much it
> > > >>>>>>>>> is covered via testing.
> > > >>>>>>>>>
> > > >>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Last think on space quota says that need doc too. See 'Space
> > > Quota'
> > > >>>>>>>> in
> > > >>>>>>>> here:
> > > >>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> > > >>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > > >>>>>>>> Does this little section need an update Josh?
> > > >>>>>>>>
> > > >>>>>>>> Thanks,
> > > >>>>>>>> S
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
> > > docs
> > > >>>>>>> already included for 17748.  Happy to put that on my plate if
> > > you're
> > > >>>>>>> good
> > > >>>>>>> with it. I can reasonably assume that no one is against it :)
> > > >>>>>>>
> > > >>>>>>> I think I had knocked out docs for the "phase 1" stuff before
> we
> > > >>>>>>> merged it in from the original feature branch. I'll double
> check
> > > and
> > > >>>>>>> update
> > > >>>>>>> the gdoc. Perhaps this was just a timing thing.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>> Thanks Josh,
> > > >>>>>> S
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

That'd be a good one to get in Ram.
S

On Mon, Oct 23, 2017 at 9:42 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:

> Hi Stack
>
> Do you want HBASE-18995 before the alpha-4 (REmoving exposed internal APIs
> from CellUtil)? Because you had mentioned no more API changes. If so I will
> start making changes and put up a patch ASAP.
>
> Regards
> Ram
>
> On Tue, Oct 24, 2017 at 3:22 AM, Stack <st...@duboce.net> wrote:
>
> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
> >
> > > +1
> > >
> > > I was trying to work on helping out on the outstanding alpha-4 stuff
> last
> > > week -- will be continuing to try to do the same this week.
> > >
> > > If you need any help, Stack, or if others need reviews where I haven't
> > > noticed on my own: feel free to @mention me.
> > >
> > >
> > Thanks for the offer Josh. All items seem assigned and are being actively
> > worked on. If you get a moment, reviews by you (or anyone else) helps
> move
> > the process along.
> >
> > We need to merge in HBASE-18410 branch to pick up Filter improvements.
> Then
> > HBASE-13346 can go in.
> >
> > You are already helping out on HBASE-18906, thanks. Looks like that will
> be
> > addressed by other alpha-4s about to land.
> >
> > St.Ack
> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > > On 10/23/17 12:53 PM, Stack wrote:
> > >
> > >> (Reviving this thread)
> > >>
> > >> Lets push out alpha-4 this week. Alpha-4 is the release that has the
> > >> refactor of the Coprocessor API shutting down access to internals
> marked
> > >> InterfaceAudience.Private.
> > >>
> > >> The outstanding list is here:
> > >> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > >>
> > >> Please push in anything marked alpha-4 that belongs to you.
> > >>
> > >> If issue, talk out loud on this thread. If you need a review to land
> an
> > >> item, shout on the issue and here; we'll help you out.
> > >>
> > >> As is, items are coming along nicely I'd say. We need to merge the
> > filter
> > >> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> > >>
> > >> Post alpha-4, we'll have to hunt down our downstreamers and help them
> > test
> > >> on top of alpha-4 so rolling into beta-1, we have confidence our
> > >> downstreamers know what to expect (or we discover what we missed
> BEFORE
> > we
> > >> beta-1).
> > >>
> > >> Thanks for time,
> > >> S
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> > >>
> > >> I'll put up an alpha3 RC Monday, probably Monday night. That should be
> > >>> time, if we all sprint, for the public-facing API fixes to be done.
> > >>>
> > >>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
> > but
> > >>> it is plain that more time is needed (in spite of valiant effort so
> far
> > >>> by
> > >>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
> > theme
> > >>> is
> > >>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> > following
> > >>> week.
> > >>>
> > >>> We should then be ready for beta (beta == no new features, no API
> > >>> changes,
> > >>> just fixes).
> > >>>
> > >>> Thanks,
> > >>> St.Ack
> > >>>
> > >>>
> > >>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
> > >>>
> > >>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
> > >>>>
> > >>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get
> a
> > >>>> release out in the next week or so.
> > >>>>
> > >>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> > >>>> interested in helping out, below are the items I think we need done
> > for
> > >>>> alpha3 (below are at least 'Critical' status, are API possibly
> > altering
> > >>>> items, and are absent those JIRAs that are making active progress,
> > i.e.
> > >>>> the
> > >>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
> > >>>> doing is
> > >>>> what Andrew did comparing 1.3. and 1.4 APIs
> > >>>>
> > >>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
> > >>>> branch-2
> > >>>> This is to do what Andrew did between 1.3 and 1.4 branches only do
> it
> > >>>> between branch-1 and branch-2.
> > >>>>
> > >>>> * HBASE-10462 Recategorize some of the client facing Public /
> Private
> > >>>> interfaces
> > >>>> This one is almost done. It could do with a finish, attention to the
> > >>>> items in last comment, and then our codebase could do with another
> > sweep
> > >>>> after the spirit of this issue since a bunch has gone in since the
> > pass
> > >>>> that was the basis of this issue.
> > >>>>
> > >>>> * HBASE-10504 Define Replication Interface
> > >>>> I was going to take a crack at this as part of the revamp forced by
> > >>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
> > but
> > >>>> if
> > >>>> anyone else is interested, be my guest.
> > >>>>
> > >>>> * HBASE-14996 Some more API cleanup for 2.0
> > >>>> Has a bunch of subtasks, some of which are being worked on. Needs
> > >>>> finishing.
> > >>>>
> > >>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
> and
> > >>>> cleanup
> > >>>> Needs a pass. Small issue I think. Could also look at new
> AsyncClient
> > >>>> and
> > >>>> make sure symmetry.
> > >>>>
> > >>>> * HBASE-15607 Remove PB references from Admin for 2.0
> > >>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
> > >>>> done.
> > >>>>
> > >>>> Rolling upgrade will have implications for our API. Would be good to
> > try
> > >>>> it and figure what needs fixup (as said above, according to trial by
> > >>>> Sean,
> > >>>> we might not be too bad here):
> > >>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
> > >>>> cluster
> > >>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
> > 1.x
> > >>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> > >>>>
> > >>>> * HBASE-17442 Move most of the replication related classes to
> > >>>> hbase-server package
> > >>>> The above would be good to do generally but it may make for ripples
> in
> > >>>> API so would be good to do now.
> > >>>>
> > >>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> > >>>> Balazs is working on this. The idea is that we avoid adding two new
> > >>>> types
> > >>>> to our API, two types that are nought but curtailed, read-only views
> > on
> > >>>> internals. Input if you have time appreciated.
> > >>>>
> > >>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
> hbase2
> > >>>> cluster; verify
> > >>>> Esteban is looking at this one
> > >>>>
> > >>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> > >>>> * HBASE-17143 Scan improvement
> > >>>>
> > >>>> Our Coprocessor Interface needs a tough edit. It exposes
> > implementations
> > >>>> marked audience Private and returns implementations rather than
> > >>>> Interfaces.
> > >>>> In a few locations, we allow returning an alternate implementation
> > >>>> altogether which is probably something we don't want a CP doing. To
> > that
> > >>>> end, the following issues started by Duo and Anoop need to be taken
> to
> > >>>> the
> > >>>> finish line; ideally they'd have an owner:
> > >>>>
> > >>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
> The
> > >>>> umbrella issue.
> > >>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> > >>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
> > >>>> RegionServerObserver
> > >>>>
> > >>>>
> > >>>> Nice-to-haves:
> > >>>>
> > >>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
> unused
> > >>>> TimeRange constructors
> > >>>>
> > >>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
> > >>>> existing in the code
> > >>>> This is the end of an old long-running project moving up on to Cell
> > >>>> Interface. We think it is done but for a few little items (deprecate
> > KV
> > >>>> methods in MR and provide Cell versions instead...)
> > >>>>
> > >>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
> needs
> > >>>> fixing
> > >>>>
> > >>>> * HBASE-13346 Clean up Filter package for post 1.0
> > >>>>
> > >>>> * HBASE-14255 Simplify Cell creation post 1.0
> > >>>> * HBASE-14997
> > >>>> Move compareOp and Comparators out of filter to client package
> > >>>>
> > >>>> * HBASE-13740 Stop using Hadoop private interfaces
> > >>>>
> > >>>> What about:
> > >>>>
> > >>>> * HBASE-18601 Remove Htrace 3.2
> > >>>> As has been noted, the HTrace API is our 'trace' API.
> > >>>>
> > >>>> If interested in any of the above and you need a legup, just ask in
> > the
> > >>>> issue and I'll be by....
> > >>>>
> > >>>> Thanks,
> > >>>> St.Ack
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
> > >>>>
> > >>>> Heads-up:
> > >>>>>
> > >>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme
> is
> > >>>>> updated dependencies, reliance on relocated popular libs (guava,
> > netty,
> > >>>>> protobuf), purge of checked-in generated src, and
> > >>>>> master-carries-no-regions
> > >>>>> by default.
> > >>>>>
> > >>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
> will
> > >>>>> be
> > >>>>> settling the APIs and compatibility (At first blush, we are not
> > >>>>> looking too
> > >>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
> > >>>>> running
> > >>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
> > >>>>> should be
> > >>>>> done by alpha3 (i.e. returning Interfaces rather than
> > Implementations,
> > >>>>> and
> > >>>>> our shutdown of CPs accessing classes in hbase marked
> > >>>>> InterfaceAudience).
> > >>>>> We'll also have purged thirdparty classes from our API; e.g. guava
> > 0.12
> > >>>>> Service showing through in our replication API and protobufs in
> Admin
> > >>>>> Interface. On alpha3, we will have to do a bunch of outreach to
> make
> > >>>>> sure
> > >>>>> our downstreamers are up on what is coming down the pipe.
> > >>>>>
> > >>>>> Beta1 in mid-September?
> > >>>>>
> > >>>>> I encourage you to check out the items marked for hbase2:
> > >>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
> Edit
> > >>>>> as
> > >>>>> you see appropriate. Punt if you know the JIRA will not get any
> > >>>>> attention
> > >>>>> in next month or so.
> > >>>>>
> > >>>>> A bunch of issues marked blocker are unassigned. I'll leave them as
> > is
> > >>>>> another while but I'll boot them soon.
> > >>>>>
> > >>>>> While I have your attention:
> > >>>>>
> > >>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
> > thrift
> > >>>>> to 0.10.0 will break existing clients. The change is easy enough if
> > >>>>> folks
> > >>>>> need to upgrade their hbase thrift. See HBASE-18591.
> > >>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> > >>>>> (0.98?).
> > >>>>>
> > >>>>> St.Ack
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> > >>>>>
> > >>>>>
> > >>>>>>
> > >>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
> > wrote:
> > >>>>>>
> > >>>>>>
> > >>>>>>>
> > >>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> > >>>>>>>
> > >>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> ...
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
> > >>>>>>>>> feature a
> > >>>>>>>>> bit more "real", but am obviously a little nervous (I have no
> > >>>>>>>>> reason
> > >>>>>>>>> to be
> > >>>>>>>>> nervous though). I am pretty happy with the feature in terms of
> > how
> > >>>>>>>>> much it
> > >>>>>>>>> is covered via testing.
> > >>>>>>>>>
> > >>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Last think on space quota says that need doc too. See 'Space
> > Quota'
> > >>>>>>>> in
> > >>>>>>>> here:
> > >>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> > >>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > >>>>>>>> Does this little section need an update Josh?
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> S
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
> > docs
> > >>>>>>> already included for 17748.  Happy to put that on my plate if
> > you're
> > >>>>>>> good
> > >>>>>>> with it. I can reasonably assume that no one is against it :)
> > >>>>>>>
> > >>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
> > >>>>>>> merged it in from the original feature branch. I'll double check
> > and
> > >>>>>>> update
> > >>>>>>> the gdoc. Perhaps this was just a timing thing.
> > >>>>>>>
> > >>>>>>>
> > >>>>>> Thanks Josh,
> > >>>>>> S
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
>

Re: Moving 2.0 forward

[ramkrishna vasudevan <ra...@gmail.com>]

Hi Stack

Do you want HBASE-18995 before the alpha-4 (REmoving exposed internal APIs
from CellUtil)? Because you had mentioned no more API changes. If so I will
start making changes and put up a patch ASAP.

Regards
Ram

On Tue, Oct 24, 2017 at 3:22 AM, Stack <st...@duboce.net> wrote:

> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>
> > +1
> >
> > I was trying to work on helping out on the outstanding alpha-4 stuff last
> > week -- will be continuing to try to do the same this week.
> >
> > If you need any help, Stack, or if others need reviews where I haven't
> > noticed on my own: feel free to @mention me.
> >
> >
> Thanks for the offer Josh. All items seem assigned and are being actively
> worked on. If you get a moment, reviews by you (or anyone else) helps move
> the process along.
>
> We need to merge in HBASE-18410 branch to pick up Filter improvements. Then
> HBASE-13346 can go in.
>
> You are already helping out on HBASE-18906, thanks. Looks like that will be
> addressed by other alpha-4s about to land.
>
> St.Ack
> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>
>
>
>
>
>
>
>
>
> > On 10/23/17 12:53 PM, Stack wrote:
> >
> >> (Reviving this thread)
> >>
> >> Lets push out alpha-4 this week. Alpha-4 is the release that has the
> >> refactor of the Coprocessor API shutting down access to internals marked
> >> InterfaceAudience.Private.
> >>
> >> The outstanding list is here:
> >> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >>
> >> Please push in anything marked alpha-4 that belongs to you.
> >>
> >> If issue, talk out loud on this thread. If you need a review to land an
> >> item, shout on the issue and here; we'll help you out.
> >>
> >> As is, items are coming along nicely I'd say. We need to merge the
> filter
> >> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> >>
> >> Post alpha-4, we'll have to hunt down our downstreamers and help them
> test
> >> on top of alpha-4 so rolling into beta-1, we have confidence our
> >> downstreamers know what to expect (or we discover what we missed BEFORE
> we
> >> beta-1).
> >>
> >> Thanks for time,
> >> S
> >>
> >>
> >>
> >>
> >>
> >> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> >>
> >> I'll put up an alpha3 RC Monday, probably Monday night. That should be
> >>> time, if we all sprint, for the public-facing API fixes to be done.
> >>>
> >>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
> but
> >>> it is plain that more time is needed (in spite of valiant effort so far
> >>> by
> >>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
> theme
> >>> is
> >>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> following
> >>> week.
> >>>
> >>> We should then be ready for beta (beta == no new features, no API
> >>> changes,
> >>> just fixes).
> >>>
> >>> Thanks,
> >>> St.Ack
> >>>
> >>>
> >>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
> >>>
> >>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
> >>>>
> >>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
> >>>> release out in the next week or so.
> >>>>
> >>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> >>>> interested in helping out, below are the items I think we need done
> for
> >>>> alpha3 (below are at least 'Critical' status, are API possibly
> altering
> >>>> items, and are absent those JIRAs that are making active progress,
> i.e.
> >>>> the
> >>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
> >>>> doing is
> >>>> what Andrew did comparing 1.3. and 1.4 APIs
> >>>>
> >>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
> >>>> branch-2
> >>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
> >>>> between branch-1 and branch-2.
> >>>>
> >>>> * HBASE-10462 Recategorize some of the client facing Public / Private
> >>>> interfaces
> >>>> This one is almost done. It could do with a finish, attention to the
> >>>> items in last comment, and then our codebase could do with another
> sweep
> >>>> after the spirit of this issue since a bunch has gone in since the
> pass
> >>>> that was the basis of this issue.
> >>>>
> >>>> * HBASE-10504 Define Replication Interface
> >>>> I was going to take a crack at this as part of the revamp forced by
> >>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
> but
> >>>> if
> >>>> anyone else is interested, be my guest.
> >>>>
> >>>> * HBASE-14996 Some more API cleanup for 2.0
> >>>> Has a bunch of subtasks, some of which are being worked on. Needs
> >>>> finishing.
> >>>>
> >>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
> >>>> cleanup
> >>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
> >>>> and
> >>>> make sure symmetry.
> >>>>
> >>>> * HBASE-15607 Remove PB references from Admin for 2.0
> >>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
> >>>> done.
> >>>>
> >>>> Rolling upgrade will have implications for our API. Would be good to
> try
> >>>> it and figure what needs fixup (as said above, according to trial by
> >>>> Sean,
> >>>> we might not be too bad here):
> >>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
> >>>> cluster
> >>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
> 1.x
> >>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> >>>>
> >>>> * HBASE-17442 Move most of the replication related classes to
> >>>> hbase-server package
> >>>> The above would be good to do generally but it may make for ripples in
> >>>> API so would be good to do now.
> >>>>
> >>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> >>>> Balazs is working on this. The idea is that we avoid adding two new
> >>>> types
> >>>> to our API, two types that are nought but curtailed, read-only views
> on
> >>>> internals. Input if you have time appreciated.
> >>>>
> >>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
> >>>> cluster; verify
> >>>> Esteban is looking at this one
> >>>>
> >>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> >>>> * HBASE-17143 Scan improvement
> >>>>
> >>>> Our Coprocessor Interface needs a tough edit. It exposes
> implementations
> >>>> marked audience Private and returns implementations rather than
> >>>> Interfaces.
> >>>> In a few locations, we allow returning an alternate implementation
> >>>> altogether which is probably something we don't want a CP doing. To
> that
> >>>> end, the following issues started by Duo and Anoop need to be taken to
> >>>> the
> >>>> finish line; ideally they'd have an owner:
> >>>>
> >>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
> >>>> umbrella issue.
> >>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> >>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
> >>>> RegionServerObserver
> >>>>
> >>>>
> >>>> Nice-to-haves:
> >>>>
> >>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
> >>>> TimeRange constructors
> >>>>
> >>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
> >>>> existing in the code
> >>>> This is the end of an old long-running project moving up on to Cell
> >>>> Interface. We think it is done but for a few little items (deprecate
> KV
> >>>> methods in MR and provide Cell versions instead...)
> >>>>
> >>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
> >>>> fixing
> >>>>
> >>>> * HBASE-13346 Clean up Filter package for post 1.0
> >>>>
> >>>> * HBASE-14255 Simplify Cell creation post 1.0
> >>>> * HBASE-14997
> >>>> Move compareOp and Comparators out of filter to client package
> >>>>
> >>>> * HBASE-13740 Stop using Hadoop private interfaces
> >>>>
> >>>> What about:
> >>>>
> >>>> * HBASE-18601 Remove Htrace 3.2
> >>>> As has been noted, the HTrace API is our 'trace' API.
> >>>>
> >>>> If interested in any of the above and you need a legup, just ask in
> the
> >>>> issue and I'll be by....
> >>>>
> >>>> Thanks,
> >>>> St.Ack
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
> >>>>
> >>>> Heads-up:
> >>>>>
> >>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
> >>>>> updated dependencies, reliance on relocated popular libs (guava,
> netty,
> >>>>> protobuf), purge of checked-in generated src, and
> >>>>> master-carries-no-regions
> >>>>> by default.
> >>>>>
> >>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme will
> >>>>> be
> >>>>> settling the APIs and compatibility (At first blush, we are not
> >>>>> looking too
> >>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
> >>>>> running
> >>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
> >>>>> should be
> >>>>> done by alpha3 (i.e. returning Interfaces rather than
> Implementations,
> >>>>> and
> >>>>> our shutdown of CPs accessing classes in hbase marked
> >>>>> InterfaceAudience).
> >>>>> We'll also have purged thirdparty classes from our API; e.g. guava
> 0.12
> >>>>> Service showing through in our replication API and protobufs in Admin
> >>>>> Interface. On alpha3, we will have to do a bunch of outreach to make
> >>>>> sure
> >>>>> our downstreamers are up on what is coming down the pipe.
> >>>>>
> >>>>> Beta1 in mid-September?
> >>>>>
> >>>>> I encourage you to check out the items marked for hbase2:
> >>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit
> >>>>> as
> >>>>> you see appropriate. Punt if you know the JIRA will not get any
> >>>>> attention
> >>>>> in next month or so.
> >>>>>
> >>>>> A bunch of issues marked blocker are unassigned. I'll leave them as
> is
> >>>>> another while but I'll boot them soon.
> >>>>>
> >>>>> While I have your attention:
> >>>>>
> >>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
> thrift
> >>>>> to 0.10.0 will break existing clients. The change is easy enough if
> >>>>> folks
> >>>>> need to upgrade their hbase thrift. See HBASE-18591.
> >>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> >>>>> (0.98?).
> >>>>>
> >>>>> St.Ack
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
> wrote:
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> >>>>>>>
> >>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> ...
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
> >>>>>>>>> feature a
> >>>>>>>>> bit more "real", but am obviously a little nervous (I have no
> >>>>>>>>> reason
> >>>>>>>>> to be
> >>>>>>>>> nervous though). I am pretty happy with the feature in terms of
> how
> >>>>>>>>> much it
> >>>>>>>>> is covered via testing.
> >>>>>>>>>
> >>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> Last think on space quota says that need doc too. See 'Space
> Quota'
> >>>>>>>> in
> >>>>>>>> here:
> >>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> >>>>>>>> Does this little section need an update Josh?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> S
> >>>>>>>>
> >>>>>>>>
> >>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
> docs
> >>>>>>> already included for 17748.  Happy to put that on my plate if
> you're
> >>>>>>> good
> >>>>>>> with it. I can reasonably assume that no one is against it :)
> >>>>>>>
> >>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
> >>>>>>> merged it in from the original feature branch. I'll double check
> and
> >>>>>>> update
> >>>>>>> the gdoc. Perhaps this was just a timing thing.
> >>>>>>>
> >>>>>>>
> >>>>>> Thanks Josh,
> >>>>>> S
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:

> +1
>
> I was trying to work on helping out on the outstanding alpha-4 stuff last
> week -- will be continuing to try to do the same this week.
>
> If you need any help, Stack, or if others need reviews where I haven't
> noticed on my own: feel free to @mention me.
>
>
Thanks for the offer Josh. All items seem assigned and are being actively
worked on. If you get a moment, reviews by you (or anyone else) helps move
the process along.

We need to merge in HBASE-18410 branch to pick up Filter improvements. Then
HBASE-13346 can go in.

You are already helping out on HBASE-18906, thanks. Looks like that will be
addressed by other alpha-4s about to land.

St.Ack
TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594









> On 10/23/17 12:53 PM, Stack wrote:
>
>> (Reviving this thread)
>>
>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>> refactor of the Coprocessor API shutting down access to internals marked
>> InterfaceAudience.Private.
>>
>> The outstanding list is here:
>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>
>> Please push in anything marked alpha-4 that belongs to you.
>>
>> If issue, talk out loud on this thread. If you need a review to land an
>> item, shout on the issue and here; we'll help you out.
>>
>> As is, items are coming along nicely I'd say. We need to merge the filter
>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>
>> Post alpha-4, we'll have to hunt down our downstreamers and help them test
>> on top of alpha-4 so rolling into beta-1, we have confidence our
>> downstreamers know what to expect (or we discover what we missed BEFORE we
>> beta-1).
>>
>> Thanks for time,
>> S
>>
>>
>>
>>
>>
>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>
>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>
>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3 but
>>> it is plain that more time is needed (in spite of valiant effort so far
>>> by
>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose theme
>>> is
>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the following
>>> week.
>>>
>>> We should then be ready for beta (beta == no new features, no API
>>> changes,
>>> just fixes).
>>>
>>> Thanks,
>>> St.Ack
>>>
>>>
>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>
>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>>>>
>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>>>> release out in the next week or so.
>>>>
>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>> interested in helping out, below are the items I think we need done for
>>>> alpha3 (below are at least 'Critical' status, are API possibly altering
>>>> items, and are absent those JIRAs that are making active progress, i.e.
>>>> the
>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>>>> doing is
>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>
>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>>> branch-2
>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
>>>> between branch-1 and branch-2.
>>>>
>>>> * HBASE-10462 Recategorize some of the client facing Public / Private
>>>> interfaces
>>>> This one is almost done. It could do with a finish, attention to the
>>>> items in last comment, and then our codebase could do with another sweep
>>>> after the spirit of this issue since a bunch has gone in since the pass
>>>> that was the basis of this issue.
>>>>
>>>> * HBASE-10504 Define Replication Interface
>>>> I was going to take a crack at this as part of the revamp forced by
>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service' but
>>>> if
>>>> anyone else is interested, be my guest.
>>>>
>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>> finishing.
>>>>
>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>>>> cleanup
>>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
>>>> and
>>>> make sure symmetry.
>>>>
>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
>>>> done.
>>>>
>>>> Rolling upgrade will have implications for our API. Would be good to try
>>>> it and figure what needs fixup (as said above, according to trial by
>>>> Sean,
>>>> we might not be too bad here):
>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>>>> cluster
>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and 1.x
>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>
>>>> * HBASE-17442 Move most of the replication related classes to
>>>> hbase-server package
>>>> The above would be good to do generally but it may make for ripples in
>>>> API so would be good to do now.
>>>>
>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>> Balazs is working on this. The idea is that we avoid adding two new
>>>> types
>>>> to our API, two types that are nought but curtailed, read-only views on
>>>> internals. Input if you have time appreciated.
>>>>
>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
>>>> cluster; verify
>>>> Esteban is looking at this one
>>>>
>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>> * HBASE-17143 Scan improvement
>>>>
>>>> Our Coprocessor Interface needs a tough edit. It exposes implementations
>>>> marked audience Private and returns implementations rather than
>>>> Interfaces.
>>>> In a few locations, we allow returning an alternate implementation
>>>> altogether which is probably something we don't want a CP doing. To that
>>>> end, the following issues started by Duo and Anoop need to be taken to
>>>> the
>>>> finish line; ideally they'd have an owner:
>>>>
>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
>>>> umbrella issue.
>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>>>> RegionServerObserver
>>>>
>>>>
>>>> Nice-to-haves:
>>>>
>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
>>>> TimeRange constructors
>>>>
>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>>> existing in the code
>>>> This is the end of an old long-running project moving up on to Cell
>>>> Interface. We think it is done but for a few little items (deprecate KV
>>>> methods in MR and provide Cell versions instead...)
>>>>
>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
>>>> fixing
>>>>
>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>
>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>> * HBASE-14997
>>>> Move compareOp and Comparators out of filter to client package
>>>>
>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>
>>>> What about:
>>>>
>>>> * HBASE-18601 Remove Htrace 3.2
>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>
>>>> If interested in any of the above and you need a legup, just ask in the
>>>> issue and I'll be by....
>>>>
>>>> Thanks,
>>>> St.Ack
>>>>
>>>>
>>>>
>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>>
>>>> Heads-up:
>>>>>
>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
>>>>> updated dependencies, reliance on relocated popular libs (guava, netty,
>>>>> protobuf), purge of checked-in generated src, and
>>>>> master-carries-no-regions
>>>>> by default.
>>>>>
>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme will
>>>>> be
>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>> looking too
>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>>>>> running
>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>>>>> should be
>>>>> done by alpha3 (i.e. returning Interfaces rather than Implementations,
>>>>> and
>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>> InterfaceAudience).
>>>>> We'll also have purged thirdparty classes from our API; e.g. guava 0.12
>>>>> Service showing through in our replication API and protobufs in Admin
>>>>> Interface. On alpha3, we will have to do a bunch of outreach to make
>>>>> sure
>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>
>>>>> Beta1 in mid-September?
>>>>>
>>>>> I encourage you to check out the items marked for hbase2:
>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit
>>>>> as
>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>> attention
>>>>> in next month or so.
>>>>>
>>>>> A bunch of issues marked blocker are unassigned. I'll leave them as is
>>>>> another while but I'll boot them soon.
>>>>>
>>>>> While I have your attention:
>>>>>
>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase thrift
>>>>> to 0.10.0 will break existing clients. The change is easy enough if
>>>>> folks
>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>> (0.98?).
>>>>>
>>>>> St.Ack
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>
>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> ...
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>> feature a
>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>>>>>>>>> reason
>>>>>>>>> to be
>>>>>>>>> nervous though). I am pretty happy with the feature in terms of how
>>>>>>>>> much it
>>>>>>>>> is covered via testing.
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Last think on space quota says that need doc too. See 'Space Quota'
>>>>>>>> in
>>>>>>>> here:
>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>> Does this little section need an update Josh?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> S
>>>>>>>>
>>>>>>>>
>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>>>>>>> already included for 17748.  Happy to put that on my plate if you're
>>>>>>> good
>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>
>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>>> merged it in from the original feature branch. I'll double check and
>>>>>>> update
>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>
>>>>>>>
>>>>>> Thanks Josh,
>>>>>> S
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>

Re: Moving 2.0 forward

[Josh Elser <el...@apache.org>]

+1

I was trying to work on helping out on the outstanding alpha-4 stuff 
last week -- will be continuing to try to do the same this week.

If you need any help, Stack, or if others need reviews where I haven't 
noticed on my own: feel free to @mention me.

On 10/23/17 12:53 PM, Stack wrote:
> (Reviving this thread)
> 
> Lets push out alpha-4 this week. Alpha-4 is the release that has the
> refactor of the Coprocessor API shutting down access to internals marked
> InterfaceAudience.Private.
> 
> The outstanding list is here:
> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> 
> Please push in anything marked alpha-4 that belongs to you.
> 
> If issue, talk out loud on this thread. If you need a review to land an
> item, shout on the issue and here; we'll help you out.
> 
> As is, items are coming along nicely I'd say. We need to merge the filter
> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> 
> Post alpha-4, we'll have to hunt down our downstreamers and help them test
> on top of alpha-4 so rolling into beta-1, we have confidence our
> downstreamers know what to expect (or we discover what we missed BEFORE we
> beta-1).
> 
> Thanks for time,
> S
> 
> 
> 
> 
> 
> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> 
>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
>> time, if we all sprint, for the public-facing API fixes to be done.
>>
>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3 but
>> it is plain that more time is needed (in spite of valiant effort so far by
>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose theme is
>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the following
>> week.
>>
>> We should then be ready for beta (beta == no new features, no API changes,
>> just fixes).
>>
>> Thanks,
>> St.Ack
>>
>>
>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>
>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>>>
>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>>> release out in the next week or so.
>>>
>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>> interested in helping out, below are the items I think we need done for
>>> alpha3 (below are at least 'Critical' status, are API possibly altering
>>> items, and are absent those JIRAs that are making active progress, i.e. the
>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs doing is
>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>
>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>> branch-2
>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
>>> between branch-1 and branch-2.
>>>
>>> * HBASE-10462 Recategorize some of the client facing Public / Private
>>> interfaces
>>> This one is almost done. It could do with a finish, attention to the
>>> items in last comment, and then our codebase could do with another sweep
>>> after the spirit of this issue since a bunch has gone in since the pass
>>> that was the basis of this issue.
>>>
>>> * HBASE-10504 Define Replication Interface
>>> I was going to take a crack at this as part of the revamp forced by
>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service' but if
>>> anyone else is interested, be my guest.
>>>
>>> * HBASE-14996 Some more API cleanup for 2.0
>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>> finishing.
>>>
>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>>> cleanup
>>> Needs a pass. Small issue I think. Could also look at new AsyncClient and
>>> make sure symmetry.
>>>
>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>> Predicated on result of an ongoing DISCUSSION thread but needs to be done.
>>>
>>> Rolling upgrade will have implications for our API. Would be good to try
>>> it and figure what needs fixup (as said above, according to trial by Sean,
>>> we might not be too bad here):
>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0 cluster
>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and 1.x
>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>
>>> * HBASE-17442 Move most of the replication related classes to
>>> hbase-server package
>>> The above would be good to do generally but it may make for ripples in
>>> API so would be good to do now.
>>>
>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>> Balazs is working on this. The idea is that we avoid adding two new types
>>> to our API, two types that are nought but curtailed, read-only views on
>>> internals. Input if you have time appreciated.
>>>
>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
>>> cluster; verify
>>> Esteban is looking at this one
>>>
>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>> * HBASE-17143 Scan improvement
>>>
>>> Our Coprocessor Interface needs a tough edit. It exposes implementations
>>> marked audience Private and returns implementations rather than Interfaces.
>>> In a few locations, we allow returning an alternate implementation
>>> altogether which is probably something we don't want a CP doing. To that
>>> end, the following issues started by Duo and Anoop need to be taken to the
>>> finish line; ideally they'd have an owner:
>>>
>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
>>> umbrella issue.
>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>>> RegionServerObserver
>>>
>>>
>>> Nice-to-haves:
>>>
>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
>>> TimeRange constructors
>>>
>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>> existing in the code
>>> This is the end of an old long-running project moving up on to Cell
>>> Interface. We think it is done but for a few little items (deprecate KV
>>> methods in MR and provide Cell versions instead...)
>>>
>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
>>> fixing
>>>
>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>
>>> * HBASE-14255 Simplify Cell creation post 1.0
>>> * HBASE-14997
>>> Move compareOp and Comparators out of filter to client package
>>>
>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>
>>> What about:
>>>
>>> * HBASE-18601 Remove Htrace 3.2
>>> As has been noted, the HTrace API is our 'trace' API.
>>>
>>> If interested in any of the above and you need a legup, just ask in the
>>> issue and I'll be by....
>>>
>>> Thanks,
>>> St.Ack
>>>
>>>
>>>
>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>
>>>> Heads-up:
>>>>
>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
>>>> updated dependencies, reliance on relocated popular libs (guava, netty,
>>>> protobuf), purge of checked-in generated src, and master-carries-no-regions
>>>> by default.
>>>>
>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme will be
>>>> settling the APIs and compatibility (At first blush, we are not looking too
>>>> bad; our Sean ran some tests over weekend that have hbase-1 client running
>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp should be
>>>> done by alpha3 (i.e. returning Interfaces rather than Implementations, and
>>>> our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
>>>> We'll also have purged thirdparty classes from our API; e.g. guava 0.12
>>>> Service showing through in our replication API and protobufs in Admin
>>>> Interface. On alpha3, we will have to do a bunch of outreach to make sure
>>>> our downstreamers are up on what is coming down the pipe.
>>>>
>>>> Beta1 in mid-September?
>>>>
>>>> I encourage you to check out the items marked for hbase2:
>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as
>>>> you see appropriate. Punt if you know the JIRA will not get any attention
>>>> in next month or so.
>>>>
>>>> A bunch of issues marked blocker are unassigned. I'll leave them as is
>>>> another while but I'll boot them soon.
>>>>
>>>> While I have your attention:
>>>>
>>>> + I think we should leave thrift version at 0.9.3. Moving hbase thrift
>>>> to 0.10.0 will break existing clients. The change is easy enough if folks
>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).
>>>>
>>>> St.Ack
>>>>
>>>>
>>>>
>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>
>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> ...
>>>>>>>>
>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>> feature a
>>>>>>>> bit more "real", but am obviously a little nervous (I have no reason
>>>>>>>> to be
>>>>>>>> nervous though). I am pretty happy with the feature in terms of how
>>>>>>>> much it
>>>>>>>> is covered via testing.
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>
>>>>>>> Last think on space quota says that need doc too. See 'Space Quota' in
>>>>>>> here:
>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>> Does this little section need an update Josh?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> S
>>>>>>>
>>>>>>
>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>>>>>> already included for 17748.  Happy to put that on my plate if you're good
>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>
>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>> merged it in from the original feature branch. I'll double check and update
>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>
>>>>>
>>>>> Thanks Josh,
>>>>> S
>>>>>
>>>>
>>>>
>>>
>>
> 

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Thu, Aug 17, 2017 at 12:45 PM, Mike Drob <md...@apache.org> wrote:

> I went ahead and tagged those as tentatively 2.0.0-alpha3 in jira so that
> we can have them all in one place later. Folks can move additional issues
> in and out as they see appropriate.
>
> Thanks M,
S



> On Thu, Aug 17, 2017 at 2:35 PM, Stack <st...@duboce.net> wrote:
>
> > I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
> >
> > For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
> > release out in the next week or so.
> >
> > I did a weeding of 2.0.0 issues over the last day. If folks are
> interested
> > in helping out, below are the items I think we need done for alpha3
> (below
> > are at least 'Critical' status, are API possibly altering items, and are
> > absent those JIRAs that are making active progress, i.e. the HTD/HCD
> revamp
> > by Chia-Ping Tsai). A project NOT listed that needs doing is what Andrew
> > did comparing 1.3. and 1.4 APIs
> >
> > * HBASE-18622 Mitigate compatibility concerns between branch-1 and
> branch-2
> > This is to do what Andrew did between 1.3 and 1.4 branches only do it
> > between branch-1 and branch-2.
> >
> > * HBASE-10462 Recategorize some of the client facing Public / Private
> > interfaces
> > This one is almost done. It could do with a finish, attention to the
> items
> > in last comment, and then our codebase could do with another sweep after
> > the spirit of this issue since a bunch has gone in since the pass that
> was
> > the basis of this issue.
> >
> > * HBASE-10504 Define Replication Interface
> > I was going to take a crack at this as part of the revamp forced by
> > 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service' but
> if
> > anyone else is interested, be my guest.
> >
> > * HBASE-14996 Some more API cleanup for 2.0
> > Has a bunch of subtasks, some of which are being worked on. Needs
> > finishing.
> >
> > * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
> > cleanup
> > Needs a pass. Small issue I think. Could also look at new AsyncClient and
> > make sure symmetry.
> >
> > * HBASE-15607 Remove PB references from Admin for 2.0
> > Predicated on result of an ongoing DISCUSSION thread but needs to be
> done.
> >
> > Rolling upgrade will have implications for our API. Would be good to try
> it
> > and figure what needs fixup (as said above, according to trial by Sean,
> we
> > might not be too bad here):
> > * HBASE-16060 1.x clients cannot access table state talking to 2.0
> cluster
> > * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and 1.x
> > RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> >
> > * HBASE-17442 Move most of the replication related classes to
> hbase-server
> > package
> > The above would be good to do generally but it may make for ripples in
> API
> > so would be good to do now.
> >
> > * HBASE-18106 Redo ProcedureInfo and LockInfo
> > Balazs is working on this. The idea is that we avoid adding two new types
> > to our API, two types that are nought but curtailed, read-only views on
> > internals. Input if you have time appreciated.
> >
> > * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
> > cluster; verify
> > Esteban is looking at this one
> >
> > * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> > * HBASE-17143 Scan improvement
> >
> > Our Coprocessor Interface needs a tough edit. It exposes implementations
> > marked audience Private and returns implementations rather than
> Interfaces.
> > In a few locations, we allow returning an alternate implementation
> > altogether which is probably something we don't want a CP doing. To that
> > end, the following issues started by Duo and Anoop need to be taken to
> the
> > finish line; ideally they'd have an owner:
> >
> > * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
> > umbrella issue.
> > * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> > * HBASE-16769 Deprecate/remove PB references from MasterObserver and
> > RegionServerObserver
> >
> >
> > Nice-to-haves:
> >
> > * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
> > TimeRange constructors
> >
> > * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
> existing
> > in the code
> > This is the end of an old long-running project moving up on to Cell
> > Interface. We think it is done but for a few little items (deprecate KV
> > methods in MR and provide Cell versions instead...)
> >
> > * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
> > fixing
> >
> > * HBASE-13346 Clean up Filter package for post 1.0
> >
> > * HBASE-14255 Simplify Cell creation post 1.0
> > * HBASE-14997
> > Move compareOp and Comparators out of filter to client package
> >
> > * HBASE-13740 Stop using Hadoop private interfaces
> >
> > What about:
> >
> > * HBASE-18601 Remove Htrace 3.2
> > As has been noted, the HTrace API is our 'trace' API.
> >
> > If interested in any of the above and you need a legup, just ask in the
> > issue and I'll be by....
> >
> > Thanks,
> > St.Ack
> >
> >
> >
> > On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
> >
> > > Heads-up:
> > >
> > > I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
> > > updated dependencies, reliance on relocated popular libs (guava, netty,
> > > protobuf), purge of checked-in generated src, and
> > master-carries-no-regions
> > > by default.
> > >
> > > alpha3 I hope will follow soon after (end-of-August?). Its theme will
> be
> > > settling the APIs and compatibility (At first blush, we are not looking
> > too
> > > bad; our Sean ran some tests over weekend that have hbase-1 client
> > running
> > > against an hbase-2 cluster....). The Coprocessor Interface revamp
> should
> > be
> > > done by alpha3 (i.e. returning Interfaces rather than Implementations,
> > and
> > > our shutdown of CPs accessing classes in hbase marked
> InterfaceAudience).
> > > We'll also have purged thirdparty classes from our API; e.g. guava 0.12
> > > Service showing through in our replication API and protobufs in Admin
> > > Interface. On alpha3, we will have to do a bunch of outreach to make
> sure
> > > our downstreamers are up on what is coming down the pipe.
> > >
> > > Beta1 in mid-September?
> > >
> > > I encourage you to check out the items marked for hbase2:
> > > https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit
> as
> > > you see appropriate. Punt if you know the JIRA will not get any
> attention
> > > in next month or so.
> > >
> > > A bunch of issues marked blocker are unassigned. I'll leave them as is
> > > another while but I'll boot them soon.
> > >
> > > While I have your attention:
> > >
> > > + I think we should leave thrift version at 0.9.3. Moving hbase thrift
> to
> > > 0.10.0 will break existing clients. The change is easy enough if folks
> > need
> > > to upgrade their hbase thrift. See HBASE-18591.
> > > + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> (0.98?).
> > >
> > > St.Ack
> > >
> > >
> > >
> > > On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> > >
> > >>
> > >>
> > >> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
> > >>
> > >>>
> > >>>
> > >>> On 7/31/17 9:00 AM, Stack wrote:
> > >>>
> > >>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
> > wrote:
> > >>>>
> > >>>> ...
> > >>>>>
> > >>>>> I like the idea of this also hitting 2.0 as it would make the
> > feature a
> > >>>>> bit more "real", but am obviously a little nervous (I have no
> reason
> > >>>>> to be
> > >>>>> nervous though). I am pretty happy with the feature in terms of how
> > >>>>> much it
> > >>>>> is covered via testing.
> > >>>>>
> > >>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> > >>>>
> > >>>> Last think on space quota says that need doc too. See 'Space Quota'
> in
> > >>>> here:
> > >>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> > >>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > >>>> Does this little section need an update Josh?
> > >>>>
> > >>>> Thanks,
> > >>>> S
> > >>>>
> > >>>
> > >>> Yep, just a couple of cherry-picks. Good test coverage and some docs
> > >>> already included for 17748.  Happy to put that on my plate if you're
> > good
> > >>> with it. I can reasonably assume that no one is against it :)
> > >>>
> > >>> I think I had knocked out docs for the "phase 1" stuff before we
> merged
> > >>> it in from the original feature branch. I'll double check and update
> > the
> > >>> gdoc. Perhaps this was just a timing thing.
> > >>>
> > >>
> > >> Thanks Josh,
> > >> S
> > >>
> > >
> > >
> >
>

Re: Moving 2.0 forward

[Mike Drob <md...@apache.org>]

I went ahead and tagged those as tentatively 2.0.0-alpha3 in jira so that
we can have them all in one place later. Folks can move additional issues
in and out as they see appropriate.

On Thu, Aug 17, 2017 at 2:35 PM, Stack <st...@duboce.net> wrote:

> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>
> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
> release out in the next week or so.
>
> I did a weeding of 2.0.0 issues over the last day. If folks are interested
> in helping out, below are the items I think we need done for alpha3 (below
> are at least 'Critical' status, are API possibly altering items, and are
> absent those JIRAs that are making active progress, i.e. the HTD/HCD revamp
> by Chia-Ping Tsai). A project NOT listed that needs doing is what Andrew
> did comparing 1.3. and 1.4 APIs
>
> * HBASE-18622 Mitigate compatibility concerns between branch-1 and branch-2
> This is to do what Andrew did between 1.3 and 1.4 branches only do it
> between branch-1 and branch-2.
>
> * HBASE-10462 Recategorize some of the client facing Public / Private
> interfaces
> This one is almost done. It could do with a finish, attention to the items
> in last comment, and then our codebase could do with another sweep after
> the spirit of this issue since a bunch has gone in since the pass that was
> the basis of this issue.
>
> * HBASE-10504 Define Replication Interface
> I was going to take a crack at this as part of the revamp forced by
> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service' but if
> anyone else is interested, be my guest.
>
> * HBASE-14996 Some more API cleanup for 2.0
> Has a bunch of subtasks, some of which are being worked on. Needs
> finishing.
>
> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
> cleanup
> Needs a pass. Small issue I think. Could also look at new AsyncClient and
> make sure symmetry.
>
> * HBASE-15607 Remove PB references from Admin for 2.0
> Predicated on result of an ongoing DISCUSSION thread but needs to be done.
>
> Rolling upgrade will have implications for our API. Would be good to try it
> and figure what needs fixup (as said above, according to trial by Sean, we
> might not be too bad here):
> * HBASE-16060 1.x clients cannot access table state talking to 2.0 cluster
> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and 1.x
> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>
> * HBASE-17442 Move most of the replication related classes to hbase-server
> package
> The above would be good to do generally but it may make for ripples in API
> so would be good to do now.
>
> * HBASE-18106 Redo ProcedureInfo and LockInfo
> Balazs is working on this. The idea is that we avoid adding two new types
> to our API, two types that are nought but curtailed, read-only views on
> internals. Input if you have time appreciated.
>
> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
> cluster; verify
> Esteban is looking at this one
>
> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> * HBASE-17143 Scan improvement
>
> Our Coprocessor Interface needs a tough edit. It exposes implementations
> marked audience Private and returns implementations rather than Interfaces.
> In a few locations, we allow returning an alternate implementation
> altogether which is probably something we don't want a CP doing. To that
> end, the following issues started by Duo and Anoop need to be taken to the
> finish line; ideally they'd have an owner:
>
> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
> umbrella issue.
> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
> RegionServerObserver
>
>
> Nice-to-haves:
>
> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
> TimeRange constructors
>
> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references existing
> in the code
> This is the end of an old long-running project moving up on to Cell
> Interface. We think it is done but for a few little items (deprecate KV
> methods in MR and provide Cell versions instead...)
>
> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
> fixing
>
> * HBASE-13346 Clean up Filter package for post 1.0
>
> * HBASE-14255 Simplify Cell creation post 1.0
> * HBASE-14997
> Move compareOp and Comparators out of filter to client package
>
> * HBASE-13740 Stop using Hadoop private interfaces
>
> What about:
>
> * HBASE-18601 Remove Htrace 3.2
> As has been noted, the HTrace API is our 'trace' API.
>
> If interested in any of the above and you need a legup, just ask in the
> issue and I'll be by....
>
> Thanks,
> St.Ack
>
>
>
> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>
> > Heads-up:
> >
> > I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
> > updated dependencies, reliance on relocated popular libs (guava, netty,
> > protobuf), purge of checked-in generated src, and
> master-carries-no-regions
> > by default.
> >
> > alpha3 I hope will follow soon after (end-of-August?). Its theme will be
> > settling the APIs and compatibility (At first blush, we are not looking
> too
> > bad; our Sean ran some tests over weekend that have hbase-1 client
> running
> > against an hbase-2 cluster....). The Coprocessor Interface revamp should
> be
> > done by alpha3 (i.e. returning Interfaces rather than Implementations,
> and
> > our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
> > We'll also have purged thirdparty classes from our API; e.g. guava 0.12
> > Service showing through in our replication API and protobufs in Admin
> > Interface. On alpha3, we will have to do a bunch of outreach to make sure
> > our downstreamers are up on what is coming down the pipe.
> >
> > Beta1 in mid-September?
> >
> > I encourage you to check out the items marked for hbase2:
> > https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as
> > you see appropriate. Punt if you know the JIRA will not get any attention
> > in next month or so.
> >
> > A bunch of issues marked blocker are unassigned. I'll leave them as is
> > another while but I'll boot them soon.
> >
> > While I have your attention:
> >
> > + I think we should leave thrift version at 0.9.3. Moving hbase thrift to
> > 0.10.0 will break existing clients. The change is easy enough if folks
> need
> > to upgrade their hbase thrift. See HBASE-18591.
> > + Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).
> >
> > St.Ack
> >
> >
> >
> > On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> >
> >>
> >>
> >> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
> >>
> >>>
> >>>
> >>> On 7/31/17 9:00 AM, Stack wrote:
> >>>
> >>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
> wrote:
> >>>>
> >>>> ...
> >>>>>
> >>>>> I like the idea of this also hitting 2.0 as it would make the
> feature a
> >>>>> bit more "real", but am obviously a little nervous (I have no reason
> >>>>> to be
> >>>>> nervous though). I am pretty happy with the feature in terms of how
> >>>>> much it
> >>>>> is covered via testing.
> >>>>>
> >>>>> https://issues.apache.org/jira/browse/HBASE-17748
> >>>>>
> >>>>>
> >>>>>
> >>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> >>>>
> >>>> Last think on space quota says that need doc too. See 'Space Quota' in
> >>>> here:
> >>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> >>>> Does this little section need an update Josh?
> >>>>
> >>>> Thanks,
> >>>> S
> >>>>
> >>>
> >>> Yep, just a couple of cherry-picks. Good test coverage and some docs
> >>> already included for 17748.  Happy to put that on my plate if you're
> good
> >>> with it. I can reasonably assume that no one is against it :)
> >>>
> >>> I think I had knocked out docs for the "phase 1" stuff before we merged
> >>> it in from the original feature branch. I'll double check and update
> the
> >>> gdoc. Perhaps this was just a timing thing.
> >>>
> >>
> >> Thanks Josh,
> >> S
> >>
> >
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

(Reviving this thread)

Lets push out alpha-4 this week. Alpha-4 is the release that has the
refactor of the Coprocessor API shutting down access to internals marked
InterfaceAudience.Private.

The outstanding list is here:
https://issues.apache.org/jira/projects/HBASE/versions/12341594

Please push in anything marked alpha-4 that belongs to you.

If issue, talk out loud on this thread. If you need a review to land an
item, shout on the issue and here; we'll help you out.

As is, items are coming along nicely I'd say. We need to merge the filter
branch -- HBASE-18410 -- so APIs are finished for hbase2.

Post alpha-4, we'll have to hunt down our downstreamers and help them test
on top of alpha-4 so rolling into beta-1, we have confidence our
downstreamers know what to expect (or we discover what we missed BEFORE we
beta-1).

Thanks for time,
S





On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:

> I'll put up an alpha3 RC Monday, probably Monday night. That should be
> time, if we all sprint, for the public-facing API fixes to be done.
>
> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3 but
> it is plain that more time is needed (in spite of valiant effort so far by
> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose theme is
> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the following
> week.
>
> We should then be ready for beta (beta == no new features, no API changes,
> just fixes).
>
> Thanks,
> St.Ack
>
>
> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>
>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>>
>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>> release out in the next week or so.
>>
>> I did a weeding of 2.0.0 issues over the last day. If folks are
>> interested in helping out, below are the items I think we need done for
>> alpha3 (below are at least 'Critical' status, are API possibly altering
>> items, and are absent those JIRAs that are making active progress, i.e. the
>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs doing is
>> what Andrew did comparing 1.3. and 1.4 APIs
>>
>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>> branch-2
>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
>> between branch-1 and branch-2.
>>
>> * HBASE-10462 Recategorize some of the client facing Public / Private
>> interfaces
>> This one is almost done. It could do with a finish, attention to the
>> items in last comment, and then our codebase could do with another sweep
>> after the spirit of this issue since a bunch has gone in since the pass
>> that was the basis of this issue.
>>
>> * HBASE-10504 Define Replication Interface
>> I was going to take a crack at this as part of the revamp forced by
>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service' but if
>> anyone else is interested, be my guest.
>>
>> * HBASE-14996 Some more API cleanup for 2.0
>> Has a bunch of subtasks, some of which are being worked on. Needs
>> finishing.
>>
>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>> cleanup
>> Needs a pass. Small issue I think. Could also look at new AsyncClient and
>> make sure symmetry.
>>
>> * HBASE-15607 Remove PB references from Admin for 2.0
>> Predicated on result of an ongoing DISCUSSION thread but needs to be done.
>>
>> Rolling upgrade will have implications for our API. Would be good to try
>> it and figure what needs fixup (as said above, according to trial by Sean,
>> we might not be too bad here):
>> * HBASE-16060 1.x clients cannot access table state talking to 2.0 cluster
>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and 1.x
>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>
>> * HBASE-17442 Move most of the replication related classes to
>> hbase-server package
>> The above would be good to do generally but it may make for ripples in
>> API so would be good to do now.
>>
>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>> Balazs is working on this. The idea is that we avoid adding two new types
>> to our API, two types that are nought but curtailed, read-only views on
>> internals. Input if you have time appreciated.
>>
>> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
>> cluster; verify
>> Esteban is looking at this one
>>
>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>> * HBASE-17143 Scan improvement
>>
>> Our Coprocessor Interface needs a tough edit. It exposes implementations
>> marked audience Private and returns implementations rather than Interfaces.
>> In a few locations, we allow returning an alternate implementation
>> altogether which is probably something we don't want a CP doing. To that
>> end, the following issues started by Duo and Anoop need to be taken to the
>> finish line; ideally they'd have an owner:
>>
>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
>> umbrella issue.
>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>> RegionServerObserver
>>
>>
>> Nice-to-haves:
>>
>> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
>> TimeRange constructors
>>
>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>> existing in the code
>> This is the end of an old long-running project moving up on to Cell
>> Interface. We think it is done but for a few little items (deprecate KV
>> methods in MR and provide Cell versions instead...)
>>
>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
>> fixing
>>
>> * HBASE-13346 Clean up Filter package for post 1.0
>>
>> * HBASE-14255 Simplify Cell creation post 1.0
>> * HBASE-14997
>> Move compareOp and Comparators out of filter to client package
>>
>> * HBASE-13740 Stop using Hadoop private interfaces
>>
>> What about:
>>
>> * HBASE-18601 Remove Htrace 3.2
>> As has been noted, the HTrace API is our 'trace' API.
>>
>> If interested in any of the above and you need a legup, just ask in the
>> issue and I'll be by....
>>
>> Thanks,
>> St.Ack
>>
>>
>>
>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>
>>> Heads-up:
>>>
>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
>>> updated dependencies, reliance on relocated popular libs (guava, netty,
>>> protobuf), purge of checked-in generated src, and master-carries-no-regions
>>> by default.
>>>
>>> alpha3 I hope will follow soon after (end-of-August?). Its theme will be
>>> settling the APIs and compatibility (At first blush, we are not looking too
>>> bad; our Sean ran some tests over weekend that have hbase-1 client running
>>> against an hbase-2 cluster....). The Coprocessor Interface revamp should be
>>> done by alpha3 (i.e. returning Interfaces rather than Implementations, and
>>> our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
>>> We'll also have purged thirdparty classes from our API; e.g. guava 0.12
>>> Service showing through in our replication API and protobufs in Admin
>>> Interface. On alpha3, we will have to do a bunch of outreach to make sure
>>> our downstreamers are up on what is coming down the pipe.
>>>
>>> Beta1 in mid-September?
>>>
>>> I encourage you to check out the items marked for hbase2:
>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as
>>> you see appropriate. Punt if you know the JIRA will not get any attention
>>> in next month or so.
>>>
>>> A bunch of issues marked blocker are unassigned. I'll leave them as is
>>> another while but I'll boot them soon.
>>>
>>> While I have your attention:
>>>
>>> + I think we should leave thrift version at 0.9.3. Moving hbase thrift
>>> to 0.10.0 will break existing clients. The change is easy enough if folks
>>> need to upgrade their hbase thrift. See HBASE-18591.
>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).
>>>
>>> St.Ack
>>>
>>>
>>>
>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
>>>>
>>>>>
>>>>>
>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>
>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> ...
>>>>>>>
>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>> feature a
>>>>>>> bit more "real", but am obviously a little nervous (I have no reason
>>>>>>> to be
>>>>>>> nervous though). I am pretty happy with the feature in terms of how
>>>>>>> much it
>>>>>>> is covered via testing.
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>
>>>>>> Last think on space quota says that need doc too. See 'Space Quota' in
>>>>>> here:
>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>> Does this little section need an update Josh?
>>>>>>
>>>>>> Thanks,
>>>>>> S
>>>>>>
>>>>>
>>>>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>>>>> already included for 17748.  Happy to put that on my plate if you're good
>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>
>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>> merged it in from the original feature branch. I'll double check and update
>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>
>>>>
>>>> Thanks Josh,
>>>> S
>>>>
>>>
>>>
>>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

I'll put up an alpha3 RC Monday, probably Monday night. That should be
time, if we all sprint, for the public-facing API fixes to be done.

I had a bunch of Coprocessor refactor and fixup scheduled for alpha3 but it
is plain that more time is needed (in spite of valiant effort so far by
Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose theme is
"Coprocessor Fixup". Hopefully we can put an alpha-4 up by the following
week.

We should then be ready for beta (beta == no new features, no API changes,
just fixes).

Thanks,
St.Ack


On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:

> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>
> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
> release out in the next week or so.
>
> I did a weeding of 2.0.0 issues over the last day. If folks are interested
> in helping out, below are the items I think we need done for alpha3 (below
> are at least 'Critical' status, are API possibly altering items, and are
> absent those JIRAs that are making active progress, i.e. the HTD/HCD revamp
> by Chia-Ping Tsai). A project NOT listed that needs doing is what Andrew
> did comparing 1.3. and 1.4 APIs
>
> * HBASE-18622 Mitigate compatibility concerns between branch-1 and branch-2
> This is to do what Andrew did between 1.3 and 1.4 branches only do it
> between branch-1 and branch-2.
>
> * HBASE-10462 Recategorize some of the client facing Public / Private
> interfaces
> This one is almost done. It could do with a finish, attention to the items
> in last comment, and then our codebase could do with another sweep after
> the spirit of this issue since a bunch has gone in since the pass that was
> the basis of this issue.
>
> * HBASE-10504 Define Replication Interface
> I was going to take a crack at this as part of the revamp forced by
> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service' but if
> anyone else is interested, be my guest.
>
> * HBASE-14996 Some more API cleanup for 2.0
> Has a bunch of subtasks, some of which are being worked on. Needs
> finishing.
>
> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
> cleanup
> Needs a pass. Small issue I think. Could also look at new AsyncClient and
> make sure symmetry.
>
> * HBASE-15607 Remove PB references from Admin for 2.0
> Predicated on result of an ongoing DISCUSSION thread but needs to be done.
>
> Rolling upgrade will have implications for our API. Would be good to try
> it and figure what needs fixup (as said above, according to trial by Sean,
> we might not be too bad here):
> * HBASE-16060 1.x clients cannot access table state talking to 2.0 cluster
> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and 1.x
> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>
> * HBASE-17442 Move most of the replication related classes to hbase-server
> package
> The above would be good to do generally but it may make for ripples in API
> so would be good to do now.
>
> * HBASE-18106 Redo ProcedureInfo and LockInfo
> Balazs is working on this. The idea is that we avoid adding two new types
> to our API, two types that are nought but curtailed, read-only views on
> internals. Input if you have time appreciated.
>
> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
> cluster; verify
> Esteban is looking at this one
>
> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> * HBASE-17143 Scan improvement
>
> Our Coprocessor Interface needs a tough edit. It exposes implementations
> marked audience Private and returns implementations rather than Interfaces.
> In a few locations, we allow returning an alternate implementation
> altogether which is probably something we don't want a CP doing. To that
> end, the following issues started by Duo and Anoop need to be taken to the
> finish line; ideally they'd have an owner:
>
> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
> umbrella issue.
> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
> RegionServerObserver
>
>
> Nice-to-haves:
>
> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
> TimeRange constructors
>
> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
> existing in the code
> This is the end of an old long-running project moving up on to Cell
> Interface. We think it is done but for a few little items (deprecate KV
> methods in MR and provide Cell versions instead...)
>
> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
> fixing
>
> * HBASE-13346 Clean up Filter package for post 1.0
>
> * HBASE-14255 Simplify Cell creation post 1.0
> * HBASE-14997
> Move compareOp and Comparators out of filter to client package
>
> * HBASE-13740 Stop using Hadoop private interfaces
>
> What about:
>
> * HBASE-18601 Remove Htrace 3.2
> As has been noted, the HTrace API is our 'trace' API.
>
> If interested in any of the above and you need a legup, just ask in the
> issue and I'll be by....
>
> Thanks,
> St.Ack
>
>
>
> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>
>> Heads-up:
>>
>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
>> updated dependencies, reliance on relocated popular libs (guava, netty,
>> protobuf), purge of checked-in generated src, and master-carries-no-regions
>> by default.
>>
>> alpha3 I hope will follow soon after (end-of-August?). Its theme will be
>> settling the APIs and compatibility (At first blush, we are not looking too
>> bad; our Sean ran some tests over weekend that have hbase-1 client running
>> against an hbase-2 cluster....). The Coprocessor Interface revamp should be
>> done by alpha3 (i.e. returning Interfaces rather than Implementations, and
>> our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
>> We'll also have purged thirdparty classes from our API; e.g. guava 0.12
>> Service showing through in our replication API and protobufs in Admin
>> Interface. On alpha3, we will have to do a bunch of outreach to make sure
>> our downstreamers are up on what is coming down the pipe.
>>
>> Beta1 in mid-September?
>>
>> I encourage you to check out the items marked for hbase2:
>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as
>> you see appropriate. Punt if you know the JIRA will not get any attention
>> in next month or so.
>>
>> A bunch of issues marked blocker are unassigned. I'll leave them as is
>> another while but I'll boot them soon.
>>
>> While I have your attention:
>>
>> + I think we should leave thrift version at 0.9.3. Moving hbase thrift to
>> 0.10.0 will break existing clients. The change is easy enough if folks need
>> to upgrade their hbase thrift. See HBASE-18591.
>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).
>>
>> St.Ack
>>
>>
>>
>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>
>>>
>>>
>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
>>>
>>>>
>>>>
>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>
>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>> wrote:
>>>>>
>>>>> ...
>>>>>>
>>>>>> I like the idea of this also hitting 2.0 as it would make the feature
>>>>>> a
>>>>>> bit more "real", but am obviously a little nervous (I have no reason
>>>>>> to be
>>>>>> nervous though). I am pretty happy with the feature in terms of how
>>>>>> much it
>>>>>> is covered via testing.
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>
>>>>>>
>>>>>>
>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>
>>>>> Last think on space quota says that need doc too. See 'Space Quota' in
>>>>> here:
>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>> Does this little section need an update Josh?
>>>>>
>>>>> Thanks,
>>>>> S
>>>>>
>>>>
>>>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>>>> already included for 17748.  Happy to put that on my plate if you're good
>>>> with it. I can reasonably assume that no one is against it :)
>>>>
>>>> I think I had knocked out docs for the "phase 1" stuff before we merged
>>>> it in from the original feature branch. I'll double check and update the
>>>> gdoc. Perhaps this was just a timing thing.
>>>>
>>>
>>> Thanks Josh,
>>> S
>>>
>>
>>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.

For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
release out in the next week or so.

I did a weeding of 2.0.0 issues over the last day. If folks are interested
in helping out, below are the items I think we need done for alpha3 (below
are at least 'Critical' status, are API possibly altering items, and are
absent those JIRAs that are making active progress, i.e. the HTD/HCD revamp
by Chia-Ping Tsai). A project NOT listed that needs doing is what Andrew
did comparing 1.3. and 1.4 APIs

* HBASE-18622 Mitigate compatibility concerns between branch-1 and branch-2
This is to do what Andrew did between 1.3 and 1.4 branches only do it
between branch-1 and branch-2.

* HBASE-10462 Recategorize some of the client facing Public / Private
interfaces
This one is almost done. It could do with a finish, attention to the items
in last comment, and then our codebase could do with another sweep after
the spirit of this issue since a bunch has gone in since the pass that was
the basis of this issue.

* HBASE-10504 Define Replication Interface
I was going to take a crack at this as part of the revamp forced by
'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service' but if
anyone else is interested, be my guest.

* HBASE-14996 Some more API cleanup for 2.0
Has a bunch of subtasks, some of which are being worked on. Needs finishing.

* HBASE-14998 Unify synchronous and asynchronous methods in Admin and
cleanup
Needs a pass. Small issue I think. Could also look at new AsyncClient and
make sure symmetry.

* HBASE-15607 Remove PB references from Admin for 2.0
Predicated on result of an ongoing DISCUSSION thread but needs to be done.

Rolling upgrade will have implications for our API. Would be good to try it
and figure what needs fixup (as said above, according to trial by Sean, we
might not be too bad here):
* HBASE-16060 1.x clients cannot access table state talking to 2.0 cluster
* HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and 1.x
RSs; i.e. support Rolling Upgrade from hbase-1 to -2.

* HBASE-17442 Move most of the replication related classes to hbase-server
package
The above would be good to do generally but it may make for ripples in API
so would be good to do now.

* HBASE-18106 Redo ProcedureInfo and LockInfo
Balazs is working on this. The idea is that we avoid adding two new types
to our API, two types that are nought but curtailed, read-only views on
internals. Input if you have time appreciated.

* HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
cluster; verify
Esteban is looking at this one

* HBASE-9417 SecureBulkLoadEndpoint should be folded in core
* HBASE-17143 Scan improvement

Our Coprocessor Interface needs a tough edit. It exposes implementations
marked audience Private and returns implementations rather than Interfaces.
In a few locations, we allow returning an alternate implementation
altogether which is probably something we don't want a CP doing. To that
end, the following issues started by Duo and Anoop need to be taken to the
finish line; ideally they'd have an owner:

* HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
umbrella issue.
* HBASE-18298 RegionServerServices Interface cleanup for CP expose
* HBASE-16769 Deprecate/remove PB references from MasterObserver and
RegionServerObserver


Nice-to-haves:

* HBASE-15284 Make TimeRange constructors IA.Private and remove unused
TimeRange constructors

* HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references existing
in the code
This is the end of an old long-running project moving up on to Cell
Interface. We think it is done but for a few little items (deprecate KV
methods in MR and provide Cell versions instead...)

* HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs fixing

* HBASE-13346 Clean up Filter package for post 1.0

* HBASE-14255 Simplify Cell creation post 1.0
* HBASE-14997
Move compareOp and Comparators out of filter to client package

* HBASE-13740 Stop using Hadoop private interfaces

What about:

* HBASE-18601 Remove Htrace 3.2
As has been noted, the HTrace API is our 'trace' API.

If interested in any of the above and you need a legup, just ask in the
issue and I'll be by....

Thanks,
St.Ack



On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:

> Heads-up:
>
> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
> updated dependencies, reliance on relocated popular libs (guava, netty,
> protobuf), purge of checked-in generated src, and master-carries-no-regions
> by default.
>
> alpha3 I hope will follow soon after (end-of-August?). Its theme will be
> settling the APIs and compatibility (At first blush, we are not looking too
> bad; our Sean ran some tests over weekend that have hbase-1 client running
> against an hbase-2 cluster....). The Coprocessor Interface revamp should be
> done by alpha3 (i.e. returning Interfaces rather than Implementations, and
> our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
> We'll also have purged thirdparty classes from our API; e.g. guava 0.12
> Service showing through in our replication API and protobufs in Admin
> Interface. On alpha3, we will have to do a bunch of outreach to make sure
> our downstreamers are up on what is coming down the pipe.
>
> Beta1 in mid-September?
>
> I encourage you to check out the items marked for hbase2:
> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as
> you see appropriate. Punt if you know the JIRA will not get any attention
> in next month or so.
>
> A bunch of issues marked blocker are unassigned. I'll leave them as is
> another while but I'll boot them soon.
>
> While I have your attention:
>
> + I think we should leave thrift version at 0.9.3. Moving hbase thrift to
> 0.10.0 will break existing clients. The change is easy enough if folks need
> to upgrade their hbase thrift. See HBASE-18591.
> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).
>
> St.Ack
>
>
>
> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>
>>
>>
>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
>>
>>>
>>>
>>> On 7/31/17 9:00 AM, Stack wrote:
>>>
>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>  wrote:
>>>>
>>>> ...
>>>>>
>>>>> I like the idea of this also hitting 2.0 as it would make the feature a
>>>>> bit more "real", but am obviously a little nervous (I have no reason
>>>>> to be
>>>>> nervous though). I am pretty happy with the feature in terms of how
>>>>> much it
>>>>> is covered via testing.
>>>>>
>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>
>>>>>
>>>>>
>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>
>>>> Last think on space quota says that need doc too. See 'Space Quota' in
>>>> here:
>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>> Does this little section need an update Josh?
>>>>
>>>> Thanks,
>>>> S
>>>>
>>>
>>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>>> already included for 17748.  Happy to put that on my plate if you're good
>>> with it. I can reasonably assume that no one is against it :)
>>>
>>> I think I had knocked out docs for the "phase 1" stuff before we merged
>>> it in from the original feature branch. I'll double check and update the
>>> gdoc. Perhaps this was just a timing thing.
>>>
>>
>> Thanks Josh,
>> S
>>
>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Wed, Aug 16, 2017 at 10:18 PM, Guanghao Zhang <zg...@gmail.com> wrote:

> About compatibility, there is one incompatible change about the replication
> TableCFs' config. The old config is a string and it concatenate the list of
> tables and column families in format "table1:cf1,cf2;table2:cfA,cfB" in
> zookeeper for table-cf to replication peer mapping. When parse the config,
> it use ":" to split the string. If table name includes namespace, it will
> be wrong (See HBASE-11386
> <https://issues.apache.org/jira/browse/HBASE-11386>). It is a problem
> since
> we support namespace (0.98). So HBASE-11393
> <https://issues.apache.org/jira/browse/HBASE-11393> (and HBASE-16653
> <https://issues.apache.org/jira/browse/HBASE-16653>) changed it to a PB
> object. When rolling update cluster, you need rolling master first. And the
> master will try to translate the string config to a PB object. But there
> are two problems.
> 1. Permission problem. The replication client can write the zookeeper
> directly. So the znode may have different owner. And master may don't have
> the write permission for the znode. It maybe failed to translate old
> table-cfs string to new PB Object. See HBASE-16938
> <https://issues.apache.org/jira/browse/HBASE-16938>
> 2. We usually keep compatibility between old client and new server. But the
> old replication client may write a string config to znode directly. Then
> the new server can't parse them.
>
> PS: HBASE-11386 is a problem since 0.98. But there are not too many users
> report this problem. So it maybe not a big incompatible issue......
>
>
Excellent writeup Guanghao. File a blocker with the above. We need to have
some facility to get folks over the above hump.
Thanks,
St.Ack



> 2017-08-15 1:54 GMT+08:00 Stack <st...@duboce.net>:
>
> > Heads-up:
> >
> > I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
> > updated dependencies, reliance on relocated popular libs (guava, netty,
> > protobuf), purge of checked-in generated src, and
> master-carries-no-regions
> > by default.
> >
> > alpha3 I hope will follow soon after (end-of-August?). Its theme will be
> > settling the APIs and compatibility (At first blush, we are not looking
> too
> > bad; our Sean ran some tests over weekend that have hbase-1 client
> running
> > against an hbase-2 cluster....). The Coprocessor Interface revamp should
> be
> > done by alpha3 (i.e. returning Interfaces rather than Implementations,
> and
> > our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
> > We'll also have purged thirdparty classes from our API; e.g. guava 0.12
> > Service showing through in our replication API and protobufs in Admin
> > Interface. On alpha3, we will have to do a bunch of outreach to make sure
> > our downstreamers are up on what is coming down the pipe.
> >
> > Beta1 in mid-September?
> >
> > I encourage you to check out the items marked for hbase2:
> > https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as
> > you
> > see appropriate. Punt if you know the JIRA will not get any attention in
> > next month or so.
> >
> > A bunch of issues marked blocker are unassigned. I'll leave them as is
> > another while but I'll boot them soon.
> >
> > While I have your attention:
> >
> > + I think we should leave thrift version at 0.9.3. Moving hbase thrift to
> > 0.10.0 will break existing clients. The change is easy enough if folks
> need
> > to upgrade their hbase thrift. See HBASE-18591.
> > + Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).
> >
> > St.Ack
> >
> >
> >
> > On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> >
> > >
> > >
> > > On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
> > >
> > >>
> > >>
> > >> On 7/31/17 9:00 AM, Stack wrote:
> > >>
> > >>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
> > wrote:
> > >>>
> > >>> ...
> > >>>>
> > >>>> I like the idea of this also hitting 2.0 as it would make the
> feature
> > a
> > >>>> bit more "real", but am obviously a little nervous (I have no reason
> > to
> > >>>> be
> > >>>> nervous though). I am pretty happy with the feature in terms of how
> > >>>> much it
> > >>>> is covered via testing.
> > >>>>
> > >>>> https://issues.apache.org/jira/browse/HBASE-17748
> > >>>>
> > >>>>
> > >>>>
> > >>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> > >>>
> > >>> Last think on space quota says that need doc too. See 'Space Quota'
> in
> > >>> here:
> > >>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> > >>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > >>> Does this little section need an update Josh?
> > >>>
> > >>> Thanks,
> > >>> S
> > >>>
> > >>
> > >> Yep, just a couple of cherry-picks. Good test coverage and some docs
> > >> already included for 17748.  Happy to put that on my plate if you're
> > good
> > >> with it. I can reasonably assume that no one is against it :)
> > >>
> > >> I think I had knocked out docs for the "phase 1" stuff before we
> merged
> > >> it in from the original feature branch. I'll double check and update
> the
> > >> gdoc. Perhaps this was just a timing thing.
> > >>
> > >
> > > Thanks Josh,
> > > S
> > >
> >
>

Re: Moving 2.0 forward

[Guanghao Zhang <zg...@gmail.com>]

About compatibility, there is one incompatible change about the replication
TableCFs' config. The old config is a string and it concatenate the list of
tables and column families in format "table1:cf1,cf2;table2:cfA,cfB" in
zookeeper for table-cf to replication peer mapping. When parse the config,
it use ":" to split the string. If table name includes namespace, it will
be wrong (See HBASE-11386
<https://issues.apache.org/jira/browse/HBASE-11386>). It is a problem since
we support namespace (0.98). So HBASE-11393
<https://issues.apache.org/jira/browse/HBASE-11393> (and HBASE-16653
<https://issues.apache.org/jira/browse/HBASE-16653>) changed it to a PB
object. When rolling update cluster, you need rolling master first. And the
master will try to translate the string config to a PB object. But there
are two problems.
1. Permission problem. The replication client can write the zookeeper
directly. So the znode may have different owner. And master may don't have
the write permission for the znode. It maybe failed to translate old
table-cfs string to new PB Object. See HBASE-16938
<https://issues.apache.org/jira/browse/HBASE-16938>
2. We usually keep compatibility between old client and new server. But the
old replication client may write a string config to znode directly. Then
the new server can't parse them.

PS: HBASE-11386 is a problem since 0.98. But there are not too many users
report this problem. So it maybe not a big incompatible issue......

2017-08-15 1:54 GMT+08:00 Stack <st...@duboce.net>:

> Heads-up:
>
> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
> updated dependencies, reliance on relocated popular libs (guava, netty,
> protobuf), purge of checked-in generated src, and master-carries-no-regions
> by default.
>
> alpha3 I hope will follow soon after (end-of-August?). Its theme will be
> settling the APIs and compatibility (At first blush, we are not looking too
> bad; our Sean ran some tests over weekend that have hbase-1 client running
> against an hbase-2 cluster....). The Coprocessor Interface revamp should be
> done by alpha3 (i.e. returning Interfaces rather than Implementations, and
> our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
> We'll also have purged thirdparty classes from our API; e.g. guava 0.12
> Service showing through in our replication API and protobufs in Admin
> Interface. On alpha3, we will have to do a bunch of outreach to make sure
> our downstreamers are up on what is coming down the pipe.
>
> Beta1 in mid-September?
>
> I encourage you to check out the items marked for hbase2:
> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as
> you
> see appropriate. Punt if you know the JIRA will not get any attention in
> next month or so.
>
> A bunch of issues marked blocker are unassigned. I'll leave them as is
> another while but I'll boot them soon.
>
> While I have your attention:
>
> + I think we should leave thrift version at 0.9.3. Moving hbase thrift to
> 0.10.0 will break existing clients. The change is easy enough if folks need
> to upgrade their hbase thrift. See HBASE-18591.
> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).
>
> St.Ack
>
>
>
> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>
> >
> >
> > On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
> >
> >>
> >>
> >> On 7/31/17 9:00 AM, Stack wrote:
> >>
> >>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
> wrote:
> >>>
> >>> ...
> >>>>
> >>>> I like the idea of this also hitting 2.0 as it would make the feature
> a
> >>>> bit more "real", but am obviously a little nervous (I have no reason
> to
> >>>> be
> >>>> nervous though). I am pretty happy with the feature in terms of how
> >>>> much it
> >>>> is covered via testing.
> >>>>
> >>>> https://issues.apache.org/jira/browse/HBASE-17748
> >>>>
> >>>>
> >>>>
> >>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> >>>
> >>> Last think on space quota says that need doc too. See 'Space Quota' in
> >>> here:
> >>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> >>> Does this little section need an update Josh?
> >>>
> >>> Thanks,
> >>> S
> >>>
> >>
> >> Yep, just a couple of cherry-picks. Good test coverage and some docs
> >> already included for 17748.  Happy to put that on my plate if you're
> good
> >> with it. I can reasonably assume that no one is against it :)
> >>
> >> I think I had knocked out docs for the "phase 1" stuff before we merged
> >> it in from the original feature branch. I'll double check and update the
> >> gdoc. Perhaps this was just a timing thing.
> >>
> >
> > Thanks Josh,
> > S
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Heads-up:

I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
updated dependencies, reliance on relocated popular libs (guava, netty,
protobuf), purge of checked-in generated src, and master-carries-no-regions
by default.

alpha3 I hope will follow soon after (end-of-August?). Its theme will be
settling the APIs and compatibility (At first blush, we are not looking too
bad; our Sean ran some tests over weekend that have hbase-1 client running
against an hbase-2 cluster....). The Coprocessor Interface revamp should be
done by alpha3 (i.e. returning Interfaces rather than Implementations, and
our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
We'll also have purged thirdparty classes from our API; e.g. guava 0.12
Service showing through in our replication API and protobufs in Admin
Interface. On alpha3, we will have to do a bunch of outreach to make sure
our downstreamers are up on what is coming down the pipe.

Beta1 in mid-September?

I encourage you to check out the items marked for hbase2:
https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as you
see appropriate. Punt if you know the JIRA will not get any attention in
next month or so.

A bunch of issues marked blocker are unassigned. I'll leave them as is
another while but I'll boot them soon.

While I have your attention:

+ I think we should leave thrift version at 0.9.3. Moving hbase thrift to
0.10.0 will break existing clients. The change is easy enough if folks need
to upgrade their hbase thrift. See HBASE-18591.
+ Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).

St.Ack



On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:

>
>
> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
>
>>
>>
>> On 7/31/17 9:00 AM, Stack wrote:
>>
>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>  wrote:
>>>
>>> ...
>>>>
>>>> I like the idea of this also hitting 2.0 as it would make the feature a
>>>> bit more "real", but am obviously a little nervous (I have no reason to
>>>> be
>>>> nervous though). I am pretty happy with the feature in terms of how
>>>> much it
>>>> is covered via testing.
>>>>
>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>
>>>>
>>>>
>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>
>>> Last think on space quota says that need doc too. See 'Space Quota' in
>>> here:
>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>> Does this little section need an update Josh?
>>>
>>> Thanks,
>>> S
>>>
>>
>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>> already included for 17748.  Happy to put that on my plate if you're good
>> with it. I can reasonably assume that no one is against it :)
>>
>> I think I had knocked out docs for the "phase 1" stuff before we merged
>> it in from the original feature branch. I'll double check and update the
>> gdoc. Perhaps this was just a timing thing.
>>
>
> Thanks Josh,
> S
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:

>
>
> On 7/31/17 9:00 AM, Stack wrote:
>
>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>  wrote:
>>
>> ...
>>>
>>> I like the idea of this also hitting 2.0 as it would make the feature a
>>> bit more "real", but am obviously a little nervous (I have no reason to
>>> be
>>> nervous though). I am pretty happy with the feature in terms of how much
>>> it
>>> is covered via testing.
>>>
>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>
>>>
>>>
>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>
>> Last think on space quota says that need doc too. See 'Space Quota' in
>> here:
>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>> Does this little section need an update Josh?
>>
>> Thanks,
>> S
>>
>
> Yep, just a couple of cherry-picks. Good test coverage and some docs
> already included for 17748.  Happy to put that on my plate if you're good
> with it. I can reasonably assume that no one is against it :)
>
> I think I had knocked out docs for the "phase 1" stuff before we merged it
> in from the original feature branch. I'll double check and update the gdoc.
> Perhaps this was just a timing thing.
>

Thanks Josh,
S

Re: Moving 2.0 forward

[Josh Elser <el...@apache.org>]

On 7/31/17 9:00 AM, Stack wrote:
> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>  wrote:
> 
>> ...
>>
>> I like the idea of this also hitting 2.0 as it would make the feature a
>> bit more "real", but am obviously a little nervous (I have no reason to be
>> nervous though). I am pretty happy with the feature in terms of how much it
>> is covered via testing.
>>
>> https://issues.apache.org/jira/browse/HBASE-17748
>>
>>
>>
> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> 
> Last think on space quota says that need doc too. See 'Space Quota' in
> here:
> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> Does this little section need an update Josh?
> 
> Thanks,
> S

Yep, just a couple of cherry-picks. Good test coverage and some docs 
already included for 17748.  Happy to put that on my plate if you're 
good with it. I can reasonably assume that no one is against it :)

I think I had knocked out docs for the "phase 1" stuff before we merged 
it in from the original feature branch. I'll double check and update the 
gdoc. Perhaps this was just a timing thing.

Re: Moving 2.0 forward

[Sean Busbey <se...@gmail.com>]

+1 on a dedicated thread for upgrade expectations.

On Mon, Jul 31, 2017 at 8:04 AM, Stack <st...@duboce.net> wrote:
> On Wed, Jul 26, 2017 at 10:57 PM, ashish singhi <as...@huawei.com>
> wrote:
>
>> One question regarding upgrade:
>> Do we expect the user to move to latest branch-1 release and then upgrade
>> to 2.0 version ?
>>
>>
> I think, as you seem to, that this is too much to ask Ashish.
>
> I think we should support at the low end going from 1.0.0 to 2.0.0; i.e.
> you'd have to upgrade to 1.x if you were running 0.98 before you could go
> to 2.0.0.
>
> What do folks think of this? I should start a separate thread on upgrade
> expectations?
>
> St.Ack
>
>
>
>> I think that should not be imposed on the user.
>>
>> Regards,
>> Ashish
>>
>> -----Original Message-----
>> From: 张铎(Duo Zhang) [mailto:palomino219@gmail.com]
>> Sent: 26 July 2017 12:10
>> To: dev@hbase.apache.org
>> Subject: Re: Moving 2.0 forward
>>
>> I set HBASE-18169 as a Blocker because I found some critial problems on
>> our current CPs. The semantics are broken. Although we are allowed to break
>> CP in a major release, I think we need to provide the same ability(in
>> another way).
>>
>> 2017-07-25 1:25 GMT+08:00 Josh Elser <el...@apache.org>:
>>
>> >
>> >
>> > On 7/21/17 12:03 PM, Stack wrote:
>> >
>> >> Status update girls and boys!
>> >>
>> >> hbase-2.0.0-alpha1 went out June 22nd.
>> >>
>> >> alpha2 has been a bit slow to follow (holidays) though there has been
>> >> steady progress closing out blockers and criticals by a bunch of you
>> all.
>> >> The plan is for a release in the first week or so of August. It
>> >> should be fully up on hbase-thirdparty using updated (and relocated)
>> >> versions of netty, guava, and protobuf as well as a default deploy
>> >> that has master-carrying-no-regions.
>> >>
>> >> alpha3 will follow soon after and will focus on making sure our
>> >> user-facing APIs are clean (branch-1 compatible, no illicit
>> >> removals/mods, and so on) and that basic upgrade 'works'.
>> >>
>> >> betas start in September?
>> >>
>> >> I've been keeping a rough general state here [1] (please update any
>> >> section that is lagging actuality) but for details on what blockers
>> >> and criticals remain, see the JIRA 2.0 view [2]. Recent
>> >> issue-gardening has brought 2.0 into better focus. Feel free to
>> >> review and punt items you think can wait till 3.0 or 2.1. If you want
>> >> to pull in more stuff, please ask first.
>> >>
>> >
>> > Chia-Ping (I think? -- JIRA is being a pain) had asked on the
>> > space-quota
>> > phase2 work (include size of hbase snapshots in a table's "quota
>> > usage") if we should try to also include that work in 2.0.
>> >
>> > I like the idea of this also hitting 2.0 as it would make the feature
>> > a bit more "real", but am obviously a little nervous (I have no reason
>> > to be nervous though). I am pretty happy with the feature in terms of
>> > how much it is covered via testing.
>> >
>> > https://issues.apache.org/jira/browse/HBASE-17748
>> >
>> >
>> > Thanks,
>> >> St.Ack
>> >>
>> >> 1.
>> >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> >> Eu_ktczrlKHK8N4SZzs/edit#
>> >> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
>> >>
>> >
>> > - Josh
>> >
>>



-- 
Sean

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Wed, Jul 26, 2017 at 10:57 PM, ashish singhi <as...@huawei.com>
wrote:

> One question regarding upgrade:
> Do we expect the user to move to latest branch-1 release and then upgrade
> to 2.0 version ?
>
>
I think, as you seem to, that this is too much to ask Ashish.

I think we should support at the low end going from 1.0.0 to 2.0.0; i.e.
you'd have to upgrade to 1.x if you were running 0.98 before you could go
to 2.0.0.

What do folks think of this? I should start a separate thread on upgrade
expectations?

St.Ack



> I think that should not be imposed on the user.
>
> Regards,
> Ashish
>
> -----Original Message-----
> From: 张铎(Duo Zhang) [mailto:palomino219@gmail.com]
> Sent: 26 July 2017 12:10
> To: dev@hbase.apache.org
> Subject: Re: Moving 2.0 forward
>
> I set HBASE-18169 as a Blocker because I found some critial problems on
> our current CPs. The semantics are broken. Although we are allowed to break
> CP in a major release, I think we need to provide the same ability(in
> another way).
>
> 2017-07-25 1:25 GMT+08:00 Josh Elser <el...@apache.org>:
>
> >
> >
> > On 7/21/17 12:03 PM, Stack wrote:
> >
> >> Status update girls and boys!
> >>
> >> hbase-2.0.0-alpha1 went out June 22nd.
> >>
> >> alpha2 has been a bit slow to follow (holidays) though there has been
> >> steady progress closing out blockers and criticals by a bunch of you
> all.
> >> The plan is for a release in the first week or so of August. It
> >> should be fully up on hbase-thirdparty using updated (and relocated)
> >> versions of netty, guava, and protobuf as well as a default deploy
> >> that has master-carrying-no-regions.
> >>
> >> alpha3 will follow soon after and will focus on making sure our
> >> user-facing APIs are clean (branch-1 compatible, no illicit
> >> removals/mods, and so on) and that basic upgrade 'works'.
> >>
> >> betas start in September?
> >>
> >> I've been keeping a rough general state here [1] (please update any
> >> section that is lagging actuality) but for details on what blockers
> >> and criticals remain, see the JIRA 2.0 view [2]. Recent
> >> issue-gardening has brought 2.0 into better focus. Feel free to
> >> review and punt items you think can wait till 3.0 or 2.1. If you want
> >> to pull in more stuff, please ask first.
> >>
> >
> > Chia-Ping (I think? -- JIRA is being a pain) had asked on the
> > space-quota
> > phase2 work (include size of hbase snapshots in a table's "quota
> > usage") if we should try to also include that work in 2.0.
> >
> > I like the idea of this also hitting 2.0 as it would make the feature
> > a bit more "real", but am obviously a little nervous (I have no reason
> > to be nervous though). I am pretty happy with the feature in terms of
> > how much it is covered via testing.
> >
> > https://issues.apache.org/jira/browse/HBASE-17748
> >
> >
> > Thanks,
> >> St.Ack
> >>
> >> 1.
> >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >> Eu_ktczrlKHK8N4SZzs/edit#
> >> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
> >>
> >
> > - Josh
> >
>

Re: Moving 2.0 forward

[karthikeyan thirunavukkarasu <me...@yahoo.com.INVALID>]

Hi All,
May be wrong forum, but how to unsubscribe my email id from this email list?
ThanksKarthik

      From: ashish singhi <as...@huawei.com>
 To: "dev@hbase.apache.org" <de...@hbase.apache.org> 
 Sent: Wednesday, July 26, 2017 8:57 PM
 Subject: RE: Moving 2.0 forward
   
One question regarding upgrade: 
Do we expect the user to move to latest branch-1 release and then upgrade to 2.0 version ?

I think that should not be imposed on the user. 

Regards,
Ashish

-----Original Message-----
From: 张铎(Duo Zhang) [mailto:palomino219@gmail.com] 
Sent: 26 July 2017 12:10
To: dev@hbase.apache.org
Subject: Re: Moving 2.0 forward

I set HBASE-18169 as a Blocker because I found some critial problems on our current CPs. The semantics are broken. Although we are allowed to break CP in a major release, I think we need to provide the same ability(in another way).

2017-07-25 1:25 GMT+08:00 Josh Elser <el...@apache.org>:

>
>
> On 7/21/17 12:03 PM, Stack wrote:
>
>> Status update girls and boys!
>>
>> hbase-2.0.0-alpha1 went out June 22nd.
>>
>> alpha2 has been a bit slow to follow (holidays) though there has been 
>> steady progress closing out blockers and criticals by a bunch of you all.
>> The plan is for a release in the first week or so of August. It 
>> should be fully up on hbase-thirdparty using updated (and relocated) 
>> versions of netty, guava, and protobuf as well as a default deploy 
>> that has master-carrying-no-regions.
>>
>> alpha3 will follow soon after and will focus on making sure our 
>> user-facing APIs are clean (branch-1 compatible, no illicit 
>> removals/mods, and so on) and that basic upgrade 'works'.
>>
>> betas start in September?
>>
>> I've been keeping a rough general state here [1] (please update any 
>> section that is lagging actuality) but for details on what blockers 
>> and criticals remain, see the JIRA 2.0 view [2]. Recent 
>> issue-gardening has brought 2.0 into better focus. Feel free to 
>> review and punt items you think can wait till 3.0 or 2.1. If you want 
>> to pull in more stuff, please ask first.
>>
>
> Chia-Ping (I think? -- JIRA is being a pain) had asked on the 
> space-quota
> phase2 work (include size of hbase snapshots in a table's "quota 
> usage") if we should try to also include that work in 2.0.
>
> I like the idea of this also hitting 2.0 as it would make the feature 
> a bit more "real", but am obviously a little nervous (I have no reason 
> to be nervous though). I am pretty happy with the feature in terms of 
> how much it is covered via testing.
>
> https://issues.apache.org/jira/browse/HBASE-17748
>
>
> Thanks,
>> St.Ack
>>
>> 1.
>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> Eu_ktczrlKHK8N4SZzs/edit#
>> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>
>
> - Josh
>


   

RE: Moving 2.0 forward

[ashish singhi <as...@huawei.com>]

One question regarding upgrade: 
Do we expect the user to move to latest branch-1 release and then upgrade to 2.0 version ?

I think that should not be imposed on the user. 

Regards,
Ashish

-----Original Message-----
From: 张铎(Duo Zhang) [mailto:palomino219@gmail.com] 
Sent: 26 July 2017 12:10
To: dev@hbase.apache.org
Subject: Re: Moving 2.0 forward

I set HBASE-18169 as a Blocker because I found some critial problems on our current CPs. The semantics are broken. Although we are allowed to break CP in a major release, I think we need to provide the same ability(in another way).

2017-07-25 1:25 GMT+08:00 Josh Elser <el...@apache.org>:

>
>
> On 7/21/17 12:03 PM, Stack wrote:
>
>> Status update girls and boys!
>>
>> hbase-2.0.0-alpha1 went out June 22nd.
>>
>> alpha2 has been a bit slow to follow (holidays) though there has been 
>> steady progress closing out blockers and criticals by a bunch of you all.
>> The plan is for a release in the first week or so of August. It 
>> should be fully up on hbase-thirdparty using updated (and relocated) 
>> versions of netty, guava, and protobuf as well as a default deploy 
>> that has master-carrying-no-regions.
>>
>> alpha3 will follow soon after and will focus on making sure our 
>> user-facing APIs are clean (branch-1 compatible, no illicit 
>> removals/mods, and so on) and that basic upgrade 'works'.
>>
>> betas start in September?
>>
>> I've been keeping a rough general state here [1] (please update any 
>> section that is lagging actuality) but for details on what blockers 
>> and criticals remain, see the JIRA 2.0 view [2]. Recent 
>> issue-gardening has brought 2.0 into better focus. Feel free to 
>> review and punt items you think can wait till 3.0 or 2.1. If you want 
>> to pull in more stuff, please ask first.
>>
>
> Chia-Ping (I think? -- JIRA is being a pain) had asked on the 
> space-quota
> phase2 work (include size of hbase snapshots in a table's "quota 
> usage") if we should try to also include that work in 2.0.
>
> I like the idea of this also hitting 2.0 as it would make the feature 
> a bit more "real", but am obviously a little nervous (I have no reason 
> to be nervous though). I am pretty happy with the feature in terms of 
> how much it is covered via testing.
>
> https://issues.apache.org/jira/browse/HBASE-17748
>
>
> Thanks,
>> St.Ack
>>
>> 1.
>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> Eu_ktczrlKHK8N4SZzs/edit#
>> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>
>
> - Josh
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Wed, Jul 26, 2017 at 1:39 AM, 张铎(Duo Zhang) <pa...@gmail.com>
wrote:

> I set HBASE-18169 as a Blocker because I found some critial problems on our
> current CPs. The semantics are broken. Although we are allowed to break CP
> in a major release, I think we need to provide the same ability(in another
> way).
>
> Thank you. Agree. Chatting w/ Anoop this morning (offline), perhaps we can
huddle at hbasecon for an hour to clean this all up.

Thanks Duo,

S



> 2017-07-25 1:25 GMT+08:00 Josh Elser <el...@apache.org>:
>
> >
> >
> > On 7/21/17 12:03 PM, Stack wrote:
> >
> >> Status update girls and boys!
> >>
> >> hbase-2.0.0-alpha1 went out June 22nd.
> >>
> >> alpha2 has been a bit slow to follow (holidays) though there has been
> >> steady progress closing out blockers and criticals by a bunch of you
> all.
> >> The plan is for a release in the first week or so of August. It should
> be
> >> fully up on hbase-thirdparty using updated (and relocated) versions of
> >> netty, guava, and protobuf as well as a default deploy that has
> >> master-carrying-no-regions.
> >>
> >> alpha3 will follow soon after and will focus on making sure our
> >> user-facing
> >> APIs are clean (branch-1 compatible, no illicit removals/mods, and so
> on)
> >> and that basic upgrade 'works'.
> >>
> >> betas start in September?
> >>
> >> I've been keeping a rough general state here [1] (please update any
> >> section
> >> that is lagging actuality) but for details on what blockers and
> criticals
> >> remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought
> 2.0
> >> into better focus. Feel free to review and punt items you think can wait
> >> till 3.0 or 2.1. If you want to pull in more stuff, please ask first.
> >>
> >
> > Chia-Ping (I think? -- JIRA is being a pain) had asked on the space-quota
> > phase2 work (include size of hbase snapshots in a table's "quota usage")
> if
> > we should try to also include that work in 2.0.
> >
> > I like the idea of this also hitting 2.0 as it would make the feature a
> > bit more "real", but am obviously a little nervous (I have no reason to
> be
> > nervous though). I am pretty happy with the feature in terms of how much
> it
> > is covered via testing.
> >
> > https://issues.apache.org/jira/browse/HBASE-17748
> >
> >
> > Thanks,
> >> St.Ack
> >>
> >> 1.
> >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >> Eu_ktczrlKHK8N4SZzs/edit#
> >> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
> >>
> >
> > - Josh
> >
>

Re: Moving 2.0 forward

["张铎 (Duo Zhang)" <pa...@gmail.com>]

I set HBASE-18169 as a Blocker because I found some critial problems on our
current CPs. The semantics are broken. Although we are allowed to break CP
in a major release, I think we need to provide the same ability(in another
way).

2017-07-25 1:25 GMT+08:00 Josh Elser <el...@apache.org>:

>
>
> On 7/21/17 12:03 PM, Stack wrote:
>
>> Status update girls and boys!
>>
>> hbase-2.0.0-alpha1 went out June 22nd.
>>
>> alpha2 has been a bit slow to follow (holidays) though there has been
>> steady progress closing out blockers and criticals by a bunch of you all.
>> The plan is for a release in the first week or so of August. It should be
>> fully up on hbase-thirdparty using updated (and relocated) versions of
>> netty, guava, and protobuf as well as a default deploy that has
>> master-carrying-no-regions.
>>
>> alpha3 will follow soon after and will focus on making sure our
>> user-facing
>> APIs are clean (branch-1 compatible, no illicit removals/mods, and so on)
>> and that basic upgrade 'works'.
>>
>> betas start in September?
>>
>> I've been keeping a rough general state here [1] (please update any
>> section
>> that is lagging actuality) but for details on what blockers and criticals
>> remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought 2.0
>> into better focus. Feel free to review and punt items you think can wait
>> till 3.0 or 2.1. If you want to pull in more stuff, please ask first.
>>
>
> Chia-Ping (I think? -- JIRA is being a pain) had asked on the space-quota
> phase2 work (include size of hbase snapshots in a table's "quota usage") if
> we should try to also include that work in 2.0.
>
> I like the idea of this also hitting 2.0 as it would make the feature a
> bit more "real", but am obviously a little nervous (I have no reason to be
> nervous though). I am pretty happy with the feature in terms of how much it
> is covered via testing.
>
> https://issues.apache.org/jira/browse/HBASE-17748
>
>
> Thanks,
>> St.Ack
>>
>> 1.
>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> Eu_ktczrlKHK8N4SZzs/edit#
>> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>
>
> - Josh
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser <el...@apache.org> wrote:

> ...
>
> I like the idea of this also hitting 2.0 as it would make the feature a
> bit more "real", but am obviously a little nervous (I have no reason to be
> nervous though). I am pretty happy with the feature in terms of how much it
> is covered via testing.
>
> https://issues.apache.org/jira/browse/HBASE-17748
>
>
>
Sounds good to me. Whats involved? Backport? If so, +1 Josh.

Last think on space quota says that need doc too. See 'Space Quota' in
here:
https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
Does this little section need an update Josh?

Thanks,
S




> Thanks,
>> St.Ack
>>
>> 1.
>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> Eu_ktczrlKHK8N4SZzs/edit#
>> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>
>
> - Josh
>

Re: Moving 2.0 forward

[Josh Elser <el...@apache.org>]

On 7/21/17 12:03 PM, Stack wrote:
> Status update girls and boys!
> 
> hbase-2.0.0-alpha1 went out June 22nd.
> 
> alpha2 has been a bit slow to follow (holidays) though there has been
> steady progress closing out blockers and criticals by a bunch of you all.
> The plan is for a release in the first week or so of August. It should be
> fully up on hbase-thirdparty using updated (and relocated) versions of
> netty, guava, and protobuf as well as a default deploy that has
> master-carrying-no-regions.
> 
> alpha3 will follow soon after and will focus on making sure our user-facing
> APIs are clean (branch-1 compatible, no illicit removals/mods, and so on)
> and that basic upgrade 'works'.
> 
> betas start in September?
> 
> I've been keeping a rough general state here [1] (please update any section
> that is lagging actuality) but for details on what blockers and criticals
> remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought 2.0
> into better focus. Feel free to review and punt items you think can wait
> till 3.0 or 2.1. If you want to pull in more stuff, please ask first.

Chia-Ping (I think? -- JIRA is being a pain) had asked on the 
space-quota phase2 work (include size of hbase snapshots in a table's 
"quota usage") if we should try to also include that work in 2.0.

I like the idea of this also hitting 2.0 as it would make the feature a 
bit more "real", but am obviously a little nervous (I have no reason to 
be nervous though). I am pretty happy with the feature in terms of how 
much it is covered via testing.

https://issues.apache.org/jira/browse/HBASE-17748

> Thanks,
> St.Ack
> 
> 1.
> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#
> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188

- Josh

Re: Moving 2.0 forward

[Mike Drob <md...@apache.org>]

Yea, beta sounds good to me. Probably we'll find more things once we start
running ITBLL and friends for long periods, but that can maybe come later.
Existing tests should be cleaned up "soon".

On Fri, Jul 21, 2017 at 3:45 PM, Andrew Purtell <ap...@apache.org> wrote:

> Beta?
>
>
> On Fri, Jul 21, 2017 at 1:16 PM, Stack <st...@duboce.net> wrote:
>
> > On Fri, Jul 21, 2017 at 5:30 PM, Mike Drob <md...@apache.org> wrote:
> >
> > >
> > > One thing that is not clear to me from your summary and the linked
> > document
> > > is which release we can block on tests? Unit tests are in bad shape
> (but
> > > getting better) and I recently started looking at ITs which also need
> > some
> > > care.
> > >
> > >
> > Dunno. What you think boss? I'd say we can't release if tests are a mess.
> > Should we block on RC (as in alpha, beta, then RC)?  Or before then? i.e.
> > no beta unless all tests (unit and IT) are passing.
> >
> > For sure testing -- as in exercising new features, changed configs, and
> > just straight longevity under stress -- is badly wanting. I should write
> up
> > a matrix I suppose? Let me work on it.
> >
> > Thanks Mike,
> > S
> >
> >
> >
> >
> > > Mike
> > >
> > > On Fri, Jul 21, 2017 at 11:03 AM, Stack <st...@duboce.net> wrote:
> > >
> > > > Status update girls and boys!
> > > >
> > > > hbase-2.0.0-alpha1 went out June 22nd.
> > > >
> > > > alpha2 has been a bit slow to follow (holidays) though there has been
> > > > steady progress closing out blockers and criticals by a bunch of you
> > all.
> > > > The plan is for a release in the first week or so of August. It
> should
> > be
> > > > fully up on hbase-thirdparty using updated (and relocated) versions
> of
> > > > netty, guava, and protobuf as well as a default deploy that has
> > > > master-carrying-no-regions.
> > > >
> > > > alpha3 will follow soon after and will focus on making sure our
> > > user-facing
> > > > APIs are clean (branch-1 compatible, no illicit removals/mods, and so
> > on)
> > > > and that basic upgrade 'works'.
> > > >
> > > > betas start in September?
> > > >
> > > > I've been keeping a rough general state here [1] (please update any
> > > section
> > > > that is lagging actuality) but for details on what blockers and
> > criticals
> > > > remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought
> > 2.0
> > > > into better focus. Feel free to review and punt items you think can
> > wait
> > > > till 3.0 or 2.1. If you want to pull in more stuff, please ask first.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > >
> > > > 1.
> > > > https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > > > ktczrlKHK8N4SZzs/edit#
> > > > 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
> > > >
> > > >
> > > > On Mon, May 15, 2017 at 4:05 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > >
> > > > > On Mon, May 15, 2017 at 1:46 AM, 张铎(Duo Zhang) <
> > palomino219@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> HBASE-18037 is a new blocker. I'm currently working on it, will be
> > > > >> finished
> > > > >> soon I think.
> > > > >>
> > > > >> I made it a blocker then and added it to our hbase2 release doc
> [1]
> > > list
> > > > > as a blocker.
> > > > >
> > > > > Thanks,
> > > > > St.Ack
> > > > >
> > > > > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > > > > ktczrlKHK8N4SZzs/edit#
> > > > >
> > > > >
> > > > >> 2017-05-15 14:12 GMT+08:00 Stack <st...@duboce.net>:
> > > > >>
> > > > >>
> > > > >> > A month on. Status.
> > > > >> >
> > > > >> > I've been working on the HBASE-14614 branch cluster testing.
> > After a
> > > > >> load
> > > > >> > of fixing, the branch passes smaller test runs (an hour or so of
> > > ITBLL
> > > > >> up
> > > > >> > to 2B rows w/ killing monkeys). When I go larger, to a scale
> I've
> > > not
> > > > >> done
> > > > >> > in a while, I start to run into other interesting issues -- some
> > of
> > > > >> which
> > > > >> > are related to AMv2 (I'm fixing), but others are not (100G WALs
> > that
> > > > >> take
> > > > >> > ten minutes to split makes for interesting cascades when monkeys
> > > kill
> > > > >> > inside the ten minutes...). I intend to keep on with this larger
> > > scale
> > > > >> > testing since it is uncovering good stuff (especially when HDFS
> is
> > > dog
> > > > >> slow
> > > > >> > because of background replications) but my thinking is that I
> > should
> > > > be
> > > > >> > large scale testing branch-2, not just HBASE-14614. I think
> > > > HBASE-14614,
> > > > >> > the new AMv2, is good enough to merge to master these times.
> Given
> > > it
> > > > is
> > > > >> > the last blocker, once in, I'll cut the hbase2 branch.
> > > > >> >
> > > > >> > I'll start up a 'Merge HBASE-14614' DISCUSSION thread in the
> next
> > > day
> > > > >> or so
> > > > >> > (I need to fix some unit tests...).
> > > > >> >
> > > > >> > The AMv2 doc is still a work in progress but should give a gist
> on
> > > > >> where we
> > > > >> > are currently[1].  There is a bunch of todo still but seems
> > > tractable;
> > > > >> e.g.
> > > > >> > rolling upgrade, finish doc., and we don't have an HBCK since it
> > > needs
> > > > >> to
> > > > >> > be recast in light of how stuff now works but a redo on HBCK is
> > > > >> premature
> > > > >> > given we don't know failure types as yet (we just fix the
> problems
> > > as
> > > > >> they
> > > > >> > come up).
> > > > >> >
> > > > >> > St.Ack
> > > > >> > 1.
> > > > >> > https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > > > >> > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Thu, Apr 13, 2017 at 1:43 PM, Stack <st...@duboce.net>
> wrote:
> > > > >> >
> > > > >> > > Some status:
> > > > >> > >
> > > > >> > > AMv2 (HBASE-14614) is near to passing all tests caveat my
> > > disabling
> > > > of
> > > > >> > all
> > > > >> > > to-do w/ fsck (fsck needs revamp) and tests that expect that
> > they
> > > > can
> > > > >> > move
> > > > >> > > hbase;meta off master (AMv2 enforces this constraint; it is
> > > supposed
> > > > >> to
> > > > >> > be
> > > > >> > > enforced on AMv1 but meta-on-master is incompletely realized
> in
> > > AMv1
> > > > >> and
> > > > >> > > AMv2). A few other tests have been disabled for various
> reasons.
> > > See
> > > > >> [1]
> > > > >> > > for full list.
> > > > >> > >
> > > > >> > > There is a hefty list of TODOs still (Again see the messy doc
> > [1])
> > > > but
> > > > >> > the
> > > > >> > > only 'blocker', IMO, is community confidence in AMv2.
> Currently,
> > > > >> cluster
> > > > >> > > tests with chaos fail (new form of 'stuck' regions). Takes
> time
> > > > >> > > investigating.
> > > > >> > >
> > > > >> > > Will keep you all posted.
> > > > >> > > St.Ack
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > > > >> > > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell <
> > > > apurtell@apache.org>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > >> +1 on branching (yay!)
> > > > >> > >>
> > > > >> > >> I have EC2 resources for running ITBLL etc.
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net>
> > wrote:
> > > > >> > >>
> > > > >> > >> > Some notes on progress toward hbase2.
> > > > >> > >> >
> > > > >> > >> > Given that stability and performance are NOT emergent
> > behaviors
> > > > but
> > > > >> > >> rather
> > > > >> > >> > projects unto themselves, my thought is that we commit all
> > that
> > > > >> we've
> > > > >> > >> > agreed as core for hbase2 (see [1]), branch, and then work
> on
> > > > >> > >> stabilizing
> > > > >> > >> > and perf rather than do stabilize, commit, and then branch.
> > > What
> > > > >> this
> > > > >> > >> means
> > > > >> > >> > in practice is that for features like Inmemory Compaction,
> we
> > > > >> commit
> > > > >> > it
> > > > >> > >> > defaulted 'on' ("BASIC" mode) which is what we want in
> > hbase2.
> > > > >> Should
> > > > >> > it
> > > > >> > >> > prove problematic under test, we disable it before release.
> > > > >> > >> >
> > > > >> > >> > Are folks good w/ this mode? I ask because, in a few issues
> > > there
> > > > >> are
> > > > >> > >> > requests for proof that a master feature is 'stable' before
> > > > commit.
> > > > >> > >> This is
> > > > >> > >> > normally a healthy request only in master's case, it is
> hard
> > to
> > > > >> > >> demonstrate
> > > > >> > >> > stability given its current state.
> > > > >> > >> >
> > > > >> > >> > Other outstanding issues such as decisions about whether
> > master
> > > > >> hosts
> > > > >> > >> > system tables only (by default), I'm thinking, we can work
> > out
> > > > post
> > > > >> > >> branch
> > > > >> > >> > in alpha/betas before release.
> > > > >> > >> >
> > > > >> > >> > The awkward item is the long-pole Assignment Manager. This
> is
> > > an
> > > > >> > >> > all-or-nothing affair. Here we are switching in a new
> Master
> > > > core.
> > > > >> > >> While I
> > > > >> > >> > think it fine that AMv2 is incomplete come branch time,
> those
> > > of
> > > > us
> > > > >> > >> working
> > > > >> > >> > on the new AM still need to demonstrate to you all that it
> > > > >> basically
> > > > >> > >> > viable.
> > > > >> > >> >
> > > > >> > >> > The point-of-no-return is commit of the patch in
> HBASE-14614.
> > > > >> > >> HBASE-14614
> > > > >> > >> > (AMv2) is coming close to passing all unit tests. We'll
> spend
> > > > some
> > > > >> > time
> > > > >> > >> > running it on a cluster to make sure it fundamentally sound
> > and
> > > > >> will
> > > > >> > >> report
> > > > >> > >> > back on our experience. There has been an ask for some dev
> > doc
> > > > and
> > > > >> > >> > low-levels on how it works (in progress). Let satisfaction
> of
> > > > these
> > > > >> > >> > requests be blockers on commit. We'll put the HBASE-14614
> > > commit
> > > > up
> > > > >> > for
> > > > >> > >> a
> > > > >> > >> > vote on dev list given its import.
> > > > >> > >> >
> > > > >> > >> > Branch will happen after HBASE-14614 goes in (or its
> > rejection)
> > > > >> with
> > > > >> > our
> > > > >> > >> > first alpha soon after. Its looking like a week or two at
> > least
> > > > >> given
> > > > >> > >> how
> > > > >> > >> > things have been going up to this.
> > > > >> > >> >
> > > > >> > >> > I intend to start in on hbase2 stability/perf projects
> after
> > we
> > > > >> > branch.
> > > > >> > >> >
> > > > >> > >> > Interested in any thoughts you all might have on the above
> > > (Would
> > > > >> also
> > > > >> > >> > appreciate updates on state in [1] if you are a feature
> > owner).
> > > > >> > >> >
> > > > >> > >> > Thanks,
> > > > >> > >> > St.Ack
> > > > >> > >> >
> > > > >> > >> > 1. https://docs.google.com/document/d/
> 1WCsVlnHjJeKUcl7wHwqb4
> > > > >> > >> > z9iEu_ktczrlKHK8N4SZzs/edit#
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <
> > elserj@apache.org
> > > >
> > > > >> > wrote:
> > > > >> > >> >
> > > > >> > >> > >
> > > > >> > >> > > Stack wrote:
> > > > >> > >> > >
> > > > >> > >> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<
> > > elserj@apache.org>
> > > > >> > >> wrote:
> > > > >> > >> > >>
> > > > >> > >> > >> Thanks for pulling in the FS Quotas work, Stack. I'm
> > trying
> > > to
> > > > >> > cross
> > > > >> > >> the
> > > > >> > >> > >>> last T's and dot the last I's.
> > > > >> > >> > >>>
> > > > >> > >> > >>> The biggest thing I know I need to do still is to
> write a
> > > new
> > > > >> > >> chapter
> > > > >> > >> > to
> > > > >> > >> > >>> the book. After that, I'd start entertaining larger
> > > > >> > >> reviews/discussions
> > > > >> > >> > >>> to
> > > > >> > >> > >>> merge the feature into master. Anyone with free time
> > > > (giggles)
> > > > >> is
> > > > >> > >> more
> > > > >> > >> > >>> than
> > > > >> > >> > >>> welcome to start perusing :)
> > > > >> > >> > >>>
> > > > >> > >> > >>>
> > > > >> > >> > >>> Out of interest, this could come in after 2.0 Josh? Any
> > 2.0
> > > > >> > specific
> > > > >> > >> > >> needs
> > > > >> > >> > >> to make this work?
> > > > >> > >> > >>
> > > > >> > >> > >> Meantime, updated the 2.0 doc 1.
> > > > >> > >> > >>
> > > > >> > >> > >> Thanks Josh,
> > > > >> > >> > >> St.Ack
> > > > >> > >> > >>
> > > > >> > >> > >> 1.
> > > > >> > >> > >> https://docs.google.com/document/d/
> > > 1WCsVlnHjJeKUcl7wHwqb4z9i
> > > > >> > >> > >> Eu_ktczrlKHK8N4SZzs/edit#
> > > > >> > >> > >>
> > > > >> > >> > >>
> > > > >> > >> > > Nope, no need to block 2.0 on this one (given the other,
> > > > related
> > > > >> > >> > chatter).
> > > > >> > >> > > Would be nice to get it in, but I completely understand
> if
> > it
> > > > >> slips
> > > > >> > :)
> > > > >> > >> > >
> > > > >> > >> > > Thanks for updating the doc for me!
> > > > >> > >> > >
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> --
> > > > >> > >> Best regards,
> > > > >> > >>
> > > > >> > >>    - Andy
> > > > >> > >>
> > > > >> > >> If you are given a choice, you believe you have acted
> freely. -
> > > > >> Raymond
> > > > >> > >> Teller (via Peter Watts)
> > > > >> > >>
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>    - A23, Crosstalk
>

Re: Moving 2.0 forward

[Andrew Purtell <ap...@apache.org>]

Beta?


On Fri, Jul 21, 2017 at 1:16 PM, Stack <st...@duboce.net> wrote:

> On Fri, Jul 21, 2017 at 5:30 PM, Mike Drob <md...@apache.org> wrote:
>
> >
> > One thing that is not clear to me from your summary and the linked
> document
> > is which release we can block on tests? Unit tests are in bad shape (but
> > getting better) and I recently started looking at ITs which also need
> some
> > care.
> >
> >
> Dunno. What you think boss? I'd say we can't release if tests are a mess.
> Should we block on RC (as in alpha, beta, then RC)?  Or before then? i.e.
> no beta unless all tests (unit and IT) are passing.
>
> For sure testing -- as in exercising new features, changed configs, and
> just straight longevity under stress -- is badly wanting. I should write up
> a matrix I suppose? Let me work on it.
>
> Thanks Mike,
> S
>
>
>
>
> > Mike
> >
> > On Fri, Jul 21, 2017 at 11:03 AM, Stack <st...@duboce.net> wrote:
> >
> > > Status update girls and boys!
> > >
> > > hbase-2.0.0-alpha1 went out June 22nd.
> > >
> > > alpha2 has been a bit slow to follow (holidays) though there has been
> > > steady progress closing out blockers and criticals by a bunch of you
> all.
> > > The plan is for a release in the first week or so of August. It should
> be
> > > fully up on hbase-thirdparty using updated (and relocated) versions of
> > > netty, guava, and protobuf as well as a default deploy that has
> > > master-carrying-no-regions.
> > >
> > > alpha3 will follow soon after and will focus on making sure our
> > user-facing
> > > APIs are clean (branch-1 compatible, no illicit removals/mods, and so
> on)
> > > and that basic upgrade 'works'.
> > >
> > > betas start in September?
> > >
> > > I've been keeping a rough general state here [1] (please update any
> > section
> > > that is lagging actuality) but for details on what blockers and
> criticals
> > > remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought
> 2.0
> > > into better focus. Feel free to review and punt items you think can
> wait
> > > till 3.0 or 2.1. If you want to pull in more stuff, please ask first.
> > >
> > > Thanks,
> > > St.Ack
> > >
> > > 1.
> > > https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > > ktczrlKHK8N4SZzs/edit#
> > > 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
> > >
> > >
> > > On Mon, May 15, 2017 at 4:05 PM, Stack <st...@duboce.net> wrote:
> > >
> > > >
> > > > On Mon, May 15, 2017 at 1:46 AM, 张铎(Duo Zhang) <
> palomino219@gmail.com>
> > > > wrote:
> > > >
> > > >> HBASE-18037 is a new blocker. I'm currently working on it, will be
> > > >> finished
> > > >> soon I think.
> > > >>
> > > >> I made it a blocker then and added it to our hbase2 release doc [1]
> > list
> > > > as a blocker.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > >
> > > > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > > > ktczrlKHK8N4SZzs/edit#
> > > >
> > > >
> > > >> 2017-05-15 14:12 GMT+08:00 Stack <st...@duboce.net>:
> > > >>
> > > >>
> > > >> > A month on. Status.
> > > >> >
> > > >> > I've been working on the HBASE-14614 branch cluster testing.
> After a
> > > >> load
> > > >> > of fixing, the branch passes smaller test runs (an hour or so of
> > ITBLL
> > > >> up
> > > >> > to 2B rows w/ killing monkeys). When I go larger, to a scale I've
> > not
> > > >> done
> > > >> > in a while, I start to run into other interesting issues -- some
> of
> > > >> which
> > > >> > are related to AMv2 (I'm fixing), but others are not (100G WALs
> that
> > > >> take
> > > >> > ten minutes to split makes for interesting cascades when monkeys
> > kill
> > > >> > inside the ten minutes...). I intend to keep on with this larger
> > scale
> > > >> > testing since it is uncovering good stuff (especially when HDFS is
> > dog
> > > >> slow
> > > >> > because of background replications) but my thinking is that I
> should
> > > be
> > > >> > large scale testing branch-2, not just HBASE-14614. I think
> > > HBASE-14614,
> > > >> > the new AMv2, is good enough to merge to master these times. Given
> > it
> > > is
> > > >> > the last blocker, once in, I'll cut the hbase2 branch.
> > > >> >
> > > >> > I'll start up a 'Merge HBASE-14614' DISCUSSION thread in the next
> > day
> > > >> or so
> > > >> > (I need to fix some unit tests...).
> > > >> >
> > > >> > The AMv2 doc is still a work in progress but should give a gist on
> > > >> where we
> > > >> > are currently[1].  There is a bunch of todo still but seems
> > tractable;
> > > >> e.g.
> > > >> > rolling upgrade, finish doc., and we don't have an HBCK since it
> > needs
> > > >> to
> > > >> > be recast in light of how stuff now works but a redo on HBCK is
> > > >> premature
> > > >> > given we don't know failure types as yet (we just fix the problems
> > as
> > > >> they
> > > >> > come up).
> > > >> >
> > > >> > St.Ack
> > > >> > 1.
> > > >> > https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > > >> > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Thu, Apr 13, 2017 at 1:43 PM, Stack <st...@duboce.net> wrote:
> > > >> >
> > > >> > > Some status:
> > > >> > >
> > > >> > > AMv2 (HBASE-14614) is near to passing all tests caveat my
> > disabling
> > > of
> > > >> > all
> > > >> > > to-do w/ fsck (fsck needs revamp) and tests that expect that
> they
> > > can
> > > >> > move
> > > >> > > hbase;meta off master (AMv2 enforces this constraint; it is
> > supposed
> > > >> to
> > > >> > be
> > > >> > > enforced on AMv1 but meta-on-master is incompletely realized in
> > AMv1
> > > >> and
> > > >> > > AMv2). A few other tests have been disabled for various reasons.
> > See
> > > >> [1]
> > > >> > > for full list.
> > > >> > >
> > > >> > > There is a hefty list of TODOs still (Again see the messy doc
> [1])
> > > but
> > > >> > the
> > > >> > > only 'blocker', IMO, is community confidence in AMv2. Currently,
> > > >> cluster
> > > >> > > tests with chaos fail (new form of 'stuck' regions). Takes time
> > > >> > > investigating.
> > > >> > >
> > > >> > > Will keep you all posted.
> > > >> > > St.Ack
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > > >> > > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell <
> > > apurtell@apache.org>
> > > >> > > wrote:
> > > >> > >
> > > >> > >> +1 on branching (yay!)
> > > >> > >>
> > > >> > >> I have EC2 resources for running ITBLL etc.
> > > >> > >>
> > > >> > >>
> > > >> > >> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net>
> wrote:
> > > >> > >>
> > > >> > >> > Some notes on progress toward hbase2.
> > > >> > >> >
> > > >> > >> > Given that stability and performance are NOT emergent
> behaviors
> > > but
> > > >> > >> rather
> > > >> > >> > projects unto themselves, my thought is that we commit all
> that
> > > >> we've
> > > >> > >> > agreed as core for hbase2 (see [1]), branch, and then work on
> > > >> > >> stabilizing
> > > >> > >> > and perf rather than do stabilize, commit, and then branch.
> > What
> > > >> this
> > > >> > >> means
> > > >> > >> > in practice is that for features like Inmemory Compaction, we
> > > >> commit
> > > >> > it
> > > >> > >> > defaulted 'on' ("BASIC" mode) which is what we want in
> hbase2.
> > > >> Should
> > > >> > it
> > > >> > >> > prove problematic under test, we disable it before release.
> > > >> > >> >
> > > >> > >> > Are folks good w/ this mode? I ask because, in a few issues
> > there
> > > >> are
> > > >> > >> > requests for proof that a master feature is 'stable' before
> > > commit.
> > > >> > >> This is
> > > >> > >> > normally a healthy request only in master's case, it is hard
> to
> > > >> > >> demonstrate
> > > >> > >> > stability given its current state.
> > > >> > >> >
> > > >> > >> > Other outstanding issues such as decisions about whether
> master
> > > >> hosts
> > > >> > >> > system tables only (by default), I'm thinking, we can work
> out
> > > post
> > > >> > >> branch
> > > >> > >> > in alpha/betas before release.
> > > >> > >> >
> > > >> > >> > The awkward item is the long-pole Assignment Manager. This is
> > an
> > > >> > >> > all-or-nothing affair. Here we are switching in a new Master
> > > core.
> > > >> > >> While I
> > > >> > >> > think it fine that AMv2 is incomplete come branch time, those
> > of
> > > us
> > > >> > >> working
> > > >> > >> > on the new AM still need to demonstrate to you all that it
> > > >> basically
> > > >> > >> > viable.
> > > >> > >> >
> > > >> > >> > The point-of-no-return is commit of the patch in HBASE-14614.
> > > >> > >> HBASE-14614
> > > >> > >> > (AMv2) is coming close to passing all unit tests. We'll spend
> > > some
> > > >> > time
> > > >> > >> > running it on a cluster to make sure it fundamentally sound
> and
> > > >> will
> > > >> > >> report
> > > >> > >> > back on our experience. There has been an ask for some dev
> doc
> > > and
> > > >> > >> > low-levels on how it works (in progress). Let satisfaction of
> > > these
> > > >> > >> > requests be blockers on commit. We'll put the HBASE-14614
> > commit
> > > up
> > > >> > for
> > > >> > >> a
> > > >> > >> > vote on dev list given its import.
> > > >> > >> >
> > > >> > >> > Branch will happen after HBASE-14614 goes in (or its
> rejection)
> > > >> with
> > > >> > our
> > > >> > >> > first alpha soon after. Its looking like a week or two at
> least
> > > >> given
> > > >> > >> how
> > > >> > >> > things have been going up to this.
> > > >> > >> >
> > > >> > >> > I intend to start in on hbase2 stability/perf projects after
> we
> > > >> > branch.
> > > >> > >> >
> > > >> > >> > Interested in any thoughts you all might have on the above
> > (Would
> > > >> also
> > > >> > >> > appreciate updates on state in [1] if you are a feature
> owner).
> > > >> > >> >
> > > >> > >> > Thanks,
> > > >> > >> > St.Ack
> > > >> > >> >
> > > >> > >> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> > > >> > >> > z9iEu_ktczrlKHK8N4SZzs/edit#
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <
> elserj@apache.org
> > >
> > > >> > wrote:
> > > >> > >> >
> > > >> > >> > >
> > > >> > >> > > Stack wrote:
> > > >> > >> > >
> > > >> > >> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<
> > elserj@apache.org>
> > > >> > >> wrote:
> > > >> > >> > >>
> > > >> > >> > >> Thanks for pulling in the FS Quotas work, Stack. I'm
> trying
> > to
> > > >> > cross
> > > >> > >> the
> > > >> > >> > >>> last T's and dot the last I's.
> > > >> > >> > >>>
> > > >> > >> > >>> The biggest thing I know I need to do still is to write a
> > new
> > > >> > >> chapter
> > > >> > >> > to
> > > >> > >> > >>> the book. After that, I'd start entertaining larger
> > > >> > >> reviews/discussions
> > > >> > >> > >>> to
> > > >> > >> > >>> merge the feature into master. Anyone with free time
> > > (giggles)
> > > >> is
> > > >> > >> more
> > > >> > >> > >>> than
> > > >> > >> > >>> welcome to start perusing :)
> > > >> > >> > >>>
> > > >> > >> > >>>
> > > >> > >> > >>> Out of interest, this could come in after 2.0 Josh? Any
> 2.0
> > > >> > specific
> > > >> > >> > >> needs
> > > >> > >> > >> to make this work?
> > > >> > >> > >>
> > > >> > >> > >> Meantime, updated the 2.0 doc 1.
> > > >> > >> > >>
> > > >> > >> > >> Thanks Josh,
> > > >> > >> > >> St.Ack
> > > >> > >> > >>
> > > >> > >> > >> 1.
> > > >> > >> > >> https://docs.google.com/document/d/
> > 1WCsVlnHjJeKUcl7wHwqb4z9i
> > > >> > >> > >> Eu_ktczrlKHK8N4SZzs/edit#
> > > >> > >> > >>
> > > >> > >> > >>
> > > >> > >> > > Nope, no need to block 2.0 on this one (given the other,
> > > related
> > > >> > >> > chatter).
> > > >> > >> > > Would be nice to get it in, but I completely understand if
> it
> > > >> slips
> > > >> > :)
> > > >> > >> > >
> > > >> > >> > > Thanks for updating the doc for me!
> > > >> > >> > >
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >> --
> > > >> > >> Best regards,
> > > >> > >>
> > > >> > >>    - Andy
> > > >> > >>
> > > >> > >> If you are given a choice, you believe you have acted freely. -
> > > >> Raymond
> > > >> > >> Teller (via Peter Watts)
> > > >> > >>
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>



-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Fri, Jul 21, 2017 at 5:30 PM, Mike Drob <md...@apache.org> wrote:

>
> One thing that is not clear to me from your summary and the linked document
> is which release we can block on tests? Unit tests are in bad shape (but
> getting better) and I recently started looking at ITs which also need some
> care.
>
>
Dunno. What you think boss? I'd say we can't release if tests are a mess.
Should we block on RC (as in alpha, beta, then RC)?  Or before then? i.e.
no beta unless all tests (unit and IT) are passing.

For sure testing -- as in exercising new features, changed configs, and
just straight longevity under stress -- is badly wanting. I should write up
a matrix I suppose? Let me work on it.

Thanks Mike,
S




> Mike
>
> On Fri, Jul 21, 2017 at 11:03 AM, Stack <st...@duboce.net> wrote:
>
> > Status update girls and boys!
> >
> > hbase-2.0.0-alpha1 went out June 22nd.
> >
> > alpha2 has been a bit slow to follow (holidays) though there has been
> > steady progress closing out blockers and criticals by a bunch of you all.
> > The plan is for a release in the first week or so of August. It should be
> > fully up on hbase-thirdparty using updated (and relocated) versions of
> > netty, guava, and protobuf as well as a default deploy that has
> > master-carrying-no-regions.
> >
> > alpha3 will follow soon after and will focus on making sure our
> user-facing
> > APIs are clean (branch-1 compatible, no illicit removals/mods, and so on)
> > and that basic upgrade 'works'.
> >
> > betas start in September?
> >
> > I've been keeping a rough general state here [1] (please update any
> section
> > that is lagging actuality) but for details on what blockers and criticals
> > remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought 2.0
> > into better focus. Feel free to review and punt items you think can wait
> > till 3.0 or 2.1. If you want to pull in more stuff, please ask first.
> >
> > Thanks,
> > St.Ack
> >
> > 1.
> > https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > ktczrlKHK8N4SZzs/edit#
> > 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
> >
> >
> > On Mon, May 15, 2017 at 4:05 PM, Stack <st...@duboce.net> wrote:
> >
> > >
> > > On Mon, May 15, 2017 at 1:46 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> > > wrote:
> > >
> > >> HBASE-18037 is a new blocker. I'm currently working on it, will be
> > >> finished
> > >> soon I think.
> > >>
> > >> I made it a blocker then and added it to our hbase2 release doc [1]
> list
> > > as a blocker.
> > >
> > > Thanks,
> > > St.Ack
> > >
> > > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > > ktczrlKHK8N4SZzs/edit#
> > >
> > >
> > >> 2017-05-15 14:12 GMT+08:00 Stack <st...@duboce.net>:
> > >>
> > >>
> > >> > A month on. Status.
> > >> >
> > >> > I've been working on the HBASE-14614 branch cluster testing. After a
> > >> load
> > >> > of fixing, the branch passes smaller test runs (an hour or so of
> ITBLL
> > >> up
> > >> > to 2B rows w/ killing monkeys). When I go larger, to a scale I've
> not
> > >> done
> > >> > in a while, I start to run into other interesting issues -- some of
> > >> which
> > >> > are related to AMv2 (I'm fixing), but others are not (100G WALs that
> > >> take
> > >> > ten minutes to split makes for interesting cascades when monkeys
> kill
> > >> > inside the ten minutes...). I intend to keep on with this larger
> scale
> > >> > testing since it is uncovering good stuff (especially when HDFS is
> dog
> > >> slow
> > >> > because of background replications) but my thinking is that I should
> > be
> > >> > large scale testing branch-2, not just HBASE-14614. I think
> > HBASE-14614,
> > >> > the new AMv2, is good enough to merge to master these times. Given
> it
> > is
> > >> > the last blocker, once in, I'll cut the hbase2 branch.
> > >> >
> > >> > I'll start up a 'Merge HBASE-14614' DISCUSSION thread in the next
> day
> > >> or so
> > >> > (I need to fix some unit tests...).
> > >> >
> > >> > The AMv2 doc is still a work in progress but should give a gist on
> > >> where we
> > >> > are currently[1].  There is a bunch of todo still but seems
> tractable;
> > >> e.g.
> > >> > rolling upgrade, finish doc., and we don't have an HBCK since it
> needs
> > >> to
> > >> > be recast in light of how stuff now works but a redo on HBCK is
> > >> premature
> > >> > given we don't know failure types as yet (we just fix the problems
> as
> > >> they
> > >> > come up).
> > >> >
> > >> > St.Ack
> > >> > 1.
> > >> > https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > >> > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
> > >> >
> > >> >
> > >> >
> > >> > On Thu, Apr 13, 2017 at 1:43 PM, Stack <st...@duboce.net> wrote:
> > >> >
> > >> > > Some status:
> > >> > >
> > >> > > AMv2 (HBASE-14614) is near to passing all tests caveat my
> disabling
> > of
> > >> > all
> > >> > > to-do w/ fsck (fsck needs revamp) and tests that expect that they
> > can
> > >> > move
> > >> > > hbase;meta off master (AMv2 enforces this constraint; it is
> supposed
> > >> to
> > >> > be
> > >> > > enforced on AMv1 but meta-on-master is incompletely realized in
> AMv1
> > >> and
> > >> > > AMv2). A few other tests have been disabled for various reasons.
> See
> > >> [1]
> > >> > > for full list.
> > >> > >
> > >> > > There is a hefty list of TODOs still (Again see the messy doc [1])
> > but
> > >> > the
> > >> > > only 'blocker', IMO, is community confidence in AMv2. Currently,
> > >> cluster
> > >> > > tests with chaos fail (new form of 'stuck' regions). Takes time
> > >> > > investigating.
> > >> > >
> > >> > > Will keep you all posted.
> > >> > > St.Ack
> > >> > >
> > >> > >
> > >> > >
> > >> > > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > >> > > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell <
> > apurtell@apache.org>
> > >> > > wrote:
> > >> > >
> > >> > >> +1 on branching (yay!)
> > >> > >>
> > >> > >> I have EC2 resources for running ITBLL etc.
> > >> > >>
> > >> > >>
> > >> > >> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> wrote:
> > >> > >>
> > >> > >> > Some notes on progress toward hbase2.
> > >> > >> >
> > >> > >> > Given that stability and performance are NOT emergent behaviors
> > but
> > >> > >> rather
> > >> > >> > projects unto themselves, my thought is that we commit all that
> > >> we've
> > >> > >> > agreed as core for hbase2 (see [1]), branch, and then work on
> > >> > >> stabilizing
> > >> > >> > and perf rather than do stabilize, commit, and then branch.
> What
> > >> this
> > >> > >> means
> > >> > >> > in practice is that for features like Inmemory Compaction, we
> > >> commit
> > >> > it
> > >> > >> > defaulted 'on' ("BASIC" mode) which is what we want in hbase2.
> > >> Should
> > >> > it
> > >> > >> > prove problematic under test, we disable it before release.
> > >> > >> >
> > >> > >> > Are folks good w/ this mode? I ask because, in a few issues
> there
> > >> are
> > >> > >> > requests for proof that a master feature is 'stable' before
> > commit.
> > >> > >> This is
> > >> > >> > normally a healthy request only in master's case, it is hard to
> > >> > >> demonstrate
> > >> > >> > stability given its current state.
> > >> > >> >
> > >> > >> > Other outstanding issues such as decisions about whether master
> > >> hosts
> > >> > >> > system tables only (by default), I'm thinking, we can work out
> > post
> > >> > >> branch
> > >> > >> > in alpha/betas before release.
> > >> > >> >
> > >> > >> > The awkward item is the long-pole Assignment Manager. This is
> an
> > >> > >> > all-or-nothing affair. Here we are switching in a new Master
> > core.
> > >> > >> While I
> > >> > >> > think it fine that AMv2 is incomplete come branch time, those
> of
> > us
> > >> > >> working
> > >> > >> > on the new AM still need to demonstrate to you all that it
> > >> basically
> > >> > >> > viable.
> > >> > >> >
> > >> > >> > The point-of-no-return is commit of the patch in HBASE-14614.
> > >> > >> HBASE-14614
> > >> > >> > (AMv2) is coming close to passing all unit tests. We'll spend
> > some
> > >> > time
> > >> > >> > running it on a cluster to make sure it fundamentally sound and
> > >> will
> > >> > >> report
> > >> > >> > back on our experience. There has been an ask for some dev doc
> > and
> > >> > >> > low-levels on how it works (in progress). Let satisfaction of
> > these
> > >> > >> > requests be blockers on commit. We'll put the HBASE-14614
> commit
> > up
> > >> > for
> > >> > >> a
> > >> > >> > vote on dev list given its import.
> > >> > >> >
> > >> > >> > Branch will happen after HBASE-14614 goes in (or its rejection)
> > >> with
> > >> > our
> > >> > >> > first alpha soon after. Its looking like a week or two at least
> > >> given
> > >> > >> how
> > >> > >> > things have been going up to this.
> > >> > >> >
> > >> > >> > I intend to start in on hbase2 stability/perf projects after we
> > >> > branch.
> > >> > >> >
> > >> > >> > Interested in any thoughts you all might have on the above
> (Would
> > >> also
> > >> > >> > appreciate updates on state in [1] if you are a feature owner).
> > >> > >> >
> > >> > >> > Thanks,
> > >> > >> > St.Ack
> > >> > >> >
> > >> > >> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> > >> > >> > z9iEu_ktczrlKHK8N4SZzs/edit#
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <elserj@apache.org
> >
> > >> > wrote:
> > >> > >> >
> > >> > >> > >
> > >> > >> > > Stack wrote:
> > >> > >> > >
> > >> > >> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<
> elserj@apache.org>
> > >> > >> wrote:
> > >> > >> > >>
> > >> > >> > >> Thanks for pulling in the FS Quotas work, Stack. I'm trying
> to
> > >> > cross
> > >> > >> the
> > >> > >> > >>> last T's and dot the last I's.
> > >> > >> > >>>
> > >> > >> > >>> The biggest thing I know I need to do still is to write a
> new
> > >> > >> chapter
> > >> > >> > to
> > >> > >> > >>> the book. After that, I'd start entertaining larger
> > >> > >> reviews/discussions
> > >> > >> > >>> to
> > >> > >> > >>> merge the feature into master. Anyone with free time
> > (giggles)
> > >> is
> > >> > >> more
> > >> > >> > >>> than
> > >> > >> > >>> welcome to start perusing :)
> > >> > >> > >>>
> > >> > >> > >>>
> > >> > >> > >>> Out of interest, this could come in after 2.0 Josh? Any 2.0
> > >> > specific
> > >> > >> > >> needs
> > >> > >> > >> to make this work?
> > >> > >> > >>
> > >> > >> > >> Meantime, updated the 2.0 doc 1.
> > >> > >> > >>
> > >> > >> > >> Thanks Josh,
> > >> > >> > >> St.Ack
> > >> > >> > >>
> > >> > >> > >> 1.
> > >> > >> > >> https://docs.google.com/document/d/
> 1WCsVlnHjJeKUcl7wHwqb4z9i
> > >> > >> > >> Eu_ktczrlKHK8N4SZzs/edit#
> > >> > >> > >>
> > >> > >> > >>
> > >> > >> > > Nope, no need to block 2.0 on this one (given the other,
> > related
> > >> > >> > chatter).
> > >> > >> > > Would be nice to get it in, but I completely understand if it
> > >> slips
> > >> > :)
> > >> > >> > >
> > >> > >> > > Thanks for updating the doc for me!
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >> --
> > >> > >> Best regards,
> > >> > >>
> > >> > >>    - Andy
> > >> > >>
> > >> > >> If you are given a choice, you believe you have acted freely. -
> > >> Raymond
> > >> > >> Teller (via Peter Watts)
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: Moving 2.0 forward

[Mike Drob <md...@apache.org>]

Thanks for tracking this, Stack!

One thing that is not clear to me from your summary and the linked document
is which release we can block on tests? Unit tests are in bad shape (but
getting better) and I recently started looking at ITs which also need some
care.

Mike

On Fri, Jul 21, 2017 at 11:03 AM, Stack <st...@duboce.net> wrote:

> Status update girls and boys!
>
> hbase-2.0.0-alpha1 went out June 22nd.
>
> alpha2 has been a bit slow to follow (holidays) though there has been
> steady progress closing out blockers and criticals by a bunch of you all.
> The plan is for a release in the first week or so of August. It should be
> fully up on hbase-thirdparty using updated (and relocated) versions of
> netty, guava, and protobuf as well as a default deploy that has
> master-carrying-no-regions.
>
> alpha3 will follow soon after and will focus on making sure our user-facing
> APIs are clean (branch-1 compatible, no illicit removals/mods, and so on)
> and that basic upgrade 'works'.
>
> betas start in September?
>
> I've been keeping a rough general state here [1] (please update any section
> that is lagging actuality) but for details on what blockers and criticals
> remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought 2.0
> into better focus. Feel free to review and punt items you think can wait
> till 3.0 or 2.1. If you want to pull in more stuff, please ask first.
>
> Thanks,
> St.Ack
>
> 1.
> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> ktczrlKHK8N4SZzs/edit#
> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
>
>
> On Mon, May 15, 2017 at 4:05 PM, Stack <st...@duboce.net> wrote:
>
> >
> > On Mon, May 15, 2017 at 1:46 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> > wrote:
> >
> >> HBASE-18037 is a new blocker. I'm currently working on it, will be
> >> finished
> >> soon I think.
> >>
> >> I made it a blocker then and added it to our hbase2 release doc [1] list
> > as a blocker.
> >
> > Thanks,
> > St.Ack
> >
> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > ktczrlKHK8N4SZzs/edit#
> >
> >
> >> 2017-05-15 14:12 GMT+08:00 Stack <st...@duboce.net>:
> >>
> >>
> >> > A month on. Status.
> >> >
> >> > I've been working on the HBASE-14614 branch cluster testing. After a
> >> load
> >> > of fixing, the branch passes smaller test runs (an hour or so of ITBLL
> >> up
> >> > to 2B rows w/ killing monkeys). When I go larger, to a scale I've not
> >> done
> >> > in a while, I start to run into other interesting issues -- some of
> >> which
> >> > are related to AMv2 (I'm fixing), but others are not (100G WALs that
> >> take
> >> > ten minutes to split makes for interesting cascades when monkeys kill
> >> > inside the ten minutes...). I intend to keep on with this larger scale
> >> > testing since it is uncovering good stuff (especially when HDFS is dog
> >> slow
> >> > because of background replications) but my thinking is that I should
> be
> >> > large scale testing branch-2, not just HBASE-14614. I think
> HBASE-14614,
> >> > the new AMv2, is good enough to merge to master these times. Given it
> is
> >> > the last blocker, once in, I'll cut the hbase2 branch.
> >> >
> >> > I'll start up a 'Merge HBASE-14614' DISCUSSION thread in the next day
> >> or so
> >> > (I need to fix some unit tests...).
> >> >
> >> > The AMv2 doc is still a work in progress but should give a gist on
> >> where we
> >> > are currently[1].  There is a bunch of todo still but seems tractable;
> >> e.g.
> >> > rolling upgrade, finish doc., and we don't have an HBCK since it needs
> >> to
> >> > be recast in light of how stuff now works but a redo on HBCK is
> >> premature
> >> > given we don't know failure types as yet (we just fix the problems as
> >> they
> >> > come up).
> >> >
> >> > St.Ack
> >> > 1.
> >> > https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> >> > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
> >> >
> >> >
> >> >
> >> > On Thu, Apr 13, 2017 at 1:43 PM, Stack <st...@duboce.net> wrote:
> >> >
> >> > > Some status:
> >> > >
> >> > > AMv2 (HBASE-14614) is near to passing all tests caveat my disabling
> of
> >> > all
> >> > > to-do w/ fsck (fsck needs revamp) and tests that expect that they
> can
> >> > move
> >> > > hbase;meta off master (AMv2 enforces this constraint; it is supposed
> >> to
> >> > be
> >> > > enforced on AMv1 but meta-on-master is incompletely realized in AMv1
> >> and
> >> > > AMv2). A few other tests have been disabled for various reasons. See
> >> [1]
> >> > > for full list.
> >> > >
> >> > > There is a hefty list of TODOs still (Again see the messy doc [1])
> but
> >> > the
> >> > > only 'blocker', IMO, is community confidence in AMv2. Currently,
> >> cluster
> >> > > tests with chaos fail (new form of 'stuck' regions). Takes time
> >> > > investigating.
> >> > >
> >> > > Will keep you all posted.
> >> > > St.Ack
> >> > >
> >> > >
> >> > >
> >> > > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> >> > > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod
> >> > >
> >> > >
> >> > >
> >> > > On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell <
> apurtell@apache.org>
> >> > > wrote:
> >> > >
> >> > >> +1 on branching (yay!)
> >> > >>
> >> > >> I have EC2 resources for running ITBLL etc.
> >> > >>
> >> > >>
> >> > >> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> wrote:
> >> > >>
> >> > >> > Some notes on progress toward hbase2.
> >> > >> >
> >> > >> > Given that stability and performance are NOT emergent behaviors
> but
> >> > >> rather
> >> > >> > projects unto themselves, my thought is that we commit all that
> >> we've
> >> > >> > agreed as core for hbase2 (see [1]), branch, and then work on
> >> > >> stabilizing
> >> > >> > and perf rather than do stabilize, commit, and then branch. What
> >> this
> >> > >> means
> >> > >> > in practice is that for features like Inmemory Compaction, we
> >> commit
> >> > it
> >> > >> > defaulted 'on' ("BASIC" mode) which is what we want in hbase2.
> >> Should
> >> > it
> >> > >> > prove problematic under test, we disable it before release.
> >> > >> >
> >> > >> > Are folks good w/ this mode? I ask because, in a few issues there
> >> are
> >> > >> > requests for proof that a master feature is 'stable' before
> commit.
> >> > >> This is
> >> > >> > normally a healthy request only in master's case, it is hard to
> >> > >> demonstrate
> >> > >> > stability given its current state.
> >> > >> >
> >> > >> > Other outstanding issues such as decisions about whether master
> >> hosts
> >> > >> > system tables only (by default), I'm thinking, we can work out
> post
> >> > >> branch
> >> > >> > in alpha/betas before release.
> >> > >> >
> >> > >> > The awkward item is the long-pole Assignment Manager. This is an
> >> > >> > all-or-nothing affair. Here we are switching in a new Master
> core.
> >> > >> While I
> >> > >> > think it fine that AMv2 is incomplete come branch time, those of
> us
> >> > >> working
> >> > >> > on the new AM still need to demonstrate to you all that it
> >> basically
> >> > >> > viable.
> >> > >> >
> >> > >> > The point-of-no-return is commit of the patch in HBASE-14614.
> >> > >> HBASE-14614
> >> > >> > (AMv2) is coming close to passing all unit tests. We'll spend
> some
> >> > time
> >> > >> > running it on a cluster to make sure it fundamentally sound and
> >> will
> >> > >> report
> >> > >> > back on our experience. There has been an ask for some dev doc
> and
> >> > >> > low-levels on how it works (in progress). Let satisfaction of
> these
> >> > >> > requests be blockers on commit. We'll put the HBASE-14614 commit
> up
> >> > for
> >> > >> a
> >> > >> > vote on dev list given its import.
> >> > >> >
> >> > >> > Branch will happen after HBASE-14614 goes in (or its rejection)
> >> with
> >> > our
> >> > >> > first alpha soon after. Its looking like a week or two at least
> >> given
> >> > >> how
> >> > >> > things have been going up to this.
> >> > >> >
> >> > >> > I intend to start in on hbase2 stability/perf projects after we
> >> > branch.
> >> > >> >
> >> > >> > Interested in any thoughts you all might have on the above (Would
> >> also
> >> > >> > appreciate updates on state in [1] if you are a feature owner).
> >> > >> >
> >> > >> > Thanks,
> >> > >> > St.Ack
> >> > >> >
> >> > >> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> >> > >> > z9iEu_ktczrlKHK8N4SZzs/edit#
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <el...@apache.org>
> >> > wrote:
> >> > >> >
> >> > >> > >
> >> > >> > > Stack wrote:
> >> > >> > >
> >> > >> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<el...@apache.org>
> >> > >> wrote:
> >> > >> > >>
> >> > >> > >> Thanks for pulling in the FS Quotas work, Stack. I'm trying to
> >> > cross
> >> > >> the
> >> > >> > >>> last T's and dot the last I's.
> >> > >> > >>>
> >> > >> > >>> The biggest thing I know I need to do still is to write a new
> >> > >> chapter
> >> > >> > to
> >> > >> > >>> the book. After that, I'd start entertaining larger
> >> > >> reviews/discussions
> >> > >> > >>> to
> >> > >> > >>> merge the feature into master. Anyone with free time
> (giggles)
> >> is
> >> > >> more
> >> > >> > >>> than
> >> > >> > >>> welcome to start perusing :)
> >> > >> > >>>
> >> > >> > >>>
> >> > >> > >>> Out of interest, this could come in after 2.0 Josh? Any 2.0
> >> > specific
> >> > >> > >> needs
> >> > >> > >> to make this work?
> >> > >> > >>
> >> > >> > >> Meantime, updated the 2.0 doc 1.
> >> > >> > >>
> >> > >> > >> Thanks Josh,
> >> > >> > >> St.Ack
> >> > >> > >>
> >> > >> > >> 1.
> >> > >> > >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >> > >> > >> Eu_ktczrlKHK8N4SZzs/edit#
> >> > >> > >>
> >> > >> > >>
> >> > >> > > Nope, no need to block 2.0 on this one (given the other,
> related
> >> > >> > chatter).
> >> > >> > > Would be nice to get it in, but I completely understand if it
> >> slips
> >> > :)
> >> > >> > >
> >> > >> > > Thanks for updating the doc for me!
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Best regards,
> >> > >>
> >> > >>    - Andy
> >> > >>
> >> > >> If you are given a choice, you believe you have acted freely. -
> >> Raymond
> >> > >> Teller (via Peter Watts)
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>