Moving 2.0 forward

[Andrew Purtell <an...@gmail.com>]

Hi all,

I've heard a rumor the co-RM situation with 2.0 may have changed. Can we get an update from co-RMs Matteo and Steven on their availability and interest in continuing in this role? 

To assist in moving 2.0 forward I intend to branch branch-2 from master next week. Unless there is an objection I will take this action under assumption of lazy consensus. Master branch will be renumbered to 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin scale tests and stabilization (via bug fixes or reverts of unfinished work) and invite interested collaborators to do the same. 

Re: Moving 2.0 forward

[Jerry He <je...@gmail.com>]

I agree we need a long and stable 1.x release. Branch-1 is a good fit for
that role.
It has the stability and compatibility of 1.x, and it has still been quite
open for flow of improvements and commits.

+1


Jerry

On Fri, Jan 6, 2017 at 1:01 PM, Mikhail Antonov <ol...@gmail.com>
wrote:

> I support that idea of cutting branch-2 early. Yes it will create some
> burden for the RM and committers to port things between the
> branches, but until the branch is cut we won't have that sense of imminense
> of approaching release, and more importantly, until
> branch is cut _all_ commits will continue to go there, making it hard to
> stabilize.
>
> Regarding branch-1 and branch-2 release lines, agree those are unrelated
> questions. I'm all for frequent and fast updates to new versions, but
> obviously we can't drop support and development on branch-1 until 2.0 is
> released and probed by early adopters, and then not until 2.0 is as stable
> as what people running late 1.* branches currently have.
>
> Thanks,
> Mikhail
>
> On Fri, Jan 6, 2017 at 10:40 AM, Andrew Purtell <an...@gmail.com>
> wrote:
>
> > Considerations for a new branch-2 and branch-1 are orthogonal in my
> > opinion.
> >
> > I intend to volunteer to be the RM for branch-1 itself (we've not had one
> > before) as necessary for it to become a stable source of incremental
> > releases for a long time, similar to how we had 0.98 active for almost
> > three years while 1.x development took place. Where I work we plan to
> have
> > branch-1 based code in production for at least one year, probably longer.
> >
> > Given the above arrangement, releases from branch-1 and branch-2 would
> > have independent roadmaps and release timelines.
> >
> > Does this sound reasonable?
> >
> >
> > > On Jan 5, 2017, at 11:51 PM, Phil Yang <ud...@gmail.com> wrote:
> > >
> > > Hi all
> > > After cutting branch-2, what will we do for branch-1? If I am not
> wrong,
> > > 1.4 may be the last 1.x release branch? Should 1.4.0 release before
> > 2.0.0?
> > > If not, will it confuse users?
> > >
> > > Thanks,
> > > Phil
> > >
> > >
> > > 2017-01-01 5:20 GMT+08:00 Andrew Purtell <an...@gmail.com>:
> > >
> > >> On the other hand branching will force the issue. There will always be
> > >> lists of issues to get in. How long have we been talking about 2.0? At
> > >> least a year and a half. At some point it's time to stop talking and
> > take
> > >> action. Let me revisit progress at the end of January and bring this
> up
> > >> again. As a member of the PMC I'm advising all concerned that 2.0 is
> > >> talking too long and I am considering steps to move it forward.
> > >>
> > >>
> > >>> On Dec 31, 2016, at 12:54 PM, Ted Yu <yu...@gmail.com> wrote:
> > >>>
> > >>> I agree with Stephen on not branching too early.
> > >>>
> > >>> When people come back from vacation, we can poll relevant parties on
> > >>> estimate of respective project to get a sense of when would be proper
> > >> time
> > >>> for branching.
> > >>>
> > >>> On Sat, Dec 31, 2016 at 12:16 PM, Stephen Jiang <
> > syuanjiangdev@gmail.com
> > >>>
> > >>> wrote:
> > >>>
> > >>>> Hello, Andrew, I was a helper on Matteo so that we can help each
> other
> > >>>> while we are focusing on the new Assignment Manager work.  Now he is
> > not
> > >>>> available (at least in the next few months).  I have to be more
> > focused
> > >> on
> > >>>> the new AM work; plus other work in my company; it would be too much
> > >> for me
> > >>>> to 2.0 RM alone.  I am happy someone would help to take primary 2.0
> RM
> > >> role
> > >>>> while I am still help to make this 2.0 release smooth.
> > >>>>
> > >>>> For branch-2, I think it is too early to cut it, as we still have a
> > lot
> > >> of
> > >>>> moving parts and on-going project that needs to be part of 2.0.  For
> > >>>> example, the mentioned new AM (and other projects, such as
> > HBASE-14414,
> > >>>> HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531,
> just
> > >> name
> > >>>> a few).  Cutting branch now would add burden to complete those
> > projects.
> > >>>>
> > >>>> thanks
> > >>>> Stephen
> > >>>>
> > >>>> On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <
> > >> andrew.purtell@gmail.com
> > >>>>>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi all,
> > >>>>>
> > >>>>> I've heard a rumor the co-RM situation with 2.0 may have changed.
> Can
> > >> we
> > >>>>> get an update from co-RMs Matteo and Steven on their availability
> and
> > >>>>> interest in continuing in this role?
> > >>>>>
> > >>>>> To assist in moving 2.0 forward I intend to branch branch-2 from
> > master
> > >>>>> next week. Unless there is an objection I will take this action
> under
> > >>>>> assumption of lazy consensus. Master branch will be renumbered to
> > >>>>> 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin
> > scale
> > >>>>> tests and stabilization (via bug fixes or reverts of unfinished
> work)
> > >> and
> > >>>>> invite interested collaborators to do the same.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>
> >
>
>
>
> --
> Thanks,
> Michael Antonov
>

Re: Moving 2.0 forward

[Mikhail Antonov <ol...@gmail.com>]

I support that idea of cutting branch-2 early. Yes it will create some
burden for the RM and committers to port things between the
branches, but until the branch is cut we won't have that sense of imminense
of approaching release, and more importantly, until
branch is cut _all_ commits will continue to go there, making it hard to
stabilize.

Regarding branch-1 and branch-2 release lines, agree those are unrelated
questions. I'm all for frequent and fast updates to new versions, but
obviously we can't drop support and development on branch-1 until 2.0 is
released and probed by early adopters, and then not until 2.0 is as stable
as what people running late 1.* branches currently have.

Thanks,
Mikhail

On Fri, Jan 6, 2017 at 10:40 AM, Andrew Purtell <an...@gmail.com>
wrote:

> Considerations for a new branch-2 and branch-1 are orthogonal in my
> opinion.
>
> I intend to volunteer to be the RM for branch-1 itself (we've not had one
> before) as necessary for it to become a stable source of incremental
> releases for a long time, similar to how we had 0.98 active for almost
> three years while 1.x development took place. Where I work we plan to have
> branch-1 based code in production for at least one year, probably longer.
>
> Given the above arrangement, releases from branch-1 and branch-2 would
> have independent roadmaps and release timelines.
>
> Does this sound reasonable?
>
>
> > On Jan 5, 2017, at 11:51 PM, Phil Yang <ud...@gmail.com> wrote:
> >
> > Hi all
> > After cutting branch-2, what will we do for branch-1? If I am not wrong,
> > 1.4 may be the last 1.x release branch? Should 1.4.0 release before
> 2.0.0?
> > If not, will it confuse users?
> >
> > Thanks,
> > Phil
> >
> >
> > 2017-01-01 5:20 GMT+08:00 Andrew Purtell <an...@gmail.com>:
> >
> >> On the other hand branching will force the issue. There will always be
> >> lists of issues to get in. How long have we been talking about 2.0? At
> >> least a year and a half. At some point it's time to stop talking and
> take
> >> action. Let me revisit progress at the end of January and bring this up
> >> again. As a member of the PMC I'm advising all concerned that 2.0 is
> >> talking too long and I am considering steps to move it forward.
> >>
> >>
> >>> On Dec 31, 2016, at 12:54 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>
> >>> I agree with Stephen on not branching too early.
> >>>
> >>> When people come back from vacation, we can poll relevant parties on
> >>> estimate of respective project to get a sense of when would be proper
> >> time
> >>> for branching.
> >>>
> >>> On Sat, Dec 31, 2016 at 12:16 PM, Stephen Jiang <
> syuanjiangdev@gmail.com
> >>>
> >>> wrote:
> >>>
> >>>> Hello, Andrew, I was a helper on Matteo so that we can help each other
> >>>> while we are focusing on the new Assignment Manager work.  Now he is
> not
> >>>> available (at least in the next few months).  I have to be more
> focused
> >> on
> >>>> the new AM work; plus other work in my company; it would be too much
> >> for me
> >>>> to 2.0 RM alone.  I am happy someone would help to take primary 2.0 RM
> >> role
> >>>> while I am still help to make this 2.0 release smooth.
> >>>>
> >>>> For branch-2, I think it is too early to cut it, as we still have a
> lot
> >> of
> >>>> moving parts and on-going project that needs to be part of 2.0.  For
> >>>> example, the mentioned new AM (and other projects, such as
> HBASE-14414,
> >>>> HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531, just
> >> name
> >>>> a few).  Cutting branch now would add burden to complete those
> projects.
> >>>>
> >>>> thanks
> >>>> Stephen
> >>>>
> >>>> On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <
> >> andrew.purtell@gmail.com
> >>>>>
> >>>> wrote:
> >>>>
> >>>>> Hi all,
> >>>>>
> >>>>> I've heard a rumor the co-RM situation with 2.0 may have changed. Can
> >> we
> >>>>> get an update from co-RMs Matteo and Steven on their availability and
> >>>>> interest in continuing in this role?
> >>>>>
> >>>>> To assist in moving 2.0 forward I intend to branch branch-2 from
> master
> >>>>> next week. Unless there is an objection I will take this action under
> >>>>> assumption of lazy consensus. Master branch will be renumbered to
> >>>>> 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin
> scale
> >>>>> tests and stabilization (via bug fixes or reverts of unfinished work)
> >> and
> >>>>> invite interested collaborators to do the same.
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
>



-- 
Thanks,
Michael Antonov

Re: Moving 2.0 forward

[Andrew Purtell <an...@gmail.com>]

Considerations for a new branch-2 and branch-1 are orthogonal in my opinion. 

I intend to volunteer to be the RM for branch-1 itself (we've not had one before) as necessary for it to become a stable source of incremental releases for a long time, similar to how we had 0.98 active for almost three years while 1.x development took place. Where I work we plan to have branch-1 based code in production for at least one year, probably longer. 

Given the above arrangement, releases from branch-1 and branch-2 would have independent roadmaps and release timelines. 

Does this sound reasonable? 


> On Jan 5, 2017, at 11:51 PM, Phil Yang <ud...@gmail.com> wrote:
> 
> Hi all
> After cutting branch-2, what will we do for branch-1? If I am not wrong,
> 1.4 may be the last 1.x release branch? Should 1.4.0 release before 2.0.0?
> If not, will it confuse users?
> 
> Thanks,
> Phil
> 
> 
> 2017-01-01 5:20 GMT+08:00 Andrew Purtell <an...@gmail.com>:
> 
>> On the other hand branching will force the issue. There will always be
>> lists of issues to get in. How long have we been talking about 2.0? At
>> least a year and a half. At some point it's time to stop talking and take
>> action. Let me revisit progress at the end of January and bring this up
>> again. As a member of the PMC I'm advising all concerned that 2.0 is
>> talking too long and I am considering steps to move it forward.
>> 
>> 
>>> On Dec 31, 2016, at 12:54 PM, Ted Yu <yu...@gmail.com> wrote:
>>> 
>>> I agree with Stephen on not branching too early.
>>> 
>>> When people come back from vacation, we can poll relevant parties on
>>> estimate of respective project to get a sense of when would be proper
>> time
>>> for branching.
>>> 
>>> On Sat, Dec 31, 2016 at 12:16 PM, Stephen Jiang <syuanjiangdev@gmail.com
>>> 
>>> wrote:
>>> 
>>>> Hello, Andrew, I was a helper on Matteo so that we can help each other
>>>> while we are focusing on the new Assignment Manager work.  Now he is not
>>>> available (at least in the next few months).  I have to be more focused
>> on
>>>> the new AM work; plus other work in my company; it would be too much
>> for me
>>>> to 2.0 RM alone.  I am happy someone would help to take primary 2.0 RM
>> role
>>>> while I am still help to make this 2.0 release smooth.
>>>> 
>>>> For branch-2, I think it is too early to cut it, as we still have a lot
>> of
>>>> moving parts and on-going project that needs to be part of 2.0.  For
>>>> example, the mentioned new AM (and other projects, such as HBASE-14414,
>>>> HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531, just
>> name
>>>> a few).  Cutting branch now would add burden to complete those projects.
>>>> 
>>>> thanks
>>>> Stephen
>>>> 
>>>> On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <
>> andrew.purtell@gmail.com
>>>>> 
>>>> wrote:
>>>> 
>>>>> Hi all,
>>>>> 
>>>>> I've heard a rumor the co-RM situation with 2.0 may have changed. Can
>> we
>>>>> get an update from co-RMs Matteo and Steven on their availability and
>>>>> interest in continuing in this role?
>>>>> 
>>>>> To assist in moving 2.0 forward I intend to branch branch-2 from master
>>>>> next week. Unless there is an objection I will take this action under
>>>>> assumption of lazy consensus. Master branch will be renumbered to
>>>>> 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin scale
>>>>> tests and stabilization (via bug fixes or reverts of unfinished work)
>> and
>>>>> invite interested collaborators to do the same.
>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 

Re: Moving 2.0 forward

[Phil Yang <ud...@gmail.com>]

Hi all
After cutting branch-2, what will we do for branch-1? If I am not wrong,
1.4 may be the last 1.x release branch? Should 1.4.0 release before 2.0.0?
If not, will it confuse users?

Thanks,
Phil


2017-01-01 5:20 GMT+08:00 Andrew Purtell <an...@gmail.com>:

> On the other hand branching will force the issue. There will always be
> lists of issues to get in. How long have we been talking about 2.0? At
> least a year and a half. At some point it's time to stop talking and take
> action. Let me revisit progress at the end of January and bring this up
> again. As a member of the PMC I'm advising all concerned that 2.0 is
> talking too long and I am considering steps to move it forward.
>
>
> > On Dec 31, 2016, at 12:54 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > I agree with Stephen on not branching too early.
> >
> > When people come back from vacation, we can poll relevant parties on
> > estimate of respective project to get a sense of when would be proper
> time
> > for branching.
> >
> > On Sat, Dec 31, 2016 at 12:16 PM, Stephen Jiang <syuanjiangdev@gmail.com
> >
> > wrote:
> >
> >> Hello, Andrew, I was a helper on Matteo so that we can help each other
> >> while we are focusing on the new Assignment Manager work.  Now he is not
> >> available (at least in the next few months).  I have to be more focused
> on
> >> the new AM work; plus other work in my company; it would be too much
> for me
> >> to 2.0 RM alone.  I am happy someone would help to take primary 2.0 RM
> role
> >> while I am still help to make this 2.0 release smooth.
> >>
> >> For branch-2, I think it is too early to cut it, as we still have a lot
> of
> >> moving parts and on-going project that needs to be part of 2.0.  For
> >> example, the mentioned new AM (and other projects, such as HBASE-14414,
> >> HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531, just
> name
> >> a few).  Cutting branch now would add burden to complete those projects.
> >>
> >> thanks
> >> Stephen
> >>
> >> On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <
> andrew.purtell@gmail.com
> >>>
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> I've heard a rumor the co-RM situation with 2.0 may have changed. Can
> we
> >>> get an update from co-RMs Matteo and Steven on their availability and
> >>> interest in continuing in this role?
> >>>
> >>> To assist in moving 2.0 forward I intend to branch branch-2 from master
> >>> next week. Unless there is an objection I will take this action under
> >>> assumption of lazy consensus. Master branch will be renumbered to
> >>> 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin scale
> >>> tests and stabilization (via bug fixes or reverts of unfinished work)
> and
> >>> invite interested collaborators to do the same.
> >>>
> >>>
> >>>
> >>
>

Re: Moving 2.0 forward

[Andrew Purtell <an...@gmail.com>]

On the other hand branching will force the issue. There will always be lists of issues to get in. How long have we been talking about 2.0? At least a year and a half. At some point it's time to stop talking and take action. Let me revisit progress at the end of January and bring this up again. As a member of the PMC I'm advising all concerned that 2.0 is talking too long and I am considering steps to move it forward. 


> On Dec 31, 2016, at 12:54 PM, Ted Yu <yu...@gmail.com> wrote:
> 
> I agree with Stephen on not branching too early.
> 
> When people come back from vacation, we can poll relevant parties on
> estimate of respective project to get a sense of when would be proper time
> for branching.
> 
> On Sat, Dec 31, 2016 at 12:16 PM, Stephen Jiang <sy...@gmail.com>
> wrote:
> 
>> Hello, Andrew, I was a helper on Matteo so that we can help each other
>> while we are focusing on the new Assignment Manager work.  Now he is not
>> available (at least in the next few months).  I have to be more focused on
>> the new AM work; plus other work in my company; it would be too much for me
>> to 2.0 RM alone.  I am happy someone would help to take primary 2.0 RM role
>> while I am still help to make this 2.0 release smooth.
>> 
>> For branch-2, I think it is too early to cut it, as we still have a lot of
>> moving parts and on-going project that needs to be part of 2.0.  For
>> example, the mentioned new AM (and other projects, such as HBASE-14414,
>> HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531, just name
>> a few).  Cutting branch now would add burden to complete those projects.
>> 
>> thanks
>> Stephen
>> 
>> On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <andrew.purtell@gmail.com
>>> 
>> wrote:
>> 
>>> Hi all,
>>> 
>>> I've heard a rumor the co-RM situation with 2.0 may have changed. Can we
>>> get an update from co-RMs Matteo and Steven on their availability and
>>> interest in continuing in this role?
>>> 
>>> To assist in moving 2.0 forward I intend to branch branch-2 from master
>>> next week. Unless there is an objection I will take this action under
>>> assumption of lazy consensus. Master branch will be renumbered to
>>> 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin scale
>>> tests and stabilization (via bug fixes or reverts of unfinished work) and
>>> invite interested collaborators to do the same.
>>> 
>>> 
>>> 
>> 

Re: Moving 2.0 forward

[Ted Yu <yu...@gmail.com>]

I agree with Stephen on not branching too early.

When people come back from vacation, we can poll relevant parties on
estimate of respective project to get a sense of when would be proper time
for branching.

On Sat, Dec 31, 2016 at 12:16 PM, Stephen Jiang <sy...@gmail.com>
wrote:

> Hello, Andrew, I was a helper on Matteo so that we can help each other
> while we are focusing on the new Assignment Manager work.  Now he is not
> available (at least in the next few months).  I have to be more focused on
> the new AM work; plus other work in my company; it would be too much for me
> to 2.0 RM alone.  I am happy someone would help to take primary 2.0 RM role
> while I am still help to make this 2.0 release smooth.
>
> For branch-2, I think it is too early to cut it, as we still have a lot of
> moving parts and on-going project that needs to be part of 2.0.  For
> example, the mentioned new AM (and other projects, such as HBASE-14414,
> HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531, just name
> a few).  Cutting branch now would add burden to complete those projects.
>
> thanks
> Stephen
>
> On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <andrew.purtell@gmail.com
> >
> wrote:
>
> > Hi all,
> >
> > I've heard a rumor the co-RM situation with 2.0 may have changed. Can we
> > get an update from co-RMs Matteo and Steven on their availability and
> > interest in continuing in this role?
> >
> > To assist in moving 2.0 forward I intend to branch branch-2 from master
> > next week. Unless there is an objection I will take this action under
> > assumption of lazy consensus. Master branch will be renumbered to
> > 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin scale
> > tests and stabilization (via bug fixes or reverts of unfinished work) and
> > invite interested collaborators to do the same.
> >
> >
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Sat, Jan 14, 2017 at 11:10 AM, Andrew Purtell <ap...@apache.org>
wrote:

> Thanks for putting that document together Stack, that was really helpful.
>
> > 1.1 New Assignment Manager, AMv2
>
> ​Can we get a virtual show of hands who is working on this and plans to
> finish it? It was Stephen and Matteo originally, right? Matteo seems
> temporarily sidelined, is that correct?
>
>
Stephen and I have this as our priority. It is about 90% code complete with
some big patches just about to land. Will need a mountain of testing
thereafter.

Yeah, our mighty Matteo, the mastermind, is no longer actively working on
the project (I think!).



> > 1.3 Offheaping of Write Path
> ​> ​
> 1.4 HBASE-11425 Offheaping of Read Path
> ​> ​
> 1.6 HBASE-15265 AsyncWAL/HBase DFSClient
> ​
> ​Maybe we can organize some efforts to test small deploys of 2.0.0-SNAPSHOT
> with these features​ enabled since they are code complete but need testing
> and more doc, which can be generated from notes from testers on setup and
> experience. I can stand up a few clusterdock-based virtual clusters on EC2
> D2-class instances running integration tests, PE, and YCSB etc; surface
> issues up into JIRA; and provide SSH access on demand. Let me see ... not
> sure if 2.0.0-SNAPSHOT is stable enough to get that far. If so hopefully
> the developers behind these features will be willing to jump on them and
> lead debugging/fix if issues are found.
>
>
This sounds good. For offheaping, we're talking about it being the default
for 2.0.0 so could do with a bit of testing first before we actually commit
(smile).

2.0.0-SNAPSHOT seems pretty stable to me in my limited testing so far. I
could publish a SNAPSHOT on a weekly basis going forward if that'd help.



> ​​> 2.3 HBASE-6721 RegionServer Group-based Assignment
>
> Same as above, although in this case I suspect interested users are on our
> own to debug/fix.
> ​
>

Agree. This would be true of all features in the Ancillary (non-core)
section I'd say (though as it happens, I'm interesting in the above in
particular).

I added a new section to the doc on "Decisions" we need to make for 2.0.0.
Primary is a conclusion to the long-running Re: [DISCUSS] No regions on
Master node in 2.0
<http://apache-hbase.679495.n3.nabble.com/DISCUSS-No-regions-on-Master-node-in-2-0-td4079067i20.html#a4084211>
 thread.

St.Ack



>
> On Fri, Jan 13, 2017 at 11:49 PM, Andrew Purtell <andrew.purtell@gmail.com
> >
> wrote:
>
> > While I don't disagree that half finished features are undesirable, I'm
> > not suggesting that as a strategy so much as we kick out stuff that just
> > doesn't seem to be getting done. Pushing 2.0 out another three months is
> > fine if there's a good chance this is realistic and we won't be having
> this
> > discussion again then. Let me have a look at the doc and return with
> > specific points for further discussion (if any).
> >
> >
> > On Jan 13, 2017, at 11:25 PM, Stack <st...@duboce.net> wrote:
> >
> > On Sat, Dec 31, 2016 at 12:16 PM, Stephen Jiang <syuanjiangdev@gmail.com
> >
> > wrote:
> >
> >> Hello, Andrew, I was a helper on Matteo so that we can help each other
> >> while we are focusing on the new Assignment Manager work.  Now he is not
> >> available (at least in the next few months).  I have to be more focused
> on
> >> the new AM work; plus other work in my company; it would be too much for
> >> me
> >> to 2.0 RM alone.  I am happy someone would help to take primary 2.0 RM
> >> role
> >> while I am still help to make this 2.0 release smooth.
> >>
> >>
> > (I could help out Stephen. We could co-RM?)
> >
> >
> >> For branch-2, I think it is too early to cut it, as we still have a lot
> of
> >> moving parts and on-going project that needs to be part of 2.0.  For
> >> example, the mentioned new AM (and other projects, such as HBASE-14414,
> >> HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531, just
> name
> >> a few).  Cutting branch now would add burden to complete those projects.
> >>
> >>
> > Agree with Stephen. A bunch of stuff is half-baked so a '2.0.0' now would
> > be all loose ends and it'd make for a messy narrative.
> >
> > I started a doc listing state of 2.0.0: https://docs.google.
> > com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit?usp=
> > sharing
> >
> > In the doc I made an estimate of what the community considers core 2.0.0
> > items based in part off old lists and after survey of current state of
> > JIRA. The doc is open for comment. Please chime in if I am off or if I am
> > missing something that should be included. I also make a rough estimate
> on
> > state of each core item.
> >
> > I intend to keep up this macro-view doc as we progress on 2.0.0 with
> > reflection where pertinent in JIRA . Suggest we branch only when code
> > compete on the core set most of which are complete or near-so.
> > End-of-February should be time enough (First 2.0.0 RC in at the start of
> > May?).
> >
> > Thanks,
> > St.Ack
> >
> >
> >
> >> thanks
> >> Stephen
> >>
> >> On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <
> >> andrew.purtell@gmail.com>
> >> wrote:
> >>
> >> > Hi all,
> >> >
> >> > I've heard a rumor the co-RM situation with 2.0 may have changed. Can
> we
> >> > get an update from co-RMs Matteo and Steven on their availability and
> >> > interest in continuing in this role?
> >> >
> >> > To assist in moving 2.0 forward I intend to branch branch-2 from
> master
> >> > next week. Unless there is an objection I will take this action under
> >> > assumption of lazy consensus. Master branch will be renumbered to
> >> > 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin scale
> >> > tests and stabilization (via bug fixes or reverts of unfinished work)
> >> and
> >> > invite interested collaborators to do the same.
> >> >
> >> >
> >> >
> >>
> >
> >
>
>
> --
> Best regards,
>
>    - Andy
>
> If you are given a choice, you believe you have acted freely. - Raymond
> Teller (via Peter Watts)
>

Re: Moving 2.0 forward

[Andrew Purtell <ap...@apache.org>]

Thanks for putting that document together Stack, that was really helpful.

> 1.1 New Assignment Manager, AMv2

​Can we get a virtual show of hands who is working on this and plans to
finish it? It was Stephen and Matteo originally, right? Matteo seems
temporarily sidelined, is that correct?

> 1.3 Offheaping of Write Path
​> ​
1.4 HBASE-11425 Offheaping of Read Path
​> ​
1.6 HBASE-15265 AsyncWAL/HBase DFSClient
​
​Maybe we can organize some efforts to test small deploys of 2.0.0-SNAPSHOT
with these features​ enabled since they are code complete but need testing
and more doc, which can be generated from notes from testers on setup and
experience. I can stand up a few clusterdock-based virtual clusters on EC2
D2-class instances running integration tests, PE, and YCSB etc; surface
issues up into JIRA; and provide SSH access on demand. Let me see ... not
sure if 2.0.0-SNAPSHOT is stable enough to get that far. If so hopefully
the developers behind these features will be willing to jump on them and
lead debugging/fix if issues are found.

​​> 2.3 HBASE-6721 RegionServer Group-based Assignment

Same as above, although in this case I suspect interested users are on our
own to debug/fix.
​

On Fri, Jan 13, 2017 at 11:49 PM, Andrew Purtell <an...@gmail.com>
wrote:

> While I don't disagree that half finished features are undesirable, I'm
> not suggesting that as a strategy so much as we kick out stuff that just
> doesn't seem to be getting done. Pushing 2.0 out another three months is
> fine if there's a good chance this is realistic and we won't be having this
> discussion again then. Let me have a look at the doc and return with
> specific points for further discussion (if any).
>
>
> On Jan 13, 2017, at 11:25 PM, Stack <st...@duboce.net> wrote:
>
> On Sat, Dec 31, 2016 at 12:16 PM, Stephen Jiang <sy...@gmail.com>
> wrote:
>
>> Hello, Andrew, I was a helper on Matteo so that we can help each other
>> while we are focusing on the new Assignment Manager work.  Now he is not
>> available (at least in the next few months).  I have to be more focused on
>> the new AM work; plus other work in my company; it would be too much for
>> me
>> to 2.0 RM alone.  I am happy someone would help to take primary 2.0 RM
>> role
>> while I am still help to make this 2.0 release smooth.
>>
>>
> (I could help out Stephen. We could co-RM?)
>
>
>> For branch-2, I think it is too early to cut it, as we still have a lot of
>> moving parts and on-going project that needs to be part of 2.0.  For
>> example, the mentioned new AM (and other projects, such as HBASE-14414,
>> HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531, just name
>> a few).  Cutting branch now would add burden to complete those projects.
>>
>>
> Agree with Stephen. A bunch of stuff is half-baked so a '2.0.0' now would
> be all loose ends and it'd make for a messy narrative.
>
> I started a doc listing state of 2.0.0: https://docs.google.
> com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit?usp=
> sharing
>
> In the doc I made an estimate of what the community considers core 2.0.0
> items based in part off old lists and after survey of current state of
> JIRA. The doc is open for comment. Please chime in if I am off or if I am
> missing something that should be included. I also make a rough estimate on
> state of each core item.
>
> I intend to keep up this macro-view doc as we progress on 2.0.0 with
> reflection where pertinent in JIRA . Suggest we branch only when code
> compete on the core set most of which are complete or near-so.
> End-of-February should be time enough (First 2.0.0 RC in at the start of
> May?).
>
> Thanks,
> St.Ack
>
>
>
>> thanks
>> Stephen
>>
>> On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <
>> andrew.purtell@gmail.com>
>> wrote:
>>
>> > Hi all,
>> >
>> > I've heard a rumor the co-RM situation with 2.0 may have changed. Can we
>> > get an update from co-RMs Matteo and Steven on their availability and
>> > interest in continuing in this role?
>> >
>> > To assist in moving 2.0 forward I intend to branch branch-2 from master
>> > next week. Unless there is an objection I will take this action under
>> > assumption of lazy consensus. Master branch will be renumbered to
>> > 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin scale
>> > tests and stabilization (via bug fixes or reverts of unfinished work)
>> and
>> > invite interested collaborators to do the same.
>> >
>> >
>> >
>>
>
>


-- 
Best regards,

   - Andy

If you are given a choice, you believe you have acted freely. - Raymond
Teller (via Peter Watts)

Re: Moving 2.0 forward

[Andrew Purtell <an...@gmail.com>]

While I don't disagree that half finished features are undesirable, I'm not suggesting that as a strategy so much as we kick out stuff that just doesn't seem to be getting done. Pushing 2.0 out another three months is fine if there's a good chance this is realistic and we won't be having this discussion again then. Let me have a look at the doc and return with specific points for further discussion (if any). 


> On Jan 13, 2017, at 11:25 PM, Stack <st...@duboce.net> wrote:
> 
>> On Sat, Dec 31, 2016 at 12:16 PM, Stephen Jiang <sy...@gmail.com> wrote:
>> Hello, Andrew, I was a helper on Matteo so that we can help each other
>> while we are focusing on the new Assignment Manager work.  Now he is not
>> available (at least in the next few months).  I have to be more focused on
>> the new AM work; plus other work in my company; it would be too much for me
>> to 2.0 RM alone.  I am happy someone would help to take primary 2.0 RM role
>> while I am still help to make this 2.0 release smooth.
>> 
> 
> (I could help out Stephen. We could co-RM?)
>  
>> For branch-2, I think it is too early to cut it, as we still have a lot of
>> moving parts and on-going project that needs to be part of 2.0.  For
>> example, the mentioned new AM (and other projects, such as HBASE-14414,
>> HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531, just name
>> a few).  Cutting branch now would add burden to complete those projects.
>> 
> 
> Agree with Stephen. A bunch of stuff is half-baked so a '2.0.0' now would be all loose ends and it'd make for a messy narrative.
> 
> I started a doc listing state of 2.0.0: https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit?usp=sharing
> 
> In the doc I made an estimate of what the community considers core 2.0.0 items based in part off old lists and after survey of current state of JIRA. The doc is open for comment. Please chime in if I am off or if I am missing something that should be included. I also make a rough estimate on state of each core item.
> 
> I intend to keep up this macro-view doc as we progress on 2.0.0 with reflection where pertinent in JIRA . Suggest we branch only when code compete on the core set most of which are complete or near-so. End-of-February should be time enough (First 2.0.0 RC in at the start of May?).
> 
> Thanks,
> St.Ack
> 
>  
>> thanks
>> Stephen
>> 
>> On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <an...@gmail.com>
>> wrote:
>> 
>> > Hi all,
>> >
>> > I've heard a rumor the co-RM situation with 2.0 may have changed. Can we
>> > get an update from co-RMs Matteo and Steven on their availability and
>> > interest in continuing in this role?
>> >
>> > To assist in moving 2.0 forward I intend to branch branch-2 from master
>> > next week. Unless there is an objection I will take this action under
>> > assumption of lazy consensus. Master branch will be renumbered to
>> > 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin scale
>> > tests and stabilization (via bug fixes or reverts of unfinished work) and
>> > invite interested collaborators to do the same.
>> >
>> >
>> >
> 

Re: Moving 2.0 forward

[Ted Yu <yu...@gmail.com>]

After HBASE-16179 gets in, we can get wider feedback from interested users in using hbase-spark module. 

We would then be able to find missing pieces. 

> On Jan 14, 2017, at 12:30 PM, Eric Charles <er...@apache.org> wrote:
> 
> I read "3.3 hbase-spark STATUS: Needs work. No one on it at mo. Doc. is just wrong. What is there is dodgy. Could get punted."
> 
> Unit tests are working and base functionality is there. Besides the doc and compilation against spark-2 (and scala-2.11), what else do you want to see?
> 
> 
>> On 14/01/17 10:29, Ted Yu wrote:
>> For 3.3, hbase-spark module, there is HBASE-16179 which enables support for Spark 2.0
>> It needs some review.
>> 
>> Cheers
>> 
>>> On Jan 13, 2017, at 11:25 PM, Stack <st...@duboce.net> wrote:
>>> 
>>> On Sat, Dec 31, 2016 at 12:16 PM, Stephen Jiang <sy...@gmail.com>
>>> wrote:
>>> 
>>>> Hello, Andrew, I was a helper on Matteo so that we can help each other
>>>> while we are focusing on the new Assignment Manager work.  Now he is not
>>>> available (at least in the next few months).  I have to be more focused on
>>>> the new AM work; plus other work in my company; it would be too much for me
>>>> to 2.0 RM alone.  I am happy someone would help to take primary 2.0 RM role
>>>> while I am still help to make this 2.0 release smooth.
>>> (I could help out Stephen. We could co-RM?)
>>> 
>>> 
>>>> For branch-2, I think it is too early to cut it, as we still have a lot of
>>>> moving parts and on-going project that needs to be part of 2.0.  For
>>>> example, the mentioned new AM (and other projects, such as HBASE-14414,
>>>> HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531, just name
>>>> a few).  Cutting branch now would add burden to complete those projects.
>>> Agree with Stephen. A bunch of stuff is half-baked so a '2.0.0' now would
>>> be all loose ends and it'd make for a messy narrative.
>>> 
>>> I started a doc listing state of 2.0.0:
>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit?usp=sharing
>>> 
>>> In the doc I made an estimate of what the community considers core 2.0.0
>>> items based in part off old lists and after survey of current state of
>>> JIRA. The doc is open for comment. Please chime in if I am off or if I am
>>> missing something that should be included. I also make a rough estimate on
>>> state of each core item.
>>> 
>>> I intend to keep up this macro-view doc as we progress on 2.0.0 with
>>> reflection where pertinent in JIRA . Suggest we branch only when code
>>> compete on the core set most of which are complete or near-so.
>>> End-of-February should be time enough (First 2.0.0 RC in at the start of
>>> May?).
>>> 
>>> Thanks,
>>> St.Ack
>>> 
>>> 
>>> 
>>>> thanks
>>>> Stephen
>>>> 
>>>> On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <andrew.purtell@gmail.com
>>>> wrote:
>>>> 
>>>>> Hi all,
>>>>> 
>>>>> I've heard a rumor the co-RM situation with 2.0 may have changed. Can we
>>>>> get an update from co-RMs Matteo and Steven on their availability and
>>>>> interest in continuing in this role?
>>>>> 
>>>>> To assist in moving 2.0 forward I intend to branch branch-2 from master
>>>>> next week. Unless there is an objection I will take this action under
>>>>> assumption of lazy consensus. Master branch will be renumbered to
>>>>> 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin scale
>>>>> tests and stabilization (via bug fixes or reverts of unfinished work) and
>>>>> invite interested collaborators to do the same.
>>>> 

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Fri, Mar 10, 2017 at 11:10 AM, Andrew Purtell <ap...@apache.org>
wrote:

> I would also like us to reexamine our branch RM model.
>
> Prior to 1.0 , branch RMs curated branches that lead to minor releases.
>
> Post 1.0, branch RMs curate branches that only lead to patch releases.
>
> This seems like a poorer use of precious resource (RM bandwidth and time),
> one we can scarcely afford. We should move back to having RMs curate
> branches that lead to minor releases.
>
> We probably have three people who can effectively serve as branch RMs for
> branch-1, branch-2, and a branch-3. No need to branch for 3.0 until ready,
> so the RM for branch-3 would be curating master.
>
>
Maybe this has been plain to others, but for me it is a mite revelatory; it
would help explain in part why branch-2 has lagged.

I like your suggestion Andrew,

St.Ack



>
>
> On Fri, Mar 10, 2017 at 11:07 AM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > I agree AMv2/Pv2 is almost finished and this makes sense as 2.0.
> >
> > Agreed, upgrade issues need to be addressed, but I have been assuming
> this
> > will be done after branching during the stabilization and polishing part.
> > Need the branch-2 first to stabilize and polish.
> >
> > Most of the rest justifies a 3.0, assuming we are agreed that a 2.0 is
> > overdue. I think we have that. If we are lamenting the long time coming
> for
> > a 2.0, why not go the other way? Push out unfinished work to 3.0, and
> > attempt to apply the same desired effort at getting that out in a 3.0
> ASAP
> > as opposed to landing them in a 2.0.
> >
> >
> > On Fri, Mar 10, 2017 at 9:54 AM, Stack <st...@duboce.net> wrote:
> >
> >> On Mon, Mar 6, 2017 at 6:15 PM, Enis Söztutar <en...@gmail.com>
> wrote:
> >>
> >> > Thanks Stack for the nice writeup.
> >> >
> >> > I think we should shoot for an alpha release sooner than 2 months. It
> >> gives
> >> > a test target, and will be a great way to test-drive and push for the
> >> > release vehicles (packing, hadoop3, license issues, etc) and also
> create
> >> > some well-deserved excitement. I can help with that.
> >> >
> >> >
> >> Looking at the list of items in the Core and Tasks (with an eye on the
> >> concurrent thread "A suggestion for releasing major versions
> faster...") ,
> >> it might be time for a branch -- end of next week or better, the
> >> end-of-the-month? We could push an Alpha soon after?
> >>
> >> As I see it, the blockers on hbase2 are:
> >>
> >> + AMv2/Pv2. Its been trickling in for a year or more now. We are close
> to
> >> throwing the switch on move up on to new AMv2, cornerstone of 1M-regions
> >> effort and fast assignment. There'll be fall-out but we'll be up on a
> more
> >> solid intent-log, no-zk basis. Could put this off to hbase-3 I suppose
> but
> >> its all-over the code base half-done; it'll rot if we just leave it.
> >> + Rolling Restart from branch-1 to branch-2. Has to work. Can't have a
> >> singularity. No work done.
> >> + Master carrying hbase:meta. Currently it does by default. We have a
> >> running thread on pros and cons still to finish. If master is to carry
> >> hbase:meta, there is work to do. If not, there is work to do.
> >> + Updating dependencies and shading the critical likely-clashing libs
> >> (netty, guava). No work done.
> >>
> >> Other super important stuff that we should fix (criticals) but that
> don't
> >> warrant hold-up of the release are:
> >>
> >> + Narrative around client operation timeout (Phil Yang doing great work
> >> here rationalizing our timeout mess)
> >> + Perf (async hdfs client, netty rpcserver, G1GC default, etc.) and
> >> updating defaults.
> >> + Hadoop3 (EC, etc.)
> >>
> >> I don't make mention of criticals in above list that I have confidence
> >> will
> >> land in time (inmemory compaction, the offheaping work). I leave aside
> >> criticals that are not getting love (hbase-replication, FS Redo, though
> it
> >> seems like hbase-spark might see some uptake -- thanks Jerry and crew).
> >>
> >> A major release is an opportunity for big changes. It'd be a pity if we
> >> missed this window to first-class sequenceid throughout or come up on
> HLC,
> >> at least for new tables, or split hbase:meta but as seems to be the push
> >> over in the concurrent thread, these can wait for hbase3.
> >>
> >> St.Ack
> >> 1.
> >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >> Eu_ktczrlKHK8N4SZzs/edit#heading=h.jxxznc91m047
> >>
> >>
> >>
> >> > Enis
> >> >
> >> > On Mon, Mar 6, 2017 at 2:54 PM, Stack <st...@duboce.net> wrote:
> >> >
> >> > > On Mon, Mar 6, 2017 at 2:50 PM, Stack <st...@duboce.net> wrote:
> >> > >
> >> > > > ...
> >> > > > + No recent work on core decision tasks (clean-up narrative around
> >> RPC
> >> > > > timeout, hbase:meta on master or not, batch vs partial semantic,
> >> etc.)
> >> > > >
> >> > > >
> >> > > Correction. batch vs partial semantic is making goodprogress
> >> > (HBASE-15484).
> >> > > S
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > > Non-criticals/Ancillaries
> >> > > >
> >> > > > + Async client and C++ client are both making good progress. Not
> >> done.
> >> > > > + Backup/Restore is making good progress
> >> > > > + RegionServer-based assignment got a bunch of scrutiny lately and
> >> is
> >> > now
> >> > > > 'done'.
> >> > > > + FileSystem Quotas making good progress.
> >> > > >
> >> > > > I'm seeing another month or two at least before branch and
> probably
> >> > > three.
> >> > > > See doc [1] for more detail.
> >> > > >
> >> > > > Yours,
> >> > > > St.Ack
> >> > > >
> >> > > > 1.  https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9
> >> > > > iEu_ktczrlKHK8N4SZzs/edit#
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Sun, Jan 29, 2017 at 9:16 PM, Stack <st...@duboce.net> wrote:
> >> > > >
> >> > > >> On Thu, Jan 26, 2017 at 11:49 PM, ramkrishna vasudevan <
> >> > > >> ramkrishna.s.vasudevan@gmail.com> wrote:
> >> > > >>
> >> > > >>> Hi All
> >> > > >>>
> >> > > >>> Thanks Stack. The doc looks great. The offheap write path/read
> >> path-
> >> > I
> >> > > >>> think from the read path perspective we have some good feedback
> >> from
> >> > > >>> Alibaba folks.
> >> > > >>>
> >> > > >>
> >> > > >> Agree.
> >> > > >>
> >> > > >>
> >> > > >>> The write path subtasks are all done. We are currently working
> on
> >> > some
> >> > > >>> perf
> >> > > >>> results that would help us to come up with some docs that
> suggests
> >> > best
> >> > > >>> configs and tunings for the offheap write path configurations.
> >> > > >>>
> >> > > >>>
> >> > > >> Thanks Ram. Would be good to hear what configs you are looking to
> >> > > >> implement as default so those of us also starting to test can
> >> enable
> >> > > them
> >> > > >> to get you feedback.
> >> > > >>
> >> > > >> Also suggest you fill the above short status into the doc (You
> are
> >> > > >> keeping up full status elsewhere). I've been trying to add status
> >> as I
> >> > > see
> >> > > >> it popping up; e.g. Enis did a nice state-of-the-C++ client
> >> recently
> >> > up
> >> > > in
> >> > > >> JIRA and I added pointer to the 2.0 doc. Anyone else working on
> 2.0
> >> > > >> features, would be good if you kept a short state in this
> overview
> >> > doc;
> >> > > >> just ask for edit perms.
> >> > > >>
> >> > > >> Thanks,
> >> > > >> St.Ack
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>>
> >> > > >>> Regards
> >> > > >>> Ram
> >> > > >>>
> >> > > >>> On Thu, Jan 19, 2017 at 5:37 AM, Andrew Purtell <
> >> > > >>> andrew.purtell@gmail.com>
> >> > > >>> wrote:
> >> > > >>>
> >> > > >>> > I'm interested in both split meta and rsgroups. Good news. I'd
> >> like
> >> > > to
> >> > > >>> > help test.
> >> > > >>> >
> >> > > >>> >
> >> > > >>> > > On Jan 18, 2017, at 2:53 PM, Stack <st...@duboce.net>
> wrote:
> >> > > >>> > >
> >> > > >>> > >> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu <
> >> toffer@apache.org
> >> > >
> >> > > >>> wrote:
> >> > > >>> > >>
> >> > > >>> > >> Hi Stack,
> >> > > >>> > >> I'd like to get split meta (HBASE-112288) in 2.x as well. I
> >> can
> >> > > >>> have a
> >> > > >>> > 2.x
> >> > > >>> > >> draft up next week. Was working on the 1.x version
> >> internally.
> >> > > >>> > >> Also if you'd like I can be the owner for rsgroups as well.
> >> > > >>> > >> Thanks,Francis
> >> > > >>> > >>
> >> > > >>> > >>
> >> > > >>> > >>
> >> > > >>> > >> I added splitting meta as a possible and had you and I as
> >> owner
> >> > on
> >> > > >>> > > rsgroups (I was doing to do a bit of testing and doc for
> this
> >> > > >>> feature).
> >> > > >>> > >
> >> > > >>> > > Would love to see splittable meta show up. Needs to be
> rolling
> >> > > >>> > upgradeable
> >> > > >>> > > though. Lets chat up on the issue.
> >> > > >>> > > St.Ack
> >> > > >>> > >
> >> > > >>> > >
> >> > > >>> > >
> >> > > >>> > >
> >> > > >>> > >>
> >> > > >>> > >>
> >> > > >>> > >>
> >> > > >>> > >>    On Wednesday, January 18, 2017 11:29 AM, Stack <
> >> > > stack@duboce.net
> >> > > >>> >
> >> > > >>> > >> wrote:
> >> > > >>> > >>
> >> > > >>> > >>
> >> > > >>> > >> Done Thiruvel (And thanks Guanghao for adding
> >> > hbase-replication).
> >> > > >>> > >> St.Ack
> >> > > >>> > >>
> >> > > >>> > >> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
> >> > > >>> > >> thiruvel@yahoo-inc.com.invalid> wrote:
> >> > > >>> > >>
> >> > > >>> > >>> Hi Stack,
> >> > > >>> > >>> I would like to add Favored Nodes to the ancillary
> section.
> >> > > >>> > >>> HBASE-15531: Favored Nodes EnhancementsStatus: Active
> >> > > >>> > development.Owner:
> >> > > >>> > >>> Thiruvel Thanks!-Thiruvel
> >> > > >>> > >>>
> >> > > >>> > >>>   On Monday, January 16, 2017 2:10 PM, Stack <
> >> stack@duboce.net
> >> > >
> >> > > >>> wrote:
> >> > > >>> > >>>
> >> > > >>> > >>>
> >> > > >>> > >>> On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <
> >> > > >>> zghaobac@gmail.com>
> >> > > >>> > >>> wrote:
> >> > > >>> > >>>
> >> > > >>> > >>>> For 6. Significant contirbs in master only, there are
> some
> >> > > issues
> >> > > >>> > about
> >> > > >>> > >>>> replication operations routed through master. They are
> >> > sub-task
> >> > > >>> > >>>> of HBASE-10504. And there are other umbrella issue for
> >> > > >>> replication,
> >> > > >>> > >> like
> >> > > >>> > >>>> HBase-14379 Replication V2 and HBASE-15867 Moving HBase
> >> > > >>> Replication
> >> > > >>> > >>>> tracking from Zookeeper to HBase. So i thought we can
> add a
> >> > new
> >> > > >>> > section
> >> > > >>> > >>>> named
> >> > > >>> > >>>> hbase-replication to possible 2.0.0s. This will help us
> to
> >> > track
> >> > > >>> the
> >> > > >>> > >>> state.
> >> > > >>> > >>>> Thanks.
> >> > > >>> > >>>>
> >> > > >>> > >>>
> >> > > >>> > >>> Thanks Guanghao Zhang. I agree. I made you an editor. If
> you
> >> > want
> >> > > >>> to
> >> > > >>> > >> have a
> >> > > >>> > >>> go at a first cut, be my guest. If nothing done in the
> next
> >> day
> >> > > or
> >> > > >>> so,
> >> > > >>> > >> I'll
> >> > > >>> > >>> add this section Sir.
> >> > > >>> > >>> Thanks,
> >> > > >>> > >>> M
> >> > > >>> > >>>
> >> > > >>> > >>>
> >> > > >>> > >>>
> >> > > >>> > >>>
> >> > > >>> > >>
> >> > > >>> > >>
> >> > > >>> > >>
> >> > > >>> > >>
> >> > > >>> >
> >> > > >>>
> >> > > >>
> >> > > >>
> >> > > >
> >> > >
> >> >
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > If you are given a choice, you believe you have acted freely. - Raymond
> > Teller (via Peter Watts)
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> If you are given a choice, you believe you have acted freely. - Raymond
> Teller (via Peter Watts)
>

Re: Moving 2.0 forward

[Andrew Purtell <ap...@apache.org>]

I would also like us to reexamine our branch RM model.

Prior to 1.0 , branch RMs curated branches that lead to minor releases.

Post 1.0, branch RMs curate branches that only lead to patch releases.

This seems like a poorer use of precious resource (RM bandwidth and time),
one we can scarcely afford. We should move back to having RMs curate
branches that lead to minor releases.

We probably have three people who can effectively serve as branch RMs for
branch-1, branch-2, and a branch-3. No need to branch for 3.0 until ready,
so the RM for branch-3 would be curating master.



On Fri, Mar 10, 2017 at 11:07 AM, Andrew Purtell <ap...@apache.org>
wrote:

> I agree AMv2/Pv2 is almost finished and this makes sense as 2.0.
>
> Agreed, upgrade issues need to be addressed, but I have been assuming this
> will be done after branching during the stabilization and polishing part.
> Need the branch-2 first to stabilize and polish.
>
> Most of the rest justifies a 3.0, assuming we are agreed that a 2.0 is
> overdue. I think we have that. If we are lamenting the long time coming for
> a 2.0, why not go the other way? Push out unfinished work to 3.0, and
> attempt to apply the same desired effort at getting that out in a 3.0 ASAP
> as opposed to landing them in a 2.0.
>
>
> On Fri, Mar 10, 2017 at 9:54 AM, Stack <st...@duboce.net> wrote:
>
>> On Mon, Mar 6, 2017 at 6:15 PM, Enis Söztutar <en...@gmail.com> wrote:
>>
>> > Thanks Stack for the nice writeup.
>> >
>> > I think we should shoot for an alpha release sooner than 2 months. It
>> gives
>> > a test target, and will be a great way to test-drive and push for the
>> > release vehicles (packing, hadoop3, license issues, etc) and also create
>> > some well-deserved excitement. I can help with that.
>> >
>> >
>> Looking at the list of items in the Core and Tasks (with an eye on the
>> concurrent thread "A suggestion for releasing major versions faster...") ,
>> it might be time for a branch -- end of next week or better, the
>> end-of-the-month? We could push an Alpha soon after?
>>
>> As I see it, the blockers on hbase2 are:
>>
>> + AMv2/Pv2. Its been trickling in for a year or more now. We are close to
>> throwing the switch on move up on to new AMv2, cornerstone of 1M-regions
>> effort and fast assignment. There'll be fall-out but we'll be up on a more
>> solid intent-log, no-zk basis. Could put this off to hbase-3 I suppose but
>> its all-over the code base half-done; it'll rot if we just leave it.
>> + Rolling Restart from branch-1 to branch-2. Has to work. Can't have a
>> singularity. No work done.
>> + Master carrying hbase:meta. Currently it does by default. We have a
>> running thread on pros and cons still to finish. If master is to carry
>> hbase:meta, there is work to do. If not, there is work to do.
>> + Updating dependencies and shading the critical likely-clashing libs
>> (netty, guava). No work done.
>>
>> Other super important stuff that we should fix (criticals) but that don't
>> warrant hold-up of the release are:
>>
>> + Narrative around client operation timeout (Phil Yang doing great work
>> here rationalizing our timeout mess)
>> + Perf (async hdfs client, netty rpcserver, G1GC default, etc.) and
>> updating defaults.
>> + Hadoop3 (EC, etc.)
>>
>> I don't make mention of criticals in above list that I have confidence
>> will
>> land in time (inmemory compaction, the offheaping work). I leave aside
>> criticals that are not getting love (hbase-replication, FS Redo, though it
>> seems like hbase-spark might see some uptake -- thanks Jerry and crew).
>>
>> A major release is an opportunity for big changes. It'd be a pity if we
>> missed this window to first-class sequenceid throughout or come up on HLC,
>> at least for new tables, or split hbase:meta but as seems to be the push
>> over in the concurrent thread, these can wait for hbase3.
>>
>> St.Ack
>> 1.
>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.jxxznc91m047
>>
>>
>>
>> > Enis
>> >
>> > On Mon, Mar 6, 2017 at 2:54 PM, Stack <st...@duboce.net> wrote:
>> >
>> > > On Mon, Mar 6, 2017 at 2:50 PM, Stack <st...@duboce.net> wrote:
>> > >
>> > > > ...
>> > > > + No recent work on core decision tasks (clean-up narrative around
>> RPC
>> > > > timeout, hbase:meta on master or not, batch vs partial semantic,
>> etc.)
>> > > >
>> > > >
>> > > Correction. batch vs partial semantic is making goodprogress
>> > (HBASE-15484).
>> > > S
>> > >
>> > >
>> > >
>> > >
>> > > > Non-criticals/Ancillaries
>> > > >
>> > > > + Async client and C++ client are both making good progress. Not
>> done.
>> > > > + Backup/Restore is making good progress
>> > > > + RegionServer-based assignment got a bunch of scrutiny lately and
>> is
>> > now
>> > > > 'done'.
>> > > > + FileSystem Quotas making good progress.
>> > > >
>> > > > I'm seeing another month or two at least before branch and probably
>> > > three.
>> > > > See doc [1] for more detail.
>> > > >
>> > > > Yours,
>> > > > St.Ack
>> > > >
>> > > > 1.  https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9
>> > > > iEu_ktczrlKHK8N4SZzs/edit#
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Sun, Jan 29, 2017 at 9:16 PM, Stack <st...@duboce.net> wrote:
>> > > >
>> > > >> On Thu, Jan 26, 2017 at 11:49 PM, ramkrishna vasudevan <
>> > > >> ramkrishna.s.vasudevan@gmail.com> wrote:
>> > > >>
>> > > >>> Hi All
>> > > >>>
>> > > >>> Thanks Stack. The doc looks great. The offheap write path/read
>> path-
>> > I
>> > > >>> think from the read path perspective we have some good feedback
>> from
>> > > >>> Alibaba folks.
>> > > >>>
>> > > >>
>> > > >> Agree.
>> > > >>
>> > > >>
>> > > >>> The write path subtasks are all done. We are currently working on
>> > some
>> > > >>> perf
>> > > >>> results that would help us to come up with some docs that suggests
>> > best
>> > > >>> configs and tunings for the offheap write path configurations.
>> > > >>>
>> > > >>>
>> > > >> Thanks Ram. Would be good to hear what configs you are looking to
>> > > >> implement as default so those of us also starting to test can
>> enable
>> > > them
>> > > >> to get you feedback.
>> > > >>
>> > > >> Also suggest you fill the above short status into the doc (You are
>> > > >> keeping up full status elsewhere). I've been trying to add status
>> as I
>> > > see
>> > > >> it popping up; e.g. Enis did a nice state-of-the-C++ client
>> recently
>> > up
>> > > in
>> > > >> JIRA and I added pointer to the 2.0 doc. Anyone else working on 2.0
>> > > >> features, would be good if you kept a short state in this overview
>> > doc;
>> > > >> just ask for edit perms.
>> > > >>
>> > > >> Thanks,
>> > > >> St.Ack
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>>
>> > > >>> Regards
>> > > >>> Ram
>> > > >>>
>> > > >>> On Thu, Jan 19, 2017 at 5:37 AM, Andrew Purtell <
>> > > >>> andrew.purtell@gmail.com>
>> > > >>> wrote:
>> > > >>>
>> > > >>> > I'm interested in both split meta and rsgroups. Good news. I'd
>> like
>> > > to
>> > > >>> > help test.
>> > > >>> >
>> > > >>> >
>> > > >>> > > On Jan 18, 2017, at 2:53 PM, Stack <st...@duboce.net> wrote:
>> > > >>> > >
>> > > >>> > >> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu <
>> toffer@apache.org
>> > >
>> > > >>> wrote:
>> > > >>> > >>
>> > > >>> > >> Hi Stack,
>> > > >>> > >> I'd like to get split meta (HBASE-112288) in 2.x as well. I
>> can
>> > > >>> have a
>> > > >>> > 2.x
>> > > >>> > >> draft up next week. Was working on the 1.x version
>> internally.
>> > > >>> > >> Also if you'd like I can be the owner for rsgroups as well.
>> > > >>> > >> Thanks,Francis
>> > > >>> > >>
>> > > >>> > >>
>> > > >>> > >>
>> > > >>> > >> I added splitting meta as a possible and had you and I as
>> owner
>> > on
>> > > >>> > > rsgroups (I was doing to do a bit of testing and doc for this
>> > > >>> feature).
>> > > >>> > >
>> > > >>> > > Would love to see splittable meta show up. Needs to be rolling
>> > > >>> > upgradeable
>> > > >>> > > though. Lets chat up on the issue.
>> > > >>> > > St.Ack
>> > > >>> > >
>> > > >>> > >
>> > > >>> > >
>> > > >>> > >
>> > > >>> > >>
>> > > >>> > >>
>> > > >>> > >>
>> > > >>> > >>    On Wednesday, January 18, 2017 11:29 AM, Stack <
>> > > stack@duboce.net
>> > > >>> >
>> > > >>> > >> wrote:
>> > > >>> > >>
>> > > >>> > >>
>> > > >>> > >> Done Thiruvel (And thanks Guanghao for adding
>> > hbase-replication).
>> > > >>> > >> St.Ack
>> > > >>> > >>
>> > > >>> > >> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
>> > > >>> > >> thiruvel@yahoo-inc.com.invalid> wrote:
>> > > >>> > >>
>> > > >>> > >>> Hi Stack,
>> > > >>> > >>> I would like to add Favored Nodes to the ancillary section.
>> > > >>> > >>> HBASE-15531: Favored Nodes EnhancementsStatus: Active
>> > > >>> > development.Owner:
>> > > >>> > >>> Thiruvel Thanks!-Thiruvel
>> > > >>> > >>>
>> > > >>> > >>>   On Monday, January 16, 2017 2:10 PM, Stack <
>> stack@duboce.net
>> > >
>> > > >>> wrote:
>> > > >>> > >>>
>> > > >>> > >>>
>> > > >>> > >>> On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <
>> > > >>> zghaobac@gmail.com>
>> > > >>> > >>> wrote:
>> > > >>> > >>>
>> > > >>> > >>>> For 6. Significant contirbs in master only, there are some
>> > > issues
>> > > >>> > about
>> > > >>> > >>>> replication operations routed through master. They are
>> > sub-task
>> > > >>> > >>>> of HBASE-10504. And there are other umbrella issue for
>> > > >>> replication,
>> > > >>> > >> like
>> > > >>> > >>>> HBase-14379 Replication V2 and HBASE-15867 Moving HBase
>> > > >>> Replication
>> > > >>> > >>>> tracking from Zookeeper to HBase. So i thought we can add a
>> > new
>> > > >>> > section
>> > > >>> > >>>> named
>> > > >>> > >>>> hbase-replication to possible 2.0.0s. This will help us to
>> > track
>> > > >>> the
>> > > >>> > >>> state.
>> > > >>> > >>>> Thanks.
>> > > >>> > >>>>
>> > > >>> > >>>
>> > > >>> > >>> Thanks Guanghao Zhang. I agree. I made you an editor. If you
>> > want
>> > > >>> to
>> > > >>> > >> have a
>> > > >>> > >>> go at a first cut, be my guest. If nothing done in the next
>> day
>> > > or
>> > > >>> so,
>> > > >>> > >> I'll
>> > > >>> > >>> add this section Sir.
>> > > >>> > >>> Thanks,
>> > > >>> > >>> M
>> > > >>> > >>>
>> > > >>> > >>>
>> > > >>> > >>>
>> > > >>> > >>>
>> > > >>> > >>
>> > > >>> > >>
>> > > >>> > >>
>> > > >>> > >>
>> > > >>> >
>> > > >>>
>> > > >>
>> > > >>
>> > > >
>> > >
>> >
>>
>
>
>
> --
> Best regards,
>
>    - Andy
>
> If you are given a choice, you believe you have acted freely. - Raymond
> Teller (via Peter Watts)
>



-- 
Best regards,

   - Andy

If you are given a choice, you believe you have acted freely. - Raymond
Teller (via Peter Watts)

Re: Moving 2.0 forward

[Andrew Purtell <ap...@apache.org>]

I agree AMv2/Pv2 is almost finished and this makes sense as 2.0.

Agreed, upgrade issues need to be addressed, but I have been assuming this
will be done after branching during the stabilization and polishing part.
Need the branch-2 first to stabilize and polish.

Most of the rest justifies a 3.0, assuming we are agreed that a 2.0 is
overdue. I think we have that. If we are lamenting the long time coming for
a 2.0, why not go the other way? Push out unfinished work to 3.0, and
attempt to apply the same desired effort at getting that out in a 3.0 ASAP
as opposed to landing them in a 2.0.


On Fri, Mar 10, 2017 at 9:54 AM, Stack <st...@duboce.net> wrote:

> On Mon, Mar 6, 2017 at 6:15 PM, Enis Söztutar <en...@gmail.com> wrote:
>
> > Thanks Stack for the nice writeup.
> >
> > I think we should shoot for an alpha release sooner than 2 months. It
> gives
> > a test target, and will be a great way to test-drive and push for the
> > release vehicles (packing, hadoop3, license issues, etc) and also create
> > some well-deserved excitement. I can help with that.
> >
> >
> Looking at the list of items in the Core and Tasks (with an eye on the
> concurrent thread "A suggestion for releasing major versions faster...") ,
> it might be time for a branch -- end of next week or better, the
> end-of-the-month? We could push an Alpha soon after?
>
> As I see it, the blockers on hbase2 are:
>
> + AMv2/Pv2. Its been trickling in for a year or more now. We are close to
> throwing the switch on move up on to new AMv2, cornerstone of 1M-regions
> effort and fast assignment. There'll be fall-out but we'll be up on a more
> solid intent-log, no-zk basis. Could put this off to hbase-3 I suppose but
> its all-over the code base half-done; it'll rot if we just leave it.
> + Rolling Restart from branch-1 to branch-2. Has to work. Can't have a
> singularity. No work done.
> + Master carrying hbase:meta. Currently it does by default. We have a
> running thread on pros and cons still to finish. If master is to carry
> hbase:meta, there is work to do. If not, there is work to do.
> + Updating dependencies and shading the critical likely-clashing libs
> (netty, guava). No work done.
>
> Other super important stuff that we should fix (criticals) but that don't
> warrant hold-up of the release are:
>
> + Narrative around client operation timeout (Phil Yang doing great work
> here rationalizing our timeout mess)
> + Perf (async hdfs client, netty rpcserver, G1GC default, etc.) and
> updating defaults.
> + Hadoop3 (EC, etc.)
>
> I don't make mention of criticals in above list that I have confidence will
> land in time (inmemory compaction, the offheaping work). I leave aside
> criticals that are not getting love (hbase-replication, FS Redo, though it
> seems like hbase-spark might see some uptake -- thanks Jerry and crew).
>
> A major release is an opportunity for big changes. It'd be a pity if we
> missed this window to first-class sequenceid throughout or come up on HLC,
> at least for new tables, or split hbase:meta but as seems to be the push
> over in the concurrent thread, these can wait for hbase3.
>
> St.Ack
> 1.
> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> ktczrlKHK8N4SZzs/edit#heading=h.jxxznc91m047
>
>
>
> > Enis
> >
> > On Mon, Mar 6, 2017 at 2:54 PM, Stack <st...@duboce.net> wrote:
> >
> > > On Mon, Mar 6, 2017 at 2:50 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > ...
> > > > + No recent work on core decision tasks (clean-up narrative around
> RPC
> > > > timeout, hbase:meta on master or not, batch vs partial semantic,
> etc.)
> > > >
> > > >
> > > Correction. batch vs partial semantic is making goodprogress
> > (HBASE-15484).
> > > S
> > >
> > >
> > >
> > >
> > > > Non-criticals/Ancillaries
> > > >
> > > > + Async client and C++ client are both making good progress. Not
> done.
> > > > + Backup/Restore is making good progress
> > > > + RegionServer-based assignment got a bunch of scrutiny lately and is
> > now
> > > > 'done'.
> > > > + FileSystem Quotas making good progress.
> > > >
> > > > I'm seeing another month or two at least before branch and probably
> > > three.
> > > > See doc [1] for more detail.
> > > >
> > > > Yours,
> > > > St.Ack
> > > >
> > > > 1.  https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9
> > > > iEu_ktczrlKHK8N4SZzs/edit#
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Sun, Jan 29, 2017 at 9:16 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > >> On Thu, Jan 26, 2017 at 11:49 PM, ramkrishna vasudevan <
> > > >> ramkrishna.s.vasudevan@gmail.com> wrote:
> > > >>
> > > >>> Hi All
> > > >>>
> > > >>> Thanks Stack. The doc looks great. The offheap write path/read
> path-
> > I
> > > >>> think from the read path perspective we have some good feedback
> from
> > > >>> Alibaba folks.
> > > >>>
> > > >>
> > > >> Agree.
> > > >>
> > > >>
> > > >>> The write path subtasks are all done. We are currently working on
> > some
> > > >>> perf
> > > >>> results that would help us to come up with some docs that suggests
> > best
> > > >>> configs and tunings for the offheap write path configurations.
> > > >>>
> > > >>>
> > > >> Thanks Ram. Would be good to hear what configs you are looking to
> > > >> implement as default so those of us also starting to test can enable
> > > them
> > > >> to get you feedback.
> > > >>
> > > >> Also suggest you fill the above short status into the doc (You are
> > > >> keeping up full status elsewhere). I've been trying to add status
> as I
> > > see
> > > >> it popping up; e.g. Enis did a nice state-of-the-C++ client recently
> > up
> > > in
> > > >> JIRA and I added pointer to the 2.0 doc. Anyone else working on 2.0
> > > >> features, would be good if you kept a short state in this overview
> > doc;
> > > >> just ask for edit perms.
> > > >>
> > > >> Thanks,
> > > >> St.Ack
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>>
> > > >>> Regards
> > > >>> Ram
> > > >>>
> > > >>> On Thu, Jan 19, 2017 at 5:37 AM, Andrew Purtell <
> > > >>> andrew.purtell@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>> > I'm interested in both split meta and rsgroups. Good news. I'd
> like
> > > to
> > > >>> > help test.
> > > >>> >
> > > >>> >
> > > >>> > > On Jan 18, 2017, at 2:53 PM, Stack <st...@duboce.net> wrote:
> > > >>> > >
> > > >>> > >> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu <
> toffer@apache.org
> > >
> > > >>> wrote:
> > > >>> > >>
> > > >>> > >> Hi Stack,
> > > >>> > >> I'd like to get split meta (HBASE-112288) in 2.x as well. I
> can
> > > >>> have a
> > > >>> > 2.x
> > > >>> > >> draft up next week. Was working on the 1.x version internally.
> > > >>> > >> Also if you'd like I can be the owner for rsgroups as well.
> > > >>> > >> Thanks,Francis
> > > >>> > >>
> > > >>> > >>
> > > >>> > >>
> > > >>> > >> I added splitting meta as a possible and had you and I as
> owner
> > on
> > > >>> > > rsgroups (I was doing to do a bit of testing and doc for this
> > > >>> feature).
> > > >>> > >
> > > >>> > > Would love to see splittable meta show up. Needs to be rolling
> > > >>> > upgradeable
> > > >>> > > though. Lets chat up on the issue.
> > > >>> > > St.Ack
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > >>
> > > >>> > >>
> > > >>> > >>
> > > >>> > >>    On Wednesday, January 18, 2017 11:29 AM, Stack <
> > > stack@duboce.net
> > > >>> >
> > > >>> > >> wrote:
> > > >>> > >>
> > > >>> > >>
> > > >>> > >> Done Thiruvel (And thanks Guanghao for adding
> > hbase-replication).
> > > >>> > >> St.Ack
> > > >>> > >>
> > > >>> > >> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
> > > >>> > >> thiruvel@yahoo-inc.com.invalid> wrote:
> > > >>> > >>
> > > >>> > >>> Hi Stack,
> > > >>> > >>> I would like to add Favored Nodes to the ancillary section.
> > > >>> > >>> HBASE-15531: Favored Nodes EnhancementsStatus: Active
> > > >>> > development.Owner:
> > > >>> > >>> Thiruvel Thanks!-Thiruvel
> > > >>> > >>>
> > > >>> > >>>   On Monday, January 16, 2017 2:10 PM, Stack <
> stack@duboce.net
> > >
> > > >>> wrote:
> > > >>> > >>>
> > > >>> > >>>
> > > >>> > >>> On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <
> > > >>> zghaobac@gmail.com>
> > > >>> > >>> wrote:
> > > >>> > >>>
> > > >>> > >>>> For 6. Significant contirbs in master only, there are some
> > > issues
> > > >>> > about
> > > >>> > >>>> replication operations routed through master. They are
> > sub-task
> > > >>> > >>>> of HBASE-10504. And there are other umbrella issue for
> > > >>> replication,
> > > >>> > >> like
> > > >>> > >>>> HBase-14379 Replication V2 and HBASE-15867 Moving HBase
> > > >>> Replication
> > > >>> > >>>> tracking from Zookeeper to HBase. So i thought we can add a
> > new
> > > >>> > section
> > > >>> > >>>> named
> > > >>> > >>>> hbase-replication to possible 2.0.0s. This will help us to
> > track
> > > >>> the
> > > >>> > >>> state.
> > > >>> > >>>> Thanks.
> > > >>> > >>>>
> > > >>> > >>>
> > > >>> > >>> Thanks Guanghao Zhang. I agree. I made you an editor. If you
> > want
> > > >>> to
> > > >>> > >> have a
> > > >>> > >>> go at a first cut, be my guest. If nothing done in the next
> day
> > > or
> > > >>> so,
> > > >>> > >> I'll
> > > >>> > >>> add this section Sir.
> > > >>> > >>> Thanks,
> > > >>> > >>> M
> > > >>> > >>>
> > > >>> > >>>
> > > >>> > >>>
> > > >>> > >>>
> > > >>> > >>
> > > >>> > >>
> > > >>> > >>
> > > >>> > >>
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>



-- 
Best regards,

   - Andy

If you are given a choice, you believe you have acted freely. - Raymond
Teller (via Peter Watts)

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Mon, Mar 6, 2017 at 6:15 PM, Enis Söztutar <en...@gmail.com> wrote:

> Thanks Stack for the nice writeup.
>
> I think we should shoot for an alpha release sooner than 2 months. It gives
> a test target, and will be a great way to test-drive and push for the
> release vehicles (packing, hadoop3, license issues, etc) and also create
> some well-deserved excitement. I can help with that.
>
>
Looking at the list of items in the Core and Tasks (with an eye on the
concurrent thread "A suggestion for releasing major versions faster...") ,
it might be time for a branch -- end of next week or better, the
end-of-the-month? We could push an Alpha soon after?

As I see it, the blockers on hbase2 are:

+ AMv2/Pv2. Its been trickling in for a year or more now. We are close to
throwing the switch on move up on to new AMv2, cornerstone of 1M-regions
effort and fast assignment. There'll be fall-out but we'll be up on a more
solid intent-log, no-zk basis. Could put this off to hbase-3 I suppose but
its all-over the code base half-done; it'll rot if we just leave it.
+ Rolling Restart from branch-1 to branch-2. Has to work. Can't have a
singularity. No work done.
+ Master carrying hbase:meta. Currently it does by default. We have a
running thread on pros and cons still to finish. If master is to carry
hbase:meta, there is work to do. If not, there is work to do.
+ Updating dependencies and shading the critical likely-clashing libs
(netty, guava). No work done.

Other super important stuff that we should fix (criticals) but that don't
warrant hold-up of the release are:

+ Narrative around client operation timeout (Phil Yang doing great work
here rationalizing our timeout mess)
+ Perf (async hdfs client, netty rpcserver, G1GC default, etc.) and
updating defaults.
+ Hadoop3 (EC, etc.)

I don't make mention of criticals in above list that I have confidence will
land in time (inmemory compaction, the offheaping work). I leave aside
criticals that are not getting love (hbase-replication, FS Redo, though it
seems like hbase-spark might see some uptake -- thanks Jerry and crew).

A major release is an opportunity for big changes. It'd be a pity if we
missed this window to first-class sequenceid throughout or come up on HLC,
at least for new tables, or split hbase:meta but as seems to be the push
over in the concurrent thread, these can wait for hbase3.

St.Ack
1.
https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.jxxznc91m047



> Enis
>
> On Mon, Mar 6, 2017 at 2:54 PM, Stack <st...@duboce.net> wrote:
>
> > On Mon, Mar 6, 2017 at 2:50 PM, Stack <st...@duboce.net> wrote:
> >
> > > ...
> > > + No recent work on core decision tasks (clean-up narrative around RPC
> > > timeout, hbase:meta on master or not, batch vs partial semantic, etc.)
> > >
> > >
> > Correction. batch vs partial semantic is making goodprogress
> (HBASE-15484).
> > S
> >
> >
> >
> >
> > > Non-criticals/Ancillaries
> > >
> > > + Async client and C++ client are both making good progress. Not done.
> > > + Backup/Restore is making good progress
> > > + RegionServer-based assignment got a bunch of scrutiny lately and is
> now
> > > 'done'.
> > > + FileSystem Quotas making good progress.
> > >
> > > I'm seeing another month or two at least before branch and probably
> > three.
> > > See doc [1] for more detail.
> > >
> > > Yours,
> > > St.Ack
> > >
> > > 1.  https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9
> > > iEu_ktczrlKHK8N4SZzs/edit#
> > >
> > >
> > >
> > >
> > >
> > > On Sun, Jan 29, 2017 at 9:16 PM, Stack <st...@duboce.net> wrote:
> > >
> > >> On Thu, Jan 26, 2017 at 11:49 PM, ramkrishna vasudevan <
> > >> ramkrishna.s.vasudevan@gmail.com> wrote:
> > >>
> > >>> Hi All
> > >>>
> > >>> Thanks Stack. The doc looks great. The offheap write path/read path-
> I
> > >>> think from the read path perspective we have some good feedback from
> > >>> Alibaba folks.
> > >>>
> > >>
> > >> Agree.
> > >>
> > >>
> > >>> The write path subtasks are all done. We are currently working on
> some
> > >>> perf
> > >>> results that would help us to come up with some docs that suggests
> best
> > >>> configs and tunings for the offheap write path configurations.
> > >>>
> > >>>
> > >> Thanks Ram. Would be good to hear what configs you are looking to
> > >> implement as default so those of us also starting to test can enable
> > them
> > >> to get you feedback.
> > >>
> > >> Also suggest you fill the above short status into the doc (You are
> > >> keeping up full status elsewhere). I've been trying to add status as I
> > see
> > >> it popping up; e.g. Enis did a nice state-of-the-C++ client recently
> up
> > in
> > >> JIRA and I added pointer to the 2.0 doc. Anyone else working on 2.0
> > >> features, would be good if you kept a short state in this overview
> doc;
> > >> just ask for edit perms.
> > >>
> > >> Thanks,
> > >> St.Ack
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>>
> > >>> Regards
> > >>> Ram
> > >>>
> > >>> On Thu, Jan 19, 2017 at 5:37 AM, Andrew Purtell <
> > >>> andrew.purtell@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > I'm interested in both split meta and rsgroups. Good news. I'd like
> > to
> > >>> > help test.
> > >>> >
> > >>> >
> > >>> > > On Jan 18, 2017, at 2:53 PM, Stack <st...@duboce.net> wrote:
> > >>> > >
> > >>> > >> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu <toffer@apache.org
> >
> > >>> wrote:
> > >>> > >>
> > >>> > >> Hi Stack,
> > >>> > >> I'd like to get split meta (HBASE-112288) in 2.x as well. I can
> > >>> have a
> > >>> > 2.x
> > >>> > >> draft up next week. Was working on the 1.x version internally.
> > >>> > >> Also if you'd like I can be the owner for rsgroups as well.
> > >>> > >> Thanks,Francis
> > >>> > >>
> > >>> > >>
> > >>> > >>
> > >>> > >> I added splitting meta as a possible and had you and I as owner
> on
> > >>> > > rsgroups (I was doing to do a bit of testing and doc for this
> > >>> feature).
> > >>> > >
> > >>> > > Would love to see splittable meta show up. Needs to be rolling
> > >>> > upgradeable
> > >>> > > though. Lets chat up on the issue.
> > >>> > > St.Ack
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > >>
> > >>> > >>
> > >>> > >>
> > >>> > >>    On Wednesday, January 18, 2017 11:29 AM, Stack <
> > stack@duboce.net
> > >>> >
> > >>> > >> wrote:
> > >>> > >>
> > >>> > >>
> > >>> > >> Done Thiruvel (And thanks Guanghao for adding
> hbase-replication).
> > >>> > >> St.Ack
> > >>> > >>
> > >>> > >> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
> > >>> > >> thiruvel@yahoo-inc.com.invalid> wrote:
> > >>> > >>
> > >>> > >>> Hi Stack,
> > >>> > >>> I would like to add Favored Nodes to the ancillary section.
> > >>> > >>> HBASE-15531: Favored Nodes EnhancementsStatus: Active
> > >>> > development.Owner:
> > >>> > >>> Thiruvel Thanks!-Thiruvel
> > >>> > >>>
> > >>> > >>>   On Monday, January 16, 2017 2:10 PM, Stack <stack@duboce.net
> >
> > >>> wrote:
> > >>> > >>>
> > >>> > >>>
> > >>> > >>> On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <
> > >>> zghaobac@gmail.com>
> > >>> > >>> wrote:
> > >>> > >>>
> > >>> > >>>> For 6. Significant contirbs in master only, there are some
> > issues
> > >>> > about
> > >>> > >>>> replication operations routed through master. They are
> sub-task
> > >>> > >>>> of HBASE-10504. And there are other umbrella issue for
> > >>> replication,
> > >>> > >> like
> > >>> > >>>> HBase-14379 Replication V2 and HBASE-15867 Moving HBase
> > >>> Replication
> > >>> > >>>> tracking from Zookeeper to HBase. So i thought we can add a
> new
> > >>> > section
> > >>> > >>>> named
> > >>> > >>>> hbase-replication to possible 2.0.0s. This will help us to
> track
> > >>> the
> > >>> > >>> state.
> > >>> > >>>> Thanks.
> > >>> > >>>>
> > >>> > >>>
> > >>> > >>> Thanks Guanghao Zhang. I agree. I made you an editor. If you
> want
> > >>> to
> > >>> > >> have a
> > >>> > >>> go at a first cut, be my guest. If nothing done in the next day
> > or
> > >>> so,
> > >>> > >> I'll
> > >>> > >>> add this section Sir.
> > >>> > >>> Thanks,
> > >>> > >>> M
> > >>> > >>>
> > >>> > >>>
> > >>> > >>>
> > >>> > >>>
> > >>> > >>
> > >>> > >>
> > >>> > >>
> > >>> > >>
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>

Re: Moving 2.0 forward

[Enis Söztutar <en...@gmail.com>]

Thanks Stack for the nice writeup.

I think we should shoot for an alpha release sooner than 2 months. It gives
a test target, and will be a great way to test-drive and push for the
release vehicles (packing, hadoop3, license issues, etc) and also create
some well-deserved excitement. I can help with that.

Enis

On Mon, Mar 6, 2017 at 2:54 PM, Stack <st...@duboce.net> wrote:

> On Mon, Mar 6, 2017 at 2:50 PM, Stack <st...@duboce.net> wrote:
>
> > ...
> > + No recent work on core decision tasks (clean-up narrative around RPC
> > timeout, hbase:meta on master or not, batch vs partial semantic, etc.)
> >
> >
> Correction. batch vs partial semantic is making goodprogress (HBASE-15484).
> S
>
>
>
>
> > Non-criticals/Ancillaries
> >
> > + Async client and C++ client are both making good progress. Not done.
> > + Backup/Restore is making good progress
> > + RegionServer-based assignment got a bunch of scrutiny lately and is now
> > 'done'.
> > + FileSystem Quotas making good progress.
> >
> > I'm seeing another month or two at least before branch and probably
> three.
> > See doc [1] for more detail.
> >
> > Yours,
> > St.Ack
> >
> > 1.  https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9
> > iEu_ktczrlKHK8N4SZzs/edit#
> >
> >
> >
> >
> >
> > On Sun, Jan 29, 2017 at 9:16 PM, Stack <st...@duboce.net> wrote:
> >
> >> On Thu, Jan 26, 2017 at 11:49 PM, ramkrishna vasudevan <
> >> ramkrishna.s.vasudevan@gmail.com> wrote:
> >>
> >>> Hi All
> >>>
> >>> Thanks Stack. The doc looks great. The offheap write path/read path- I
> >>> think from the read path perspective we have some good feedback from
> >>> Alibaba folks.
> >>>
> >>
> >> Agree.
> >>
> >>
> >>> The write path subtasks are all done. We are currently working on some
> >>> perf
> >>> results that would help us to come up with some docs that suggests best
> >>> configs and tunings for the offheap write path configurations.
> >>>
> >>>
> >> Thanks Ram. Would be good to hear what configs you are looking to
> >> implement as default so those of us also starting to test can enable
> them
> >> to get you feedback.
> >>
> >> Also suggest you fill the above short status into the doc (You are
> >> keeping up full status elsewhere). I've been trying to add status as I
> see
> >> it popping up; e.g. Enis did a nice state-of-the-C++ client recently up
> in
> >> JIRA and I added pointer to the 2.0 doc. Anyone else working on 2.0
> >> features, would be good if you kept a short state in this overview doc;
> >> just ask for edit perms.
> >>
> >> Thanks,
> >> St.Ack
> >>
> >>
> >>
> >>
> >>
> >>>
> >>> Regards
> >>> Ram
> >>>
> >>> On Thu, Jan 19, 2017 at 5:37 AM, Andrew Purtell <
> >>> andrew.purtell@gmail.com>
> >>> wrote:
> >>>
> >>> > I'm interested in both split meta and rsgroups. Good news. I'd like
> to
> >>> > help test.
> >>> >
> >>> >
> >>> > > On Jan 18, 2017, at 2:53 PM, Stack <st...@duboce.net> wrote:
> >>> > >
> >>> > >> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu <to...@apache.org>
> >>> wrote:
> >>> > >>
> >>> > >> Hi Stack,
> >>> > >> I'd like to get split meta (HBASE-112288) in 2.x as well. I can
> >>> have a
> >>> > 2.x
> >>> > >> draft up next week. Was working on the 1.x version internally.
> >>> > >> Also if you'd like I can be the owner for rsgroups as well.
> >>> > >> Thanks,Francis
> >>> > >>
> >>> > >>
> >>> > >>
> >>> > >> I added splitting meta as a possible and had you and I as owner on
> >>> > > rsgroups (I was doing to do a bit of testing and doc for this
> >>> feature).
> >>> > >
> >>> > > Would love to see splittable meta show up. Needs to be rolling
> >>> > upgradeable
> >>> > > though. Lets chat up on the issue.
> >>> > > St.Ack
> >>> > >
> >>> > >
> >>> > >
> >>> > >
> >>> > >>
> >>> > >>
> >>> > >>
> >>> > >>    On Wednesday, January 18, 2017 11:29 AM, Stack <
> stack@duboce.net
> >>> >
> >>> > >> wrote:
> >>> > >>
> >>> > >>
> >>> > >> Done Thiruvel (And thanks Guanghao for adding hbase-replication).
> >>> > >> St.Ack
> >>> > >>
> >>> > >> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
> >>> > >> thiruvel@yahoo-inc.com.invalid> wrote:
> >>> > >>
> >>> > >>> Hi Stack,
> >>> > >>> I would like to add Favored Nodes to the ancillary section.
> >>> > >>> HBASE-15531: Favored Nodes EnhancementsStatus: Active
> >>> > development.Owner:
> >>> > >>> Thiruvel Thanks!-Thiruvel
> >>> > >>>
> >>> > >>>   On Monday, January 16, 2017 2:10 PM, Stack <st...@duboce.net>
> >>> wrote:
> >>> > >>>
> >>> > >>>
> >>> > >>> On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <
> >>> zghaobac@gmail.com>
> >>> > >>> wrote:
> >>> > >>>
> >>> > >>>> For 6. Significant contirbs in master only, there are some
> issues
> >>> > about
> >>> > >>>> replication operations routed through master. They are sub-task
> >>> > >>>> of HBASE-10504. And there are other umbrella issue for
> >>> replication,
> >>> > >> like
> >>> > >>>> HBase-14379 Replication V2 and HBASE-15867 Moving HBase
> >>> Replication
> >>> > >>>> tracking from Zookeeper to HBase. So i thought we can add a new
> >>> > section
> >>> > >>>> named
> >>> > >>>> hbase-replication to possible 2.0.0s. This will help us to track
> >>> the
> >>> > >>> state.
> >>> > >>>> Thanks.
> >>> > >>>>
> >>> > >>>
> >>> > >>> Thanks Guanghao Zhang. I agree. I made you an editor. If you want
> >>> to
> >>> > >> have a
> >>> > >>> go at a first cut, be my guest. If nothing done in the next day
> or
> >>> so,
> >>> > >> I'll
> >>> > >>> add this section Sir.
> >>> > >>> Thanks,
> >>> > >>> M
> >>> > >>>
> >>> > >>>
> >>> > >>>
> >>> > >>>
> >>> > >>
> >>> > >>
> >>> > >>
> >>> > >>
> >>> >
> >>>
> >>
> >>
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Mon, Mar 6, 2017 at 2:50 PM, Stack <st...@duboce.net> wrote:

> ...
> + No recent work on core decision tasks (clean-up narrative around RPC
> timeout, hbase:meta on master or not, batch vs partial semantic, etc.)
>
>
Correction. batch vs partial semantic is making goodprogress (HBASE-15484).
S




> Non-criticals/Ancillaries
>
> + Async client and C++ client are both making good progress. Not done.
> + Backup/Restore is making good progress
> + RegionServer-based assignment got a bunch of scrutiny lately and is now
> 'done'.
> + FileSystem Quotas making good progress.
>
> I'm seeing another month or two at least before branch and probably three.
> See doc [1] for more detail.
>
> Yours,
> St.Ack
>
> 1.  https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9
> iEu_ktczrlKHK8N4SZzs/edit#
>
>
>
>
>
> On Sun, Jan 29, 2017 at 9:16 PM, Stack <st...@duboce.net> wrote:
>
>> On Thu, Jan 26, 2017 at 11:49 PM, ramkrishna vasudevan <
>> ramkrishna.s.vasudevan@gmail.com> wrote:
>>
>>> Hi All
>>>
>>> Thanks Stack. The doc looks great. The offheap write path/read path- I
>>> think from the read path perspective we have some good feedback from
>>> Alibaba folks.
>>>
>>
>> Agree.
>>
>>
>>> The write path subtasks are all done. We are currently working on some
>>> perf
>>> results that would help us to come up with some docs that suggests best
>>> configs and tunings for the offheap write path configurations.
>>>
>>>
>> Thanks Ram. Would be good to hear what configs you are looking to
>> implement as default so those of us also starting to test can enable them
>> to get you feedback.
>>
>> Also suggest you fill the above short status into the doc (You are
>> keeping up full status elsewhere). I've been trying to add status as I see
>> it popping up; e.g. Enis did a nice state-of-the-C++ client recently up in
>> JIRA and I added pointer to the 2.0 doc. Anyone else working on 2.0
>> features, would be good if you kept a short state in this overview doc;
>> just ask for edit perms.
>>
>> Thanks,
>> St.Ack
>>
>>
>>
>>
>>
>>>
>>> Regards
>>> Ram
>>>
>>> On Thu, Jan 19, 2017 at 5:37 AM, Andrew Purtell <
>>> andrew.purtell@gmail.com>
>>> wrote:
>>>
>>> > I'm interested in both split meta and rsgroups. Good news. I'd like to
>>> > help test.
>>> >
>>> >
>>> > > On Jan 18, 2017, at 2:53 PM, Stack <st...@duboce.net> wrote:
>>> > >
>>> > >> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu <to...@apache.org>
>>> wrote:
>>> > >>
>>> > >> Hi Stack,
>>> > >> I'd like to get split meta (HBASE-112288) in 2.x as well. I can
>>> have a
>>> > 2.x
>>> > >> draft up next week. Was working on the 1.x version internally.
>>> > >> Also if you'd like I can be the owner for rsgroups as well.
>>> > >> Thanks,Francis
>>> > >>
>>> > >>
>>> > >>
>>> > >> I added splitting meta as a possible and had you and I as owner on
>>> > > rsgroups (I was doing to do a bit of testing and doc for this
>>> feature).
>>> > >
>>> > > Would love to see splittable meta show up. Needs to be rolling
>>> > upgradeable
>>> > > though. Lets chat up on the issue.
>>> > > St.Ack
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >>
>>> > >>
>>> > >>
>>> > >>    On Wednesday, January 18, 2017 11:29 AM, Stack <stack@duboce.net
>>> >
>>> > >> wrote:
>>> > >>
>>> > >>
>>> > >> Done Thiruvel (And thanks Guanghao for adding hbase-replication).
>>> > >> St.Ack
>>> > >>
>>> > >> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
>>> > >> thiruvel@yahoo-inc.com.invalid> wrote:
>>> > >>
>>> > >>> Hi Stack,
>>> > >>> I would like to add Favored Nodes to the ancillary section.
>>> > >>> HBASE-15531: Favored Nodes EnhancementsStatus: Active
>>> > development.Owner:
>>> > >>> Thiruvel Thanks!-Thiruvel
>>> > >>>
>>> > >>>   On Monday, January 16, 2017 2:10 PM, Stack <st...@duboce.net>
>>> wrote:
>>> > >>>
>>> > >>>
>>> > >>> On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <
>>> zghaobac@gmail.com>
>>> > >>> wrote:
>>> > >>>
>>> > >>>> For 6. Significant contirbs in master only, there are some issues
>>> > about
>>> > >>>> replication operations routed through master. They are sub-task
>>> > >>>> of HBASE-10504. And there are other umbrella issue for
>>> replication,
>>> > >> like
>>> > >>>> HBase-14379 Replication V2 and HBASE-15867 Moving HBase
>>> Replication
>>> > >>>> tracking from Zookeeper to HBase. So i thought we can add a new
>>> > section
>>> > >>>> named
>>> > >>>> hbase-replication to possible 2.0.0s. This will help us to track
>>> the
>>> > >>> state.
>>> > >>>> Thanks.
>>> > >>>>
>>> > >>>
>>> > >>> Thanks Guanghao Zhang. I agree. I made you an editor. If you want
>>> to
>>> > >> have a
>>> > >>> go at a first cut, be my guest. If nothing done in the next day or
>>> so,
>>> > >> I'll
>>> > >>> add this section Sir.
>>> > >>> Thanks,
>>> > >>> M
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>
>>> > >>
>>> > >>
>>> > >>
>>> >
>>>
>>
>>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Wed, Mar 8, 2017 at 4:06 PM, Jerry He <je...@gmail.com> wrote:

> Thanks for the write-up, Stack.
>
> I take the liberty to make the Spark item as:
>
> 4.3 hbase-spark
> <https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> ktczrlKHK8N4SZzs/edit#heading=h.24l014zh4tmp>
>
> 4.3.1 Status=
> <https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> ktczrlKHK8N4SZzs/edit#heading=h.qx14vj8gbs1e>
> IN_PROGRESS
>
> 4.3.2 Owner=
> <https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> ktczrlKHK8N4SZzs/edit#heading=h.h3ncb1tfs5e8>Jerry
> and team
>
> We will see how much we can dot to fix-up and improve.
>
>
I love it Jerry.

What you thinking regards fixup? Talk out loud I'd say because others
interested.

Thanks for edit on doc.

St.Ack



> Thanks.
>
> Jerry
>

Re: Moving 2.0 forward

[Jerry He <je...@gmail.com>]

Thanks for the write-up, Stack.

I take the liberty to make the Spark item as:

4.3 hbase-spark
<https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.24l014zh4tmp>

4.3.1 Status=
<https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.qx14vj8gbs1e>
IN_PROGRESS

4.3.2 Owner=
<https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.h3ncb1tfs5e8>Jerry
and team

We will see how much we can dot to fix-up and improve.

Thanks.

Jerry

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

In another thread in this list I flag the coming 2.0.0-beta-2. Look for an
RC in the next day or so.

After 2.0.0-beta-2 makes it out, I'll cut branch-2.0. Soon after will come
our first hbase-2.0.0 release candidate. Expect this within a week or so
after beta-2 after remaining blockers are addressed (At this stage,
blockers are mostly documentation). Scream if there is something that has
to make 2.0.0 that I may have missed, but at this point, only critical
bug-fixes or compatibility breakages will be entertained.

I just did a pass over the issues filed against hbase-2.0.0. I did a pretty
brutal edit, sparse on commentary mostly because the count of issues
against 2.0.0 was high. There were a fair amount filed against '2.0.0' just
because 2.0.0 used to be thought of as the end of the rainbow, an event
that was way off on the horizon that would never arrive[1]. I unscheduled
the bulk of these. Reschedule against 2.0.0 if I got it wrong.  A good
bunch were great ideas almost- or half-done or never started (HBASE-14055
RegionServer Refactor, HBASE-15531 Favored Nodes, Andy's security
improvements, etc.). These were also unscheduled. Others that were really
old without recent updates I resolved. A few had been realized via other
channels.

Here is the current list [2]. Help on blockers would be much appreciated.

Your 2.0.0RM

P.S.No scale testing, perf compare, compatibility, or exercise of new
features (at least to my knowledge) has been done. We could do with some
exercise along any of these dimensions if any of you have a few spare
cycles. Thanks.

1. hbase-2.0.0 still has this smell to it, but we do our best.
2.
https://issues.apache.org/jira/projects/HBASE/versions/12327188#release-report-tab-body


On Wed, Dec 27, 2017 at 9:54 AM, Stack <st...@duboce.net> wrote:

> Heads-up, I'm working on cutting a beta-1. Shout if there is something you
> think needs to make it in (The Chia-Ping Tsai cleanups look good). I'm
> mainly waiting on the hbase-thirdparty changes to go in. The rest can wait
> it.
>
> Thanks,
> S
>
>
> On Wed, Dec 13, 2017 at 11:28 PM, Stack <st...@duboce.net> wrote:
>
>> Update. See below.
>>
>> On Tue, Dec 5, 2017 at 10:43 AM, Stack <st...@duboce.net> wrote:
>>
>>> Update on hbase-2.0.0-beta-1. I'm going to put up a beta-1 RC before
>>> Christmas day.
>>>
>>>
>> Still aiming for an hbase2 beta-1 showing up in your christmas stocking.
>>
>>
>>> Here is the list [1]. We're doing pretty good. We've fixed a load since
>>> the last update including some nice cleanups  two weeks ago (e.g. undoing
>>> client dependency on curator/zk watching making the dependence read-only)
>>> but there are still a few hairy ones hanging out there.
>>>
>>>  * HBASE-18946 Stochastic load balancer assigns replica regions to the
>>> same RS
>>>    This one is proving a little tough. Ram has been banging his head on
>>> it. We'll be able to clear a bunch of test failures once this is done.
>>>
>>>
>> This should be going in tomorrow.
>>
>>
>>
>>>  * HBASE-19112 Suspect methods on Cell to be deprecated
>>>    Good discussion up on the JIRA. We need to be crystal clear around
>>> Cell and derivatives API and Audience when we ship 2.0.
>>>
>>>
>> Ram, Chia-Ping Tsai, and Anoop are doing a nice job here and it should be
>> done in next day or so.
>>
>>
>>
>>>  * HBASE-17204 Make L2 off heap cache default ON
>>>    We need to try this at least so offheaping can be default.
>>>
>>>
>> Chatted with Anoop and Ram. Because it is so late in the day and because
>> it critical that the user have a good experience out of the box -- which
>> requires time and testing -- we are going to punt this from 2.0. Will look
>> at it for 2.1 (or 3.0 if soon). 2.0 will still be carrying all of the boys
>> offheap read/write path goodness.
>>
>>
>>
>>> Making asyncfswal default is also ongoing, HBASE-16890
>>> <https://issues.apache.org/jira/browse/HBASE-16890>, making good
>>> progress.
>>>
>>>
>>
>> Good progress here. In perf testing asyncfswal makes us generally faster.
>> Duo making good progress on last few tests that are in the way of our
>> making this default for hbase2.
>>
>> A load of goodness has been landing these last few weeks. Keep up the
>> great work.
>>
>> Your hbase-2 RM.
>>
>>
>>
>>
>>> I punted HBASE-19147 "All branch-2 unit tests pass" out to beta-2. Our
>>> unit test story has gotten better and a few of us are actively working on
>>> flakies [2] but we may make a beta-1 w/o shutting down all failures.
>>>
>>> Speak up if you need help on an issue or if you think we are missing
>>> items from our list.
>>>
>>> Thanks for all the hard work,
>>>
>>> The hbase-2.0.0 RM.
>>>
>>>
>>> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>>> 2. https://builds.apache.org/job/HBASE-Find-Flaky-Tests/last
>>> SuccessfulBuild/artifact/dashboard.html
>>>
>>> On Thu, Nov 16, 2017 at 3:48 PM, Stack <st...@duboce.net> wrote:
>>>
>>>> hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
>>>> remaining features and apply final polish to API. There will be a beta-2
>>>> but it is about upgrade/rolling-upgrade and bug fixes ONLY).
>>>>
>>>> Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1
>>>> list this morning. See here [1].
>>>>
>>>> There are about ~12 issues in progress with most of these about to
>>>> land. There are 37 TODO. Many of these are tests we need to run, some are
>>>> related to the backup/restore, but a good few are meaty w/o assignees.
>>>>
>>>> The awkward outstanding ones as I see it are the below:
>>>>
>>>> HBASE-18946 Stochastic load balancer assigns replica regions to the
>>>> same RS
>>>> HBASE-17204 Make L2 off heap cache default ON
>>>> HBASE-19112 Suspect methods on Cell to be deprecated
>>>> HBASE-19147 All branch-2 unit tests pass
>>>>
>>>> We need to make progress on the above or punt on them (can't punt on
>>>> the last one though).
>>>>
>>>> Any ideas on what configs we should update in hbase2? Dump ideas into:
>>>> HBASE-19148 Edit of default configuration
>>>>
>>>> Still hoping for an early December beta-1 RC. beta-2 hopefully will be
>>>> close behind.
>>>>
>>>> Comments? Thoughts?
>>>>
>>>> Thanks all,
>>>> S
>>>>
>>>> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>>>>
>>>> On Tue, Oct 31, 2017 at 2:56 PM, Stack <st...@duboce.net> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:
>>>>>
>>>>>> Hoping to keep momentum going from our Stack working on alpha4, I
>>>>>> tried to
>>>>>> take a stab at triaging some of the open beta-1 issues.
>>>>>>
>>>>>> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done
>>>>>> sooner
>>>>>> then I'm happy to see it pulled back in. Trying to balance optimism
>>>>>> with
>>>>>> realism here, and knowing that documentation unfortunately often gets
>>>>>> pushed to the back-burner.
>>>>>>
>>>>>> Also, I poked some folks on unassigned issues that they've filed for
>>>>>> beta-1, especially in the last few days. If issues don't have an
>>>>>> owner they
>>>>>> are unlikely to get worked. I chatted with stack and he agreed to
>>>>>> take on
>>>>>> some of the tasks, but there's a lot of surface area to cover.
>>>>>>
>>>>>> If you you're working on issues that are critical for beta-1, please
>>>>>> mark
>>>>>> them as such. Then the rest of the community will know to help
>>>>>> prioritize
>>>>>> feedback and reviews there.
>>>>>>
>>>>>> Do we have a general theme for the betas like we did with the alphas?
>>>>>>
>>>>>> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
>>>>>> well? Continue to work on tests throughout?
>>>>>>
>>>>>>
>>>>> Thanks Mike for helping to kick off the beta-1 train.
>>>>>
>>>>> Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what
>>>>> we are going to ship (beta-2 is rolling upgrade and any minor items turned
>>>>> up in testing/burn-in).
>>>>>
>>>>> Thanks,
>>>>> S
>>>>>
>>>>>
>>>>>
>>>>>> Mike
>>>>>>
>>>>>> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> > +1 go from my POV.
>>>>>> >
>>>>>> >
>>>>>> > On 10/31/17 10:07 AM, Stack wrote:
>>>>>> >
>>>>>> >> I want to push an alpha-4 today. A few items didn't make it
>>>>>> (HBASE-19092).
>>>>>> >> They need more time. We'll pull them in for beta-1. CP API is
>>>>>> basically
>>>>>> >> done. There may be some changes for beta-1 but hopefully only
>>>>>> changes
>>>>>> >> informed by experience trying to port an existing Coprocessor to
>>>>>> hbase2.
>>>>>> >>
>>>>>> >> Shout if there is anything that needs to make alpha-4.
>>>>>> >>
>>>>>> >> Thanks,
>>>>>> >> St.Ack
>>>>>> >>
>>>>>> >>
>>>>>> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> Yup, that was going to be my plan, Mike!
>>>>>> >>>
>>>>>> >>> Making a pass now, and will check back later tonight again. I see
>>>>>> others
>>>>>> >>> have already done some work today on this front.
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
>>>>>> >>>
>>>>>> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know
>>>>>> how to do
>>>>>> >>>> it directly on the jenkins job, so you don't have to bother with
>>>>>> JIRA
>>>>>> >>>> uploads)
>>>>>> >>>>
>>>>>> >>>> If you're busy, then I can make time tomorrow or Sunday to kick
>>>>>> off
>>>>>> >>>> jobs,
>>>>>> >>>> but I want to make sure we're not duplicating effort and jenkins
>>>>>> cycles.
>>>>>> >>>>
>>>>>> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
>>>>>> wrote:
>>>>>> >>>>
>>>>>> >>>> My turn to bump ;)
>>>>>> >>>>
>>>>>> >>>>>
>>>>>> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues
>>>>>> that remain
>>>>>> >>>>> needing some more work. The rest are just awaiting a good QA
>>>>>> run.
>>>>>> >>>>>
>>>>>> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over
>>>>>> the
>>>>>> >>>>> weekend, bump them along as necessary, and get them committed.
>>>>>> Would be
>>>>>> >>>>> great to be able get a vote up on Monday.
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>> On 10/24/17 6:03 PM, Stack wrote:
>>>>>> >>>>>
>>>>>> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back
>>>>>> and forth
>>>>>> >>>>>
>>>>>> >>>>>> what remains to be done. Surfacing our thoughts here so you
>>>>>> all clued
>>>>>> >>>>>> in....Or if you think otherwise, please speak up.
>>>>>> >>>>>>
>>>>>> >>>>>> We have ~13 issues to land:
>>>>>> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341
>>>>>> 594 About
>>>>>> >>>>>> two
>>>>>> >>>>>> are meta-issues that are about process which leaves 11.
>>>>>> >>>>>>
>>>>>> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>>>>> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because
>>>>>> some
>>>>>> >>>>>> changed
>>>>>> >>>>>> API/semantic that we need to get out earlier rather than later.
>>>>>> >>>>>>
>>>>>> >>>>>> Once the above is merged, HBASE-13346, change of Filter method
>>>>>> names
>>>>>> >>>>>> to
>>>>>> >>>>>> mention Cell instead of KeyValue can land.
>>>>>> >>>>>>
>>>>>> >>>>>> HBASE-199048 needs a review (Anoop will probably do it),
>>>>>> removing
>>>>>> >>>>>> IA.Private objects as params to MasterObserver... Hopefully
>>>>>> this goes
>>>>>> >>>>>> in
>>>>>> >>>>>> soon.
>>>>>> >>>>>>
>>>>>> >>>>>> Duo is hard at work on trackers for flush and compaction for
>>>>>> CPs
>>>>>> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for
>>>>>> Tephra)?
>>>>>> >>>>>>
>>>>>> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate
>>>>>> after
>>>>>> >>>>>> Duo
>>>>>> >>>>>> is
>>>>>> >>>>>> done w/ his work above.
>>>>>> >>>>>>
>>>>>> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity
>>>>>> after
>>>>>> >>>>>> all
>>>>>> >>>>>> the
>>>>>> >>>>>> purges allowing CPs do direct calls against Regions in same
>>>>>> Host.
>>>>>> >>>>>>
>>>>>> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>>>>> >>>>>>
>>>>>> >>>>>> Another day or two?
>>>>>> >>>>>>
>>>>>> >>>>>> St.Ack
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net>
>>>>>> wrote:
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <
>>>>>> elserj@apache.org>
>>>>>> >>>>>> wrote:
>>>>>> >>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>> +1
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>> I was trying to work on helping out on the outstanding
>>>>>> alpha-4 stuff
>>>>>> >>>>>>>> last
>>>>>> >>>>>>>> week -- will be continuing to try to do the same this week.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> If you need any help, Stack, or if others need reviews where
>>>>>> I
>>>>>> >>>>>>>> haven't
>>>>>> >>>>>>>> noticed on my own: feel free to @mention me.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are
>>>>>> being
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> actively
>>>>>> >>>>>>> worked on. If you get a moment, reviews by you (or anyone
>>>>>> else) helps
>>>>>> >>>>>>> move
>>>>>> >>>>>>> the process along.
>>>>>> >>>>>>>
>>>>>> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
>>>>>> >>>>>>> improvements.
>>>>>> >>>>>>> Then HBASE-13346 can go in.
>>>>>> >>>>>>>
>>>>>> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks
>>>>>> like that
>>>>>> >>>>>>> will
>>>>>> >>>>>>> be addressed by other alpha-4s about to land.
>>>>>> >>>>>>>
>>>>>> >>>>>>> St.Ack
>>>>>> >>>>>>> TODOs: https://issues.apache.org/jira
>>>>>> /projects/HBASE/versions/12341
>>>>>> >>>>>>> 594
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>> (Reviving this thread)
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>
>>>>>> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release
>>>>>> that has
>>>>>> >>>>>>>>> the
>>>>>> >>>>>>>>> refactor of the Coprocessor API shutting down access to
>>>>>> internals
>>>>>> >>>>>>>>> marked
>>>>>> >>>>>>>>> InterfaceAudience.Private.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> The outstanding list is here:
>>>>>> >>>>>>>>> https://issues.apache.org/jira
>>>>>> /projects/HBASE/versions/12341594
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> If issue, talk out loud on this thread. If you need a
>>>>>> review to
>>>>>> >>>>>>>>> land
>>>>>> >>>>>>>>> an
>>>>>> >>>>>>>>> item, shout on the issue and here; we'll help you out.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> As is, items are coming along nicely I'd say. We need to
>>>>>> merge the
>>>>>> >>>>>>>>> filter
>>>>>> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and
>>>>>> help
>>>>>> >>>>>>>>> them
>>>>>> >>>>>>>>> test
>>>>>> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have
>>>>>> confidence our
>>>>>> >>>>>>>>> downstreamers know what to expect (or we discover what we
>>>>>> missed
>>>>>> >>>>>>>>> BEFORE
>>>>>> >>>>>>>>> we
>>>>>> >>>>>>>>> beta-1).
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Thanks for time,
>>>>>> >>>>>>>>> S
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net>
>>>>>> wrote:
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night.
>>>>>> That should
>>>>>> >>>>>>>>> be
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to
>>>>>> be done.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled
>>>>>> for
>>>>>> >>>>>>>>>> alpha3
>>>>>> >>>>>>>>>> but
>>>>>> >>>>>>>>>> it is plain that more time is needed (in spite of valiant
>>>>>> effort
>>>>>> >>>>>>>>>> so
>>>>>> >>>>>>>>>> far
>>>>>> >>>>>>>>>> by
>>>>>> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a
>>>>>> 2.0.0-alpha-4 whose
>>>>>> >>>>>>>>>> theme is
>>>>>> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by
>>>>>> the
>>>>>> >>>>>>>>>> following
>>>>>> >>>>>>>>>> week.
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> We should then be ready for beta (beta == no new features,
>>>>>> no API
>>>>>> >>>>>>>>>> changes,
>>>>>> >>>>>>>>>> just fixes).
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> Thanks,
>>>>>> >>>>>>>>>> St.Ack
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
>>>>>> wrote:
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please
>>>>>> vote on
>>>>>> >>>>>>>>>> it.
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I
>>>>>> hope to
>>>>>> >>>>>>>>>>> get a
>>>>>> >>>>>>>>>>> release out in the next week or so.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If
>>>>>> folks are
>>>>>> >>>>>>>>>>> interested in helping out, below are the items I think we
>>>>>> need
>>>>>> >>>>>>>>>>> done
>>>>>> >>>>>>>>>>> for
>>>>>> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API
>>>>>> possibly
>>>>>> >>>>>>>>>>> altering
>>>>>> >>>>>>>>>>> items, and are absent those JIRAs that are making active
>>>>>> >>>>>>>>>>> progress,
>>>>>> >>>>>>>>>>> i.e. the
>>>>>> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed
>>>>>> that
>>>>>> >>>>>>>>>>> needs
>>>>>> >>>>>>>>>>> doing is
>>>>>> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between
>>>>>> branch-1
>>>>>> >>>>>>>>>>> and
>>>>>> >>>>>>>>>>> branch-2
>>>>>> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4
>>>>>> branches only
>>>>>> >>>>>>>>>>> do
>>>>>> >>>>>>>>>>> it
>>>>>> >>>>>>>>>>> between branch-1 and branch-2.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing
>>>>>> Public /
>>>>>> >>>>>>>>>>> Private
>>>>>> >>>>>>>>>>> interfaces
>>>>>> >>>>>>>>>>> This one is almost done. It could do with a finish,
>>>>>> attention to
>>>>>> >>>>>>>>>>> the
>>>>>> >>>>>>>>>>> items in last comment, and then our codebase could do with
>>>>>> >>>>>>>>>>> another
>>>>>> >>>>>>>>>>> sweep
>>>>>> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in
>>>>>> since
>>>>>> >>>>>>>>>>> the
>>>>>> >>>>>>>>>>> pass
>>>>>> >>>>>>>>>>> that was the basis of this issue.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
>>>>>> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
>>>>>> forced
>>>>>> >>>>>>>>>>> by
>>>>>> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
>>>>>> >>>>>>>>>>> Service'
>>>>>> >>>>>>>>>>> but if
>>>>>> >>>>>>>>>>> anyone else is interested, be my guest.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>>> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked
>>>>>> on. Needs
>>>>>> >>>>>>>>>>> finishing.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods
>>>>>> in Admin
>>>>>> >>>>>>>>>>> and
>>>>>> >>>>>>>>>>> cleanup
>>>>>> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>>>>>> >>>>>>>>>>> AsyncClient
>>>>>> >>>>>>>>>>> and
>>>>>> >>>>>>>>>>> make sure symmetry.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>>> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but
>>>>>> needs to
>>>>>> >>>>>>>>>>> be
>>>>>> >>>>>>>>>>> done.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would
>>>>>> be good
>>>>>> >>>>>>>>>>> to
>>>>>> >>>>>>>>>>> try
>>>>>> >>>>>>>>>>> it and figure what needs fixup (as said above, according
>>>>>> to trial
>>>>>> >>>>>>>>>>> by
>>>>>> >>>>>>>>>>> Sean,
>>>>>> >>>>>>>>>>> we might not be too bad here):
>>>>>> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state
>>>>>> talking to
>>>>>> >>>>>>>>>>> 2.0
>>>>>> >>>>>>>>>>> cluster
>>>>>> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
>>>>>> Master
>>>>>> >>>>>>>>>>> and
>>>>>> >>>>>>>>>>> 1.x
>>>>>> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-17442 Move most of the replication related
>>>>>> classes to
>>>>>> >>>>>>>>>>> hbase-server package
>>>>>> >>>>>>>>>>> The above would be good to do generally but it may make
>>>>>> for
>>>>>> >>>>>>>>>>> ripples
>>>>>> >>>>>>>>>>> in
>>>>>> >>>>>>>>>>> API so would be good to do now.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>>> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid
>>>>>> adding two
>>>>>> >>>>>>>>>>> new
>>>>>> >>>>>>>>>>> types
>>>>>> >>>>>>>>>>> to our API, two types that are nought but curtailed,
>>>>>> read-only
>>>>>> >>>>>>>>>>> views
>>>>>> >>>>>>>>>>> on
>>>>>> >>>>>>>>>>> internals. Input if you have time appreciated.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to
>>>>>> replicate to a
>>>>>> >>>>>>>>>>> hbase2
>>>>>> >>>>>>>>>>> cluster; verify
>>>>>> >>>>>>>>>>> Esteban is looking at this one
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in
>>>>>> core
>>>>>> >>>>>>>>>>> * HBASE-17143 Scan improvement
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>>> >>>>>>>>>>> implementations
>>>>>> >>>>>>>>>>> marked audience Private and returns implementations
>>>>>> rather than
>>>>>> >>>>>>>>>>> Interfaces.
>>>>>> >>>>>>>>>>> In a few locations, we allow returning an alternate
>>>>>> >>>>>>>>>>> implementation
>>>>>> >>>>>>>>>>> altogether which is probably something we don't want a CP
>>>>>> doing.
>>>>>> >>>>>>>>>>> To
>>>>>> >>>>>>>>>>> that
>>>>>> >>>>>>>>>>> end, the following issues started by Duo and Anoop need
>>>>>> to be
>>>>>> >>>>>>>>>>> taken
>>>>>> >>>>>>>>>>> to
>>>>>> >>>>>>>>>>> the
>>>>>> >>>>>>>>>>> finish line; ideally they'd have an owner:
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
>>>>>> release <=
>>>>>> >>>>>>>>>>> The
>>>>>> >>>>>>>>>>> umbrella issue.
>>>>>> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for
>>>>>> CP
>>>>>> >>>>>>>>>>> expose
>>>>>> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
>>>>>> MasterObserver
>>>>>> >>>>>>>>>>> and
>>>>>> >>>>>>>>>>> RegionServerObserver
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Nice-to-haves:
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
>>>>>> remove
>>>>>> >>>>>>>>>>> unused
>>>>>> >>>>>>>>>>> TimeRange constructors
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
>>>>>> >>>>>>>>>>> references
>>>>>> >>>>>>>>>>> existing in the code
>>>>>> >>>>>>>>>>> This is the end of an old long-running project moving up
>>>>>> on to
>>>>>> >>>>>>>>>>> Cell
>>>>>> >>>>>>>>>>> Interface. We think it is done but for a few little items
>>>>>> >>>>>>>>>>> (deprecate
>>>>>> >>>>>>>>>>> KV
>>>>>> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is
>>>>>> indeterminate;
>>>>>> >>>>>>>>>>> needs
>>>>>> >>>>>>>>>>> fixing
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>>> >>>>>>>>>>> * HBASE-14997
>>>>>> >>>>>>>>>>> Move compareOp and Comparators out of filter to client
>>>>>> package
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> What about:
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>>> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> If interested in any of the above and you need a legup,
>>>>>> just ask
>>>>>> >>>>>>>>>>> in
>>>>>> >>>>>>>>>>> the
>>>>>> >>>>>>>>>>> issue and I'll be by....
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Thanks,
>>>>>> >>>>>>>>>>> St.Ack
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <stack@duboce.net
>>>>>> >
>>>>>> >>>>>>>>>>> wrote:
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Heads-up:
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release
>>>>>> Candidate.
>>>>>> >>>>>>>>>>>> Theme
>>>>>> >>>>>>>>>>>> is
>>>>>> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
>>>>>> (guava,
>>>>>> >>>>>>>>>>>> netty,
>>>>>> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
>>>>>> >>>>>>>>>>>> master-carries-no-regions
>>>>>> >>>>>>>>>>>> by default.
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?).
>>>>>> Its theme
>>>>>> >>>>>>>>>>>> will
>>>>>> >>>>>>>>>>>> be
>>>>>> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we
>>>>>> are not
>>>>>> >>>>>>>>>>>> looking too
>>>>>> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have
>>>>>> hbase-1
>>>>>> >>>>>>>>>>>> client
>>>>>> >>>>>>>>>>>> running
>>>>>> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor
>>>>>> Interface
>>>>>> >>>>>>>>>>>> revamp
>>>>>> >>>>>>>>>>>> should be
>>>>>> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>> >>>>>>>>>>>> Implementations, and
>>>>>> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>> >>>>>>>>>>>> InterfaceAudience).
>>>>>> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API;
>>>>>> e.g.
>>>>>> >>>>>>>>>>>> guava
>>>>>> >>>>>>>>>>>> 0.12
>>>>>> >>>>>>>>>>>> Service showing through in our replication API and
>>>>>> protobufs in
>>>>>> >>>>>>>>>>>> Admin
>>>>>> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of
>>>>>> outreach to
>>>>>> >>>>>>>>>>>> make
>>>>>> >>>>>>>>>>>> sure
>>>>>> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> Beta1 in mid-September?
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>> >>>>>>>>>>>> https://issues.apache.org/jira
>>>>>> /projects/HBASE/versions/12327188
>>>>>> >>>>>>>>>>>> Edit
>>>>>> >>>>>>>>>>>> as
>>>>>> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not
>>>>>> get any
>>>>>> >>>>>>>>>>>> attention
>>>>>> >>>>>>>>>>>> in next month or so.
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll
>>>>>> leave them
>>>>>> >>>>>>>>>>>> as
>>>>>> >>>>>>>>>>>> is
>>>>>> >>>>>>>>>>>> another while but I'll boot them soon.
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> While I have your attention:
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3.
>>>>>> Moving hbase
>>>>>> >>>>>>>>>>>> thrift
>>>>>> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is
>>>>>> easy enough
>>>>>> >>>>>>>>>>>> if
>>>>>> >>>>>>>>>>>> folks
>>>>>> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to
>>>>>> 1.0 first
>>>>>> >>>>>>>>>>>> (0.98?).
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> St.Ack
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
>>>>>> wrote:
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
>>>>>> elserj@apache.org>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> wrote:
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
>>>>>> >>>>>>>>>>>>>> elserj@apache.org>
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>> wrote:
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>> ...
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would
>>>>>> make the
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>> feature a
>>>>>> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous
>>>>>> (I have
>>>>>> >>>>>>>>>>>>>>>> no
>>>>>> >>>>>>>>>>>>>>>> reason
>>>>>> >>>>>>>>>>>>>>>> to be
>>>>>> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature
>>>>>> in terms
>>>>>> >>>>>>>>>>>>>>>> of
>>>>>> >>>>>>>>>>>>>>>> how
>>>>>> >>>>>>>>>>>>>>>> much it
>>>>>> >>>>>>>>>>>>>>>> is covered via testing.
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so,
>>>>>> +1 Josh.
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too.
>>>>>> See 'Space
>>>>>> >>>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>> Quota' in
>>>>>> >>>>>>>>>>>>>>> here:
>>>>>> >>>>>>>>>>>>>>> https://docs.google.com/docume
>>>>>> nt/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>> >>>>>>>>>>>>>>> Does this little section need an update Josh?
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>> Thanks,
>>>>>> >>>>>>>>>>>>>>> S
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test
>>>>>> coverage and
>>>>>> >>>>>>>>>>>>>>> some
>>>>>> >>>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>> docs
>>>>>> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my
>>>>>> plate if
>>>>>> >>>>>>>>>>>>>> you're good
>>>>>> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is
>>>>>> against it :)
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
>>>>>> before
>>>>>> >>>>>>>>>>>>>> we
>>>>>> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll
>>>>>> double
>>>>>> >>>>>>>>>>>>>> check
>>>>>> >>>>>>>>>>>>>> and update
>>>>>> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>> Thanks Josh,
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>> S
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>
>>>>>> >>>>
>>>>>> >>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Heads-up, I'm working on cutting a beta-1. Shout if there is something you
think needs to make it in (The Chia-Ping Tsai cleanups look good). I'm
mainly waiting on the hbase-thirdparty changes to go in. The rest can wait
it.

Thanks,
S


On Wed, Dec 13, 2017 at 11:28 PM, Stack <st...@duboce.net> wrote:

> Update. See below.
>
> On Tue, Dec 5, 2017 at 10:43 AM, Stack <st...@duboce.net> wrote:
>
>> Update on hbase-2.0.0-beta-1. I'm going to put up a beta-1 RC before
>> Christmas day.
>>
>>
> Still aiming for an hbase2 beta-1 showing up in your christmas stocking.
>
>
>> Here is the list [1]. We're doing pretty good. We've fixed a load since
>> the last update including some nice cleanups  two weeks ago (e.g. undoing
>> client dependency on curator/zk watching making the dependence read-only)
>> but there are still a few hairy ones hanging out there.
>>
>>  * HBASE-18946 Stochastic load balancer assigns replica regions to the
>> same RS
>>    This one is proving a little tough. Ram has been banging his head on
>> it. We'll be able to clear a bunch of test failures once this is done.
>>
>>
> This should be going in tomorrow.
>
>
>
>>  * HBASE-19112 Suspect methods on Cell to be deprecated
>>    Good discussion up on the JIRA. We need to be crystal clear around
>> Cell and derivatives API and Audience when we ship 2.0.
>>
>>
> Ram, Chia-Ping Tsai, and Anoop are doing a nice job here and it should be
> done in next day or so.
>
>
>
>>  * HBASE-17204 Make L2 off heap cache default ON
>>    We need to try this at least so offheaping can be default.
>>
>>
> Chatted with Anoop and Ram. Because it is so late in the day and because
> it critical that the user have a good experience out of the box -- which
> requires time and testing -- we are going to punt this from 2.0. Will look
> at it for 2.1 (or 3.0 if soon). 2.0 will still be carrying all of the boys
> offheap read/write path goodness.
>
>
>
>> Making asyncfswal default is also ongoing, HBASE-16890
>> <https://issues.apache.org/jira/browse/HBASE-16890>, making good
>> progress.
>>
>>
>
> Good progress here. In perf testing asyncfswal makes us generally faster.
> Duo making good progress on last few tests that are in the way of our
> making this default for hbase2.
>
> A load of goodness has been landing these last few weeks. Keep up the
> great work.
>
> Your hbase-2 RM.
>
>
>
>
>> I punted HBASE-19147 "All branch-2 unit tests pass" out to beta-2. Our
>> unit test story has gotten better and a few of us are actively working on
>> flakies [2] but we may make a beta-1 w/o shutting down all failures.
>>
>> Speak up if you need help on an issue or if you think we are missing
>> items from our list.
>>
>> Thanks for all the hard work,
>>
>> The hbase-2.0.0 RM.
>>
>>
>> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>> 2. https://builds.apache.org/job/HBASE-Find-Flaky-Tests/last
>> SuccessfulBuild/artifact/dashboard.html
>>
>> On Thu, Nov 16, 2017 at 3:48 PM, Stack <st...@duboce.net> wrote:
>>
>>> hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
>>> remaining features and apply final polish to API. There will be a beta-2
>>> but it is about upgrade/rolling-upgrade and bug fixes ONLY).
>>>
>>> Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1
>>> list this morning. See here [1].
>>>
>>> There are about ~12 issues in progress with most of these about to land.
>>> There are 37 TODO. Many of these are tests we need to run, some are related
>>> to the backup/restore, but a good few are meaty w/o assignees.
>>>
>>> The awkward outstanding ones as I see it are the below:
>>>
>>> HBASE-18946 Stochastic load balancer assigns replica regions to the same
>>> RS
>>> HBASE-17204 Make L2 off heap cache default ON
>>> HBASE-19112 Suspect methods on Cell to be deprecated
>>> HBASE-19147 All branch-2 unit tests pass
>>>
>>> We need to make progress on the above or punt on them (can't punt on the
>>> last one though).
>>>
>>> Any ideas on what configs we should update in hbase2? Dump ideas into:
>>> HBASE-19148 Edit of default configuration
>>>
>>> Still hoping for an early December beta-1 RC. beta-2 hopefully will be
>>> close behind.
>>>
>>> Comments? Thoughts?
>>>
>>> Thanks all,
>>> S
>>>
>>> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>>>
>>> On Tue, Oct 31, 2017 at 2:56 PM, Stack <st...@duboce.net> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:
>>>>
>>>>> Hoping to keep momentum going from our Stack working on alpha4, I
>>>>> tried to
>>>>> take a stab at triaging some of the open beta-1 issues.
>>>>>
>>>>> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done
>>>>> sooner
>>>>> then I'm happy to see it pulled back in. Trying to balance optimism
>>>>> with
>>>>> realism here, and knowing that documentation unfortunately often gets
>>>>> pushed to the back-burner.
>>>>>
>>>>> Also, I poked some folks on unassigned issues that they've filed for
>>>>> beta-1, especially in the last few days. If issues don't have an owner
>>>>> they
>>>>> are unlikely to get worked. I chatted with stack and he agreed to take
>>>>> on
>>>>> some of the tasks, but there's a lot of surface area to cover.
>>>>>
>>>>> If you you're working on issues that are critical for beta-1, please
>>>>> mark
>>>>> them as such. Then the rest of the community will know to help
>>>>> prioritize
>>>>> feedback and reviews there.
>>>>>
>>>>> Do we have a general theme for the betas like we did with the alphas?
>>>>>
>>>>> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
>>>>> well? Continue to work on tests throughout?
>>>>>
>>>>>
>>>> Thanks Mike for helping to kick off the beta-1 train.
>>>>
>>>> Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what
>>>> we are going to ship (beta-2 is rolling upgrade and any minor items turned
>>>> up in testing/burn-in).
>>>>
>>>> Thanks,
>>>> S
>>>>
>>>>
>>>>
>>>>> Mike
>>>>>
>>>>> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org>
>>>>> wrote:
>>>>>
>>>>> > +1 go from my POV.
>>>>> >
>>>>> >
>>>>> > On 10/31/17 10:07 AM, Stack wrote:
>>>>> >
>>>>> >> I want to push an alpha-4 today. A few items didn't make it
>>>>> (HBASE-19092).
>>>>> >> They need more time. We'll pull them in for beta-1. CP API is
>>>>> basically
>>>>> >> done. There may be some changes for beta-1 but hopefully only
>>>>> changes
>>>>> >> informed by experience trying to port an existing Coprocessor to
>>>>> hbase2.
>>>>> >>
>>>>> >> Shout if there is anything that needs to make alpha-4.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> St.Ack
>>>>> >>
>>>>> >>
>>>>> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
>>>>> wrote:
>>>>> >>
>>>>> >> Yup, that was going to be my plan, Mike!
>>>>> >>>
>>>>> >>> Making a pass now, and will check back later tonight again. I see
>>>>> others
>>>>> >>> have already done some work today on this front.
>>>>> >>>
>>>>> >>>
>>>>> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
>>>>> >>>
>>>>> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know
>>>>> how to do
>>>>> >>>> it directly on the jenkins job, so you don't have to bother with
>>>>> JIRA
>>>>> >>>> uploads)
>>>>> >>>>
>>>>> >>>> If you're busy, then I can make time tomorrow or Sunday to kick
>>>>> off
>>>>> >>>> jobs,
>>>>> >>>> but I want to make sure we're not duplicating effort and jenkins
>>>>> cycles.
>>>>> >>>>
>>>>> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>> My turn to bump ;)
>>>>> >>>>
>>>>> >>>>>
>>>>> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that
>>>>> remain
>>>>> >>>>> needing some more work. The rest are just awaiting a good QA run.
>>>>> >>>>>
>>>>> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over
>>>>> the
>>>>> >>>>> weekend, bump them along as necessary, and get them committed.
>>>>> Would be
>>>>> >>>>> great to be able get a vote up on Monday.
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> On 10/24/17 6:03 PM, Stack wrote:
>>>>> >>>>>
>>>>> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back
>>>>> and forth
>>>>> >>>>>
>>>>> >>>>>> what remains to be done. Surfacing our thoughts here so you all
>>>>> clued
>>>>> >>>>>> in....Or if you think otherwise, please speak up.
>>>>> >>>>>>
>>>>> >>>>>> We have ~13 issues to land:
>>>>> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>> About
>>>>> >>>>>> two
>>>>> >>>>>> are meta-issues that are about process which leaves 11.
>>>>> >>>>>>
>>>>> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>>>> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because
>>>>> some
>>>>> >>>>>> changed
>>>>> >>>>>> API/semantic that we need to get out earlier rather than later.
>>>>> >>>>>>
>>>>> >>>>>> Once the above is merged, HBASE-13346, change of Filter method
>>>>> names
>>>>> >>>>>> to
>>>>> >>>>>> mention Cell instead of KeyValue can land.
>>>>> >>>>>>
>>>>> >>>>>> HBASE-199048 needs a review (Anoop will probably do it),
>>>>> removing
>>>>> >>>>>> IA.Private objects as params to MasterObserver... Hopefully
>>>>> this goes
>>>>> >>>>>> in
>>>>> >>>>>> soon.
>>>>> >>>>>>
>>>>> >>>>>> Duo is hard at work on trackers for flush and compaction for CPs
>>>>> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for
>>>>> Tephra)?
>>>>> >>>>>>
>>>>> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate
>>>>> after
>>>>> >>>>>> Duo
>>>>> >>>>>> is
>>>>> >>>>>> done w/ his work above.
>>>>> >>>>>>
>>>>> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity
>>>>> after
>>>>> >>>>>> all
>>>>> >>>>>> the
>>>>> >>>>>> purges allowing CPs do direct calls against Regions in same
>>>>> Host.
>>>>> >>>>>>
>>>>> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>>>> >>>>>>
>>>>> >>>>>> Another day or two?
>>>>> >>>>>>
>>>>> >>>>>> St.Ack
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net>
>>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <elserj@apache.org
>>>>> >
>>>>> >>>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>> +1
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>> I was trying to work on helping out on the outstanding
>>>>> alpha-4 stuff
>>>>> >>>>>>>> last
>>>>> >>>>>>>> week -- will be continuing to try to do the same this week.
>>>>> >>>>>>>>
>>>>> >>>>>>>> If you need any help, Stack, or if others need reviews where I
>>>>> >>>>>>>> haven't
>>>>> >>>>>>>> noticed on my own: feel free to @mention me.
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are
>>>>> being
>>>>> >>>>>>>>
>>>>> >>>>>>>> actively
>>>>> >>>>>>> worked on. If you get a moment, reviews by you (or anyone
>>>>> else) helps
>>>>> >>>>>>> move
>>>>> >>>>>>> the process along.
>>>>> >>>>>>>
>>>>> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
>>>>> >>>>>>> improvements.
>>>>> >>>>>>> Then HBASE-13346 can go in.
>>>>> >>>>>>>
>>>>> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks like
>>>>> that
>>>>> >>>>>>> will
>>>>> >>>>>>> be addressed by other alpha-4s about to land.
>>>>> >>>>>>>
>>>>> >>>>>>> St.Ack
>>>>> >>>>>>> TODOs: https://issues.apache.org/jira
>>>>> /projects/HBASE/versions/12341
>>>>> >>>>>>> 594
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>> (Reviving this thread)
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that
>>>>> has
>>>>> >>>>>>>>> the
>>>>> >>>>>>>>> refactor of the Coprocessor API shutting down access to
>>>>> internals
>>>>> >>>>>>>>> marked
>>>>> >>>>>>>>> InterfaceAudience.Private.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> The outstanding list is here:
>>>>> >>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341
>>>>> 594
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> If issue, talk out loud on this thread. If you need a review
>>>>> to
>>>>> >>>>>>>>> land
>>>>> >>>>>>>>> an
>>>>> >>>>>>>>> item, shout on the issue and here; we'll help you out.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> As is, items are coming along nicely I'd say. We need to
>>>>> merge the
>>>>> >>>>>>>>> filter
>>>>> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and
>>>>> help
>>>>> >>>>>>>>> them
>>>>> >>>>>>>>> test
>>>>> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence
>>>>> our
>>>>> >>>>>>>>> downstreamers know what to expect (or we discover what we
>>>>> missed
>>>>> >>>>>>>>> BEFORE
>>>>> >>>>>>>>> we
>>>>> >>>>>>>>> beta-1).
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Thanks for time,
>>>>> >>>>>>>>> S
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net>
>>>>> wrote:
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That
>>>>> should
>>>>> >>>>>>>>> be
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to
>>>>> be done.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled
>>>>> for
>>>>> >>>>>>>>>> alpha3
>>>>> >>>>>>>>>> but
>>>>> >>>>>>>>>> it is plain that more time is needed (in spite of valiant
>>>>> effort
>>>>> >>>>>>>>>> so
>>>>> >>>>>>>>>> far
>>>>> >>>>>>>>>> by
>>>>> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a
>>>>> 2.0.0-alpha-4 whose
>>>>> >>>>>>>>>> theme is
>>>>> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by
>>>>> the
>>>>> >>>>>>>>>> following
>>>>> >>>>>>>>>> week.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> We should then be ready for beta (beta == no new features,
>>>>> no API
>>>>> >>>>>>>>>> changes,
>>>>> >>>>>>>>>> just fixes).
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Thanks,
>>>>> >>>>>>>>>> St.Ack
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
>>>>> wrote:
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please
>>>>> vote on
>>>>> >>>>>>>>>> it.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I
>>>>> hope to
>>>>> >>>>>>>>>>> get a
>>>>> >>>>>>>>>>> release out in the next week or so.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If
>>>>> folks are
>>>>> >>>>>>>>>>> interested in helping out, below are the items I think we
>>>>> need
>>>>> >>>>>>>>>>> done
>>>>> >>>>>>>>>>> for
>>>>> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API
>>>>> possibly
>>>>> >>>>>>>>>>> altering
>>>>> >>>>>>>>>>> items, and are absent those JIRAs that are making active
>>>>> >>>>>>>>>>> progress,
>>>>> >>>>>>>>>>> i.e. the
>>>>> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed
>>>>> that
>>>>> >>>>>>>>>>> needs
>>>>> >>>>>>>>>>> doing is
>>>>> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between
>>>>> branch-1
>>>>> >>>>>>>>>>> and
>>>>> >>>>>>>>>>> branch-2
>>>>> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches
>>>>> only
>>>>> >>>>>>>>>>> do
>>>>> >>>>>>>>>>> it
>>>>> >>>>>>>>>>> between branch-1 and branch-2.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing
>>>>> Public /
>>>>> >>>>>>>>>>> Private
>>>>> >>>>>>>>>>> interfaces
>>>>> >>>>>>>>>>> This one is almost done. It could do with a finish,
>>>>> attention to
>>>>> >>>>>>>>>>> the
>>>>> >>>>>>>>>>> items in last comment, and then our codebase could do with
>>>>> >>>>>>>>>>> another
>>>>> >>>>>>>>>>> sweep
>>>>> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in
>>>>> since
>>>>> >>>>>>>>>>> the
>>>>> >>>>>>>>>>> pass
>>>>> >>>>>>>>>>> that was the basis of this issue.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
>>>>> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
>>>>> forced
>>>>> >>>>>>>>>>> by
>>>>> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
>>>>> >>>>>>>>>>> Service'
>>>>> >>>>>>>>>>> but if
>>>>> >>>>>>>>>>> anyone else is interested, be my guest.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked
>>>>> on. Needs
>>>>> >>>>>>>>>>> finishing.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods
>>>>> in Admin
>>>>> >>>>>>>>>>> and
>>>>> >>>>>>>>>>> cleanup
>>>>> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>>>>> >>>>>>>>>>> AsyncClient
>>>>> >>>>>>>>>>> and
>>>>> >>>>>>>>>>> make sure symmetry.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but
>>>>> needs to
>>>>> >>>>>>>>>>> be
>>>>> >>>>>>>>>>> done.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would
>>>>> be good
>>>>> >>>>>>>>>>> to
>>>>> >>>>>>>>>>> try
>>>>> >>>>>>>>>>> it and figure what needs fixup (as said above, according
>>>>> to trial
>>>>> >>>>>>>>>>> by
>>>>> >>>>>>>>>>> Sean,
>>>>> >>>>>>>>>>> we might not be too bad here):
>>>>> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state
>>>>> talking to
>>>>> >>>>>>>>>>> 2.0
>>>>> >>>>>>>>>>> cluster
>>>>> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
>>>>> Master
>>>>> >>>>>>>>>>> and
>>>>> >>>>>>>>>>> 1.x
>>>>> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-17442 Move most of the replication related classes
>>>>> to
>>>>> >>>>>>>>>>> hbase-server package
>>>>> >>>>>>>>>>> The above would be good to do generally but it may make for
>>>>> >>>>>>>>>>> ripples
>>>>> >>>>>>>>>>> in
>>>>> >>>>>>>>>>> API so would be good to do now.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid
>>>>> adding two
>>>>> >>>>>>>>>>> new
>>>>> >>>>>>>>>>> types
>>>>> >>>>>>>>>>> to our API, two types that are nought but curtailed,
>>>>> read-only
>>>>> >>>>>>>>>>> views
>>>>> >>>>>>>>>>> on
>>>>> >>>>>>>>>>> internals. Input if you have time appreciated.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate
>>>>> to a
>>>>> >>>>>>>>>>> hbase2
>>>>> >>>>>>>>>>> cluster; verify
>>>>> >>>>>>>>>>> Esteban is looking at this one
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in
>>>>> core
>>>>> >>>>>>>>>>> * HBASE-17143 Scan improvement
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>> >>>>>>>>>>> implementations
>>>>> >>>>>>>>>>> marked audience Private and returns implementations rather
>>>>> than
>>>>> >>>>>>>>>>> Interfaces.
>>>>> >>>>>>>>>>> In a few locations, we allow returning an alternate
>>>>> >>>>>>>>>>> implementation
>>>>> >>>>>>>>>>> altogether which is probably something we don't want a CP
>>>>> doing.
>>>>> >>>>>>>>>>> To
>>>>> >>>>>>>>>>> that
>>>>> >>>>>>>>>>> end, the following issues started by Duo and Anoop need to
>>>>> be
>>>>> >>>>>>>>>>> taken
>>>>> >>>>>>>>>>> to
>>>>> >>>>>>>>>>> the
>>>>> >>>>>>>>>>> finish line; ideally they'd have an owner:
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
>>>>> release <=
>>>>> >>>>>>>>>>> The
>>>>> >>>>>>>>>>> umbrella issue.
>>>>> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
>>>>> >>>>>>>>>>> expose
>>>>> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
>>>>> MasterObserver
>>>>> >>>>>>>>>>> and
>>>>> >>>>>>>>>>> RegionServerObserver
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Nice-to-haves:
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
>>>>> remove
>>>>> >>>>>>>>>>> unused
>>>>> >>>>>>>>>>> TimeRange constructors
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
>>>>> >>>>>>>>>>> references
>>>>> >>>>>>>>>>> existing in the code
>>>>> >>>>>>>>>>> This is the end of an old long-running project moving up
>>>>> on to
>>>>> >>>>>>>>>>> Cell
>>>>> >>>>>>>>>>> Interface. We think it is done but for a few little items
>>>>> >>>>>>>>>>> (deprecate
>>>>> >>>>>>>>>>> KV
>>>>> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is
>>>>> indeterminate;
>>>>> >>>>>>>>>>> needs
>>>>> >>>>>>>>>>> fixing
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>> >>>>>>>>>>> * HBASE-14997
>>>>> >>>>>>>>>>> Move compareOp and Comparators out of filter to client
>>>>> package
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> What about:
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> If interested in any of the above and you need a legup,
>>>>> just ask
>>>>> >>>>>>>>>>> in
>>>>> >>>>>>>>>>> the
>>>>> >>>>>>>>>>> issue and I'll be by....
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Thanks,
>>>>> >>>>>>>>>>> St.Ack
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
>>>>> >>>>>>>>>>> wrote:
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Heads-up:
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release
>>>>> Candidate.
>>>>> >>>>>>>>>>>> Theme
>>>>> >>>>>>>>>>>> is
>>>>> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
>>>>> (guava,
>>>>> >>>>>>>>>>>> netty,
>>>>> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
>>>>> >>>>>>>>>>>> master-carries-no-regions
>>>>> >>>>>>>>>>>> by default.
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?).
>>>>> Its theme
>>>>> >>>>>>>>>>>> will
>>>>> >>>>>>>>>>>> be
>>>>> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we
>>>>> are not
>>>>> >>>>>>>>>>>> looking too
>>>>> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have
>>>>> hbase-1
>>>>> >>>>>>>>>>>> client
>>>>> >>>>>>>>>>>> running
>>>>> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface
>>>>> >>>>>>>>>>>> revamp
>>>>> >>>>>>>>>>>> should be
>>>>> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>> >>>>>>>>>>>> Implementations, and
>>>>> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>> >>>>>>>>>>>> InterfaceAudience).
>>>>> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API;
>>>>> e.g.
>>>>> >>>>>>>>>>>> guava
>>>>> >>>>>>>>>>>> 0.12
>>>>> >>>>>>>>>>>> Service showing through in our replication API and
>>>>> protobufs in
>>>>> >>>>>>>>>>>> Admin
>>>>> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of
>>>>> outreach to
>>>>> >>>>>>>>>>>> make
>>>>> >>>>>>>>>>>> sure
>>>>> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> Beta1 in mid-September?
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>> >>>>>>>>>>>> https://issues.apache.org/jira
>>>>> /projects/HBASE/versions/12327188
>>>>> >>>>>>>>>>>> Edit
>>>>> >>>>>>>>>>>> as
>>>>> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not
>>>>> get any
>>>>> >>>>>>>>>>>> attention
>>>>> >>>>>>>>>>>> in next month or so.
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll
>>>>> leave them
>>>>> >>>>>>>>>>>> as
>>>>> >>>>>>>>>>>> is
>>>>> >>>>>>>>>>>> another while but I'll boot them soon.
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> While I have your attention:
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving
>>>>> hbase
>>>>> >>>>>>>>>>>> thrift
>>>>> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy
>>>>> enough
>>>>> >>>>>>>>>>>> if
>>>>> >>>>>>>>>>>> folks
>>>>> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
>>>>> first
>>>>> >>>>>>>>>>>> (0.98?).
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> St.Ack
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
>>>>> wrote:
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
>>>>> elserj@apache.org>
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>>> wrote:
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
>>>>> >>>>>>>>>>>>>> elserj@apache.org>
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>> wrote:
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>> ...
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would
>>>>> make the
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>> feature a
>>>>> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I
>>>>> have
>>>>> >>>>>>>>>>>>>>>> no
>>>>> >>>>>>>>>>>>>>>> reason
>>>>> >>>>>>>>>>>>>>>> to be
>>>>> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature
>>>>> in terms
>>>>> >>>>>>>>>>>>>>>> of
>>>>> >>>>>>>>>>>>>>>> how
>>>>> >>>>>>>>>>>>>>>> much it
>>>>> >>>>>>>>>>>>>>>> is covered via testing.
>>>>> >>>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>> >>>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so,
>>>>> +1 Josh.
>>>>> >>>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See
>>>>> 'Space
>>>>> >>>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>> Quota' in
>>>>> >>>>>>>>>>>>>>> here:
>>>>> >>>>>>>>>>>>>>> https://docs.google.com/docume
>>>>> nt/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>> >>>>>>>>>>>>>>> Does this little section need an update Josh?
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>> Thanks,
>>>>> >>>>>>>>>>>>>>> S
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage
>>>>> and
>>>>> >>>>>>>>>>>>>>> some
>>>>> >>>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>> docs
>>>>> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my
>>>>> plate if
>>>>> >>>>>>>>>>>>>> you're good
>>>>> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is against
>>>>> it :)
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
>>>>> before
>>>>> >>>>>>>>>>>>>> we
>>>>> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll
>>>>> double
>>>>> >>>>>>>>>>>>>> check
>>>>> >>>>>>>>>>>>>> and update
>>>>> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>> Thanks Josh,
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>> S
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>
>>>>> >>>>
>>>>> >>
>>>>>
>>>>
>>>>
>>>
>>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Update. See below.

On Tue, Dec 5, 2017 at 10:43 AM, Stack <st...@duboce.net> wrote:

> Update on hbase-2.0.0-beta-1. I'm going to put up a beta-1 RC before
> Christmas day.
>
>
Still aiming for an hbase2 beta-1 showing up in your christmas stocking.


> Here is the list [1]. We're doing pretty good. We've fixed a load since
> the last update including some nice cleanups  two weeks ago (e.g. undoing
> client dependency on curator/zk watching making the dependence read-only)
> but there are still a few hairy ones hanging out there.
>
>  * HBASE-18946 Stochastic load balancer assigns replica regions to the
> same RS
>    This one is proving a little tough. Ram has been banging his head on
> it. We'll be able to clear a bunch of test failures once this is done.
>
>
This should be going in tomorrow.



>  * HBASE-19112 Suspect methods on Cell to be deprecated
>    Good discussion up on the JIRA. We need to be crystal clear around Cell
> and derivatives API and Audience when we ship 2.0.
>
>
Ram, Chia-Ping Tsai, and Anoop are doing a nice job here and it should be
done in next day or so.



>  * HBASE-17204 Make L2 off heap cache default ON
>    We need to try this at least so offheaping can be default.
>
>
Chatted with Anoop and Ram. Because it is so late in the day and because it
critical that the user have a good experience out of the box -- which
requires time and testing -- we are going to punt this from 2.0. Will look
at it for 2.1 (or 3.0 if soon). 2.0 will still be carrying all of the boys
offheap read/write path goodness.



> Making asyncfswal default is also ongoing, HBASE-16890
> <https://issues.apache.org/jira/browse/HBASE-16890>, making good progress.
>
>

Good progress here. In perf testing asyncfswal makes us generally faster.
Duo making good progress on last few tests that are in the way of our
making this default for hbase2.

A load of goodness has been landing these last few weeks. Keep up the great
work.

Your hbase-2 RM.




> I punted HBASE-19147 "All branch-2 unit tests pass" out to beta-2. Our
> unit test story has gotten better and a few of us are actively working on
> flakies [2] but we may make a beta-1 w/o shutting down all failures.
>
> Speak up if you need help on an issue or if you think we are missing items
> from our list.
>
> Thanks for all the hard work,
>
> The hbase-2.0.0 RM.
>
>
> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
> 2. https://builds.apache.org/job/HBASE-Find-Flaky-Tests/
> lastSuccessfulBuild/artifact/dashboard.html
>
> On Thu, Nov 16, 2017 at 3:48 PM, Stack <st...@duboce.net> wrote:
>
>> hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
>> remaining features and apply final polish to API. There will be a beta-2
>> but it is about upgrade/rolling-upgrade and bug fixes ONLY).
>>
>> Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1
>> list this morning. See here [1].
>>
>> There are about ~12 issues in progress with most of these about to land.
>> There are 37 TODO. Many of these are tests we need to run, some are related
>> to the backup/restore, but a good few are meaty w/o assignees.
>>
>> The awkward outstanding ones as I see it are the below:
>>
>> HBASE-18946 Stochastic load balancer assigns replica regions to the same
>> RS
>> HBASE-17204 Make L2 off heap cache default ON
>> HBASE-19112 Suspect methods on Cell to be deprecated
>> HBASE-19147 All branch-2 unit tests pass
>>
>> We need to make progress on the above or punt on them (can't punt on the
>> last one though).
>>
>> Any ideas on what configs we should update in hbase2? Dump ideas into:
>> HBASE-19148 Edit of default configuration
>>
>> Still hoping for an early December beta-1 RC. beta-2 hopefully will be
>> close behind.
>>
>> Comments? Thoughts?
>>
>> Thanks all,
>> S
>>
>> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>>
>> On Tue, Oct 31, 2017 at 2:56 PM, Stack <st...@duboce.net> wrote:
>>
>>>
>>>
>>> On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:
>>>
>>>> Hoping to keep momentum going from our Stack working on alpha4, I tried
>>>> to
>>>> take a stab at triaging some of the open beta-1 issues.
>>>>
>>>> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done
>>>> sooner
>>>> then I'm happy to see it pulled back in. Trying to balance optimism with
>>>> realism here, and knowing that documentation unfortunately often gets
>>>> pushed to the back-burner.
>>>>
>>>> Also, I poked some folks on unassigned issues that they've filed for
>>>> beta-1, especially in the last few days. If issues don't have an owner
>>>> they
>>>> are unlikely to get worked. I chatted with stack and he agreed to take
>>>> on
>>>> some of the tasks, but there's a lot of surface area to cover.
>>>>
>>>> If you you're working on issues that are critical for beta-1, please
>>>> mark
>>>> them as such. Then the rest of the community will know to help
>>>> prioritize
>>>> feedback and reviews there.
>>>>
>>>> Do we have a general theme for the betas like we did with the alphas?
>>>>
>>>> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
>>>> well? Continue to work on tests throughout?
>>>>
>>>>
>>> Thanks Mike for helping to kick off the beta-1 train.
>>>
>>> Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what
>>> we are going to ship (beta-2 is rolling upgrade and any minor items turned
>>> up in testing/burn-in).
>>>
>>> Thanks,
>>> S
>>>
>>>
>>>
>>>> Mike
>>>>
>>>> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org> wrote:
>>>>
>>>> > +1 go from my POV.
>>>> >
>>>> >
>>>> > On 10/31/17 10:07 AM, Stack wrote:
>>>> >
>>>> >> I want to push an alpha-4 today. A few items didn't make it
>>>> (HBASE-19092).
>>>> >> They need more time. We'll pull them in for beta-1. CP API is
>>>> basically
>>>> >> done. There may be some changes for beta-1 but hopefully only changes
>>>> >> informed by experience trying to port an existing Coprocessor to
>>>> hbase2.
>>>> >>
>>>> >> Shout if there is anything that needs to make alpha-4.
>>>> >>
>>>> >> Thanks,
>>>> >> St.Ack
>>>> >>
>>>> >>
>>>> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
>>>> wrote:
>>>> >>
>>>> >> Yup, that was going to be my plan, Mike!
>>>> >>>
>>>> >>> Making a pass now, and will check back later tonight again. I see
>>>> others
>>>> >>> have already done some work today on this front.
>>>> >>>
>>>> >>>
>>>> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
>>>> >>>
>>>> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how
>>>> to do
>>>> >>>> it directly on the jenkins job, so you don't have to bother with
>>>> JIRA
>>>> >>>> uploads)
>>>> >>>>
>>>> >>>> If you're busy, then I can make time tomorrow or Sunday to kick off
>>>> >>>> jobs,
>>>> >>>> but I want to make sure we're not duplicating effort and jenkins
>>>> cycles.
>>>> >>>>
>>>> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
>>>> wrote:
>>>> >>>>
>>>> >>>> My turn to bump ;)
>>>> >>>>
>>>> >>>>>
>>>> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that
>>>> remain
>>>> >>>>> needing some more work. The rest are just awaiting a good QA run.
>>>> >>>>>
>>>> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over
>>>> the
>>>> >>>>> weekend, bump them along as necessary, and get them committed.
>>>> Would be
>>>> >>>>> great to be able get a vote up on Monday.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On 10/24/17 6:03 PM, Stack wrote:
>>>> >>>>>
>>>> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back and
>>>> forth
>>>> >>>>>
>>>> >>>>>> what remains to be done. Surfacing our thoughts here so you all
>>>> clued
>>>> >>>>>> in....Or if you think otherwise, please speak up.
>>>> >>>>>>
>>>> >>>>>> We have ~13 issues to land:
>>>> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>> About
>>>> >>>>>> two
>>>> >>>>>> are meta-issues that are about process which leaves 11.
>>>> >>>>>>
>>>> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>>> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
>>>> >>>>>> changed
>>>> >>>>>> API/semantic that we need to get out earlier rather than later.
>>>> >>>>>>
>>>> >>>>>> Once the above is merged, HBASE-13346, change of Filter method
>>>> names
>>>> >>>>>> to
>>>> >>>>>> mention Cell instead of KeyValue can land.
>>>> >>>>>>
>>>> >>>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
>>>> >>>>>> IA.Private objects as params to MasterObserver... Hopefully this
>>>> goes
>>>> >>>>>> in
>>>> >>>>>> soon.
>>>> >>>>>>
>>>> >>>>>> Duo is hard at work on trackers for flush and compaction for CPs
>>>> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for
>>>> Tephra)?
>>>> >>>>>>
>>>> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate
>>>> after
>>>> >>>>>> Duo
>>>> >>>>>> is
>>>> >>>>>> done w/ his work above.
>>>> >>>>>>
>>>> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity
>>>> after
>>>> >>>>>> all
>>>> >>>>>> the
>>>> >>>>>> purges allowing CPs do direct calls against Regions in same Host.
>>>> >>>>>>
>>>> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>>> >>>>>>
>>>> >>>>>> Another day or two?
>>>> >>>>>>
>>>> >>>>>> St.Ack
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
>>>> >>>>>> wrote:
>>>> >>>>>>
>>>> >>>>>>>
>>>> >>>>>>> +1
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>> I was trying to work on helping out on the outstanding alpha-4
>>>> stuff
>>>> >>>>>>>> last
>>>> >>>>>>>> week -- will be continuing to try to do the same this week.
>>>> >>>>>>>>
>>>> >>>>>>>> If you need any help, Stack, or if others need reviews where I
>>>> >>>>>>>> haven't
>>>> >>>>>>>> noticed on my own: feel free to @mention me.
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are
>>>> being
>>>> >>>>>>>>
>>>> >>>>>>>> actively
>>>> >>>>>>> worked on. If you get a moment, reviews by you (or anyone else)
>>>> helps
>>>> >>>>>>> move
>>>> >>>>>>> the process along.
>>>> >>>>>>>
>>>> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
>>>> >>>>>>> improvements.
>>>> >>>>>>> Then HBASE-13346 can go in.
>>>> >>>>>>>
>>>> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks like
>>>> that
>>>> >>>>>>> will
>>>> >>>>>>> be addressed by other alpha-4s about to land.
>>>> >>>>>>>
>>>> >>>>>>> St.Ack
>>>> >>>>>>> TODOs: https://issues.apache.org/jira
>>>> /projects/HBASE/versions/12341
>>>> >>>>>>> 594
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>> (Reviving this thread)
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that
>>>> has
>>>> >>>>>>>>> the
>>>> >>>>>>>>> refactor of the Coprocessor API shutting down access to
>>>> internals
>>>> >>>>>>>>> marked
>>>> >>>>>>>>> InterfaceAudience.Private.
>>>> >>>>>>>>>
>>>> >>>>>>>>> The outstanding list is here:
>>>> >>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341
>>>> 594
>>>> >>>>>>>>>
>>>> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>> >>>>>>>>>
>>>> >>>>>>>>> If issue, talk out loud on this thread. If you need a review
>>>> to
>>>> >>>>>>>>> land
>>>> >>>>>>>>> an
>>>> >>>>>>>>> item, shout on the issue and here; we'll help you out.
>>>> >>>>>>>>>
>>>> >>>>>>>>> As is, items are coming along nicely I'd say. We need to
>>>> merge the
>>>> >>>>>>>>> filter
>>>> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>> >>>>>>>>>
>>>> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and
>>>> help
>>>> >>>>>>>>> them
>>>> >>>>>>>>> test
>>>> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence
>>>> our
>>>> >>>>>>>>> downstreamers know what to expect (or we discover what we
>>>> missed
>>>> >>>>>>>>> BEFORE
>>>> >>>>>>>>> we
>>>> >>>>>>>>> beta-1).
>>>> >>>>>>>>>
>>>> >>>>>>>>> Thanks for time,
>>>> >>>>>>>>> S
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net>
>>>> wrote:
>>>> >>>>>>>>>
>>>> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That
>>>> should
>>>> >>>>>>>>> be
>>>> >>>>>>>>>
>>>> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to be
>>>> done.
>>>> >>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for
>>>> >>>>>>>>>> alpha3
>>>> >>>>>>>>>> but
>>>> >>>>>>>>>> it is plain that more time is needed (in spite of valiant
>>>> effort
>>>> >>>>>>>>>> so
>>>> >>>>>>>>>> far
>>>> >>>>>>>>>> by
>>>> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
>>>> whose
>>>> >>>>>>>>>> theme is
>>>> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by
>>>> the
>>>> >>>>>>>>>> following
>>>> >>>>>>>>>> week.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> We should then be ready for beta (beta == no new features,
>>>> no API
>>>> >>>>>>>>>> changes,
>>>> >>>>>>>>>> just fixes).
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Thanks,
>>>> >>>>>>>>>> St.Ack
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
>>>> wrote:
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please
>>>> vote on
>>>> >>>>>>>>>> it.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope
>>>> to
>>>> >>>>>>>>>>> get a
>>>> >>>>>>>>>>> release out in the next week or so.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks
>>>> are
>>>> >>>>>>>>>>> interested in helping out, below are the items I think we
>>>> need
>>>> >>>>>>>>>>> done
>>>> >>>>>>>>>>> for
>>>> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API
>>>> possibly
>>>> >>>>>>>>>>> altering
>>>> >>>>>>>>>>> items, and are absent those JIRAs that are making active
>>>> >>>>>>>>>>> progress,
>>>> >>>>>>>>>>> i.e. the
>>>> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
>>>> >>>>>>>>>>> needs
>>>> >>>>>>>>>>> doing is
>>>> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between
>>>> branch-1
>>>> >>>>>>>>>>> and
>>>> >>>>>>>>>>> branch-2
>>>> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches
>>>> only
>>>> >>>>>>>>>>> do
>>>> >>>>>>>>>>> it
>>>> >>>>>>>>>>> between branch-1 and branch-2.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public
>>>> /
>>>> >>>>>>>>>>> Private
>>>> >>>>>>>>>>> interfaces
>>>> >>>>>>>>>>> This one is almost done. It could do with a finish,
>>>> attention to
>>>> >>>>>>>>>>> the
>>>> >>>>>>>>>>> items in last comment, and then our codebase could do with
>>>> >>>>>>>>>>> another
>>>> >>>>>>>>>>> sweep
>>>> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in
>>>> since
>>>> >>>>>>>>>>> the
>>>> >>>>>>>>>>> pass
>>>> >>>>>>>>>>> that was the basis of this issue.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
>>>> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
>>>> forced
>>>> >>>>>>>>>>> by
>>>> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
>>>> >>>>>>>>>>> Service'
>>>> >>>>>>>>>>> but if
>>>> >>>>>>>>>>> anyone else is interested, be my guest.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked on.
>>>> Needs
>>>> >>>>>>>>>>> finishing.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
>>>> Admin
>>>> >>>>>>>>>>> and
>>>> >>>>>>>>>>> cleanup
>>>> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>>>> >>>>>>>>>>> AsyncClient
>>>> >>>>>>>>>>> and
>>>> >>>>>>>>>>> make sure symmetry.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but
>>>> needs to
>>>> >>>>>>>>>>> be
>>>> >>>>>>>>>>> done.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would
>>>> be good
>>>> >>>>>>>>>>> to
>>>> >>>>>>>>>>> try
>>>> >>>>>>>>>>> it and figure what needs fixup (as said above, according to
>>>> trial
>>>> >>>>>>>>>>> by
>>>> >>>>>>>>>>> Sean,
>>>> >>>>>>>>>>> we might not be too bad here):
>>>> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking
>>>> to
>>>> >>>>>>>>>>> 2.0
>>>> >>>>>>>>>>> cluster
>>>> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
>>>> Master
>>>> >>>>>>>>>>> and
>>>> >>>>>>>>>>> 1.x
>>>> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-17442 Move most of the replication related classes
>>>> to
>>>> >>>>>>>>>>> hbase-server package
>>>> >>>>>>>>>>> The above would be good to do generally but it may make for
>>>> >>>>>>>>>>> ripples
>>>> >>>>>>>>>>> in
>>>> >>>>>>>>>>> API so would be good to do now.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid adding
>>>> two
>>>> >>>>>>>>>>> new
>>>> >>>>>>>>>>> types
>>>> >>>>>>>>>>> to our API, two types that are nought but curtailed,
>>>> read-only
>>>> >>>>>>>>>>> views
>>>> >>>>>>>>>>> on
>>>> >>>>>>>>>>> internals. Input if you have time appreciated.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate
>>>> to a
>>>> >>>>>>>>>>> hbase2
>>>> >>>>>>>>>>> cluster; verify
>>>> >>>>>>>>>>> Esteban is looking at this one
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>> >>>>>>>>>>> * HBASE-17143 Scan improvement
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>> >>>>>>>>>>> implementations
>>>> >>>>>>>>>>> marked audience Private and returns implementations rather
>>>> than
>>>> >>>>>>>>>>> Interfaces.
>>>> >>>>>>>>>>> In a few locations, we allow returning an alternate
>>>> >>>>>>>>>>> implementation
>>>> >>>>>>>>>>> altogether which is probably something we don't want a CP
>>>> doing.
>>>> >>>>>>>>>>> To
>>>> >>>>>>>>>>> that
>>>> >>>>>>>>>>> end, the following issues started by Duo and Anoop need to
>>>> be
>>>> >>>>>>>>>>> taken
>>>> >>>>>>>>>>> to
>>>> >>>>>>>>>>> the
>>>> >>>>>>>>>>> finish line; ideally they'd have an owner:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
>>>> release <=
>>>> >>>>>>>>>>> The
>>>> >>>>>>>>>>> umbrella issue.
>>>> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
>>>> >>>>>>>>>>> expose
>>>> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
>>>> MasterObserver
>>>> >>>>>>>>>>> and
>>>> >>>>>>>>>>> RegionServerObserver
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Nice-to-haves:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
>>>> remove
>>>> >>>>>>>>>>> unused
>>>> >>>>>>>>>>> TimeRange constructors
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
>>>> >>>>>>>>>>> references
>>>> >>>>>>>>>>> existing in the code
>>>> >>>>>>>>>>> This is the end of an old long-running project moving up on
>>>> to
>>>> >>>>>>>>>>> Cell
>>>> >>>>>>>>>>> Interface. We think it is done but for a few little items
>>>> >>>>>>>>>>> (deprecate
>>>> >>>>>>>>>>> KV
>>>> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is
>>>> indeterminate;
>>>> >>>>>>>>>>> needs
>>>> >>>>>>>>>>> fixing
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>> >>>>>>>>>>> * HBASE-14997
>>>> >>>>>>>>>>> Move compareOp and Comparators out of filter to client
>>>> package
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> What about:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> If interested in any of the above and you need a legup,
>>>> just ask
>>>> >>>>>>>>>>> in
>>>> >>>>>>>>>>> the
>>>> >>>>>>>>>>> issue and I'll be by....
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Thanks,
>>>> >>>>>>>>>>> St.Ack
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
>>>> >>>>>>>>>>> wrote:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Heads-up:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
>>>> >>>>>>>>>>>> Theme
>>>> >>>>>>>>>>>> is
>>>> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
>>>> (guava,
>>>> >>>>>>>>>>>> netty,
>>>> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
>>>> >>>>>>>>>>>> master-carries-no-regions
>>>> >>>>>>>>>>>> by default.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
>>>> theme
>>>> >>>>>>>>>>>> will
>>>> >>>>>>>>>>>> be
>>>> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we
>>>> are not
>>>> >>>>>>>>>>>> looking too
>>>> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1
>>>> >>>>>>>>>>>> client
>>>> >>>>>>>>>>>> running
>>>> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface
>>>> >>>>>>>>>>>> revamp
>>>> >>>>>>>>>>>> should be
>>>> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>> >>>>>>>>>>>> Implementations, and
>>>> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>> >>>>>>>>>>>> InterfaceAudience).
>>>> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API;
>>>> e.g.
>>>> >>>>>>>>>>>> guava
>>>> >>>>>>>>>>>> 0.12
>>>> >>>>>>>>>>>> Service showing through in our replication API and
>>>> protobufs in
>>>> >>>>>>>>>>>> Admin
>>>> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of
>>>> outreach to
>>>> >>>>>>>>>>>> make
>>>> >>>>>>>>>>>> sure
>>>> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> Beta1 in mid-September?
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>> >>>>>>>>>>>> https://issues.apache.org/jira
>>>> /projects/HBASE/versions/12327188
>>>> >>>>>>>>>>>> Edit
>>>> >>>>>>>>>>>> as
>>>> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not
>>>> get any
>>>> >>>>>>>>>>>> attention
>>>> >>>>>>>>>>>> in next month or so.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll
>>>> leave them
>>>> >>>>>>>>>>>> as
>>>> >>>>>>>>>>>> is
>>>> >>>>>>>>>>>> another while but I'll boot them soon.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> While I have your attention:
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving
>>>> hbase
>>>> >>>>>>>>>>>> thrift
>>>> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy
>>>> enough
>>>> >>>>>>>>>>>> if
>>>> >>>>>>>>>>>> folks
>>>> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
>>>> first
>>>> >>>>>>>>>>>> (0.98?).
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> St.Ack
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
>>>> wrote:
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
>>>> elserj@apache.org>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>> wrote:
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
>>>> >>>>>>>>>>>>>> elserj@apache.org>
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> wrote:
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>> ...
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would
>>>> make the
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>> feature a
>>>> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I
>>>> have
>>>> >>>>>>>>>>>>>>>> no
>>>> >>>>>>>>>>>>>>>> reason
>>>> >>>>>>>>>>>>>>>> to be
>>>> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in
>>>> terms
>>>> >>>>>>>>>>>>>>>> of
>>>> >>>>>>>>>>>>>>>> how
>>>> >>>>>>>>>>>>>>>> much it
>>>> >>>>>>>>>>>>>>>> is covered via testing.
>>>> >>>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>> >>>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
>>>> Josh.
>>>> >>>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See
>>>> 'Space
>>>> >>>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>> Quota' in
>>>> >>>>>>>>>>>>>>> here:
>>>> >>>>>>>>>>>>>>> https://docs.google.com/docume
>>>> nt/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>> >>>>>>>>>>>>>>> Does this little section need an update Josh?
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>> Thanks,
>>>> >>>>>>>>>>>>>>> S
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage
>>>> and
>>>> >>>>>>>>>>>>>>> some
>>>> >>>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>> docs
>>>> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my
>>>> plate if
>>>> >>>>>>>>>>>>>> you're good
>>>> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is against
>>>> it :)
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
>>>> before
>>>> >>>>>>>>>>>>>> we
>>>> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll
>>>> double
>>>> >>>>>>>>>>>>>> check
>>>> >>>>>>>>>>>>>> and update
>>>> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> Thanks Josh,
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> S
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>>
>>>
>>>
>>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Update on hbase-2.0.0-beta-1. I'm going to put up a beta-1 RC before
Christmas day.

Here is the list [1]. We're doing pretty good. We've fixed a load since the
last update including some nice cleanups  two weeks ago (e.g. undoing
client dependency on curator/zk watching making the dependence read-only)
but there are still a few hairy ones hanging out there.

 * HBASE-18946 Stochastic load balancer assigns replica regions to the same
RS
   This one is proving a little tough. Ram has been banging his head on it.
We'll be able to clear a bunch of test failures once this is done.

 * HBASE-19112 Suspect methods on Cell to be deprecated
   Good discussion up on the JIRA. We need to be crystal clear around Cell
and derivatives API and Audience when we ship 2.0.

 * HBASE-17204 Make L2 off heap cache default ON
   We need to try this at least so offheaping can be default.

Making asyncfswal default is also ongoing, HBASE-16890
<https://issues.apache.org/jira/browse/HBASE-16890>, making good progress.

I punted HBASE-19147 "All branch-2 unit tests pass" out to beta-2. Our unit
test story has gotten better and a few of us are actively working on
flakies [2] but we may make a beta-1 w/o shutting down all failures.

Speak up if you need help on an issue or if you think we are missing items
from our list.

Thanks for all the hard work,

The hbase-2.0.0 RM.


1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
2.
https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html

On Thu, Nov 16, 2017 at 3:48 PM, Stack <st...@duboce.net> wrote:

> hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
> remaining features and apply final polish to API. There will be a beta-2
> but it is about upgrade/rolling-upgrade and bug fixes ONLY).
>
> Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1
> list this morning. See here [1].
>
> There are about ~12 issues in progress with most of these about to land.
> There are 37 TODO. Many of these are tests we need to run, some are related
> to the backup/restore, but a good few are meaty w/o assignees.
>
> The awkward outstanding ones as I see it are the below:
>
> HBASE-18946 Stochastic load balancer assigns replica regions to the same RS
> HBASE-17204 Make L2 off heap cache default ON
> HBASE-19112 Suspect methods on Cell to be deprecated
> HBASE-19147 All branch-2 unit tests pass
>
> We need to make progress on the above or punt on them (can't punt on the
> last one though).
>
> Any ideas on what configs we should update in hbase2? Dump ideas into:
> HBASE-19148 Edit of default configuration
>
> Still hoping for an early December beta-1 RC. beta-2 hopefully will be
> close behind.
>
> Comments? Thoughts?
>
> Thanks all,
> S
>
> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>
> On Tue, Oct 31, 2017 at 2:56 PM, Stack <st...@duboce.net> wrote:
>
>>
>>
>> On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:
>>
>>> Hoping to keep momentum going from our Stack working on alpha4, I tried
>>> to
>>> take a stab at triaging some of the open beta-1 issues.
>>>
>>> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done sooner
>>> then I'm happy to see it pulled back in. Trying to balance optimism with
>>> realism here, and knowing that documentation unfortunately often gets
>>> pushed to the back-burner.
>>>
>>> Also, I poked some folks on unassigned issues that they've filed for
>>> beta-1, especially in the last few days. If issues don't have an owner
>>> they
>>> are unlikely to get worked. I chatted with stack and he agreed to take on
>>> some of the tasks, but there's a lot of surface area to cover.
>>>
>>> If you you're working on issues that are critical for beta-1, please mark
>>> them as such. Then the rest of the community will know to help prioritize
>>> feedback and reviews there.
>>>
>>> Do we have a general theme for the betas like we did with the alphas?
>>>
>>> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
>>> well? Continue to work on tests throughout?
>>>
>>>
>> Thanks Mike for helping to kick off the beta-1 train.
>>
>> Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what we
>> are going to ship (beta-2 is rolling upgrade and any minor items turned up
>> in testing/burn-in).
>>
>> Thanks,
>> S
>>
>>
>>
>>> Mike
>>>
>>> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org> wrote:
>>>
>>> > +1 go from my POV.
>>> >
>>> >
>>> > On 10/31/17 10:07 AM, Stack wrote:
>>> >
>>> >> I want to push an alpha-4 today. A few items didn't make it
>>> (HBASE-19092).
>>> >> They need more time. We'll pull them in for beta-1. CP API is
>>> basically
>>> >> done. There may be some changes for beta-1 but hopefully only changes
>>> >> informed by experience trying to port an existing Coprocessor to
>>> hbase2.
>>> >>
>>> >> Shout if there is anything that needs to make alpha-4.
>>> >>
>>> >> Thanks,
>>> >> St.Ack
>>> >>
>>> >>
>>> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
>>> wrote:
>>> >>
>>> >> Yup, that was going to be my plan, Mike!
>>> >>>
>>> >>> Making a pass now, and will check back later tonight again. I see
>>> others
>>> >>> have already done some work today on this front.
>>> >>>
>>> >>>
>>> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
>>> >>>
>>> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how
>>> to do
>>> >>>> it directly on the jenkins job, so you don't have to bother with
>>> JIRA
>>> >>>> uploads)
>>> >>>>
>>> >>>> If you're busy, then I can make time tomorrow or Sunday to kick off
>>> >>>> jobs,
>>> >>>> but I want to make sure we're not duplicating effort and jenkins
>>> cycles.
>>> >>>>
>>> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
>>> wrote:
>>> >>>>
>>> >>>> My turn to bump ;)
>>> >>>>
>>> >>>>>
>>> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that
>>> remain
>>> >>>>> needing some more work. The rest are just awaiting a good QA run.
>>> >>>>>
>>> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over the
>>> >>>>> weekend, bump them along as necessary, and get them committed.
>>> Would be
>>> >>>>> great to be able get a vote up on Monday.
>>> >>>>>
>>> >>>>>
>>> >>>>> On 10/24/17 6:03 PM, Stack wrote:
>>> >>>>>
>>> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back and
>>> forth
>>> >>>>>
>>> >>>>>> what remains to be done. Surfacing our thoughts here so you all
>>> clued
>>> >>>>>> in....Or if you think otherwise, please speak up.
>>> >>>>>>
>>> >>>>>> We have ~13 issues to land:
>>> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>> About
>>> >>>>>> two
>>> >>>>>> are meta-issues that are about process which leaves 11.
>>> >>>>>>
>>> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
>>> >>>>>> changed
>>> >>>>>> API/semantic that we need to get out earlier rather than later.
>>> >>>>>>
>>> >>>>>> Once the above is merged, HBASE-13346, change of Filter method
>>> names
>>> >>>>>> to
>>> >>>>>> mention Cell instead of KeyValue can land.
>>> >>>>>>
>>> >>>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
>>> >>>>>> IA.Private objects as params to MasterObserver... Hopefully this
>>> goes
>>> >>>>>> in
>>> >>>>>> soon.
>>> >>>>>>
>>> >>>>>> Duo is hard at work on trackers for flush and compaction for CPs
>>> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for
>>> Tephra)?
>>> >>>>>>
>>> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate
>>> after
>>> >>>>>> Duo
>>> >>>>>> is
>>> >>>>>> done w/ his work above.
>>> >>>>>>
>>> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity
>>> after
>>> >>>>>> all
>>> >>>>>> the
>>> >>>>>> purges allowing CPs do direct calls against Regions in same Host.
>>> >>>>>>
>>> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>> >>>>>>
>>> >>>>>> Another day or two?
>>> >>>>>>
>>> >>>>>> St.Ack
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>>>
>>> >>>>>>> +1
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>> I was trying to work on helping out on the outstanding alpha-4
>>> stuff
>>> >>>>>>>> last
>>> >>>>>>>> week -- will be continuing to try to do the same this week.
>>> >>>>>>>>
>>> >>>>>>>> If you need any help, Stack, or if others need reviews where I
>>> >>>>>>>> haven't
>>> >>>>>>>> noticed on my own: feel free to @mention me.
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are being
>>> >>>>>>>>
>>> >>>>>>>> actively
>>> >>>>>>> worked on. If you get a moment, reviews by you (or anyone else)
>>> helps
>>> >>>>>>> move
>>> >>>>>>> the process along.
>>> >>>>>>>
>>> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
>>> >>>>>>> improvements.
>>> >>>>>>> Then HBASE-13346 can go in.
>>> >>>>>>>
>>> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks like
>>> that
>>> >>>>>>> will
>>> >>>>>>> be addressed by other alpha-4s about to land.
>>> >>>>>>>
>>> >>>>>>> St.Ack
>>> >>>>>>> TODOs: https://issues.apache.org/jira
>>> /projects/HBASE/versions/12341
>>> >>>>>>> 594
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>> (Reviving this thread)
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that
>>> has
>>> >>>>>>>>> the
>>> >>>>>>>>> refactor of the Coprocessor API shutting down access to
>>> internals
>>> >>>>>>>>> marked
>>> >>>>>>>>> InterfaceAudience.Private.
>>> >>>>>>>>>
>>> >>>>>>>>> The outstanding list is here:
>>> >>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341
>>> 594
>>> >>>>>>>>>
>>> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>> >>>>>>>>>
>>> >>>>>>>>> If issue, talk out loud on this thread. If you need a review to
>>> >>>>>>>>> land
>>> >>>>>>>>> an
>>> >>>>>>>>> item, shout on the issue and here; we'll help you out.
>>> >>>>>>>>>
>>> >>>>>>>>> As is, items are coming along nicely I'd say. We need to merge
>>> the
>>> >>>>>>>>> filter
>>> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>> >>>>>>>>>
>>> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and
>>> help
>>> >>>>>>>>> them
>>> >>>>>>>>> test
>>> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence
>>> our
>>> >>>>>>>>> downstreamers know what to expect (or we discover what we
>>> missed
>>> >>>>>>>>> BEFORE
>>> >>>>>>>>> we
>>> >>>>>>>>> beta-1).
>>> >>>>>>>>>
>>> >>>>>>>>> Thanks for time,
>>> >>>>>>>>> S
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net>
>>> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That
>>> should
>>> >>>>>>>>> be
>>> >>>>>>>>>
>>> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to be
>>> done.
>>> >>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for
>>> >>>>>>>>>> alpha3
>>> >>>>>>>>>> but
>>> >>>>>>>>>> it is plain that more time is needed (in spite of valiant
>>> effort
>>> >>>>>>>>>> so
>>> >>>>>>>>>> far
>>> >>>>>>>>>> by
>>> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
>>> whose
>>> >>>>>>>>>> theme is
>>> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>>> >>>>>>>>>> following
>>> >>>>>>>>>> week.
>>> >>>>>>>>>>
>>> >>>>>>>>>> We should then be ready for beta (beta == no new features, no
>>> API
>>> >>>>>>>>>> changes,
>>> >>>>>>>>>> just fixes).
>>> >>>>>>>>>>
>>> >>>>>>>>>> Thanks,
>>> >>>>>>>>>> St.Ack
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
>>> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please
>>> vote on
>>> >>>>>>>>>> it.
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope
>>> to
>>> >>>>>>>>>>> get a
>>> >>>>>>>>>>> release out in the next week or so.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks
>>> are
>>> >>>>>>>>>>> interested in helping out, below are the items I think we
>>> need
>>> >>>>>>>>>>> done
>>> >>>>>>>>>>> for
>>> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API
>>> possibly
>>> >>>>>>>>>>> altering
>>> >>>>>>>>>>> items, and are absent those JIRAs that are making active
>>> >>>>>>>>>>> progress,
>>> >>>>>>>>>>> i.e. the
>>> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
>>> >>>>>>>>>>> needs
>>> >>>>>>>>>>> doing is
>>> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between
>>> branch-1
>>> >>>>>>>>>>> and
>>> >>>>>>>>>>> branch-2
>>> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches
>>> only
>>> >>>>>>>>>>> do
>>> >>>>>>>>>>> it
>>> >>>>>>>>>>> between branch-1 and branch-2.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public /
>>> >>>>>>>>>>> Private
>>> >>>>>>>>>>> interfaces
>>> >>>>>>>>>>> This one is almost done. It could do with a finish,
>>> attention to
>>> >>>>>>>>>>> the
>>> >>>>>>>>>>> items in last comment, and then our codebase could do with
>>> >>>>>>>>>>> another
>>> >>>>>>>>>>> sweep
>>> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in
>>> since
>>> >>>>>>>>>>> the
>>> >>>>>>>>>>> pass
>>> >>>>>>>>>>> that was the basis of this issue.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
>>> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
>>> forced
>>> >>>>>>>>>>> by
>>> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
>>> >>>>>>>>>>> Service'
>>> >>>>>>>>>>> but if
>>> >>>>>>>>>>> anyone else is interested, be my guest.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked on.
>>> Needs
>>> >>>>>>>>>>> finishing.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
>>> Admin
>>> >>>>>>>>>>> and
>>> >>>>>>>>>>> cleanup
>>> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>>> >>>>>>>>>>> AsyncClient
>>> >>>>>>>>>>> and
>>> >>>>>>>>>>> make sure symmetry.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but
>>> needs to
>>> >>>>>>>>>>> be
>>> >>>>>>>>>>> done.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would be
>>> good
>>> >>>>>>>>>>> to
>>> >>>>>>>>>>> try
>>> >>>>>>>>>>> it and figure what needs fixup (as said above, according to
>>> trial
>>> >>>>>>>>>>> by
>>> >>>>>>>>>>> Sean,
>>> >>>>>>>>>>> we might not be too bad here):
>>> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking
>>> to
>>> >>>>>>>>>>> 2.0
>>> >>>>>>>>>>> cluster
>>> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
>>> Master
>>> >>>>>>>>>>> and
>>> >>>>>>>>>>> 1.x
>>> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-17442 Move most of the replication related classes to
>>> >>>>>>>>>>> hbase-server package
>>> >>>>>>>>>>> The above would be good to do generally but it may make for
>>> >>>>>>>>>>> ripples
>>> >>>>>>>>>>> in
>>> >>>>>>>>>>> API so would be good to do now.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid adding
>>> two
>>> >>>>>>>>>>> new
>>> >>>>>>>>>>> types
>>> >>>>>>>>>>> to our API, two types that are nought but curtailed,
>>> read-only
>>> >>>>>>>>>>> views
>>> >>>>>>>>>>> on
>>> >>>>>>>>>>> internals. Input if you have time appreciated.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate
>>> to a
>>> >>>>>>>>>>> hbase2
>>> >>>>>>>>>>> cluster; verify
>>> >>>>>>>>>>> Esteban is looking at this one
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>> >>>>>>>>>>> * HBASE-17143 Scan improvement
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>> >>>>>>>>>>> implementations
>>> >>>>>>>>>>> marked audience Private and returns implementations rather
>>> than
>>> >>>>>>>>>>> Interfaces.
>>> >>>>>>>>>>> In a few locations, we allow returning an alternate
>>> >>>>>>>>>>> implementation
>>> >>>>>>>>>>> altogether which is probably something we don't want a CP
>>> doing.
>>> >>>>>>>>>>> To
>>> >>>>>>>>>>> that
>>> >>>>>>>>>>> end, the following issues started by Duo and Anoop need to be
>>> >>>>>>>>>>> taken
>>> >>>>>>>>>>> to
>>> >>>>>>>>>>> the
>>> >>>>>>>>>>> finish line; ideally they'd have an owner:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
>>> release <=
>>> >>>>>>>>>>> The
>>> >>>>>>>>>>> umbrella issue.
>>> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
>>> >>>>>>>>>>> expose
>>> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
>>> MasterObserver
>>> >>>>>>>>>>> and
>>> >>>>>>>>>>> RegionServerObserver
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Nice-to-haves:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
>>> remove
>>> >>>>>>>>>>> unused
>>> >>>>>>>>>>> TimeRange constructors
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
>>> >>>>>>>>>>> references
>>> >>>>>>>>>>> existing in the code
>>> >>>>>>>>>>> This is the end of an old long-running project moving up on
>>> to
>>> >>>>>>>>>>> Cell
>>> >>>>>>>>>>> Interface. We think it is done but for a few little items
>>> >>>>>>>>>>> (deprecate
>>> >>>>>>>>>>> KV
>>> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is
>>> indeterminate;
>>> >>>>>>>>>>> needs
>>> >>>>>>>>>>> fixing
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>> >>>>>>>>>>> * HBASE-14997
>>> >>>>>>>>>>> Move compareOp and Comparators out of filter to client
>>> package
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> What about:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> If interested in any of the above and you need a legup, just
>>> ask
>>> >>>>>>>>>>> in
>>> >>>>>>>>>>> the
>>> >>>>>>>>>>> issue and I'll be by....
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Thanks,
>>> >>>>>>>>>>> St.Ack
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
>>> >>>>>>>>>>> wrote:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Heads-up:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
>>> >>>>>>>>>>>> Theme
>>> >>>>>>>>>>>> is
>>> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
>>> (guava,
>>> >>>>>>>>>>>> netty,
>>> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
>>> >>>>>>>>>>>> master-carries-no-regions
>>> >>>>>>>>>>>> by default.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
>>> theme
>>> >>>>>>>>>>>> will
>>> >>>>>>>>>>>> be
>>> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we are
>>> not
>>> >>>>>>>>>>>> looking too
>>> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1
>>> >>>>>>>>>>>> client
>>> >>>>>>>>>>>> running
>>> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface
>>> >>>>>>>>>>>> revamp
>>> >>>>>>>>>>>> should be
>>> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>> >>>>>>>>>>>> Implementations, and
>>> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>> >>>>>>>>>>>> InterfaceAudience).
>>> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API; e.g.
>>> >>>>>>>>>>>> guava
>>> >>>>>>>>>>>> 0.12
>>> >>>>>>>>>>>> Service showing through in our replication API and
>>> protobufs in
>>> >>>>>>>>>>>> Admin
>>> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of
>>> outreach to
>>> >>>>>>>>>>>> make
>>> >>>>>>>>>>>> sure
>>> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> Beta1 in mid-September?
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>> >>>>>>>>>>>> https://issues.apache.org/jira
>>> /projects/HBASE/versions/12327188
>>> >>>>>>>>>>>> Edit
>>> >>>>>>>>>>>> as
>>> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get
>>> any
>>> >>>>>>>>>>>> attention
>>> >>>>>>>>>>>> in next month or so.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave
>>> them
>>> >>>>>>>>>>>> as
>>> >>>>>>>>>>>> is
>>> >>>>>>>>>>>> another while but I'll boot them soon.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> While I have your attention:
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving
>>> hbase
>>> >>>>>>>>>>>> thrift
>>> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy
>>> enough
>>> >>>>>>>>>>>> if
>>> >>>>>>>>>>>> folks
>>> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
>>> first
>>> >>>>>>>>>>>> (0.98?).
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> St.Ack
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
>>> wrote:
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
>>> elserj@apache.org>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
>>> >>>>>>>>>>>>>> elserj@apache.org>
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> ...
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would
>>> make the
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> feature a
>>> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I
>>> have
>>> >>>>>>>>>>>>>>>> no
>>> >>>>>>>>>>>>>>>> reason
>>> >>>>>>>>>>>>>>>> to be
>>> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in
>>> terms
>>> >>>>>>>>>>>>>>>> of
>>> >>>>>>>>>>>>>>>> how
>>> >>>>>>>>>>>>>>>> much it
>>> >>>>>>>>>>>>>>>> is covered via testing.
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
>>> Josh.
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See
>>> 'Space
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Quota' in
>>> >>>>>>>>>>>>>>> here:
>>> >>>>>>>>>>>>>>> https://docs.google.com/docume
>>> nt/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>> >>>>>>>>>>>>>>> Does this little section need an update Josh?
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Thanks,
>>> >>>>>>>>>>>>>>> S
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage
>>> and
>>> >>>>>>>>>>>>>>> some
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> docs
>>> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my
>>> plate if
>>> >>>>>>>>>>>>>> you're good
>>> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is against
>>> it :)
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
>>> before
>>> >>>>>>>>>>>>>> we
>>> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll double
>>> >>>>>>>>>>>>>> check
>>> >>>>>>>>>>>>>> and update
>>> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> Thanks Josh,
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> S
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>>
>>
>>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Thu, Nov 16, 2017 at 8:08 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:

> HBASE-18946 Stochastic load balancer assigns replica regions to the same RS
>
> Am on this today and probably will put up another patch.
>
> Regards
> Ram
>
> Thanks Ram,
St.Ack



> On Fri, Nov 17, 2017 at 5:18 AM, Stack <st...@duboce.net> wrote:
>
> > hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
> > remaining features and apply final polish to API. There will be a beta-2
> > but it is about upgrade/rolling-upgrade and bug fixes ONLY).
> >
> > Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1
> list
> > this morning. See here [1].
> >
> > There are about ~12 issues in progress with most of these about to land.
> > There are 37 TODO. Many of these are tests we need to run, some are
> related
> > to the backup/restore, but a good few are meaty w/o assignees.
> >
> > The awkward outstanding ones as I see it are the below:
> >
> > HBASE-18946 Stochastic load balancer assigns replica regions to the same
> RS
> > HBASE-17204 Make L2 off heap cache default ON
> > HBASE-19112 Suspect methods on Cell to be deprecated
> > HBASE-19147 All branch-2 unit tests pass
> >
> > We need to make progress on the above or punt on them (can't punt on the
> > last one though).
> >
> > Any ideas on what configs we should update in hbase2? Dump ideas into:
> > HBASE-19148 Edit of default configuration
> >
> > Still hoping for an early December beta-1 RC. beta-2 hopefully will be
> > close behind.
> >
> > Comments? Thoughts?
> >
> > Thanks all,
> > S
> >
> > 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
> >
> > On Tue, Oct 31, 2017 at 2:56 PM, Stack <st...@duboce.net> wrote:
> >
> > >
> > >
> > > On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:
> > >
> > >> Hoping to keep momentum going from our Stack working on alpha4, I
> tried
> > to
> > >> take a stab at triaging some of the open beta-1 issues.
> > >>
> > >> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done
> > sooner
> > >> then I'm happy to see it pulled back in. Trying to balance optimism
> with
> > >> realism here, and knowing that documentation unfortunately often gets
> > >> pushed to the back-burner.
> > >>
> > >> Also, I poked some folks on unassigned issues that they've filed for
> > >> beta-1, especially in the last few days. If issues don't have an owner
> > >> they
> > >> are unlikely to get worked. I chatted with stack and he agreed to take
> > on
> > >> some of the tasks, but there's a lot of surface area to cover.
> > >>
> > >> If you you're working on issues that are critical for beta-1, please
> > mark
> > >> them as such. Then the rest of the community will know to help
> > prioritize
> > >> feedback and reviews there.
> > >>
> > >> Do we have a general theme for the betas like we did with the alphas?
> > >>
> > >> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
> > >> well? Continue to work on tests throughout?
> > >>
> > >>
> > > Thanks Mike for helping to kick off the beta-1 train.
> > >
> > > Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what
> we
> > > are going to ship (beta-2 is rolling upgrade and any minor items turned
> > up
> > > in testing/burn-in).
> > >
> > > Thanks,
> > > S
> > >
> > >
> > >
> > >> Mike
> > >>
> > >> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org>
> wrote:
> > >>
> > >> > +1 go from my POV.
> > >> >
> > >> >
> > >> > On 10/31/17 10:07 AM, Stack wrote:
> > >> >
> > >> >> I want to push an alpha-4 today. A few items didn't make it
> > >> (HBASE-19092).
> > >> >> They need more time. We'll pull them in for beta-1. CP API is
> > basically
> > >> >> done. There may be some changes for beta-1 but hopefully only
> changes
> > >> >> informed by experience trying to port an existing Coprocessor to
> > >> hbase2.
> > >> >>
> > >> >> Shout if there is anything that needs to make alpha-4.
> > >> >>
> > >> >> Thanks,
> > >> >> St.Ack
> > >> >>
> > >> >>
> > >> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
> > >> wrote:
> > >> >>
> > >> >> Yup, that was going to be my plan, Mike!
> > >> >>>
> > >> >>> Making a pass now, and will check back later tonight again. I see
> > >> others
> > >> >>> have already done some work today on this front.
> > >> >>>
> > >> >>>
> > >> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
> > >> >>>
> > >> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know
> how
> > >> to do
> > >> >>>> it directly on the jenkins job, so you don't have to bother with
> > JIRA
> > >> >>>> uploads)
> > >> >>>>
> > >> >>>> If you're busy, then I can make time tomorrow or Sunday to kick
> off
> > >> >>>> jobs,
> > >> >>>> but I want to make sure we're not duplicating effort and jenkins
> > >> cycles.
> > >> >>>>
> > >> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
> > >> wrote:
> > >> >>>>
> > >> >>>> My turn to bump ;)
> > >> >>>>
> > >> >>>>>
> > >> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that
> > >> remain
> > >> >>>>> needing some more work. The rest are just awaiting a good QA
> run.
> > >> >>>>>
> > >> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over
> > the
> > >> >>>>> weekend, bump them along as necessary, and get them committed.
> > >> Would be
> > >> >>>>> great to be able get a vote up on Monday.
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> On 10/24/17 6:03 PM, Stack wrote:
> > >> >>>>>
> > >> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back
> and
> > >> forth
> > >> >>>>>
> > >> >>>>>> what remains to be done. Surfacing our thoughts here so you all
> > >> clued
> > >> >>>>>> in....Or if you think otherwise, please speak up.
> > >> >>>>>>
> > >> >>>>>> We have ~13 issues to land:
> > >> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> 12341594
> > >> About
> > >> >>>>>> two
> > >> >>>>>> are meta-issues that are about process which leaves 11.
> > >> >>>>>>
> > >> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
> > >> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because
> some
> > >> >>>>>> changed
> > >> >>>>>> API/semantic that we need to get out earlier rather than later.
> > >> >>>>>>
> > >> >>>>>> Once the above is merged, HBASE-13346, change of Filter method
> > >> names
> > >> >>>>>> to
> > >> >>>>>> mention Cell instead of KeyValue can land.
> > >> >>>>>>
> > >> >>>>>> HBASE-199048 needs a review (Anoop will probably do it),
> removing
> > >> >>>>>> IA.Private objects as params to MasterObserver... Hopefully
> this
> > >> goes
> > >> >>>>>> in
> > >> >>>>>> soon.
> > >> >>>>>>
> > >> >>>>>> Duo is hard at work on trackers for flush and compaction for
> CPs
> > >> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for
> > >> Tephra)?
> > >> >>>>>>
> > >> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate
> > >> after
> > >> >>>>>> Duo
> > >> >>>>>> is
> > >> >>>>>> done w/ his work above.
> > >> >>>>>>
> > >> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity
> > after
> > >> >>>>>> all
> > >> >>>>>> the
> > >> >>>>>> purges allowing CPs do direct calls against Regions in same
> Host.
> > >> >>>>>>
> > >> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> > >> >>>>>>
> > >> >>>>>> Another day or two?
> > >> >>>>>>
> > >> >>>>>> St.Ack
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net>
> wrote:
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <
> elserj@apache.org>
> > >> >>>>>> wrote:
> > >> >>>>>>
> > >> >>>>>>>
> > >> >>>>>>> +1
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>> I was trying to work on helping out on the outstanding
> alpha-4
> > >> stuff
> > >> >>>>>>>> last
> > >> >>>>>>>> week -- will be continuing to try to do the same this week.
> > >> >>>>>>>>
> > >> >>>>>>>> If you need any help, Stack, or if others need reviews where
> I
> > >> >>>>>>>> haven't
> > >> >>>>>>>> noticed on my own: feel free to @mention me.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are
> > being
> > >> >>>>>>>>
> > >> >>>>>>>> actively
> > >> >>>>>>> worked on. If you get a moment, reviews by you (or anyone
> else)
> > >> helps
> > >> >>>>>>> move
> > >> >>>>>>> the process along.
> > >> >>>>>>>
> > >> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
> > >> >>>>>>> improvements.
> > >> >>>>>>> Then HBASE-13346 can go in.
> > >> >>>>>>>
> > >> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks like
> > >> that
> > >> >>>>>>> will
> > >> >>>>>>> be addressed by other alpha-4s about to land.
> > >> >>>>>>>
> > >> >>>>>>> St.Ack
> > >> >>>>>>> TODOs: https://issues.apache.org/jira
> > >> /projects/HBASE/versions/12341
> > >> >>>>>>> 594
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>> (Reviving this thread)
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that
> > has
> > >> >>>>>>>>> the
> > >> >>>>>>>>> refactor of the Coprocessor API shutting down access to
> > >> internals
> > >> >>>>>>>>> marked
> > >> >>>>>>>>> InterfaceAudience.Private.
> > >> >>>>>>>>>
> > >> >>>>>>>>> The outstanding list is here:
> > >> >>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> > 12341594
> > >> >>>>>>>>>
> > >> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
> > >> >>>>>>>>>
> > >> >>>>>>>>> If issue, talk out loud on this thread. If you need a review
> > to
> > >> >>>>>>>>> land
> > >> >>>>>>>>> an
> > >> >>>>>>>>> item, shout on the issue and here; we'll help you out.
> > >> >>>>>>>>>
> > >> >>>>>>>>> As is, items are coming along nicely I'd say. We need to
> merge
> > >> the
> > >> >>>>>>>>> filter
> > >> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> > >> >>>>>>>>>
> > >> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and
> > help
> > >> >>>>>>>>> them
> > >> >>>>>>>>> test
> > >> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence
> > our
> > >> >>>>>>>>> downstreamers know what to expect (or we discover what we
> > missed
> > >> >>>>>>>>> BEFORE
> > >> >>>>>>>>> we
> > >> >>>>>>>>> beta-1).
> > >> >>>>>>>>>
> > >> >>>>>>>>> Thanks for time,
> > >> >>>>>>>>> S
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net>
> > wrote:
> > >> >>>>>>>>>
> > >> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That
> > >> should
> > >> >>>>>>>>> be
> > >> >>>>>>>>>
> > >> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to
> be
> > >> done.
> > >> >>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled
> for
> > >> >>>>>>>>>> alpha3
> > >> >>>>>>>>>> but
> > >> >>>>>>>>>> it is plain that more time is needed (in spite of valiant
> > >> effort
> > >> >>>>>>>>>> so
> > >> >>>>>>>>>> far
> > >> >>>>>>>>>> by
> > >> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a
> 2.0.0-alpha-4
> > >> whose
> > >> >>>>>>>>>> theme is
> > >> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by
> > the
> > >> >>>>>>>>>> following
> > >> >>>>>>>>>> week.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> We should then be ready for beta (beta == no new features,
> no
> > >> API
> > >> >>>>>>>>>> changes,
> > >> >>>>>>>>>> just fixes).
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Thanks,
> > >> >>>>>>>>>> St.Ack
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
> > >> wrote:
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please
> > vote
> > >> on
> > >> >>>>>>>>>> it.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I
> hope
> > to
> > >> >>>>>>>>>>> get a
> > >> >>>>>>>>>>> release out in the next week or so.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If
> folks
> > >> are
> > >> >>>>>>>>>>> interested in helping out, below are the items I think we
> > need
> > >> >>>>>>>>>>> done
> > >> >>>>>>>>>>> for
> > >> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API
> > possibly
> > >> >>>>>>>>>>> altering
> > >> >>>>>>>>>>> items, and are absent those JIRAs that are making active
> > >> >>>>>>>>>>> progress,
> > >> >>>>>>>>>>> i.e. the
> > >> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed
> that
> > >> >>>>>>>>>>> needs
> > >> >>>>>>>>>>> doing is
> > >> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between
> > branch-1
> > >> >>>>>>>>>>> and
> > >> >>>>>>>>>>> branch-2
> > >> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches
> > >> only
> > >> >>>>>>>>>>> do
> > >> >>>>>>>>>>> it
> > >> >>>>>>>>>>> between branch-1 and branch-2.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing
> Public
> > /
> > >> >>>>>>>>>>> Private
> > >> >>>>>>>>>>> interfaces
> > >> >>>>>>>>>>> This one is almost done. It could do with a finish,
> > attention
> > >> to
> > >> >>>>>>>>>>> the
> > >> >>>>>>>>>>> items in last comment, and then our codebase could do with
> > >> >>>>>>>>>>> another
> > >> >>>>>>>>>>> sweep
> > >> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in
> > since
> > >> >>>>>>>>>>> the
> > >> >>>>>>>>>>> pass
> > >> >>>>>>>>>>> that was the basis of this issue.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
> > >> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
> > >> forced
> > >> >>>>>>>>>>> by
> > >> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> > >> >>>>>>>>>>> Service'
> > >> >>>>>>>>>>> but if
> > >> >>>>>>>>>>> anyone else is interested, be my guest.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
> > >> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked
> on.
> > >> Needs
> > >> >>>>>>>>>>> finishing.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods
> in
> > >> Admin
> > >> >>>>>>>>>>> and
> > >> >>>>>>>>>>> cleanup
> > >> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
> > >> >>>>>>>>>>> AsyncClient
> > >> >>>>>>>>>>> and
> > >> >>>>>>>>>>> make sure symmetry.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> > >> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but
> > >> needs to
> > >> >>>>>>>>>>> be
> > >> >>>>>>>>>>> done.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would
> be
> > >> good
> > >> >>>>>>>>>>> to
> > >> >>>>>>>>>>> try
> > >> >>>>>>>>>>> it and figure what needs fixup (as said above, according
> to
> > >> trial
> > >> >>>>>>>>>>> by
> > >> >>>>>>>>>>> Sean,
> > >> >>>>>>>>>>> we might not be too bad here):
> > >> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state
> talking
> > to
> > >> >>>>>>>>>>> 2.0
> > >> >>>>>>>>>>> cluster
> > >> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
> > >> Master
> > >> >>>>>>>>>>> and
> > >> >>>>>>>>>>> 1.x
> > >> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-17442 Move most of the replication related classes
> > to
> > >> >>>>>>>>>>> hbase-server package
> > >> >>>>>>>>>>> The above would be good to do generally but it may make
> for
> > >> >>>>>>>>>>> ripples
> > >> >>>>>>>>>>> in
> > >> >>>>>>>>>>> API so would be good to do now.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> > >> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid
> adding
> > >> two
> > >> >>>>>>>>>>> new
> > >> >>>>>>>>>>> types
> > >> >>>>>>>>>>> to our API, two types that are nought but curtailed,
> > read-only
> > >> >>>>>>>>>>> views
> > >> >>>>>>>>>>> on
> > >> >>>>>>>>>>> internals. Input if you have time appreciated.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate
> > to
> > >> a
> > >> >>>>>>>>>>> hbase2
> > >> >>>>>>>>>>> cluster; verify
> > >> >>>>>>>>>>> Esteban is looking at this one
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in
> core
> > >> >>>>>>>>>>> * HBASE-17143 Scan improvement
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
> > >> >>>>>>>>>>> implementations
> > >> >>>>>>>>>>> marked audience Private and returns implementations rather
> > >> than
> > >> >>>>>>>>>>> Interfaces.
> > >> >>>>>>>>>>> In a few locations, we allow returning an alternate
> > >> >>>>>>>>>>> implementation
> > >> >>>>>>>>>>> altogether which is probably something we don't want a CP
> > >> doing.
> > >> >>>>>>>>>>> To
> > >> >>>>>>>>>>> that
> > >> >>>>>>>>>>> end, the following issues started by Duo and Anoop need to
> > be
> > >> >>>>>>>>>>> taken
> > >> >>>>>>>>>>> to
> > >> >>>>>>>>>>> the
> > >> >>>>>>>>>>> finish line; ideally they'd have an owner:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
> > >> release <=
> > >> >>>>>>>>>>> The
> > >> >>>>>>>>>>> umbrella issue.
> > >> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for
> CP
> > >> >>>>>>>>>>> expose
> > >> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
> > >> MasterObserver
> > >> >>>>>>>>>>> and
> > >> >>>>>>>>>>> RegionServerObserver
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Nice-to-haves:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
> > >> remove
> > >> >>>>>>>>>>> unused
> > >> >>>>>>>>>>> TimeRange constructors
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
> > >> >>>>>>>>>>> references
> > >> >>>>>>>>>>> existing in the code
> > >> >>>>>>>>>>> This is the end of an old long-running project moving up
> on
> > to
> > >> >>>>>>>>>>> Cell
> > >> >>>>>>>>>>> Interface. We think it is done but for a few little items
> > >> >>>>>>>>>>> (deprecate
> > >> >>>>>>>>>>> KV
> > >> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is
> > >> indeterminate;
> > >> >>>>>>>>>>> needs
> > >> >>>>>>>>>>> fixing
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
> > >> >>>>>>>>>>> * HBASE-14997
> > >> >>>>>>>>>>> Move compareOp and Comparators out of filter to client
> > package
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> What about:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
> > >> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> If interested in any of the above and you need a legup,
> just
> > >> ask
> > >> >>>>>>>>>>> in
> > >> >>>>>>>>>>> the
> > >> >>>>>>>>>>> issue and I'll be by....
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Thanks,
> > >> >>>>>>>>>>> St.Ack
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <stack@duboce.net
> >
> > >> >>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Heads-up:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release
> Candidate.
> > >> >>>>>>>>>>>> Theme
> > >> >>>>>>>>>>>> is
> > >> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
> > >> (guava,
> > >> >>>>>>>>>>>> netty,
> > >> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
> > >> >>>>>>>>>>>> master-carries-no-regions
> > >> >>>>>>>>>>>> by default.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?).
> Its
> > >> theme
> > >> >>>>>>>>>>>> will
> > >> >>>>>>>>>>>> be
> > >> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we
> are
> > >> not
> > >> >>>>>>>>>>>> looking too
> > >> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have
> hbase-1
> > >> >>>>>>>>>>>> client
> > >> >>>>>>>>>>>> running
> > >> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor
> Interface
> > >> >>>>>>>>>>>> revamp
> > >> >>>>>>>>>>>> should be
> > >> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
> > >> >>>>>>>>>>>> Implementations, and
> > >> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
> > >> >>>>>>>>>>>> InterfaceAudience).
> > >> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API;
> > e.g.
> > >> >>>>>>>>>>>> guava
> > >> >>>>>>>>>>>> 0.12
> > >> >>>>>>>>>>>> Service showing through in our replication API and
> > protobufs
> > >> in
> > >> >>>>>>>>>>>> Admin
> > >> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of
> > outreach
> > >> to
> > >> >>>>>>>>>>>> make
> > >> >>>>>>>>>>>> sure
> > >> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> Beta1 in mid-September?
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
> > >> >>>>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> > 12327
> > >> 188
> > >> >>>>>>>>>>>> Edit
> > >> >>>>>>>>>>>> as
> > >> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not
> get
> > >> any
> > >> >>>>>>>>>>>> attention
> > >> >>>>>>>>>>>> in next month or so.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll
> leave
> > >> them
> > >> >>>>>>>>>>>> as
> > >> >>>>>>>>>>>> is
> > >> >>>>>>>>>>>> another while but I'll boot them soon.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> While I have your attention:
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving
> > >> hbase
> > >> >>>>>>>>>>>> thrift
> > >> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy
> > >> enough
> > >> >>>>>>>>>>>> if
> > >> >>>>>>>>>>>> folks
> > >> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> > >> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
> > >> first
> > >> >>>>>>>>>>>> (0.98?).
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> St.Ack
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
> > >> wrote:
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
> > >> elserj@apache.org>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
> > >> >>>>>>>>>>>>>> elserj@apache.org>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> ...
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would
> > make
> > >> the
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> feature a
> > >> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I
> > >> have
> > >> >>>>>>>>>>>>>>>> no
> > >> >>>>>>>>>>>>>>>> reason
> > >> >>>>>>>>>>>>>>>> to be
> > >> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature
> in
> > >> terms
> > >> >>>>>>>>>>>>>>>> of
> > >> >>>>>>>>>>>>>>>> how
> > >> >>>>>>>>>>>>>>>> much it
> > >> >>>>>>>>>>>>>>>> is covered via testing.
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so,
> +1
> > >> Josh.
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See
> > >> 'Space
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> Quota' in
> > >> >>>>>>>>>>>>>>> here:
> > >> >>>>>>>>>>>>>>> https://docs.google.com/docume
> > >> nt/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> > >> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > >> >>>>>>>>>>>>>>> Does this little section need an update Josh?
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> Thanks,
> > >> >>>>>>>>>>>>>>> S
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage
> > and
> > >> >>>>>>>>>>>>>>> some
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> docs
> > >> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my
> > plate
> > >> if
> > >> >>>>>>>>>>>>>> you're good
> > >> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is against
> > it
> > >> :)
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
> > >> before
> > >> >>>>>>>>>>>>>> we
> > >> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll
> > double
> > >> >>>>>>>>>>>>>> check
> > >> >>>>>>>>>>>>>> and update
> > >> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> Thanks Josh,
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> S
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>
> > >> >>>>>>
> > >> >>>>
> > >> >>
> > >>
> > >
> > >
> >
>

Re: Moving 2.0 forward

[ramkrishna vasudevan <ra...@gmail.com>]

HBASE-18946 Stochastic load balancer assigns replica regions to the same RS

Am on this today and probably will put up another patch.

Regards
Ram

On Fri, Nov 17, 2017 at 5:18 AM, Stack <st...@duboce.net> wrote:

> hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
> remaining features and apply final polish to API. There will be a beta-2
> but it is about upgrade/rolling-upgrade and bug fixes ONLY).
>
> Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1 list
> this morning. See here [1].
>
> There are about ~12 issues in progress with most of these about to land.
> There are 37 TODO. Many of these are tests we need to run, some are related
> to the backup/restore, but a good few are meaty w/o assignees.
>
> The awkward outstanding ones as I see it are the below:
>
> HBASE-18946 Stochastic load balancer assigns replica regions to the same RS
> HBASE-17204 Make L2 off heap cache default ON
> HBASE-19112 Suspect methods on Cell to be deprecated
> HBASE-19147 All branch-2 unit tests pass
>
> We need to make progress on the above or punt on them (can't punt on the
> last one though).
>
> Any ideas on what configs we should update in hbase2? Dump ideas into:
> HBASE-19148 Edit of default configuration
>
> Still hoping for an early December beta-1 RC. beta-2 hopefully will be
> close behind.
>
> Comments? Thoughts?
>
> Thanks all,
> S
>
> 1. https://issues.apache.org/jira/projects/HBASE/versions/12340861
>
> On Tue, Oct 31, 2017 at 2:56 PM, Stack <st...@duboce.net> wrote:
>
> >
> >
> > On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:
> >
> >> Hoping to keep momentum going from our Stack working on alpha4, I tried
> to
> >> take a stab at triaging some of the open beta-1 issues.
> >>
> >> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done
> sooner
> >> then I'm happy to see it pulled back in. Trying to balance optimism with
> >> realism here, and knowing that documentation unfortunately often gets
> >> pushed to the back-burner.
> >>
> >> Also, I poked some folks on unassigned issues that they've filed for
> >> beta-1, especially in the last few days. If issues don't have an owner
> >> they
> >> are unlikely to get worked. I chatted with stack and he agreed to take
> on
> >> some of the tasks, but there's a lot of surface area to cover.
> >>
> >> If you you're working on issues that are critical for beta-1, please
> mark
> >> them as such. Then the rest of the community will know to help
> prioritize
> >> feedback and reviews there.
> >>
> >> Do we have a general theme for the betas like we did with the alphas?
> >>
> >> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
> >> well? Continue to work on tests throughout?
> >>
> >>
> > Thanks Mike for helping to kick off the beta-1 train.
> >
> > Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what we
> > are going to ship (beta-2 is rolling upgrade and any minor items turned
> up
> > in testing/burn-in).
> >
> > Thanks,
> > S
> >
> >
> >
> >> Mike
> >>
> >> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org> wrote:
> >>
> >> > +1 go from my POV.
> >> >
> >> >
> >> > On 10/31/17 10:07 AM, Stack wrote:
> >> >
> >> >> I want to push an alpha-4 today. A few items didn't make it
> >> (HBASE-19092).
> >> >> They need more time. We'll pull them in for beta-1. CP API is
> basically
> >> >> done. There may be some changes for beta-1 but hopefully only changes
> >> >> informed by experience trying to port an existing Coprocessor to
> >> hbase2.
> >> >>
> >> >> Shout if there is anything that needs to make alpha-4.
> >> >>
> >> >> Thanks,
> >> >> St.Ack
> >> >>
> >> >>
> >> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
> >> wrote:
> >> >>
> >> >> Yup, that was going to be my plan, Mike!
> >> >>>
> >> >>> Making a pass now, and will check back later tonight again. I see
> >> others
> >> >>> have already done some work today on this front.
> >> >>>
> >> >>>
> >> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
> >> >>>
> >> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how
> >> to do
> >> >>>> it directly on the jenkins job, so you don't have to bother with
> JIRA
> >> >>>> uploads)
> >> >>>>
> >> >>>> If you're busy, then I can make time tomorrow or Sunday to kick off
> >> >>>> jobs,
> >> >>>> but I want to make sure we're not duplicating effort and jenkins
> >> cycles.
> >> >>>>
> >> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
> >> wrote:
> >> >>>>
> >> >>>> My turn to bump ;)
> >> >>>>
> >> >>>>>
> >> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that
> >> remain
> >> >>>>> needing some more work. The rest are just awaiting a good QA run.
> >> >>>>>
> >> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over
> the
> >> >>>>> weekend, bump them along as necessary, and get them committed.
> >> Would be
> >> >>>>> great to be able get a vote up on Monday.
> >> >>>>>
> >> >>>>>
> >> >>>>> On 10/24/17 6:03 PM, Stack wrote:
> >> >>>>>
> >> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back and
> >> forth
> >> >>>>>
> >> >>>>>> what remains to be done. Surfacing our thoughts here so you all
> >> clued
> >> >>>>>> in....Or if you think otherwise, please speak up.
> >> >>>>>>
> >> >>>>>> We have ~13 issues to land:
> >> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >> About
> >> >>>>>> two
> >> >>>>>> are meta-issues that are about process which leaves 11.
> >> >>>>>>
> >> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
> >> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
> >> >>>>>> changed
> >> >>>>>> API/semantic that we need to get out earlier rather than later.
> >> >>>>>>
> >> >>>>>> Once the above is merged, HBASE-13346, change of Filter method
> >> names
> >> >>>>>> to
> >> >>>>>> mention Cell instead of KeyValue can land.
> >> >>>>>>
> >> >>>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
> >> >>>>>> IA.Private objects as params to MasterObserver... Hopefully this
> >> goes
> >> >>>>>> in
> >> >>>>>> soon.
> >> >>>>>>
> >> >>>>>> Duo is hard at work on trackers for flush and compaction for CPs
> >> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for
> >> Tephra)?
> >> >>>>>>
> >> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate
> >> after
> >> >>>>>> Duo
> >> >>>>>> is
> >> >>>>>> done w/ his work above.
> >> >>>>>>
> >> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity
> after
> >> >>>>>> all
> >> >>>>>> the
> >> >>>>>> purges allowing CPs do direct calls against Regions in same Host.
> >> >>>>>>
> >> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> >> >>>>>>
> >> >>>>>> Another day or two?
> >> >>>>>>
> >> >>>>>> St.Ack
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
> >> >>>>>> wrote:
> >> >>>>>>
> >> >>>>>>>
> >> >>>>>>> +1
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>> I was trying to work on helping out on the outstanding alpha-4
> >> stuff
> >> >>>>>>>> last
> >> >>>>>>>> week -- will be continuing to try to do the same this week.
> >> >>>>>>>>
> >> >>>>>>>> If you need any help, Stack, or if others need reviews where I
> >> >>>>>>>> haven't
> >> >>>>>>>> noticed on my own: feel free to @mention me.
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are
> being
> >> >>>>>>>>
> >> >>>>>>>> actively
> >> >>>>>>> worked on. If you get a moment, reviews by you (or anyone else)
> >> helps
> >> >>>>>>> move
> >> >>>>>>> the process along.
> >> >>>>>>>
> >> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
> >> >>>>>>> improvements.
> >> >>>>>>> Then HBASE-13346 can go in.
> >> >>>>>>>
> >> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks like
> >> that
> >> >>>>>>> will
> >> >>>>>>> be addressed by other alpha-4s about to land.
> >> >>>>>>>
> >> >>>>>>> St.Ack
> >> >>>>>>> TODOs: https://issues.apache.org/jira
> >> /projects/HBASE/versions/12341
> >> >>>>>>> 594
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>> (Reviving this thread)
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that
> has
> >> >>>>>>>>> the
> >> >>>>>>>>> refactor of the Coprocessor API shutting down access to
> >> internals
> >> >>>>>>>>> marked
> >> >>>>>>>>> InterfaceAudience.Private.
> >> >>>>>>>>>
> >> >>>>>>>>> The outstanding list is here:
> >> >>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> 12341594
> >> >>>>>>>>>
> >> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
> >> >>>>>>>>>
> >> >>>>>>>>> If issue, talk out loud on this thread. If you need a review
> to
> >> >>>>>>>>> land
> >> >>>>>>>>> an
> >> >>>>>>>>> item, shout on the issue and here; we'll help you out.
> >> >>>>>>>>>
> >> >>>>>>>>> As is, items are coming along nicely I'd say. We need to merge
> >> the
> >> >>>>>>>>> filter
> >> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> >> >>>>>>>>>
> >> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and
> help
> >> >>>>>>>>> them
> >> >>>>>>>>> test
> >> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence
> our
> >> >>>>>>>>> downstreamers know what to expect (or we discover what we
> missed
> >> >>>>>>>>> BEFORE
> >> >>>>>>>>> we
> >> >>>>>>>>> beta-1).
> >> >>>>>>>>>
> >> >>>>>>>>> Thanks for time,
> >> >>>>>>>>> S
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net>
> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That
> >> should
> >> >>>>>>>>> be
> >> >>>>>>>>>
> >> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to be
> >> done.
> >> >>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for
> >> >>>>>>>>>> alpha3
> >> >>>>>>>>>> but
> >> >>>>>>>>>> it is plain that more time is needed (in spite of valiant
> >> effort
> >> >>>>>>>>>> so
> >> >>>>>>>>>> far
> >> >>>>>>>>>> by
> >> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
> >> whose
> >> >>>>>>>>>> theme is
> >> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by
> the
> >> >>>>>>>>>> following
> >> >>>>>>>>>> week.
> >> >>>>>>>>>>
> >> >>>>>>>>>> We should then be ready for beta (beta == no new features, no
> >> API
> >> >>>>>>>>>> changes,
> >> >>>>>>>>>> just fixes).
> >> >>>>>>>>>>
> >> >>>>>>>>>> Thanks,
> >> >>>>>>>>>> St.Ack
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
> >> wrote:
> >> >>>>>>>>>>
> >> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please
> vote
> >> on
> >> >>>>>>>>>> it.
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope
> to
> >> >>>>>>>>>>> get a
> >> >>>>>>>>>>> release out in the next week or so.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks
> >> are
> >> >>>>>>>>>>> interested in helping out, below are the items I think we
> need
> >> >>>>>>>>>>> done
> >> >>>>>>>>>>> for
> >> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API
> possibly
> >> >>>>>>>>>>> altering
> >> >>>>>>>>>>> items, and are absent those JIRAs that are making active
> >> >>>>>>>>>>> progress,
> >> >>>>>>>>>>> i.e. the
> >> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
> >> >>>>>>>>>>> needs
> >> >>>>>>>>>>> doing is
> >> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between
> branch-1
> >> >>>>>>>>>>> and
> >> >>>>>>>>>>> branch-2
> >> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches
> >> only
> >> >>>>>>>>>>> do
> >> >>>>>>>>>>> it
> >> >>>>>>>>>>> between branch-1 and branch-2.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public
> /
> >> >>>>>>>>>>> Private
> >> >>>>>>>>>>> interfaces
> >> >>>>>>>>>>> This one is almost done. It could do with a finish,
> attention
> >> to
> >> >>>>>>>>>>> the
> >> >>>>>>>>>>> items in last comment, and then our codebase could do with
> >> >>>>>>>>>>> another
> >> >>>>>>>>>>> sweep
> >> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in
> since
> >> >>>>>>>>>>> the
> >> >>>>>>>>>>> pass
> >> >>>>>>>>>>> that was the basis of this issue.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
> >> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
> >> forced
> >> >>>>>>>>>>> by
> >> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> >> >>>>>>>>>>> Service'
> >> >>>>>>>>>>> but if
> >> >>>>>>>>>>> anyone else is interested, be my guest.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
> >> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked on.
> >> Needs
> >> >>>>>>>>>>> finishing.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
> >> Admin
> >> >>>>>>>>>>> and
> >> >>>>>>>>>>> cleanup
> >> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
> >> >>>>>>>>>>> AsyncClient
> >> >>>>>>>>>>> and
> >> >>>>>>>>>>> make sure symmetry.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> >> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but
> >> needs to
> >> >>>>>>>>>>> be
> >> >>>>>>>>>>> done.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would be
> >> good
> >> >>>>>>>>>>> to
> >> >>>>>>>>>>> try
> >> >>>>>>>>>>> it and figure what needs fixup (as said above, according to
> >> trial
> >> >>>>>>>>>>> by
> >> >>>>>>>>>>> Sean,
> >> >>>>>>>>>>> we might not be too bad here):
> >> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking
> to
> >> >>>>>>>>>>> 2.0
> >> >>>>>>>>>>> cluster
> >> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
> >> Master
> >> >>>>>>>>>>> and
> >> >>>>>>>>>>> 1.x
> >> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-17442 Move most of the replication related classes
> to
> >> >>>>>>>>>>> hbase-server package
> >> >>>>>>>>>>> The above would be good to do generally but it may make for
> >> >>>>>>>>>>> ripples
> >> >>>>>>>>>>> in
> >> >>>>>>>>>>> API so would be good to do now.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> >> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid adding
> >> two
> >> >>>>>>>>>>> new
> >> >>>>>>>>>>> types
> >> >>>>>>>>>>> to our API, two types that are nought but curtailed,
> read-only
> >> >>>>>>>>>>> views
> >> >>>>>>>>>>> on
> >> >>>>>>>>>>> internals. Input if you have time appreciated.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate
> to
> >> a
> >> >>>>>>>>>>> hbase2
> >> >>>>>>>>>>> cluster; verify
> >> >>>>>>>>>>> Esteban is looking at this one
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> >> >>>>>>>>>>> * HBASE-17143 Scan improvement
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
> >> >>>>>>>>>>> implementations
> >> >>>>>>>>>>> marked audience Private and returns implementations rather
> >> than
> >> >>>>>>>>>>> Interfaces.
> >> >>>>>>>>>>> In a few locations, we allow returning an alternate
> >> >>>>>>>>>>> implementation
> >> >>>>>>>>>>> altogether which is probably something we don't want a CP
> >> doing.
> >> >>>>>>>>>>> To
> >> >>>>>>>>>>> that
> >> >>>>>>>>>>> end, the following issues started by Duo and Anoop need to
> be
> >> >>>>>>>>>>> taken
> >> >>>>>>>>>>> to
> >> >>>>>>>>>>> the
> >> >>>>>>>>>>> finish line; ideally they'd have an owner:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
> >> release <=
> >> >>>>>>>>>>> The
> >> >>>>>>>>>>> umbrella issue.
> >> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
> >> >>>>>>>>>>> expose
> >> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
> >> MasterObserver
> >> >>>>>>>>>>> and
> >> >>>>>>>>>>> RegionServerObserver
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Nice-to-haves:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
> >> remove
> >> >>>>>>>>>>> unused
> >> >>>>>>>>>>> TimeRange constructors
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
> >> >>>>>>>>>>> references
> >> >>>>>>>>>>> existing in the code
> >> >>>>>>>>>>> This is the end of an old long-running project moving up on
> to
> >> >>>>>>>>>>> Cell
> >> >>>>>>>>>>> Interface. We think it is done but for a few little items
> >> >>>>>>>>>>> (deprecate
> >> >>>>>>>>>>> KV
> >> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is
> >> indeterminate;
> >> >>>>>>>>>>> needs
> >> >>>>>>>>>>> fixing
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
> >> >>>>>>>>>>> * HBASE-14997
> >> >>>>>>>>>>> Move compareOp and Comparators out of filter to client
> package
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> What about:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
> >> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> If interested in any of the above and you need a legup, just
> >> ask
> >> >>>>>>>>>>> in
> >> >>>>>>>>>>> the
> >> >>>>>>>>>>> issue and I'll be by....
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Thanks,
> >> >>>>>>>>>>> St.Ack
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
> >> >>>>>>>>>>> wrote:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Heads-up:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
> >> >>>>>>>>>>>> Theme
> >> >>>>>>>>>>>> is
> >> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
> >> (guava,
> >> >>>>>>>>>>>> netty,
> >> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
> >> >>>>>>>>>>>> master-carries-no-regions
> >> >>>>>>>>>>>> by default.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
> >> theme
> >> >>>>>>>>>>>> will
> >> >>>>>>>>>>>> be
> >> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we are
> >> not
> >> >>>>>>>>>>>> looking too
> >> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1
> >> >>>>>>>>>>>> client
> >> >>>>>>>>>>>> running
> >> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface
> >> >>>>>>>>>>>> revamp
> >> >>>>>>>>>>>> should be
> >> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
> >> >>>>>>>>>>>> Implementations, and
> >> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
> >> >>>>>>>>>>>> InterfaceAudience).
> >> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API;
> e.g.
> >> >>>>>>>>>>>> guava
> >> >>>>>>>>>>>> 0.12
> >> >>>>>>>>>>>> Service showing through in our replication API and
> protobufs
> >> in
> >> >>>>>>>>>>>> Admin
> >> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of
> outreach
> >> to
> >> >>>>>>>>>>>> make
> >> >>>>>>>>>>>> sure
> >> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Beta1 in mid-September?
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
> >> >>>>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> 12327
> >> 188
> >> >>>>>>>>>>>> Edit
> >> >>>>>>>>>>>> as
> >> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get
> >> any
> >> >>>>>>>>>>>> attention
> >> >>>>>>>>>>>> in next month or so.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave
> >> them
> >> >>>>>>>>>>>> as
> >> >>>>>>>>>>>> is
> >> >>>>>>>>>>>> another while but I'll boot them soon.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> While I have your attention:
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving
> >> hbase
> >> >>>>>>>>>>>> thrift
> >> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy
> >> enough
> >> >>>>>>>>>>>> if
> >> >>>>>>>>>>>> folks
> >> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> >> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
> >> first
> >> >>>>>>>>>>>> (0.98?).
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> St.Ack
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
> >> wrote:
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
> >> elserj@apache.org>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
> >> >>>>>>>>>>>>>> elserj@apache.org>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> ...
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would
> make
> >> the
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> feature a
> >> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I
> >> have
> >> >>>>>>>>>>>>>>>> no
> >> >>>>>>>>>>>>>>>> reason
> >> >>>>>>>>>>>>>>>> to be
> >> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in
> >> terms
> >> >>>>>>>>>>>>>>>> of
> >> >>>>>>>>>>>>>>>> how
> >> >>>>>>>>>>>>>>>> much it
> >> >>>>>>>>>>>>>>>> is covered via testing.
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
> >> Josh.
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See
> >> 'Space
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Quota' in
> >> >>>>>>>>>>>>>>> here:
> >> >>>>>>>>>>>>>>> https://docs.google.com/docume
> >> nt/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> >> >>>>>>>>>>>>>>> Does this little section need an update Josh?
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Thanks,
> >> >>>>>>>>>>>>>>> S
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage
> and
> >> >>>>>>>>>>>>>>> some
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> docs
> >> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my
> plate
> >> if
> >> >>>>>>>>>>>>>> you're good
> >> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is against
> it
> >> :)
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
> >> before
> >> >>>>>>>>>>>>>> we
> >> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll
> double
> >> >>>>>>>>>>>>>> check
> >> >>>>>>>>>>>>>> and update
> >> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> Thanks Josh,
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> S
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>
> >> >>>>>>
> >> >>>>
> >> >>
> >>
> >
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

hbase-2.0.0-beta-1 update (Reminder, beta-1 is where we finish last
remaining features and apply final polish to API. There will be a beta-2
but it is about upgrade/rolling-upgrade and bug fixes ONLY).

Myself and Mr. Drob did a pass over the outstanding hbase-2.0.0-beta-1 list
this morning. See here [1].

There are about ~12 issues in progress with most of these about to land.
There are 37 TODO. Many of these are tests we need to run, some are related
to the backup/restore, but a good few are meaty w/o assignees.

The awkward outstanding ones as I see it are the below:

HBASE-18946 Stochastic load balancer assigns replica regions to the same RS
HBASE-17204 Make L2 off heap cache default ON
HBASE-19112 Suspect methods on Cell to be deprecated
HBASE-19147 All branch-2 unit tests pass

We need to make progress on the above or punt on them (can't punt on the
last one though).

Any ideas on what configs we should update in hbase2? Dump ideas into:
HBASE-19148 Edit of default configuration

Still hoping for an early December beta-1 RC. beta-2 hopefully will be
close behind.

Comments? Thoughts?

Thanks all,
S

1. https://issues.apache.org/jira/projects/HBASE/versions/12340861

On Tue, Oct 31, 2017 at 2:56 PM, Stack <st...@duboce.net> wrote:

>
>
> On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:
>
>> Hoping to keep momentum going from our Stack working on alpha4, I tried to
>> take a stab at triaging some of the open beta-1 issues.
>>
>> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done sooner
>> then I'm happy to see it pulled back in. Trying to balance optimism with
>> realism here, and knowing that documentation unfortunately often gets
>> pushed to the back-burner.
>>
>> Also, I poked some folks on unassigned issues that they've filed for
>> beta-1, especially in the last few days. If issues don't have an owner
>> they
>> are unlikely to get worked. I chatted with stack and he agreed to take on
>> some of the tasks, but there's a lot of surface area to cover.
>>
>> If you you're working on issues that are critical for beta-1, please mark
>> them as such. Then the rest of the community will know to help prioritize
>> feedback and reviews there.
>>
>> Do we have a general theme for the betas like we did with the alphas?
>>
>> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
>> well? Continue to work on tests throughout?
>>
>>
> Thanks Mike for helping to kick off the beta-1 train.
>
> Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what we
> are going to ship (beta-2 is rolling upgrade and any minor items turned up
> in testing/burn-in).
>
> Thanks,
> S
>
>
>
>> Mike
>>
>> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org> wrote:
>>
>> > +1 go from my POV.
>> >
>> >
>> > On 10/31/17 10:07 AM, Stack wrote:
>> >
>> >> I want to push an alpha-4 today. A few items didn't make it
>> (HBASE-19092).
>> >> They need more time. We'll pull them in for beta-1. CP API is basically
>> >> done. There may be some changes for beta-1 but hopefully only changes
>> >> informed by experience trying to port an existing Coprocessor to
>> hbase2.
>> >>
>> >> Shout if there is anything that needs to make alpha-4.
>> >>
>> >> Thanks,
>> >> St.Ack
>> >>
>> >>
>> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
>> wrote:
>> >>
>> >> Yup, that was going to be my plan, Mike!
>> >>>
>> >>> Making a pass now, and will check back later tonight again. I see
>> others
>> >>> have already done some work today on this front.
>> >>>
>> >>>
>> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
>> >>>
>> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how
>> to do
>> >>>> it directly on the jenkins job, so you don't have to bother with JIRA
>> >>>> uploads)
>> >>>>
>> >>>> If you're busy, then I can make time tomorrow or Sunday to kick off
>> >>>> jobs,
>> >>>> but I want to make sure we're not duplicating effort and jenkins
>> cycles.
>> >>>>
>> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
>> wrote:
>> >>>>
>> >>>> My turn to bump ;)
>> >>>>
>> >>>>>
>> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that
>> remain
>> >>>>> needing some more work. The rest are just awaiting a good QA run.
>> >>>>>
>> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over the
>> >>>>> weekend, bump them along as necessary, and get them committed.
>> Would be
>> >>>>> great to be able get a vote up on Monday.
>> >>>>>
>> >>>>>
>> >>>>> On 10/24/17 6:03 PM, Stack wrote:
>> >>>>>
>> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back and
>> forth
>> >>>>>
>> >>>>>> what remains to be done. Surfacing our thoughts here so you all
>> clued
>> >>>>>> in....Or if you think otherwise, please speak up.
>> >>>>>>
>> >>>>>> We have ~13 issues to land:
>> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>> About
>> >>>>>> two
>> >>>>>> are meta-issues that are about process which leaves 11.
>> >>>>>>
>> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
>> >>>>>> changed
>> >>>>>> API/semantic that we need to get out earlier rather than later.
>> >>>>>>
>> >>>>>> Once the above is merged, HBASE-13346, change of Filter method
>> names
>> >>>>>> to
>> >>>>>> mention Cell instead of KeyValue can land.
>> >>>>>>
>> >>>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
>> >>>>>> IA.Private objects as params to MasterObserver... Hopefully this
>> goes
>> >>>>>> in
>> >>>>>> soon.
>> >>>>>>
>> >>>>>> Duo is hard at work on trackers for flush and compaction for CPs
>> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for
>> Tephra)?
>> >>>>>>
>> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate
>> after
>> >>>>>> Duo
>> >>>>>> is
>> >>>>>> done w/ his work above.
>> >>>>>>
>> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after
>> >>>>>> all
>> >>>>>> the
>> >>>>>> purges allowing CPs do direct calls against Regions in same Host.
>> >>>>>>
>> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>> >>>>>>
>> >>>>>> Another day or two?
>> >>>>>>
>> >>>>>> St.Ack
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>> >>>>>>
>> >>>>>>
>> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>>
>> >>>>>>> +1
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>> I was trying to work on helping out on the outstanding alpha-4
>> stuff
>> >>>>>>>> last
>> >>>>>>>> week -- will be continuing to try to do the same this week.
>> >>>>>>>>
>> >>>>>>>> If you need any help, Stack, or if others need reviews where I
>> >>>>>>>> haven't
>> >>>>>>>> noticed on my own: feel free to @mention me.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are being
>> >>>>>>>>
>> >>>>>>>> actively
>> >>>>>>> worked on. If you get a moment, reviews by you (or anyone else)
>> helps
>> >>>>>>> move
>> >>>>>>> the process along.
>> >>>>>>>
>> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
>> >>>>>>> improvements.
>> >>>>>>> Then HBASE-13346 can go in.
>> >>>>>>>
>> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks like
>> that
>> >>>>>>> will
>> >>>>>>> be addressed by other alpha-4s about to land.
>> >>>>>>>
>> >>>>>>> St.Ack
>> >>>>>>> TODOs: https://issues.apache.org/jira
>> /projects/HBASE/versions/12341
>> >>>>>>> 594
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>> (Reviving this thread)
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has
>> >>>>>>>>> the
>> >>>>>>>>> refactor of the Coprocessor API shutting down access to
>> internals
>> >>>>>>>>> marked
>> >>>>>>>>> InterfaceAudience.Private.
>> >>>>>>>>>
>> >>>>>>>>> The outstanding list is here:
>> >>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>> >>>>>>>>>
>> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>> >>>>>>>>>
>> >>>>>>>>> If issue, talk out loud on this thread. If you need a review to
>> >>>>>>>>> land
>> >>>>>>>>> an
>> >>>>>>>>> item, shout on the issue and here; we'll help you out.
>> >>>>>>>>>
>> >>>>>>>>> As is, items are coming along nicely I'd say. We need to merge
>> the
>> >>>>>>>>> filter
>> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>> >>>>>>>>>
>> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and help
>> >>>>>>>>> them
>> >>>>>>>>> test
>> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>> >>>>>>>>> downstreamers know what to expect (or we discover what we missed
>> >>>>>>>>> BEFORE
>> >>>>>>>>> we
>> >>>>>>>>> beta-1).
>> >>>>>>>>>
>> >>>>>>>>> Thanks for time,
>> >>>>>>>>> S
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>> >>>>>>>>>
>> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That
>> should
>> >>>>>>>>> be
>> >>>>>>>>>
>> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to be
>> done.
>> >>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for
>> >>>>>>>>>> alpha3
>> >>>>>>>>>> but
>> >>>>>>>>>> it is plain that more time is needed (in spite of valiant
>> effort
>> >>>>>>>>>> so
>> >>>>>>>>>> far
>> >>>>>>>>>> by
>> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
>> whose
>> >>>>>>>>>> theme is
>> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>> >>>>>>>>>> following
>> >>>>>>>>>> week.
>> >>>>>>>>>>
>> >>>>>>>>>> We should then be ready for beta (beta == no new features, no
>> API
>> >>>>>>>>>> changes,
>> >>>>>>>>>> just fixes).
>> >>>>>>>>>>
>> >>>>>>>>>> Thanks,
>> >>>>>>>>>> St.Ack
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote
>> on
>> >>>>>>>>>> it.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
>> >>>>>>>>>>> get a
>> >>>>>>>>>>> release out in the next week or so.
>> >>>>>>>>>>>
>> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks
>> are
>> >>>>>>>>>>> interested in helping out, below are the items I think we need
>> >>>>>>>>>>> done
>> >>>>>>>>>>> for
>> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API possibly
>> >>>>>>>>>>> altering
>> >>>>>>>>>>> items, and are absent those JIRAs that are making active
>> >>>>>>>>>>> progress,
>> >>>>>>>>>>> i.e. the
>> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
>> >>>>>>>>>>> needs
>> >>>>>>>>>>> doing is
>> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1
>> >>>>>>>>>>> and
>> >>>>>>>>>>> branch-2
>> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches
>> only
>> >>>>>>>>>>> do
>> >>>>>>>>>>> it
>> >>>>>>>>>>> between branch-1 and branch-2.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public /
>> >>>>>>>>>>> Private
>> >>>>>>>>>>> interfaces
>> >>>>>>>>>>> This one is almost done. It could do with a finish, attention
>> to
>> >>>>>>>>>>> the
>> >>>>>>>>>>> items in last comment, and then our codebase could do with
>> >>>>>>>>>>> another
>> >>>>>>>>>>> sweep
>> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in since
>> >>>>>>>>>>> the
>> >>>>>>>>>>> pass
>> >>>>>>>>>>> that was the basis of this issue.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
>> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
>> forced
>> >>>>>>>>>>> by
>> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
>> >>>>>>>>>>> Service'
>> >>>>>>>>>>> but if
>> >>>>>>>>>>> anyone else is interested, be my guest.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked on.
>> Needs
>> >>>>>>>>>>> finishing.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
>> Admin
>> >>>>>>>>>>> and
>> >>>>>>>>>>> cleanup
>> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>> >>>>>>>>>>> AsyncClient
>> >>>>>>>>>>> and
>> >>>>>>>>>>> make sure symmetry.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but
>> needs to
>> >>>>>>>>>>> be
>> >>>>>>>>>>> done.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would be
>> good
>> >>>>>>>>>>> to
>> >>>>>>>>>>> try
>> >>>>>>>>>>> it and figure what needs fixup (as said above, according to
>> trial
>> >>>>>>>>>>> by
>> >>>>>>>>>>> Sean,
>> >>>>>>>>>>> we might not be too bad here):
>> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking to
>> >>>>>>>>>>> 2.0
>> >>>>>>>>>>> cluster
>> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
>> Master
>> >>>>>>>>>>> and
>> >>>>>>>>>>> 1.x
>> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-17442 Move most of the replication related classes to
>> >>>>>>>>>>> hbase-server package
>> >>>>>>>>>>> The above would be good to do generally but it may make for
>> >>>>>>>>>>> ripples
>> >>>>>>>>>>> in
>> >>>>>>>>>>> API so would be good to do now.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid adding
>> two
>> >>>>>>>>>>> new
>> >>>>>>>>>>> types
>> >>>>>>>>>>> to our API, two types that are nought but curtailed, read-only
>> >>>>>>>>>>> views
>> >>>>>>>>>>> on
>> >>>>>>>>>>> internals. Input if you have time appreciated.
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to
>> a
>> >>>>>>>>>>> hbase2
>> >>>>>>>>>>> cluster; verify
>> >>>>>>>>>>> Esteban is looking at this one
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>> >>>>>>>>>>> * HBASE-17143 Scan improvement
>> >>>>>>>>>>>
>> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>> >>>>>>>>>>> implementations
>> >>>>>>>>>>> marked audience Private and returns implementations rather
>> than
>> >>>>>>>>>>> Interfaces.
>> >>>>>>>>>>> In a few locations, we allow returning an alternate
>> >>>>>>>>>>> implementation
>> >>>>>>>>>>> altogether which is probably something we don't want a CP
>> doing.
>> >>>>>>>>>>> To
>> >>>>>>>>>>> that
>> >>>>>>>>>>> end, the following issues started by Duo and Anoop need to be
>> >>>>>>>>>>> taken
>> >>>>>>>>>>> to
>> >>>>>>>>>>> the
>> >>>>>>>>>>> finish line; ideally they'd have an owner:
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0
>> release <=
>> >>>>>>>>>>> The
>> >>>>>>>>>>> umbrella issue.
>> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
>> >>>>>>>>>>> expose
>> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
>> MasterObserver
>> >>>>>>>>>>> and
>> >>>>>>>>>>> RegionServerObserver
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Nice-to-haves:
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
>> remove
>> >>>>>>>>>>> unused
>> >>>>>>>>>>> TimeRange constructors
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
>> >>>>>>>>>>> references
>> >>>>>>>>>>> existing in the code
>> >>>>>>>>>>> This is the end of an old long-running project moving up on to
>> >>>>>>>>>>> Cell
>> >>>>>>>>>>> Interface. We think it is done but for a few little items
>> >>>>>>>>>>> (deprecate
>> >>>>>>>>>>> KV
>> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is
>> indeterminate;
>> >>>>>>>>>>> needs
>> >>>>>>>>>>> fixing
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>> >>>>>>>>>>> * HBASE-14997
>> >>>>>>>>>>> Move compareOp and Comparators out of filter to client package
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>> >>>>>>>>>>>
>> >>>>>>>>>>> What about:
>> >>>>>>>>>>>
>> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>> >>>>>>>>>>>
>> >>>>>>>>>>> If interested in any of the above and you need a legup, just
>> ask
>> >>>>>>>>>>> in
>> >>>>>>>>>>> the
>> >>>>>>>>>>> issue and I'll be by....
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thanks,
>> >>>>>>>>>>> St.Ack
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Heads-up:
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
>> >>>>>>>>>>>> Theme
>> >>>>>>>>>>>> is
>> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
>> (guava,
>> >>>>>>>>>>>> netty,
>> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
>> >>>>>>>>>>>> master-carries-no-regions
>> >>>>>>>>>>>> by default.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
>> theme
>> >>>>>>>>>>>> will
>> >>>>>>>>>>>> be
>> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we are
>> not
>> >>>>>>>>>>>> looking too
>> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1
>> >>>>>>>>>>>> client
>> >>>>>>>>>>>> running
>> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface
>> >>>>>>>>>>>> revamp
>> >>>>>>>>>>>> should be
>> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>> >>>>>>>>>>>> Implementations, and
>> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>> >>>>>>>>>>>> InterfaceAudience).
>> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API; e.g.
>> >>>>>>>>>>>> guava
>> >>>>>>>>>>>> 0.12
>> >>>>>>>>>>>> Service showing through in our replication API and protobufs
>> in
>> >>>>>>>>>>>> Admin
>> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach
>> to
>> >>>>>>>>>>>> make
>> >>>>>>>>>>>> sure
>> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Beta1 in mid-September?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>> >>>>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327
>> 188
>> >>>>>>>>>>>> Edit
>> >>>>>>>>>>>> as
>> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get
>> any
>> >>>>>>>>>>>> attention
>> >>>>>>>>>>>> in next month or so.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave
>> them
>> >>>>>>>>>>>> as
>> >>>>>>>>>>>> is
>> >>>>>>>>>>>> another while but I'll boot them soon.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> While I have your attention:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving
>> hbase
>> >>>>>>>>>>>> thrift
>> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy
>> enough
>> >>>>>>>>>>>> if
>> >>>>>>>>>>>> folks
>> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
>> first
>> >>>>>>>>>>>> (0.98?).
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> St.Ack
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
>> elserj@apache.org>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
>> >>>>>>>>>>>>>> elserj@apache.org>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> ...
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make
>> the
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> feature a
>> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I
>> have
>> >>>>>>>>>>>>>>>> no
>> >>>>>>>>>>>>>>>> reason
>> >>>>>>>>>>>>>>>> to be
>> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in
>> terms
>> >>>>>>>>>>>>>>>> of
>> >>>>>>>>>>>>>>>> how
>> >>>>>>>>>>>>>>>> much it
>> >>>>>>>>>>>>>>>> is covered via testing.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
>> Josh.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See
>> 'Space
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Quota' in
>> >>>>>>>>>>>>>>> here:
>> >>>>>>>>>>>>>>> https://docs.google.com/docume
>> nt/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>> >>>>>>>>>>>>>>> Does this little section need an update Josh?
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>>>> S
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and
>> >>>>>>>>>>>>>>> some
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> docs
>> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my plate
>> if
>> >>>>>>>>>>>>>> you're good
>> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is against it
>> :)
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
>> before
>> >>>>>>>>>>>>>> we
>> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll double
>> >>>>>>>>>>>>>> check
>> >>>>>>>>>>>>>> and update
>> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thanks Josh,
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> S
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>
>> >>
>>
>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Tue, Oct 31, 2017 at 2:31 PM, Mike Drob <md...@apache.org> wrote:

> Hoping to keep momentum going from our Stack working on alpha4, I tried to
> take a stab at triaging some of the open beta-1 issues.
>
> I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done sooner
> then I'm happy to see it pulled back in. Trying to balance optimism with
> realism here, and knowing that documentation unfortunately often gets
> pushed to the back-burner.
>
> Also, I poked some folks on unassigned issues that they've filed for
> beta-1, especially in the last few days. If issues don't have an owner they
> are unlikely to get worked. I chatted with stack and he agreed to take on
> some of the tasks, but there's a lot of surface area to cover.
>
> If you you're working on issues that are critical for beta-1, please mark
> them as such. Then the rest of the community will know to help prioritize
> feedback and reviews there.
>
> Do we have a general theme for the betas like we did with the alphas?
>
> Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
> well? Continue to work on tests throughout?
>
>
Thanks Mike for helping to kick off the beta-1 train.

Regards a theme for beta-1, I'd like it to be 'finish'; beta-1 is what we
are going to ship (beta-2 is rolling upgrade and any minor items turned up
in testing/burn-in).

Thanks,
S



> Mike
>
> On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org> wrote:
>
> > +1 go from my POV.
> >
> >
> > On 10/31/17 10:07 AM, Stack wrote:
> >
> >> I want to push an alpha-4 today. A few items didn't make it
> (HBASE-19092).
> >> They need more time. We'll pull them in for beta-1. CP API is basically
> >> done. There may be some changes for beta-1 but hopefully only changes
> >> informed by experience trying to port an existing Coprocessor to hbase2.
> >>
> >> Shout if there is anything that needs to make alpha-4.
> >>
> >> Thanks,
> >> St.Ack
> >>
> >>
> >> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com>
> wrote:
> >>
> >> Yup, that was going to be my plan, Mike!
> >>>
> >>> Making a pass now, and will check back later tonight again. I see
> others
> >>> have already done some work today on this front.
> >>>
> >>>
> >>> On 10/27/17 11:38 PM, Mike Drob wrote:
> >>>
> >>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how to
> do
> >>>> it directly on the jenkins job, so you don't have to bother with JIRA
> >>>> uploads)
> >>>>
> >>>> If you're busy, then I can make time tomorrow or Sunday to kick off
> >>>> jobs,
> >>>> but I want to make sure we're not duplicating effort and jenkins
> cycles.
> >>>>
> >>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org>
> wrote:
> >>>>
> >>>> My turn to bump ;)
> >>>>
> >>>>>
> >>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that
> remain
> >>>>> needing some more work. The rest are just awaiting a good QA run.
> >>>>>
> >>>>> Unless I hear otherwise, I'll try to keep an eye on things over the
> >>>>> weekend, bump them along as necessary, and get them committed. Would
> be
> >>>>> great to be able get a vote up on Monday.
> >>>>>
> >>>>>
> >>>>> On 10/24/17 6:03 PM, Stack wrote:
> >>>>>
> >>>>> Chatting with my coworker Mr. Mike Drob, we were batting back and
> forth
> >>>>>
> >>>>>> what remains to be done. Surfacing our thoughts here so you all
> clued
> >>>>>> in....Or if you think otherwise, please speak up.
> >>>>>>
> >>>>>> We have ~13 issues to land:
> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> About
> >>>>>> two
> >>>>>> are meta-issues that are about process which leaves 11.
> >>>>>>
> >>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
> >>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
> >>>>>> changed
> >>>>>> API/semantic that we need to get out earlier rather than later.
> >>>>>>
> >>>>>> Once the above is merged, HBASE-13346, change of Filter method names
> >>>>>> to
> >>>>>> mention Cell instead of KeyValue can land.
> >>>>>>
> >>>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
> >>>>>> IA.Private objects as params to MasterObserver... Hopefully this
> goes
> >>>>>> in
> >>>>>> soon.
> >>>>>>
> >>>>>> Duo is hard at work on trackers for flush and compaction for CPs
> >>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
> >>>>>>
> >>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after
> >>>>>> Duo
> >>>>>> is
> >>>>>> done w/ his work above.
> >>>>>>
> >>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after
> >>>>>> all
> >>>>>> the
> >>>>>> purges allowing CPs do direct calls against Regions in same Host.
> >>>>>>
> >>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> >>>>>>
> >>>>>> Another day or two?
> >>>>>>
> >>>>>> St.Ack
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>>
> >>>>>>> +1
> >>>>>>>
> >>>>>>>
> >>>>>>>> I was trying to work on helping out on the outstanding alpha-4
> stuff
> >>>>>>>> last
> >>>>>>>> week -- will be continuing to try to do the same this week.
> >>>>>>>>
> >>>>>>>> If you need any help, Stack, or if others need reviews where I
> >>>>>>>> haven't
> >>>>>>>> noticed on my own: feel free to @mention me.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Thanks for the offer Josh. All items seem assigned and are being
> >>>>>>>>
> >>>>>>>> actively
> >>>>>>> worked on. If you get a moment, reviews by you (or anyone else)
> helps
> >>>>>>> move
> >>>>>>> the process along.
> >>>>>>>
> >>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
> >>>>>>> improvements.
> >>>>>>> Then HBASE-13346 can go in.
> >>>>>>>
> >>>>>>> You are already helping out on HBASE-18906, thanks. Looks like that
> >>>>>>> will
> >>>>>>> be addressed by other alpha-4s about to land.
> >>>>>>>
> >>>>>>> St.Ack
> >>>>>>> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/
> 12341
> >>>>>>> 594
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 10/23/17 12:53 PM, Stack wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>> (Reviving this thread)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has
> >>>>>>>>> the
> >>>>>>>>> refactor of the Coprocessor API shutting down access to internals
> >>>>>>>>> marked
> >>>>>>>>> InterfaceAudience.Private.
> >>>>>>>>>
> >>>>>>>>> The outstanding list is here:
> >>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >>>>>>>>>
> >>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
> >>>>>>>>>
> >>>>>>>>> If issue, talk out loud on this thread. If you need a review to
> >>>>>>>>> land
> >>>>>>>>> an
> >>>>>>>>> item, shout on the issue and here; we'll help you out.
> >>>>>>>>>
> >>>>>>>>> As is, items are coming along nicely I'd say. We need to merge
> the
> >>>>>>>>> filter
> >>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> >>>>>>>>>
> >>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and help
> >>>>>>>>> them
> >>>>>>>>> test
> >>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
> >>>>>>>>> downstreamers know what to expect (or we discover what we missed
> >>>>>>>>> BEFORE
> >>>>>>>>> we
> >>>>>>>>> beta-1).
> >>>>>>>>>
> >>>>>>>>> Thanks for time,
> >>>>>>>>> S
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> >>>>>>>>>
> >>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That
> should
> >>>>>>>>> be
> >>>>>>>>>
> >>>>>>>>> time, if we all sprint, for the public-facing API fixes to be
> done.
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for
> >>>>>>>>>> alpha3
> >>>>>>>>>> but
> >>>>>>>>>> it is plain that more time is needed (in spite of valiant effort
> >>>>>>>>>> so
> >>>>>>>>>> far
> >>>>>>>>>> by
> >>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
> whose
> >>>>>>>>>> theme is
> >>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> >>>>>>>>>> following
> >>>>>>>>>> week.
> >>>>>>>>>>
> >>>>>>>>>> We should then be ready for beta (beta == no new features, no
> API
> >>>>>>>>>> changes,
> >>>>>>>>>> just fixes).
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> St.Ack
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote
> on
> >>>>>>>>>> it.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
> >>>>>>>>>>> get a
> >>>>>>>>>>> release out in the next week or so.
> >>>>>>>>>>>
> >>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> >>>>>>>>>>> interested in helping out, below are the items I think we need
> >>>>>>>>>>> done
> >>>>>>>>>>> for
> >>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API possibly
> >>>>>>>>>>> altering
> >>>>>>>>>>> items, and are absent those JIRAs that are making active
> >>>>>>>>>>> progress,
> >>>>>>>>>>> i.e. the
> >>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
> >>>>>>>>>>> needs
> >>>>>>>>>>> doing is
> >>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1
> >>>>>>>>>>> and
> >>>>>>>>>>> branch-2
> >>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only
> >>>>>>>>>>> do
> >>>>>>>>>>> it
> >>>>>>>>>>> between branch-1 and branch-2.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public /
> >>>>>>>>>>> Private
> >>>>>>>>>>> interfaces
> >>>>>>>>>>> This one is almost done. It could do with a finish, attention
> to
> >>>>>>>>>>> the
> >>>>>>>>>>> items in last comment, and then our codebase could do with
> >>>>>>>>>>> another
> >>>>>>>>>>> sweep
> >>>>>>>>>>> after the spirit of this issue since a bunch has gone in since
> >>>>>>>>>>> the
> >>>>>>>>>>> pass
> >>>>>>>>>>> that was the basis of this issue.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-10504 Define Replication Interface
> >>>>>>>>>>> I was going to take a crack at this as part of the revamp
> forced
> >>>>>>>>>>> by
> >>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> >>>>>>>>>>> Service'
> >>>>>>>>>>> but if
> >>>>>>>>>>> anyone else is interested, be my guest.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
> >>>>>>>>>>> Has a bunch of subtasks, some of which are being worked on.
> Needs
> >>>>>>>>>>> finishing.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
> Admin
> >>>>>>>>>>> and
> >>>>>>>>>>> cleanup
> >>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
> >>>>>>>>>>> AsyncClient
> >>>>>>>>>>> and
> >>>>>>>>>>> make sure symmetry.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> >>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but needs
> to
> >>>>>>>>>>> be
> >>>>>>>>>>> done.
> >>>>>>>>>>>
> >>>>>>>>>>> Rolling upgrade will have implications for our API. Would be
> good
> >>>>>>>>>>> to
> >>>>>>>>>>> try
> >>>>>>>>>>> it and figure what needs fixup (as said above, according to
> trial
> >>>>>>>>>>> by
> >>>>>>>>>>> Sean,
> >>>>>>>>>>> we might not be too bad here):
> >>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking to
> >>>>>>>>>>> 2.0
> >>>>>>>>>>> cluster
> >>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
> Master
> >>>>>>>>>>> and
> >>>>>>>>>>> 1.x
> >>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-17442 Move most of the replication related classes to
> >>>>>>>>>>> hbase-server package
> >>>>>>>>>>> The above would be good to do generally but it may make for
> >>>>>>>>>>> ripples
> >>>>>>>>>>> in
> >>>>>>>>>>> API so would be good to do now.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> >>>>>>>>>>> Balazs is working on this. The idea is that we avoid adding two
> >>>>>>>>>>> new
> >>>>>>>>>>> types
> >>>>>>>>>>> to our API, two types that are nought but curtailed, read-only
> >>>>>>>>>>> views
> >>>>>>>>>>> on
> >>>>>>>>>>> internals. Input if you have time appreciated.
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
> >>>>>>>>>>> hbase2
> >>>>>>>>>>> cluster; verify
> >>>>>>>>>>> Esteban is looking at this one
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> >>>>>>>>>>> * HBASE-17143 Scan improvement
> >>>>>>>>>>>
> >>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
> >>>>>>>>>>> implementations
> >>>>>>>>>>> marked audience Private and returns implementations rather than
> >>>>>>>>>>> Interfaces.
> >>>>>>>>>>> In a few locations, we allow returning an alternate
> >>>>>>>>>>> implementation
> >>>>>>>>>>> altogether which is probably something we don't want a CP
> doing.
> >>>>>>>>>>> To
> >>>>>>>>>>> that
> >>>>>>>>>>> end, the following issues started by Duo and Anoop need to be
> >>>>>>>>>>> taken
> >>>>>>>>>>> to
> >>>>>>>>>>> the
> >>>>>>>>>>> finish line; ideally they'd have an owner:
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release
> <=
> >>>>>>>>>>> The
> >>>>>>>>>>> umbrella issue.
> >>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
> >>>>>>>>>>> expose
> >>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from
> MasterObserver
> >>>>>>>>>>> and
> >>>>>>>>>>> RegionServerObserver
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Nice-to-haves:
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
> >>>>>>>>>>> unused
> >>>>>>>>>>> TimeRange constructors
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
> >>>>>>>>>>> references
> >>>>>>>>>>> existing in the code
> >>>>>>>>>>> This is the end of an old long-running project moving up on to
> >>>>>>>>>>> Cell
> >>>>>>>>>>> Interface. We think it is done but for a few little items
> >>>>>>>>>>> (deprecate
> >>>>>>>>>>> KV
> >>>>>>>>>>> methods in MR and provide Cell versions instead...)
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
> >>>>>>>>>>> needs
> >>>>>>>>>>> fixing
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
> >>>>>>>>>>> * HBASE-14997
> >>>>>>>>>>> Move compareOp and Comparators out of filter to client package
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
> >>>>>>>>>>>
> >>>>>>>>>>> What about:
> >>>>>>>>>>>
> >>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
> >>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
> >>>>>>>>>>>
> >>>>>>>>>>> If interested in any of the above and you need a legup, just
> ask
> >>>>>>>>>>> in
> >>>>>>>>>>> the
> >>>>>>>>>>> issue and I'll be by....
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> St.Ack
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Heads-up:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
> >>>>>>>>>>>> Theme
> >>>>>>>>>>>> is
> >>>>>>>>>>>> updated dependencies, reliance on relocated popular libs
> (guava,
> >>>>>>>>>>>> netty,
> >>>>>>>>>>>> protobuf), purge of checked-in generated src, and
> >>>>>>>>>>>> master-carries-no-regions
> >>>>>>>>>>>> by default.
> >>>>>>>>>>>>
> >>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
> theme
> >>>>>>>>>>>> will
> >>>>>>>>>>>> be
> >>>>>>>>>>>> settling the APIs and compatibility (At first blush, we are
> not
> >>>>>>>>>>>> looking too
> >>>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1
> >>>>>>>>>>>> client
> >>>>>>>>>>>> running
> >>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface
> >>>>>>>>>>>> revamp
> >>>>>>>>>>>> should be
> >>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
> >>>>>>>>>>>> Implementations, and
> >>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
> >>>>>>>>>>>> InterfaceAudience).
> >>>>>>>>>>>> We'll also have purged thirdparty classes from our API; e.g.
> >>>>>>>>>>>> guava
> >>>>>>>>>>>> 0.12
> >>>>>>>>>>>> Service showing through in our replication API and protobufs
> in
> >>>>>>>>>>>> Admin
> >>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach
> to
> >>>>>>>>>>>> make
> >>>>>>>>>>>> sure
> >>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Beta1 in mid-September?
> >>>>>>>>>>>>
> >>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
> >>>>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> 12327188
> >>>>>>>>>>>> Edit
> >>>>>>>>>>>> as
> >>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get
> any
> >>>>>>>>>>>> attention
> >>>>>>>>>>>> in next month or so.
> >>>>>>>>>>>>
> >>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave
> them
> >>>>>>>>>>>> as
> >>>>>>>>>>>> is
> >>>>>>>>>>>> another while but I'll boot them soon.
> >>>>>>>>>>>>
> >>>>>>>>>>>> While I have your attention:
> >>>>>>>>>>>>
> >>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving
> hbase
> >>>>>>>>>>>> thrift
> >>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy
> enough
> >>>>>>>>>>>> if
> >>>>>>>>>>>> folks
> >>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> >>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
> first
> >>>>>>>>>>>> (0.98?).
> >>>>>>>>>>>>
> >>>>>>>>>>>> St.Ack
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <elserj@apache.org
> >
> >>>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
> >>>>>>>>>>>>>> elserj@apache.org>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make
> the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> feature a
> >>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I have
> >>>>>>>>>>>>>>>> no
> >>>>>>>>>>>>>>>> reason
> >>>>>>>>>>>>>>>> to be
> >>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in
> terms
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>> how
> >>>>>>>>>>>>>>>> much it
> >>>>>>>>>>>>>>>> is covered via testing.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
> Josh.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See
> 'Space
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Quota' in
> >>>>>>>>>>>>>>> here:
> >>>>>>>>>>>>>>> https://docs.google.com/document/d/
> 1WCsVlnHjJeKUcl7wHwqb4z9i
> >>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> >>>>>>>>>>>>>>> Does this little section need an update Josh?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> S
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and
> >>>>>>>>>>>>>>> some
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> docs
> >>>>>>>>>>>>>> already included for 17748.  Happy to put that on my plate
> if
> >>>>>>>>>>>>>> you're good
> >>>>>>>>>>>>>> with it. I can reasonably assume that no one is against it
> :)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff
> before
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>> merged it in from the original feature branch. I'll double
> >>>>>>>>>>>>>> check
> >>>>>>>>>>>>>> and update
> >>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks Josh,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> S
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
>

Re: Moving 2.0 forward

[Mike Drob <md...@apache.org>]

Hoping to keep momentum going from our Stack working on alpha4, I tried to
take a stab at triaging some of the open beta-1 issues.

I moved some docs stuff out form beta-1 to 2.0-GA, if it gets done sooner
then I'm happy to see it pulled back in. Trying to balance optimism with
realism here, and knowing that documentation unfortunately often gets
pushed to the back-burner.

Also, I poked some folks on unassigned issues that they've filed for
beta-1, especially in the last few days. If issues don't have an owner they
are unlikely to get worked. I chatted with stack and he agreed to take on
some of the tasks, but there's a lot of surface area to cover.

If you you're working on issues that are critical for beta-1, please mark
them as such. Then the rest of the community will know to help prioritize
feedback and reviews there.

Do we have a general theme for the betas like we did with the alphas?

Beta1 is upgrades work from branch1, beta2 is rolling upgrades work as
well? Continue to work on tests throughout?

Mike

On Tue, Oct 31, 2017 at 10:04 AM, Josh Elser <el...@apache.org> wrote:

> +1 go from my POV.
>
>
> On 10/31/17 10:07 AM, Stack wrote:
>
>> I want to push an alpha-4 today. A few items didn't make it (HBASE-19092).
>> They need more time. We'll pull them in for beta-1. CP API is basically
>> done. There may be some changes for beta-1 but hopefully only changes
>> informed by experience trying to port an existing Coprocessor to hbase2.
>>
>> Shout if there is anything that needs to make alpha-4.
>>
>> Thanks,
>> St.Ack
>>
>>
>> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com> wrote:
>>
>> Yup, that was going to be my plan, Mike!
>>>
>>> Making a pass now, and will check back later tonight again. I see others
>>> have already done some work today on this front.
>>>
>>>
>>> On 10/27/17 11:38 PM, Mike Drob wrote:
>>>
>>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how to do
>>>> it directly on the jenkins job, so you don't have to bother with JIRA
>>>> uploads)
>>>>
>>>> If you're busy, then I can make time tomorrow or Sunday to kick off
>>>> jobs,
>>>> but I want to make sure we're not duplicating effort and jenkins cycles.
>>>>
>>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org> wrote:
>>>>
>>>> My turn to bump ;)
>>>>
>>>>>
>>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that remain
>>>>> needing some more work. The rest are just awaiting a good QA run.
>>>>>
>>>>> Unless I hear otherwise, I'll try to keep an eye on things over the
>>>>> weekend, bump them along as necessary, and get them committed. Would be
>>>>> great to be able get a vote up on Monday.
>>>>>
>>>>>
>>>>> On 10/24/17 6:03 PM, Stack wrote:
>>>>>
>>>>> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
>>>>>
>>>>>> what remains to be done. Surfacing our thoughts here so you all clued
>>>>>> in....Or if you think otherwise, please speak up.
>>>>>>
>>>>>> We have ~13 issues to land:
>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About
>>>>>> two
>>>>>> are meta-issues that are about process which leaves 11.
>>>>>>
>>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
>>>>>> changed
>>>>>> API/semantic that we need to get out earlier rather than later.
>>>>>>
>>>>>> Once the above is merged, HBASE-13346, change of Filter method names
>>>>>> to
>>>>>> mention Cell instead of KeyValue can land.
>>>>>>
>>>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
>>>>>> IA.Private objects as params to MasterObserver... Hopefully this goes
>>>>>> in
>>>>>> soon.
>>>>>>
>>>>>> Duo is hard at work on trackers for flush and compaction for CPs
>>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>>>>>>
>>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after
>>>>>> Duo
>>>>>> is
>>>>>> done w/ his work above.
>>>>>>
>>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after
>>>>>> all
>>>>>> the
>>>>>> purges allowing CPs do direct calls against Regions in same Host.
>>>>>>
>>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>>>>>
>>>>>> Another day or two?
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>>
>>>>>>>> I was trying to work on helping out on the outstanding alpha-4 stuff
>>>>>>>> last
>>>>>>>> week -- will be continuing to try to do the same this week.
>>>>>>>>
>>>>>>>> If you need any help, Stack, or if others need reviews where I
>>>>>>>> haven't
>>>>>>>> noticed on my own: feel free to @mention me.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks for the offer Josh. All items seem assigned and are being
>>>>>>>>
>>>>>>>> actively
>>>>>>> worked on. If you get a moment, reviews by you (or anyone else) helps
>>>>>>> move
>>>>>>> the process along.
>>>>>>>
>>>>>>> We need to merge in HBASE-18410 branch to pick up Filter
>>>>>>> improvements.
>>>>>>> Then HBASE-13346 can go in.
>>>>>>>
>>>>>>> You are already helping out on HBASE-18906, thanks. Looks like that
>>>>>>> will
>>>>>>> be addressed by other alpha-4s about to land.
>>>>>>>
>>>>>>> St.Ack
>>>>>>> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341
>>>>>>> 594
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 10/23/17 12:53 PM, Stack wrote:
>>>>>>>
>>>>>>>
>>>>>>>> (Reviving this thread)
>>>>>>>>
>>>>>>>>
>>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has
>>>>>>>>> the
>>>>>>>>> refactor of the Coprocessor API shutting down access to internals
>>>>>>>>> marked
>>>>>>>>> InterfaceAudience.Private.
>>>>>>>>>
>>>>>>>>> The outstanding list is here:
>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>>>>>>
>>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>>>>>>
>>>>>>>>> If issue, talk out loud on this thread. If you need a review to
>>>>>>>>> land
>>>>>>>>> an
>>>>>>>>> item, shout on the issue and here; we'll help you out.
>>>>>>>>>
>>>>>>>>> As is, items are coming along nicely I'd say. We need to merge the
>>>>>>>>> filter
>>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>>>>>>
>>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and help
>>>>>>>>> them
>>>>>>>>> test
>>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>>>>>>>>> downstreamers know what to expect (or we discover what we missed
>>>>>>>>> BEFORE
>>>>>>>>> we
>>>>>>>>> beta-1).
>>>>>>>>>
>>>>>>>>> Thanks for time,
>>>>>>>>> S
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>>>>>>>>
>>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That should
>>>>>>>>> be
>>>>>>>>>
>>>>>>>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for
>>>>>>>>>> alpha3
>>>>>>>>>> but
>>>>>>>>>> it is plain that more time is needed (in spite of valiant effort
>>>>>>>>>> so
>>>>>>>>>> far
>>>>>>>>>> by
>>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>>>>>>>>>> theme is
>>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>>>>>>>>>> following
>>>>>>>>>> week.
>>>>>>>>>>
>>>>>>>>>> We should then be ready for beta (beta == no new features, no API
>>>>>>>>>> changes,
>>>>>>>>>> just fixes).
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> St.Ack
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>>>>>>>>
>>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on
>>>>>>>>>> it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
>>>>>>>>>>> get a
>>>>>>>>>>> release out in the next week or so.
>>>>>>>>>>>
>>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>>>>>>>>> interested in helping out, below are the items I think we need
>>>>>>>>>>> done
>>>>>>>>>>> for
>>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API possibly
>>>>>>>>>>> altering
>>>>>>>>>>> items, and are absent those JIRAs that are making active
>>>>>>>>>>> progress,
>>>>>>>>>>> i.e. the
>>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
>>>>>>>>>>> needs
>>>>>>>>>>> doing is
>>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1
>>>>>>>>>>> and
>>>>>>>>>>> branch-2
>>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only
>>>>>>>>>>> do
>>>>>>>>>>> it
>>>>>>>>>>> between branch-1 and branch-2.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public /
>>>>>>>>>>> Private
>>>>>>>>>>> interfaces
>>>>>>>>>>> This one is almost done. It could do with a finish, attention to
>>>>>>>>>>> the
>>>>>>>>>>> items in last comment, and then our codebase could do with
>>>>>>>>>>> another
>>>>>>>>>>> sweep
>>>>>>>>>>> after the spirit of this issue since a bunch has gone in since
>>>>>>>>>>> the
>>>>>>>>>>> pass
>>>>>>>>>>> that was the basis of this issue.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-10504 Define Replication Interface
>>>>>>>>>>> I was going to take a crack at this as part of the revamp forced
>>>>>>>>>>> by
>>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
>>>>>>>>>>> Service'
>>>>>>>>>>> but if
>>>>>>>>>>> anyone else is interested, be my guest.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>>>>>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>>>>>>>>> finishing.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
>>>>>>>>>>> and
>>>>>>>>>>> cleanup
>>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>>>>>>>>>>> AsyncClient
>>>>>>>>>>> and
>>>>>>>>>>> make sure symmetry.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but needs to
>>>>>>>>>>> be
>>>>>>>>>>> done.
>>>>>>>>>>>
>>>>>>>>>>> Rolling upgrade will have implications for our API. Would be good
>>>>>>>>>>> to
>>>>>>>>>>> try
>>>>>>>>>>> it and figure what needs fixup (as said above, according to trial
>>>>>>>>>>> by
>>>>>>>>>>> Sean,
>>>>>>>>>>> we might not be too bad here):
>>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking to
>>>>>>>>>>> 2.0
>>>>>>>>>>> cluster
>>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master
>>>>>>>>>>> and
>>>>>>>>>>> 1.x
>>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-17442 Move most of the replication related classes to
>>>>>>>>>>> hbase-server package
>>>>>>>>>>> The above would be good to do generally but it may make for
>>>>>>>>>>> ripples
>>>>>>>>>>> in
>>>>>>>>>>> API so would be good to do now.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>>>>>>>> Balazs is working on this. The idea is that we avoid adding two
>>>>>>>>>>> new
>>>>>>>>>>> types
>>>>>>>>>>> to our API, two types that are nought but curtailed, read-only
>>>>>>>>>>> views
>>>>>>>>>>> on
>>>>>>>>>>> internals. Input if you have time appreciated.
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
>>>>>>>>>>> hbase2
>>>>>>>>>>> cluster; verify
>>>>>>>>>>> Esteban is looking at this one
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>>>>>>>>> * HBASE-17143 Scan improvement
>>>>>>>>>>>
>>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>>>>>>>> implementations
>>>>>>>>>>> marked audience Private and returns implementations rather than
>>>>>>>>>>> Interfaces.
>>>>>>>>>>> In a few locations, we allow returning an alternate
>>>>>>>>>>> implementation
>>>>>>>>>>> altogether which is probably something we don't want a CP doing.
>>>>>>>>>>> To
>>>>>>>>>>> that
>>>>>>>>>>> end, the following issues started by Duo and Anoop need to be
>>>>>>>>>>> taken
>>>>>>>>>>> to
>>>>>>>>>>> the
>>>>>>>>>>> finish line; ideally they'd have an owner:
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
>>>>>>>>>>> The
>>>>>>>>>>> umbrella issue.
>>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
>>>>>>>>>>> expose
>>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver
>>>>>>>>>>> and
>>>>>>>>>>> RegionServerObserver
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Nice-to-haves:
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
>>>>>>>>>>> unused
>>>>>>>>>>> TimeRange constructors
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
>>>>>>>>>>> references
>>>>>>>>>>> existing in the code
>>>>>>>>>>> This is the end of an old long-running project moving up on to
>>>>>>>>>>> Cell
>>>>>>>>>>> Interface. We think it is done but for a few little items
>>>>>>>>>>> (deprecate
>>>>>>>>>>> KV
>>>>>>>>>>> methods in MR and provide Cell versions instead...)
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
>>>>>>>>>>> needs
>>>>>>>>>>> fixing
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>>>>>>>> * HBASE-14997
>>>>>>>>>>> Move compareOp and Comparators out of filter to client package
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>>>>>>>
>>>>>>>>>>> What about:
>>>>>>>>>>>
>>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>>>>>>>
>>>>>>>>>>> If interested in any of the above and you need a legup, just ask
>>>>>>>>>>> in
>>>>>>>>>>> the
>>>>>>>>>>> issue and I'll be by....
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> St.Ack
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Heads-up:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
>>>>>>>>>>>> Theme
>>>>>>>>>>>> is
>>>>>>>>>>>> updated dependencies, reliance on relocated popular libs (guava,
>>>>>>>>>>>> netty,
>>>>>>>>>>>> protobuf), purge of checked-in generated src, and
>>>>>>>>>>>> master-carries-no-regions
>>>>>>>>>>>> by default.
>>>>>>>>>>>>
>>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
>>>>>>>>>>>> will
>>>>>>>>>>>> be
>>>>>>>>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>>>>>>>>> looking too
>>>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1
>>>>>>>>>>>> client
>>>>>>>>>>>> running
>>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface
>>>>>>>>>>>> revamp
>>>>>>>>>>>> should be
>>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>>>>>>>> Implementations, and
>>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>>>>>>>> InterfaceAudience).
>>>>>>>>>>>> We'll also have purged thirdparty classes from our API; e.g.
>>>>>>>>>>>> guava
>>>>>>>>>>>> 0.12
>>>>>>>>>>>> Service showing through in our replication API and protobufs in
>>>>>>>>>>>> Admin
>>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach to
>>>>>>>>>>>> make
>>>>>>>>>>>> sure
>>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>>>>>>>>
>>>>>>>>>>>> Beta1 in mid-September?
>>>>>>>>>>>>
>>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>>>>>>>>>>> Edit
>>>>>>>>>>>> as
>>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>>>>>>>>> attention
>>>>>>>>>>>> in next month or so.
>>>>>>>>>>>>
>>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave them
>>>>>>>>>>>> as
>>>>>>>>>>>> is
>>>>>>>>>>>> another while but I'll boot them soon.
>>>>>>>>>>>>
>>>>>>>>>>>> While I have your attention:
>>>>>>>>>>>>
>>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
>>>>>>>>>>>> thrift
>>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy enough
>>>>>>>>>>>> if
>>>>>>>>>>>> folks
>>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>>>>>>>>> (0.98?).
>>>>>>>>>>>>
>>>>>>>>>>>> St.Ack
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
>>>>>>>>>>>>>> elserj@apache.org>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> feature a
>>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I have
>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>> reason
>>>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in terms
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>> much it
>>>>>>>>>>>>>>>> is covered via testing.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See 'Space
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Quota' in
>>>>>>>>>>>>>>> here:
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>>>>>>>>> Does this little section need an update Josh?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> S
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and
>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> docs
>>>>>>>>>>>>>> already included for 17748.  Happy to put that on my plate if
>>>>>>>>>>>>>> you're good
>>>>>>>>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff before
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> merged it in from the original feature branch. I'll double
>>>>>>>>>>>>>> check
>>>>>>>>>>>>>> and update
>>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks Josh,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> S
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>

Re: Moving 2.0 forward

[Josh Elser <el...@apache.org>]

+1 go from my POV.

On 10/31/17 10:07 AM, Stack wrote:
> I want to push an alpha-4 today. A few items didn't make it (HBASE-19092).
> They need more time. We'll pull them in for beta-1. CP API is basically
> done. There may be some changes for beta-1 but hopefully only changes
> informed by experience trying to port an existing Coprocessor to hbase2.
> 
> Shout if there is anything that needs to make alpha-4.
> 
> Thanks,
> St.Ack
> 
> 
> On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com> wrote:
> 
>> Yup, that was going to be my plan, Mike!
>>
>> Making a pass now, and will check back later tonight again. I see others
>> have already done some work today on this front.
>>
>>
>> On 10/27/17 11:38 PM, Mike Drob wrote:
>>
>>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how to do
>>> it directly on the jenkins job, so you don't have to bother with JIRA
>>> uploads)
>>>
>>> If you're busy, then I can make time tomorrow or Sunday to kick off jobs,
>>> but I want to make sure we're not duplicating effort and jenkins cycles.
>>>
>>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org> wrote:
>>>
>>> My turn to bump ;)
>>>>
>>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that remain
>>>> needing some more work. The rest are just awaiting a good QA run.
>>>>
>>>> Unless I hear otherwise, I'll try to keep an eye on things over the
>>>> weekend, bump them along as necessary, and get them committed. Would be
>>>> great to be able get a vote up on Monday.
>>>>
>>>>
>>>> On 10/24/17 6:03 PM, Stack wrote:
>>>>
>>>> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
>>>>> what remains to be done. Surfacing our thoughts here so you all clued
>>>>> in....Or if you think otherwise, please speak up.
>>>>>
>>>>> We have ~13 issues to land:
>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About
>>>>> two
>>>>> are meta-issues that are about process which leaves 11.
>>>>>
>>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
>>>>> changed
>>>>> API/semantic that we need to get out earlier rather than later.
>>>>>
>>>>> Once the above is merged, HBASE-13346, change of Filter method names to
>>>>> mention Cell instead of KeyValue can land.
>>>>>
>>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
>>>>> IA.Private objects as params to MasterObserver... Hopefully this goes in
>>>>> soon.
>>>>>
>>>>> Duo is hard at work on trackers for flush and compaction for CPs
>>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>>>>>
>>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo
>>>>> is
>>>>> done w/ his work above.
>>>>>
>>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all
>>>>> the
>>>>> purges allowing CPs do direct calls against Regions in same Host.
>>>>>
>>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>>>>
>>>>> Another day or two?
>>>>>
>>>>> St.Ack
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>>>>
>>>>>
>>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>>>>>>
>>>>>> +1
>>>>>>
>>>>>>>
>>>>>>> I was trying to work on helping out on the outstanding alpha-4 stuff
>>>>>>> last
>>>>>>> week -- will be continuing to try to do the same this week.
>>>>>>>
>>>>>>> If you need any help, Stack, or if others need reviews where I haven't
>>>>>>> noticed on my own: feel free to @mention me.
>>>>>>>
>>>>>>>
>>>>>>> Thanks for the offer Josh. All items seem assigned and are being
>>>>>>>
>>>>>> actively
>>>>>> worked on. If you get a moment, reviews by you (or anyone else) helps
>>>>>> move
>>>>>> the process along.
>>>>>>
>>>>>> We need to merge in HBASE-18410 branch to pick up Filter improvements.
>>>>>> Then HBASE-13346 can go in.
>>>>>>
>>>>>> You are already helping out on HBASE-18906, thanks. Looks like that
>>>>>> will
>>>>>> be addressed by other alpha-4s about to land.
>>>>>>
>>>>>> St.Ack
>>>>>> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 10/23/17 12:53 PM, Stack wrote:
>>>>>>
>>>>>>>
>>>>>>> (Reviving this thread)
>>>>>>>
>>>>>>>>
>>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>>>>>>>> refactor of the Coprocessor API shutting down access to internals
>>>>>>>> marked
>>>>>>>> InterfaceAudience.Private.
>>>>>>>>
>>>>>>>> The outstanding list is here:
>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>>>>>
>>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>>>>>
>>>>>>>> If issue, talk out loud on this thread. If you need a review to land
>>>>>>>> an
>>>>>>>> item, shout on the issue and here; we'll help you out.
>>>>>>>>
>>>>>>>> As is, items are coming along nicely I'd say. We need to merge the
>>>>>>>> filter
>>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>>>>>
>>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and help them
>>>>>>>> test
>>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>>>>>>>> downstreamers know what to expect (or we discover what we missed
>>>>>>>> BEFORE
>>>>>>>> we
>>>>>>>> beta-1).
>>>>>>>>
>>>>>>>> Thanks for time,
>>>>>>>> S
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>>>>>>>
>>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That should
>>>>>>>> be
>>>>>>>>
>>>>>>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>>>>>>>
>>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
>>>>>>>>> but
>>>>>>>>> it is plain that more time is needed (in spite of valiant effort so
>>>>>>>>> far
>>>>>>>>> by
>>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>>>>>>>>> theme is
>>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>>>>>>>>> following
>>>>>>>>> week.
>>>>>>>>>
>>>>>>>>> We should then be ready for beta (beta == no new features, no API
>>>>>>>>> changes,
>>>>>>>>> just fixes).
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> St.Ack
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>>>>>>>
>>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on
>>>>>>>>> it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
>>>>>>>>>> get a
>>>>>>>>>> release out in the next week or so.
>>>>>>>>>>
>>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>>>>>>>> interested in helping out, below are the items I think we need done
>>>>>>>>>> for
>>>>>>>>>> alpha3 (below are at least 'Critical' status, are API possibly
>>>>>>>>>> altering
>>>>>>>>>> items, and are absent those JIRAs that are making active progress,
>>>>>>>>>> i.e. the
>>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>>>>>>>>>> doing is
>>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>>>>>>
>>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>>>>>>>>> branch-2
>>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do
>>>>>>>>>> it
>>>>>>>>>> between branch-1 and branch-2.
>>>>>>>>>>
>>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public /
>>>>>>>>>> Private
>>>>>>>>>> interfaces
>>>>>>>>>> This one is almost done. It could do with a finish, attention to
>>>>>>>>>> the
>>>>>>>>>> items in last comment, and then our codebase could do with another
>>>>>>>>>> sweep
>>>>>>>>>> after the spirit of this issue since a bunch has gone in since the
>>>>>>>>>> pass
>>>>>>>>>> that was the basis of this issue.
>>>>>>>>>>
>>>>>>>>>> * HBASE-10504 Define Replication Interface
>>>>>>>>>> I was going to take a crack at this as part of the revamp forced by
>>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
>>>>>>>>>> but if
>>>>>>>>>> anyone else is interested, be my guest.
>>>>>>>>>>
>>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>>>>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>>>>>>>> finishing.
>>>>>>>>>>
>>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
>>>>>>>>>> and
>>>>>>>>>> cleanup
>>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>>>>>>>>>> AsyncClient
>>>>>>>>>> and
>>>>>>>>>> make sure symmetry.
>>>>>>>>>>
>>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but needs to
>>>>>>>>>> be
>>>>>>>>>> done.
>>>>>>>>>>
>>>>>>>>>> Rolling upgrade will have implications for our API. Would be good
>>>>>>>>>> to
>>>>>>>>>> try
>>>>>>>>>> it and figure what needs fixup (as said above, according to trial
>>>>>>>>>> by
>>>>>>>>>> Sean,
>>>>>>>>>> we might not be too bad here):
>>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>>>>>>>>>> cluster
>>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master
>>>>>>>>>> and
>>>>>>>>>> 1.x
>>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>>>>>>
>>>>>>>>>> * HBASE-17442 Move most of the replication related classes to
>>>>>>>>>> hbase-server package
>>>>>>>>>> The above would be good to do generally but it may make for ripples
>>>>>>>>>> in
>>>>>>>>>> API so would be good to do now.
>>>>>>>>>>
>>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>>>>>>> Balazs is working on this. The idea is that we avoid adding two new
>>>>>>>>>> types
>>>>>>>>>> to our API, two types that are nought but curtailed, read-only
>>>>>>>>>> views
>>>>>>>>>> on
>>>>>>>>>> internals. Input if you have time appreciated.
>>>>>>>>>>
>>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
>>>>>>>>>> hbase2
>>>>>>>>>> cluster; verify
>>>>>>>>>> Esteban is looking at this one
>>>>>>>>>>
>>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>>>>>>>> * HBASE-17143 Scan improvement
>>>>>>>>>>
>>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>>>>>>> implementations
>>>>>>>>>> marked audience Private and returns implementations rather than
>>>>>>>>>> Interfaces.
>>>>>>>>>> In a few locations, we allow returning an alternate implementation
>>>>>>>>>> altogether which is probably something we don't want a CP doing. To
>>>>>>>>>> that
>>>>>>>>>> end, the following issues started by Duo and Anoop need to be taken
>>>>>>>>>> to
>>>>>>>>>> the
>>>>>>>>>> finish line; ideally they'd have an owner:
>>>>>>>>>>
>>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
>>>>>>>>>> The
>>>>>>>>>> umbrella issue.
>>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver
>>>>>>>>>> and
>>>>>>>>>> RegionServerObserver
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Nice-to-haves:
>>>>>>>>>>
>>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
>>>>>>>>>> unused
>>>>>>>>>> TimeRange constructors
>>>>>>>>>>
>>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>>>>>>>>> existing in the code
>>>>>>>>>> This is the end of an old long-running project moving up on to Cell
>>>>>>>>>> Interface. We think it is done but for a few little items
>>>>>>>>>> (deprecate
>>>>>>>>>> KV
>>>>>>>>>> methods in MR and provide Cell versions instead...)
>>>>>>>>>>
>>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
>>>>>>>>>> needs
>>>>>>>>>> fixing
>>>>>>>>>>
>>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>>>>>>
>>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>>>>>>> * HBASE-14997
>>>>>>>>>> Move compareOp and Comparators out of filter to client package
>>>>>>>>>>
>>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>>>>>>
>>>>>>>>>> What about:
>>>>>>>>>>
>>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>>>>>>
>>>>>>>>>> If interested in any of the above and you need a legup, just ask in
>>>>>>>>>> the
>>>>>>>>>> issue and I'll be by....
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> St.Ack
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>>>
>>>>>>>>>> Heads-up:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme
>>>>>>>>>>> is
>>>>>>>>>>> updated dependencies, reliance on relocated popular libs (guava,
>>>>>>>>>>> netty,
>>>>>>>>>>> protobuf), purge of checked-in generated src, and
>>>>>>>>>>> master-carries-no-regions
>>>>>>>>>>> by default.
>>>>>>>>>>>
>>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
>>>>>>>>>>> will
>>>>>>>>>>> be
>>>>>>>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>>>>>>>> looking too
>>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>>>>>>>>>>> running
>>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>>>>>>>>>>> should be
>>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>>>>>>> Implementations, and
>>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>>>>>>> InterfaceAudience).
>>>>>>>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
>>>>>>>>>>> 0.12
>>>>>>>>>>> Service showing through in our replication API and protobufs in
>>>>>>>>>>> Admin
>>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach to
>>>>>>>>>>> make
>>>>>>>>>>> sure
>>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>>>>>>>
>>>>>>>>>>> Beta1 in mid-September?
>>>>>>>>>>>
>>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>>>>>>>>>> Edit
>>>>>>>>>>> as
>>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>>>>>>>> attention
>>>>>>>>>>> in next month or so.
>>>>>>>>>>>
>>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave them
>>>>>>>>>>> as
>>>>>>>>>>> is
>>>>>>>>>>> another while but I'll boot them soon.
>>>>>>>>>>>
>>>>>>>>>>> While I have your attention:
>>>>>>>>>>>
>>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
>>>>>>>>>>> thrift
>>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy enough
>>>>>>>>>>> if
>>>>>>>>>>> folks
>>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>>>>>>>> (0.98?).
>>>>>>>>>>>
>>>>>>>>>>> St.Ack
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>>>>>>>> feature a
>>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>>>>>>>>>>>>>>> reason
>>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in terms
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>> much it
>>>>>>>>>>>>>>> is covered via testing.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Last think on space quota says that need doc too. See 'Space
>>>>>>>>>>>>>> Quota' in
>>>>>>>>>>>>>> here:
>>>>>>>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>>>>>>>> Does this little section need an update Josh?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> S
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
>>>>>>>>>>>>>>
>>>>>>>>>>>>> docs
>>>>>>>>>>>>> already included for 17748.  Happy to put that on my plate if
>>>>>>>>>>>>> you're good
>>>>>>>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>>>>>>>>> merged it in from the original feature branch. I'll double check
>>>>>>>>>>>>> and update
>>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Josh,
>>>>>>>>>>>>>
>>>>>>>>>>>> S
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>
> 

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

I want to push an alpha-4 today. A few items didn't make it (HBASE-19092).
They need more time. We'll pull them in for beta-1. CP API is basically
done. There may be some changes for beta-1 but hopefully only changes
informed by experience trying to port an existing Coprocessor to hbase2.

Shout if there is anything that needs to make alpha-4.

Thanks,
St.Ack


On Sat, Oct 28, 2017 at 2:48 PM, Josh Elser <jo...@gmail.com> wrote:

> Yup, that was going to be my plan, Mike!
>
> Making a pass now, and will check back later tonight again. I see others
> have already done some work today on this front.
>
>
> On 10/27/17 11:38 PM, Mike Drob wrote:
>
>> Josh - Do you want to kick off a bunch of QA runs? (Do you know how to do
>> it directly on the jenkins job, so you don't have to bother with JIRA
>> uploads)
>>
>> If you're busy, then I can make time tomorrow or Sunday to kick off jobs,
>> but I want to make sure we're not duplicating effort and jenkins cycles.
>>
>> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org> wrote:
>>
>> My turn to bump ;)
>>>
>>> By my take: HBASE-18770 and HBASE-19092 are the only issues that remain
>>> needing some more work. The rest are just awaiting a good QA run.
>>>
>>> Unless I hear otherwise, I'll try to keep an eye on things over the
>>> weekend, bump them along as necessary, and get them committed. Would be
>>> great to be able get a vote up on Monday.
>>>
>>>
>>> On 10/24/17 6:03 PM, Stack wrote:
>>>
>>> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
>>>> what remains to be done. Surfacing our thoughts here so you all clued
>>>> in....Or if you think otherwise, please speak up.
>>>>
>>>> We have ~13 issues to land:
>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About
>>>> two
>>>> are meta-issues that are about process which leaves 11.
>>>>
>>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
>>>> changed
>>>> API/semantic that we need to get out earlier rather than later.
>>>>
>>>> Once the above is merged, HBASE-13346, change of Filter method names to
>>>> mention Cell instead of KeyValue can land.
>>>>
>>>> HBASE-199048 needs a review (Anoop will probably do it), removing
>>>> IA.Private objects as params to MasterObserver... Hopefully this goes in
>>>> soon.
>>>>
>>>> Duo is hard at work on trackers for flush and compaction for CPs
>>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>>>>
>>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo
>>>> is
>>>> done w/ his work above.
>>>>
>>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all
>>>> the
>>>> purges allowing CPs do direct calls against Regions in same Host.
>>>>
>>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>>>
>>>> Another day or two?
>>>>
>>>> St.Ack
>>>>
>>>>
>>>>
>>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>>>
>>>>
>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>>>>>
>>>>> +1
>>>>>
>>>>>>
>>>>>> I was trying to work on helping out on the outstanding alpha-4 stuff
>>>>>> last
>>>>>> week -- will be continuing to try to do the same this week.
>>>>>>
>>>>>> If you need any help, Stack, or if others need reviews where I haven't
>>>>>> noticed on my own: feel free to @mention me.
>>>>>>
>>>>>>
>>>>>> Thanks for the offer Josh. All items seem assigned and are being
>>>>>>
>>>>> actively
>>>>> worked on. If you get a moment, reviews by you (or anyone else) helps
>>>>> move
>>>>> the process along.
>>>>>
>>>>> We need to merge in HBASE-18410 branch to pick up Filter improvements.
>>>>> Then HBASE-13346 can go in.
>>>>>
>>>>> You are already helping out on HBASE-18906, thanks. Looks like that
>>>>> will
>>>>> be addressed by other alpha-4s about to land.
>>>>>
>>>>> St.Ack
>>>>> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 10/23/17 12:53 PM, Stack wrote:
>>>>>
>>>>>>
>>>>>> (Reviving this thread)
>>>>>>
>>>>>>>
>>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>>>>>>> refactor of the Coprocessor API shutting down access to internals
>>>>>>> marked
>>>>>>> InterfaceAudience.Private.
>>>>>>>
>>>>>>> The outstanding list is here:
>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>>>>
>>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>>>>
>>>>>>> If issue, talk out loud on this thread. If you need a review to land
>>>>>>> an
>>>>>>> item, shout on the issue and here; we'll help you out.
>>>>>>>
>>>>>>> As is, items are coming along nicely I'd say. We need to merge the
>>>>>>> filter
>>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>>>>
>>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and help them
>>>>>>> test
>>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>>>>>>> downstreamers know what to expect (or we discover what we missed
>>>>>>> BEFORE
>>>>>>> we
>>>>>>> beta-1).
>>>>>>>
>>>>>>> Thanks for time,
>>>>>>> S
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That should
>>>>>>> be
>>>>>>>
>>>>>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>>>>>>
>>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
>>>>>>>> but
>>>>>>>> it is plain that more time is needed (in spite of valiant effort so
>>>>>>>> far
>>>>>>>> by
>>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>>>>>>>> theme is
>>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>>>>>>>> following
>>>>>>>> week.
>>>>>>>>
>>>>>>>> We should then be ready for beta (beta == no new features, no API
>>>>>>>> changes,
>>>>>>>> just fixes).
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> St.Ack
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>>>>>>
>>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on
>>>>>>>> it.
>>>>>>>>
>>>>>>>>
>>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
>>>>>>>>> get a
>>>>>>>>> release out in the next week or so.
>>>>>>>>>
>>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>>>>>>> interested in helping out, below are the items I think we need done
>>>>>>>>> for
>>>>>>>>> alpha3 (below are at least 'Critical' status, are API possibly
>>>>>>>>> altering
>>>>>>>>> items, and are absent those JIRAs that are making active progress,
>>>>>>>>> i.e. the
>>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>>>>>>>>> doing is
>>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>>>>>
>>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>>>>>>>> branch-2
>>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do
>>>>>>>>> it
>>>>>>>>> between branch-1 and branch-2.
>>>>>>>>>
>>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public /
>>>>>>>>> Private
>>>>>>>>> interfaces
>>>>>>>>> This one is almost done. It could do with a finish, attention to
>>>>>>>>> the
>>>>>>>>> items in last comment, and then our codebase could do with another
>>>>>>>>> sweep
>>>>>>>>> after the spirit of this issue since a bunch has gone in since the
>>>>>>>>> pass
>>>>>>>>> that was the basis of this issue.
>>>>>>>>>
>>>>>>>>> * HBASE-10504 Define Replication Interface
>>>>>>>>> I was going to take a crack at this as part of the revamp forced by
>>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
>>>>>>>>> but if
>>>>>>>>> anyone else is interested, be my guest.
>>>>>>>>>
>>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>>>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>>>>>>> finishing.
>>>>>>>>>
>>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
>>>>>>>>> and
>>>>>>>>> cleanup
>>>>>>>>> Needs a pass. Small issue I think. Could also look at new
>>>>>>>>> AsyncClient
>>>>>>>>> and
>>>>>>>>> make sure symmetry.
>>>>>>>>>
>>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but needs to
>>>>>>>>> be
>>>>>>>>> done.
>>>>>>>>>
>>>>>>>>> Rolling upgrade will have implications for our API. Would be good
>>>>>>>>> to
>>>>>>>>> try
>>>>>>>>> it and figure what needs fixup (as said above, according to trial
>>>>>>>>> by
>>>>>>>>> Sean,
>>>>>>>>> we might not be too bad here):
>>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>>>>>>>>> cluster
>>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master
>>>>>>>>> and
>>>>>>>>> 1.x
>>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>>>>>
>>>>>>>>> * HBASE-17442 Move most of the replication related classes to
>>>>>>>>> hbase-server package
>>>>>>>>> The above would be good to do generally but it may make for ripples
>>>>>>>>> in
>>>>>>>>> API so would be good to do now.
>>>>>>>>>
>>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>>>>>> Balazs is working on this. The idea is that we avoid adding two new
>>>>>>>>> types
>>>>>>>>> to our API, two types that are nought but curtailed, read-only
>>>>>>>>> views
>>>>>>>>> on
>>>>>>>>> internals. Input if you have time appreciated.
>>>>>>>>>
>>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
>>>>>>>>> hbase2
>>>>>>>>> cluster; verify
>>>>>>>>> Esteban is looking at this one
>>>>>>>>>
>>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>>>>>>> * HBASE-17143 Scan improvement
>>>>>>>>>
>>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>>>>>> implementations
>>>>>>>>> marked audience Private and returns implementations rather than
>>>>>>>>> Interfaces.
>>>>>>>>> In a few locations, we allow returning an alternate implementation
>>>>>>>>> altogether which is probably something we don't want a CP doing. To
>>>>>>>>> that
>>>>>>>>> end, the following issues started by Duo and Anoop need to be taken
>>>>>>>>> to
>>>>>>>>> the
>>>>>>>>> finish line; ideally they'd have an owner:
>>>>>>>>>
>>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
>>>>>>>>> The
>>>>>>>>> umbrella issue.
>>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>>>>>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver
>>>>>>>>> and
>>>>>>>>> RegionServerObserver
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Nice-to-haves:
>>>>>>>>>
>>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
>>>>>>>>> unused
>>>>>>>>> TimeRange constructors
>>>>>>>>>
>>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>>>>>>>> existing in the code
>>>>>>>>> This is the end of an old long-running project moving up on to Cell
>>>>>>>>> Interface. We think it is done but for a few little items
>>>>>>>>> (deprecate
>>>>>>>>> KV
>>>>>>>>> methods in MR and provide Cell versions instead...)
>>>>>>>>>
>>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
>>>>>>>>> needs
>>>>>>>>> fixing
>>>>>>>>>
>>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>>>>>
>>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>>>>>> * HBASE-14997
>>>>>>>>> Move compareOp and Comparators out of filter to client package
>>>>>>>>>
>>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>>>>>
>>>>>>>>> What about:
>>>>>>>>>
>>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>>>>>
>>>>>>>>> If interested in any of the above and you need a legup, just ask in
>>>>>>>>> the
>>>>>>>>> issue and I'll be by....
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> St.Ack
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>>
>>>>>>>>> Heads-up:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme
>>>>>>>>>> is
>>>>>>>>>> updated dependencies, reliance on relocated popular libs (guava,
>>>>>>>>>> netty,
>>>>>>>>>> protobuf), purge of checked-in generated src, and
>>>>>>>>>> master-carries-no-regions
>>>>>>>>>> by default.
>>>>>>>>>>
>>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
>>>>>>>>>> will
>>>>>>>>>> be
>>>>>>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>>>>>>> looking too
>>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>>>>>>>>>> running
>>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>>>>>>>>>> should be
>>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>>>>>> Implementations, and
>>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>>>>>> InterfaceAudience).
>>>>>>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
>>>>>>>>>> 0.12
>>>>>>>>>> Service showing through in our replication API and protobufs in
>>>>>>>>>> Admin
>>>>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach to
>>>>>>>>>> make
>>>>>>>>>> sure
>>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>>>>>>
>>>>>>>>>> Beta1 in mid-September?
>>>>>>>>>>
>>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>>>>>>>>> Edit
>>>>>>>>>> as
>>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>>>>>>> attention
>>>>>>>>>> in next month or so.
>>>>>>>>>>
>>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave them
>>>>>>>>>> as
>>>>>>>>>> is
>>>>>>>>>> another while but I'll boot them soon.
>>>>>>>>>>
>>>>>>>>>> While I have your attention:
>>>>>>>>>>
>>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
>>>>>>>>>> thrift
>>>>>>>>>> to 0.10.0 will break existing clients. The change is easy enough
>>>>>>>>>> if
>>>>>>>>>> folks
>>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>>>>>>> (0.98?).
>>>>>>>>>>
>>>>>>>>>> St.Ack
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>>>>>>> feature a
>>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>>>>>>>>>>>>>> reason
>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in terms
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> how
>>>>>>>>>>>>>> much it
>>>>>>>>>>>>>> is covered via testing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Last think on space quota says that need doc too. See 'Space
>>>>>>>>>>>>> Quota' in
>>>>>>>>>>>>> here:
>>>>>>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>>>>>>> Does this little section need an update Josh?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> S
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
>>>>>>>>>>>>>
>>>>>>>>>>>> docs
>>>>>>>>>>>> already included for 17748.  Happy to put that on my plate if
>>>>>>>>>>>> you're good
>>>>>>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>>>>>>
>>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>>>>>>>> merged it in from the original feature branch. I'll double check
>>>>>>>>>>>> and update
>>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks Josh,
>>>>>>>>>>>>
>>>>>>>>>>> S
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>

Re: Moving 2.0 forward

[Josh Elser <jo...@gmail.com>]

Yup, that was going to be my plan, Mike!

Making a pass now, and will check back later tonight again. I see others 
have already done some work today on this front.

On 10/27/17 11:38 PM, Mike Drob wrote:
> Josh - Do you want to kick off a bunch of QA runs? (Do you know how to do
> it directly on the jenkins job, so you don't have to bother with JIRA
> uploads)
> 
> If you're busy, then I can make time tomorrow or Sunday to kick off jobs,
> but I want to make sure we're not duplicating effort and jenkins cycles.
> 
> On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org> wrote:
> 
>> My turn to bump ;)
>>
>> By my take: HBASE-18770 and HBASE-19092 are the only issues that remain
>> needing some more work. The rest are just awaiting a good QA run.
>>
>> Unless I hear otherwise, I'll try to keep an eye on things over the
>> weekend, bump them along as necessary, and get them committed. Would be
>> great to be able get a vote up on Monday.
>>
>>
>> On 10/24/17 6:03 PM, Stack wrote:
>>
>>> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
>>> what remains to be done. Surfacing our thoughts here so you all clued
>>> in....Or if you think otherwise, please speak up.
>>>
>>> We have ~13 issues to land:
>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About two
>>> are meta-issues that are about process which leaves 11.
>>>
>>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some changed
>>> API/semantic that we need to get out earlier rather than later.
>>>
>>> Once the above is merged, HBASE-13346, change of Filter method names to
>>> mention Cell instead of KeyValue can land.
>>>
>>> HBASE-199048 needs a review (Anoop will probably do it), removing
>>> IA.Private objects as params to MasterObserver... Hopefully this goes in
>>> soon.
>>>
>>> Duo is hard at work on trackers for flush and compaction for CPs
>>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>>>
>>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo
>>> is
>>> done w/ his work above.
>>>
>>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all
>>> the
>>> purges allowing CPs do direct calls against Regions in same Host.
>>>
>>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>>
>>> Another day or two?
>>>
>>> St.Ack
>>>
>>>
>>>
>>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>>
>>>
>>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>>>>
>>>> +1
>>>>>
>>>>> I was trying to work on helping out on the outstanding alpha-4 stuff
>>>>> last
>>>>> week -- will be continuing to try to do the same this week.
>>>>>
>>>>> If you need any help, Stack, or if others need reviews where I haven't
>>>>> noticed on my own: feel free to @mention me.
>>>>>
>>>>>
>>>>> Thanks for the offer Josh. All items seem assigned and are being
>>>> actively
>>>> worked on. If you get a moment, reviews by you (or anyone else) helps
>>>> move
>>>> the process along.
>>>>
>>>> We need to merge in HBASE-18410 branch to pick up Filter improvements.
>>>> Then HBASE-13346 can go in.
>>>>
>>>> You are already helping out on HBASE-18906, thanks. Looks like that will
>>>> be addressed by other alpha-4s about to land.
>>>>
>>>> St.Ack
>>>> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 10/23/17 12:53 PM, Stack wrote:
>>>>>
>>>>> (Reviving this thread)
>>>>>>
>>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>>>>>> refactor of the Coprocessor API shutting down access to internals
>>>>>> marked
>>>>>> InterfaceAudience.Private.
>>>>>>
>>>>>> The outstanding list is here:
>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>>>
>>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>>>
>>>>>> If issue, talk out loud on this thread. If you need a review to land an
>>>>>> item, shout on the issue and here; we'll help you out.
>>>>>>
>>>>>> As is, items are coming along nicely I'd say. We need to merge the
>>>>>> filter
>>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>>>
>>>>>> Post alpha-4, we'll have to hunt down our downstreamers and help them
>>>>>> test
>>>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>>>>>> downstreamers know what to expect (or we discover what we missed BEFORE
>>>>>> we
>>>>>> beta-1).
>>>>>>
>>>>>> Thanks for time,
>>>>>> S
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>>>>>
>>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
>>>>>>
>>>>>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>>>>>
>>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
>>>>>>> but
>>>>>>> it is plain that more time is needed (in spite of valiant effort so
>>>>>>> far
>>>>>>> by
>>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>>>>>>> theme is
>>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>>>>>>> following
>>>>>>> week.
>>>>>>>
>>>>>>> We should then be ready for beta (beta == no new features, no API
>>>>>>> changes,
>>>>>>> just fixes).
>>>>>>>
>>>>>>> Thanks,
>>>>>>> St.Ack
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>>>>>>>
>>>>>>>>
>>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>>>>>>>> release out in the next week or so.
>>>>>>>>
>>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>>>>>> interested in helping out, below are the items I think we need done
>>>>>>>> for
>>>>>>>> alpha3 (below are at least 'Critical' status, are API possibly
>>>>>>>> altering
>>>>>>>> items, and are absent those JIRAs that are making active progress,
>>>>>>>> i.e. the
>>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>>>>>>>> doing is
>>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>>>>
>>>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>>>>>>> branch-2
>>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
>>>>>>>> between branch-1 and branch-2.
>>>>>>>>
>>>>>>>> * HBASE-10462 Recategorize some of the client facing Public / Private
>>>>>>>> interfaces
>>>>>>>> This one is almost done. It could do with a finish, attention to the
>>>>>>>> items in last comment, and then our codebase could do with another
>>>>>>>> sweep
>>>>>>>> after the spirit of this issue since a bunch has gone in since the
>>>>>>>> pass
>>>>>>>> that was the basis of this issue.
>>>>>>>>
>>>>>>>> * HBASE-10504 Define Replication Interface
>>>>>>>> I was going to take a crack at this as part of the revamp forced by
>>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
>>>>>>>> but if
>>>>>>>> anyone else is interested, be my guest.
>>>>>>>>
>>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>>>>>> finishing.
>>>>>>>>
>>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>>>>>>>> cleanup
>>>>>>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
>>>>>>>> and
>>>>>>>> make sure symmetry.
>>>>>>>>
>>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
>>>>>>>> done.
>>>>>>>>
>>>>>>>> Rolling upgrade will have implications for our API. Would be good to
>>>>>>>> try
>>>>>>>> it and figure what needs fixup (as said above, according to trial by
>>>>>>>> Sean,
>>>>>>>> we might not be too bad here):
>>>>>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>>>>>>>> cluster
>>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
>>>>>>>> 1.x
>>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>>>>
>>>>>>>> * HBASE-17442 Move most of the replication related classes to
>>>>>>>> hbase-server package
>>>>>>>> The above would be good to do generally but it may make for ripples
>>>>>>>> in
>>>>>>>> API so would be good to do now.
>>>>>>>>
>>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>>>>> Balazs is working on this. The idea is that we avoid adding two new
>>>>>>>> types
>>>>>>>> to our API, two types that are nought but curtailed, read-only views
>>>>>>>> on
>>>>>>>> internals. Input if you have time appreciated.
>>>>>>>>
>>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
>>>>>>>> hbase2
>>>>>>>> cluster; verify
>>>>>>>> Esteban is looking at this one
>>>>>>>>
>>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>>>>>> * HBASE-17143 Scan improvement
>>>>>>>>
>>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>>>>> implementations
>>>>>>>> marked audience Private and returns implementations rather than
>>>>>>>> Interfaces.
>>>>>>>> In a few locations, we allow returning an alternate implementation
>>>>>>>> altogether which is probably something we don't want a CP doing. To
>>>>>>>> that
>>>>>>>> end, the following issues started by Duo and Anoop need to be taken
>>>>>>>> to
>>>>>>>> the
>>>>>>>> finish line; ideally they'd have an owner:
>>>>>>>>
>>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
>>>>>>>> umbrella issue.
>>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>>>>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>>>>>>>> RegionServerObserver
>>>>>>>>
>>>>>>>>
>>>>>>>> Nice-to-haves:
>>>>>>>>
>>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
>>>>>>>> unused
>>>>>>>> TimeRange constructors
>>>>>>>>
>>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>>>>>>> existing in the code
>>>>>>>> This is the end of an old long-running project moving up on to Cell
>>>>>>>> Interface. We think it is done but for a few little items (deprecate
>>>>>>>> KV
>>>>>>>> methods in MR and provide Cell versions instead...)
>>>>>>>>
>>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
>>>>>>>> fixing
>>>>>>>>
>>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>>>>
>>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>>>>> * HBASE-14997
>>>>>>>> Move compareOp and Comparators out of filter to client package
>>>>>>>>
>>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>>>>
>>>>>>>> What about:
>>>>>>>>
>>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>>>>
>>>>>>>> If interested in any of the above and you need a legup, just ask in
>>>>>>>> the
>>>>>>>> issue and I'll be by....
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> St.Ack
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>
>>>>>>>> Heads-up:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme
>>>>>>>>> is
>>>>>>>>> updated dependencies, reliance on relocated popular libs (guava,
>>>>>>>>> netty,
>>>>>>>>> protobuf), purge of checked-in generated src, and
>>>>>>>>> master-carries-no-regions
>>>>>>>>> by default.
>>>>>>>>>
>>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
>>>>>>>>> will
>>>>>>>>> be
>>>>>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>>>>>> looking too
>>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>>>>>>>>> running
>>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>>>>>>>>> should be
>>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>>>>> Implementations, and
>>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>>>>> InterfaceAudience).
>>>>>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
>>>>>>>>> 0.12
>>>>>>>>> Service showing through in our replication API and protobufs in
>>>>>>>>> Admin
>>>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach to make
>>>>>>>>> sure
>>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>>>>>
>>>>>>>>> Beta1 in mid-September?
>>>>>>>>>
>>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>>>>>>>> Edit
>>>>>>>>> as
>>>>>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>>>>>> attention
>>>>>>>>> in next month or so.
>>>>>>>>>
>>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave them as
>>>>>>>>> is
>>>>>>>>> another while but I'll boot them soon.
>>>>>>>>>
>>>>>>>>> While I have your attention:
>>>>>>>>>
>>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
>>>>>>>>> thrift
>>>>>>>>> to 0.10.0 will break existing clients. The change is easy enough if
>>>>>>>>> folks
>>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>>>>>> (0.98?).
>>>>>>>>>
>>>>>>>>> St.Ack
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> ...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>>>>>> feature a
>>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>>>>>>>>>>>>> reason
>>>>>>>>>>>>> to be
>>>>>>>>>>>>> nervous though). I am pretty happy with the feature in terms of
>>>>>>>>>>>>> how
>>>>>>>>>>>>> much it
>>>>>>>>>>>>> is covered via testing.
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Last think on space quota says that need doc too. See 'Space
>>>>>>>>>>>> Quota' in
>>>>>>>>>>>> here:
>>>>>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>>>>>> Does this little section need an update Josh?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> S
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
>>>>>>>>>>> docs
>>>>>>>>>>> already included for 17748.  Happy to put that on my plate if
>>>>>>>>>>> you're good
>>>>>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>>>>>
>>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>>>>>>> merged it in from the original feature branch. I'll double check
>>>>>>>>>>> and update
>>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks Josh,
>>>>>>>>>> S
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
> 

Re: Moving 2.0 forward

[Mike Drob <md...@apache.org>]

Josh - Do you want to kick off a bunch of QA runs? (Do you know how to do
it directly on the jenkins job, so you don't have to bother with JIRA
uploads)

If you're busy, then I can make time tomorrow or Sunday to kick off jobs,
but I want to make sure we're not duplicating effort and jenkins cycles.

On Fri, Oct 27, 2017 at 7:10 PM, Josh Elser <el...@apache.org> wrote:

> My turn to bump ;)
>
> By my take: HBASE-18770 and HBASE-19092 are the only issues that remain
> needing some more work. The rest are just awaiting a good QA run.
>
> Unless I hear otherwise, I'll try to keep an eye on things over the
> weekend, bump them along as necessary, and get them committed. Would be
> great to be able get a vote up on Monday.
>
>
> On 10/24/17 6:03 PM, Stack wrote:
>
>> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
>> what remains to be done. Surfacing our thoughts here so you all clued
>> in....Or if you think otherwise, please speak up.
>>
>> We have ~13 issues to land:
>> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About two
>> are meta-issues that are about process which leaves 11.
>>
>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some changed
>> API/semantic that we need to get out earlier rather than later.
>>
>> Once the above is merged, HBASE-13346, change of Filter method names to
>> mention Cell instead of KeyValue can land.
>>
>> HBASE-199048 needs a review (Anoop will probably do it), removing
>> IA.Private objects as params to MasterObserver... Hopefully this goes in
>> soon.
>>
>> Duo is hard at work on trackers for flush and compaction for CPs
>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>>
>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo
>> is
>> done w/ his work above.
>>
>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all
>> the
>> purges allowing CPs do direct calls against Regions in same Host.
>>
>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>
>> Another day or two?
>>
>> St.Ack
>>
>>
>>
>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>
>>
>>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>>>
>>> +1
>>>>
>>>> I was trying to work on helping out on the outstanding alpha-4 stuff
>>>> last
>>>> week -- will be continuing to try to do the same this week.
>>>>
>>>> If you need any help, Stack, or if others need reviews where I haven't
>>>> noticed on my own: feel free to @mention me.
>>>>
>>>>
>>>> Thanks for the offer Josh. All items seem assigned and are being
>>> actively
>>> worked on. If you get a moment, reviews by you (or anyone else) helps
>>> move
>>> the process along.
>>>
>>> We need to merge in HBASE-18410 branch to pick up Filter improvements.
>>> Then HBASE-13346 can go in.
>>>
>>> You are already helping out on HBASE-18906, thanks. Looks like that will
>>> be addressed by other alpha-4s about to land.
>>>
>>> St.Ack
>>> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 10/23/17 12:53 PM, Stack wrote:
>>>>
>>>> (Reviving this thread)
>>>>>
>>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>>>>> refactor of the Coprocessor API shutting down access to internals
>>>>> marked
>>>>> InterfaceAudience.Private.
>>>>>
>>>>> The outstanding list is here:
>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>>
>>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>>
>>>>> If issue, talk out loud on this thread. If you need a review to land an
>>>>> item, shout on the issue and here; we'll help you out.
>>>>>
>>>>> As is, items are coming along nicely I'd say. We need to merge the
>>>>> filter
>>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>>
>>>>> Post alpha-4, we'll have to hunt down our downstreamers and help them
>>>>> test
>>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>>>>> downstreamers know what to expect (or we discover what we missed BEFORE
>>>>> we
>>>>> beta-1).
>>>>>
>>>>> Thanks for time,
>>>>> S
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>>>>
>>>>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
>>>>>
>>>>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>>>>
>>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
>>>>>> but
>>>>>> it is plain that more time is needed (in spite of valiant effort so
>>>>>> far
>>>>>> by
>>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>>>>>> theme is
>>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>>>>>> following
>>>>>> week.
>>>>>>
>>>>>> We should then be ready for beta (beta == no new features, no API
>>>>>> changes,
>>>>>> just fixes).
>>>>>>
>>>>>> Thanks,
>>>>>> St.Ack
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>>>>
>>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>>>>>>
>>>>>>>
>>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>>>>>>> release out in the next week or so.
>>>>>>>
>>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>>>>> interested in helping out, below are the items I think we need done
>>>>>>> for
>>>>>>> alpha3 (below are at least 'Critical' status, are API possibly
>>>>>>> altering
>>>>>>> items, and are absent those JIRAs that are making active progress,
>>>>>>> i.e. the
>>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>>>>>>> doing is
>>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>>>
>>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>>>>>> branch-2
>>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
>>>>>>> between branch-1 and branch-2.
>>>>>>>
>>>>>>> * HBASE-10462 Recategorize some of the client facing Public / Private
>>>>>>> interfaces
>>>>>>> This one is almost done. It could do with a finish, attention to the
>>>>>>> items in last comment, and then our codebase could do with another
>>>>>>> sweep
>>>>>>> after the spirit of this issue since a bunch has gone in since the
>>>>>>> pass
>>>>>>> that was the basis of this issue.
>>>>>>>
>>>>>>> * HBASE-10504 Define Replication Interface
>>>>>>> I was going to take a crack at this as part of the revamp forced by
>>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
>>>>>>> but if
>>>>>>> anyone else is interested, be my guest.
>>>>>>>
>>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>>>>> finishing.
>>>>>>>
>>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>>>>>>> cleanup
>>>>>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
>>>>>>> and
>>>>>>> make sure symmetry.
>>>>>>>
>>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
>>>>>>> done.
>>>>>>>
>>>>>>> Rolling upgrade will have implications for our API. Would be good to
>>>>>>> try
>>>>>>> it and figure what needs fixup (as said above, according to trial by
>>>>>>> Sean,
>>>>>>> we might not be too bad here):
>>>>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>>>>>>> cluster
>>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
>>>>>>> 1.x
>>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>>>
>>>>>>> * HBASE-17442 Move most of the replication related classes to
>>>>>>> hbase-server package
>>>>>>> The above would be good to do generally but it may make for ripples
>>>>>>> in
>>>>>>> API so would be good to do now.
>>>>>>>
>>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>>>> Balazs is working on this. The idea is that we avoid adding two new
>>>>>>> types
>>>>>>> to our API, two types that are nought but curtailed, read-only views
>>>>>>> on
>>>>>>> internals. Input if you have time appreciated.
>>>>>>>
>>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
>>>>>>> hbase2
>>>>>>> cluster; verify
>>>>>>> Esteban is looking at this one
>>>>>>>
>>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>>>>> * HBASE-17143 Scan improvement
>>>>>>>
>>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>>>> implementations
>>>>>>> marked audience Private and returns implementations rather than
>>>>>>> Interfaces.
>>>>>>> In a few locations, we allow returning an alternate implementation
>>>>>>> altogether which is probably something we don't want a CP doing. To
>>>>>>> that
>>>>>>> end, the following issues started by Duo and Anoop need to be taken
>>>>>>> to
>>>>>>> the
>>>>>>> finish line; ideally they'd have an owner:
>>>>>>>
>>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
>>>>>>> umbrella issue.
>>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>>>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>>>>>>> RegionServerObserver
>>>>>>>
>>>>>>>
>>>>>>> Nice-to-haves:
>>>>>>>
>>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
>>>>>>> unused
>>>>>>> TimeRange constructors
>>>>>>>
>>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>>>>>> existing in the code
>>>>>>> This is the end of an old long-running project moving up on to Cell
>>>>>>> Interface. We think it is done but for a few little items (deprecate
>>>>>>> KV
>>>>>>> methods in MR and provide Cell versions instead...)
>>>>>>>
>>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
>>>>>>> fixing
>>>>>>>
>>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>>>
>>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>>>> * HBASE-14997
>>>>>>> Move compareOp and Comparators out of filter to client package
>>>>>>>
>>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>>>
>>>>>>> What about:
>>>>>>>
>>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>>>
>>>>>>> If interested in any of the above and you need a legup, just ask in
>>>>>>> the
>>>>>>> issue and I'll be by....
>>>>>>>
>>>>>>> Thanks,
>>>>>>> St.Ack
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>> Heads-up:
>>>>>>>
>>>>>>>>
>>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme
>>>>>>>> is
>>>>>>>> updated dependencies, reliance on relocated popular libs (guava,
>>>>>>>> netty,
>>>>>>>> protobuf), purge of checked-in generated src, and
>>>>>>>> master-carries-no-regions
>>>>>>>> by default.
>>>>>>>>
>>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
>>>>>>>> will
>>>>>>>> be
>>>>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>>>>> looking too
>>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>>>>>>>> running
>>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>>>>>>>> should be
>>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>>>> Implementations, and
>>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>>>> InterfaceAudience).
>>>>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
>>>>>>>> 0.12
>>>>>>>> Service showing through in our replication API and protobufs in
>>>>>>>> Admin
>>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach to make
>>>>>>>> sure
>>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>>>>
>>>>>>>> Beta1 in mid-September?
>>>>>>>>
>>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>>>>>>> Edit
>>>>>>>> as
>>>>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>>>>> attention
>>>>>>>> in next month or so.
>>>>>>>>
>>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave them as
>>>>>>>> is
>>>>>>>> another while but I'll boot them soon.
>>>>>>>>
>>>>>>>> While I have your attention:
>>>>>>>>
>>>>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
>>>>>>>> thrift
>>>>>>>> to 0.10.0 will break existing clients. The change is easy enough if
>>>>>>>> folks
>>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>>>>> (0.98?).
>>>>>>>>
>>>>>>>> St.Ack
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>>>>
>>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> ...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>>>>> feature a
>>>>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>>>>>>>>>>>> reason
>>>>>>>>>>>> to be
>>>>>>>>>>>> nervous though). I am pretty happy with the feature in terms of
>>>>>>>>>>>> how
>>>>>>>>>>>> much it
>>>>>>>>>>>> is covered via testing.
>>>>>>>>>>>>
>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Last think on space quota says that need doc too. See 'Space
>>>>>>>>>>> Quota' in
>>>>>>>>>>> here:
>>>>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>>>>> Does this little section need an update Josh?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> S
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
>>>>>>>>>> docs
>>>>>>>>>> already included for 17748.  Happy to put that on my plate if
>>>>>>>>>> you're good
>>>>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>>>>
>>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>>>>>> merged it in from the original feature branch. I'll double check
>>>>>>>>>> and update
>>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks Josh,
>>>>>>>>> S
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>>

Re: Moving 2.0 forward

[Josh Elser <el...@apache.org>]

My turn to bump ;)

By my take: HBASE-18770 and HBASE-19092 are the only issues that remain 
needing some more work. The rest are just awaiting a good QA run.

Unless I hear otherwise, I'll try to keep an eye on things over the 
weekend, bump them along as necessary, and get them committed. Would be 
great to be able get a vote up on Monday.

On 10/24/17 6:03 PM, Stack wrote:
> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
> what remains to be done. Surfacing our thoughts here so you all clued
> in....Or if you think otherwise, please speak up.
> 
> We have ~13 issues to land:
> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About two
> are meta-issues that are about process which leaves 11.
> 
> Duo and Zheng Hu are to merge the FilterList fixes improvements
> (HBASE-19057, HBASE-18410 et al.). These are blocker because some changed
> API/semantic that we need to get out earlier rather than later.
> 
> Once the above is merged, HBASE-13346, change of Filter method names to
> mention Cell instead of KeyValue can land.
> 
> HBASE-199048 needs a review (Anoop will probably do it), removing
> IA.Private objects as params to MasterObserver... Hopefully this goes in
> soon.
> 
> Duo is hard at work on trackers for flush and compaction for CPs
> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
> 
> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo is
> done w/ his work above.
> 
> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all the
> purges allowing CPs do direct calls against Regions in same Host.
> 
> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> 
> Another day or two?
> 
> St.Ack
> 
> 
> 
> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
> 
>>
>> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>>
>>> +1
>>>
>>> I was trying to work on helping out on the outstanding alpha-4 stuff last
>>> week -- will be continuing to try to do the same this week.
>>>
>>> If you need any help, Stack, or if others need reviews where I haven't
>>> noticed on my own: feel free to @mention me.
>>>
>>>
>> Thanks for the offer Josh. All items seem assigned and are being actively
>> worked on. If you get a moment, reviews by you (or anyone else) helps move
>> the process along.
>>
>> We need to merge in HBASE-18410 branch to pick up Filter improvements.
>> Then HBASE-13346 can go in.
>>
>> You are already helping out on HBASE-18906, thanks. Looks like that will
>> be addressed by other alpha-4s about to land.
>>
>> St.Ack
>> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>> On 10/23/17 12:53 PM, Stack wrote:
>>>
>>>> (Reviving this thread)
>>>>
>>>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>>>> refactor of the Coprocessor API shutting down access to internals marked
>>>> InterfaceAudience.Private.
>>>>
>>>> The outstanding list is here:
>>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>>
>>>> Please push in anything marked alpha-4 that belongs to you.
>>>>
>>>> If issue, talk out loud on this thread. If you need a review to land an
>>>> item, shout on the issue and here; we'll help you out.
>>>>
>>>> As is, items are coming along nicely I'd say. We need to merge the filter
>>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>>
>>>> Post alpha-4, we'll have to hunt down our downstreamers and help them
>>>> test
>>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>>>> downstreamers know what to expect (or we discover what we missed BEFORE
>>>> we
>>>> beta-1).
>>>>
>>>> Thanks for time,
>>>> S
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>>>
>>>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
>>>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>>>
>>>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3 but
>>>>> it is plain that more time is needed (in spite of valiant effort so far
>>>>> by
>>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>>>>> theme is
>>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the following
>>>>> week.
>>>>>
>>>>> We should then be ready for beta (beta == no new features, no API
>>>>> changes,
>>>>> just fixes).
>>>>>
>>>>> Thanks,
>>>>> St.Ack
>>>>>
>>>>>
>>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>>>
>>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>>>>>>
>>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>>>>>> release out in the next week or so.
>>>>>>
>>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>>>> interested in helping out, below are the items I think we need done for
>>>>>> alpha3 (below are at least 'Critical' status, are API possibly altering
>>>>>> items, and are absent those JIRAs that are making active progress,
>>>>>> i.e. the
>>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>>>>>> doing is
>>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>>
>>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>>>>> branch-2
>>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
>>>>>> between branch-1 and branch-2.
>>>>>>
>>>>>> * HBASE-10462 Recategorize some of the client facing Public / Private
>>>>>> interfaces
>>>>>> This one is almost done. It could do with a finish, attention to the
>>>>>> items in last comment, and then our codebase could do with another
>>>>>> sweep
>>>>>> after the spirit of this issue since a bunch has gone in since the pass
>>>>>> that was the basis of this issue.
>>>>>>
>>>>>> * HBASE-10504 Define Replication Interface
>>>>>> I was going to take a crack at this as part of the revamp forced by
>>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
>>>>>> but if
>>>>>> anyone else is interested, be my guest.
>>>>>>
>>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>>>> finishing.
>>>>>>
>>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>>>>>> cleanup
>>>>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
>>>>>> and
>>>>>> make sure symmetry.
>>>>>>
>>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
>>>>>> done.
>>>>>>
>>>>>> Rolling upgrade will have implications for our API. Would be good to
>>>>>> try
>>>>>> it and figure what needs fixup (as said above, according to trial by
>>>>>> Sean,
>>>>>> we might not be too bad here):
>>>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>>>>>> cluster
>>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
>>>>>> 1.x
>>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>>
>>>>>> * HBASE-17442 Move most of the replication related classes to
>>>>>> hbase-server package
>>>>>> The above would be good to do generally but it may make for ripples in
>>>>>> API so would be good to do now.
>>>>>>
>>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>>> Balazs is working on this. The idea is that we avoid adding two new
>>>>>> types
>>>>>> to our API, two types that are nought but curtailed, read-only views on
>>>>>> internals. Input if you have time appreciated.
>>>>>>
>>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
>>>>>> cluster; verify
>>>>>> Esteban is looking at this one
>>>>>>
>>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>>>> * HBASE-17143 Scan improvement
>>>>>>
>>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>>> implementations
>>>>>> marked audience Private and returns implementations rather than
>>>>>> Interfaces.
>>>>>> In a few locations, we allow returning an alternate implementation
>>>>>> altogether which is probably something we don't want a CP doing. To
>>>>>> that
>>>>>> end, the following issues started by Duo and Anoop need to be taken to
>>>>>> the
>>>>>> finish line; ideally they'd have an owner:
>>>>>>
>>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
>>>>>> umbrella issue.
>>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>>>>>> RegionServerObserver
>>>>>>
>>>>>>
>>>>>> Nice-to-haves:
>>>>>>
>>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
>>>>>> TimeRange constructors
>>>>>>
>>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>>>>> existing in the code
>>>>>> This is the end of an old long-running project moving up on to Cell
>>>>>> Interface. We think it is done but for a few little items (deprecate KV
>>>>>> methods in MR and provide Cell versions instead...)
>>>>>>
>>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
>>>>>> fixing
>>>>>>
>>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>>
>>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>>> * HBASE-14997
>>>>>> Move compareOp and Comparators out of filter to client package
>>>>>>
>>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>>
>>>>>> What about:
>>>>>>
>>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>>
>>>>>> If interested in any of the above and you need a legup, just ask in the
>>>>>> issue and I'll be by....
>>>>>>
>>>>>> Thanks,
>>>>>> St.Ack
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>>>>
>>>>>> Heads-up:
>>>>>>>
>>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
>>>>>>> updated dependencies, reliance on relocated popular libs (guava,
>>>>>>> netty,
>>>>>>> protobuf), purge of checked-in generated src, and
>>>>>>> master-carries-no-regions
>>>>>>> by default.
>>>>>>>
>>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme will
>>>>>>> be
>>>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>>>> looking too
>>>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>>>>>>> running
>>>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>>>>>>> should be
>>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>>> Implementations, and
>>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>>> InterfaceAudience).
>>>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
>>>>>>> 0.12
>>>>>>> Service showing through in our replication API and protobufs in Admin
>>>>>>> Interface. On alpha3, we will have to do a bunch of outreach to make
>>>>>>> sure
>>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>>>
>>>>>>> Beta1 in mid-September?
>>>>>>>
>>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit
>>>>>>> as
>>>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>>>> attention
>>>>>>> in next month or so.
>>>>>>>
>>>>>>> A bunch of issues marked blocker are unassigned. I'll leave them as is
>>>>>>> another while but I'll boot them soon.
>>>>>>>
>>>>>>> While I have your attention:
>>>>>>>
>>>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase thrift
>>>>>>> to 0.10.0 will break existing clients. The change is easy enough if
>>>>>>> folks
>>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>>>> (0.98?).
>>>>>>>
>>>>>>> St.Ack
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>>>
>>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>>>> feature a
>>>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>>>>>>>>>>> reason
>>>>>>>>>>> to be
>>>>>>>>>>> nervous though). I am pretty happy with the feature in terms of
>>>>>>>>>>> how
>>>>>>>>>>> much it
>>>>>>>>>>> is covered via testing.
>>>>>>>>>>>
>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Last think on space quota says that need doc too. See 'Space
>>>>>>>>>> Quota' in
>>>>>>>>>> here:
>>>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>>>> Does this little section need an update Josh?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> S
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>>>>>>>>> already included for 17748.  Happy to put that on my plate if
>>>>>>>>> you're good
>>>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>>>
>>>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>>>>> merged it in from the original feature branch. I'll double check
>>>>>>>>> and update
>>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Thanks Josh,
>>>>>>>> S
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
> 

Re: Moving 2.0 forward

[Guanghao Zhang <zg...@gmail.com>]

>
> For public API we have rules on how to keep compatible so I think it is
> less hurt for users, beta1 is fine.
>
Got it. Thanks for you explanation.

When you think you'd be done w/ the sync of Admin and AsyncAdmin
> Interfaces? Are the changes to Admin or to AsyncAdmin?

There are two issues (HBASE-19009 and HBASE-18950) need more review.  After
resolve these issues, I will upload a new patch for HBASE-18911. Thanks.

2017-10-27 0:14 GMT+08:00 Stack <st...@duboce.net>:

> On Thu, Oct 26, 2017 at 2:27 AM, Guanghao Zhang <zg...@gmail.com>
> wrote:
>
> > I saw beta == no new features, no API changes, just fixes. And I am
> working
> > on HBASE-18805 to unify Admin and AsyncAdmin methods. The fix version was
> > 2.0-beta-1. But I thought this will introduce API change(deprecate some
> API
> > and introduce new one). So should I change the fix versions to
> 2.0-alpha-4
> > and finish it before we release 2.0-alpha-4?
> >
> > Issue HBASE-18978 (Align the methods in Table and AsyncTable) may have
> same
> > problem. Thanks.
> >
> >
> Yeah, we were supposed to be done w/ public API changes when we shipped
> alpha-3 (smile).
>
> For Admin Interface, semantic versioning applies as per Duo comment. I see
> some polish still being done on the Admin/Table APIs (removal of deprecated
> methods, alignment of Admin and AsyncAdmin APIs). Hard to say no to this
> clean up on our most public facing Interfaces.
>
> When you think you'd be done w/ the sync of Admin and AsyncAdmin
> Interfaces? Are the changes to Admin or to AsyncAdmin?
>
> Thank you Guanghao Zhang,
>
> St.Ack
>
>
> > 2017-10-26 9:51 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
> >
> > > OK, skimmed,  we are in trouble! The in memory compaction just use the
> > same
> > > constructor with normal compaction to construct a StoreScanner, and use
> > it
> > > to do compaction...
> > >
> > > We have to provide several preXXX and postXXX for it, at least we
> should
> > > allow user reset TTL and max versions, and also do filtering on the
> > > scanner.
> > >
> > > Thanks.
> > >
> > >
> > >
> > > 2017-10-26 9:41 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
> > >
> > > > When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
> > > > HBASE-19033, I found that there maybe a hole in our CP hools. It is
> the
> > > in
> > > > memory compaction.
> > > >
> > > > As Anoop suggested in HBASE-19001, we still need to give user the
> > ability
> > > > to extend the max versions and TTL config, so I plan to add back the
> > > hooks
> > > > above to let CP users can change the max versions and TTL of a
> ScanInfo
> > > > object. But I'm not sure whether in memory compaction will also
> discard
> > > > expired cells, if so then we are in trouble...
> > > >
> > > > Thanks.
> > > >
> > > > 2017-10-25 6:03 GMT+08:00 Stack <st...@duboce.net>:
> > > >
> > > >> Chatting with my coworker Mr. Mike Drob, we were batting back and
> > forth
> > > >> what remains to be done. Surfacing our thoughts here so you all
> clued
> > > >> in....Or if you think otherwise, please speak up.
> > > >>
> > > >> We have ~13 issues to land:
> > > >> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> About
> > > two
> > > >> are meta-issues that are about process which leaves 11.
> > > >>
> > > >> Duo and Zheng Hu are to merge the FilterList fixes improvements
> > > >> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
> > > changed
> > > >> API/semantic that we need to get out earlier rather than later.
> > > >>
> > > >> Once the above is merged, HBASE-13346, change of Filter method names
> > to
> > > >> mention Cell instead of KeyValue can land.
> > > >>
> > > >> HBASE-199048 needs a review (Anoop will probably do it), removing
> > > >> IA.Private objects as params to MasterObserver... Hopefully this
> goes
> > in
> > > >> soon.
> > > >>
> > > >> Duo is hard at work on trackers for flush and compaction for CPs
> > > >> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
> > > >>
> > > >> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after
> > Duo
> > > >> is
> > > >> done w/ his work above.
> > > >>
> > > >> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after
> > all
> > > >> the
> > > >> purges allowing CPs do direct calls against Regions in same Host.
> > > >>
> > > >> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> > > >>
> > > >> Another day or two?
> > > >>
> > > >> St.Ack
> > > >>
> > > >>
> > > >>
> > > >> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
> > > >>
> > > >> >
> > > >> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
> > > wrote:
> > > >> >
> > > >> >> +1
> > > >> >>
> > > >> >> I was trying to work on helping out on the outstanding alpha-4
> > stuff
> > > >> last
> > > >> >> week -- will be continuing to try to do the same this week.
> > > >> >>
> > > >> >> If you need any help, Stack, or if others need reviews where I
> > > haven't
> > > >> >> noticed on my own: feel free to @mention me.
> > > >> >>
> > > >> >>
> > > >> > Thanks for the offer Josh. All items seem assigned and are being
> > > >> actively
> > > >> > worked on. If you get a moment, reviews by you (or anyone else)
> > helps
> > > >> move
> > > >> > the process along.
> > > >> >
> > > >> > We need to merge in HBASE-18410 branch to pick up Filter
> > improvements.
> > > >> > Then HBASE-13346 can go in.
> > > >> >
> > > >> > You are already helping out on HBASE-18906, thanks. Looks like
> that
> > > will
> > > >> > be addressed by other alpha-4s about to land.
> > > >> >
> > > >> > St.Ack
> > > >> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/
> > > 12341594
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >> On 10/23/17 12:53 PM, Stack wrote:
> > > >> >>
> > > >> >>> (Reviving this thread)
> > > >> >>>
> > > >> >>> Lets push out alpha-4 this week. Alpha-4 is the release that has
> > the
> > > >> >>> refactor of the Coprocessor API shutting down access to
> internals
> > > >> marked
> > > >> >>> InterfaceAudience.Private.
> > > >> >>>
> > > >> >>> The outstanding list is here:
> > > >> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > > >> >>>
> > > >> >>> Please push in anything marked alpha-4 that belongs to you.
> > > >> >>>
> > > >> >>> If issue, talk out loud on this thread. If you need a review to
> > land
> > > >> an
> > > >> >>> item, shout on the issue and here; we'll help you out.
> > > >> >>>
> > > >> >>> As is, items are coming along nicely I'd say. We need to merge
> the
> > > >> filter
> > > >> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> > > >> >>>
> > > >> >>> Post alpha-4, we'll have to hunt down our downstreamers and help
> > > them
> > > >> >>> test
> > > >> >>> on top of alpha-4 so rolling into beta-1, we have confidence our
> > > >> >>> downstreamers know what to expect (or we discover what we missed
> > > >> BEFORE
> > > >> >>> we
> > > >> >>> beta-1).
> > > >> >>>
> > > >> >>> Thanks for time,
> > > >> >>> S
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> > > >> >>>
> > > >> >>> I'll put up an alpha3 RC Monday, probably Monday night. That
> > should
> > > be
> > > >> >>>> time, if we all sprint, for the public-facing API fixes to be
> > done.
> > > >> >>>>
> > > >> >>>> I had a bunch of Coprocessor refactor and fixup scheduled for
> > > alpha3
> > > >> but
> > > >> >>>> it is plain that more time is needed (in spite of valiant
> effort
> > so
> > > >> far
> > > >> >>>> by
> > > >> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
> > whose
> > > >> >>>> theme is
> > > >> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> > > >> following
> > > >> >>>> week.
> > > >> >>>>
> > > >> >>>> We should then be ready for beta (beta == no new features, no
> API
> > > >> >>>> changes,
> > > >> >>>> just fixes).
> > > >> >>>>
> > > >> >>>> Thanks,
> > > >> >>>> St.Ack
> > > >> >>>>
> > > >> >>>>
> > > >> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
> > wrote:
> > > >> >>>>
> > > >> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote
> on
> > > it.
> > > >> >>>>>
> > > >> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope
> to
> > > get
> > > >> a
> > > >> >>>>> release out in the next week or so.
> > > >> >>>>>
> > > >> >>>>> I did a weeding of 2.0.0 issues over the last day. If folks
> are
> > > >> >>>>> interested in helping out, below are the items I think we need
> > > done
> > > >> for
> > > >> >>>>> alpha3 (below are at least 'Critical' status, are API possibly
> > > >> altering
> > > >> >>>>> items, and are absent those JIRAs that are making active
> > progress,
> > > >> >>>>> i.e. the
> > > >> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
> > needs
> > > >> >>>>> doing is
> > > >> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
> > > >> >>>>>
> > > >> >>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1
> > and
> > > >> >>>>> branch-2
> > > >> >>>>> This is to do what Andrew did between 1.3 and 1.4 branches
> only
> > do
> > > >> it
> > > >> >>>>> between branch-1 and branch-2.
> > > >> >>>>>
> > > >> >>>>> * HBASE-10462 Recategorize some of the client facing Public /
> > > >> Private
> > > >> >>>>> interfaces
> > > >> >>>>> This one is almost done. It could do with a finish, attention
> to
> > > the
> > > >> >>>>> items in last comment, and then our codebase could do with
> > another
> > > >> >>>>> sweep
> > > >> >>>>> after the spirit of this issue since a bunch has gone in since
> > the
> > > >> pass
> > > >> >>>>> that was the basis of this issue.
> > > >> >>>>>
> > > >> >>>>> * HBASE-10504 Define Replication Interface
> > > >> >>>>> I was going to take a crack at this as part of the revamp
> forced
> > > by
> > > >> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> > > Service'
> > > >> >>>>> but if
> > > >> >>>>> anyone else is interested, be my guest.
> > > >> >>>>>
> > > >> >>>>> * HBASE-14996 Some more API cleanup for 2.0
> > > >> >>>>> Has a bunch of subtasks, some of which are being worked on.
> > Needs
> > > >> >>>>> finishing.
> > > >> >>>>>
> > > >> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
> > Admin
> > > >> and
> > > >> >>>>> cleanup
> > > >> >>>>> Needs a pass. Small issue I think. Could also look at new
> > > >> AsyncClient
> > > >> >>>>> and
> > > >> >>>>> make sure symmetry.
> > > >> >>>>>
> > > >> >>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> > > >> >>>>> Predicated on result of an ongoing DISCUSSION thread but needs
> > to
> > > be
> > > >> >>>>> done.
> > > >> >>>>>
> > > >> >>>>> Rolling upgrade will have implications for our API. Would be
> > good
> > > to
> > > >> >>>>> try
> > > >> >>>>> it and figure what needs fixup (as said above, according to
> > trial
> > > by
> > > >> >>>>> Sean,
> > > >> >>>>> we might not be too bad here):
> > > >> >>>>> * HBASE-16060 1.x clients cannot access table state talking to
> > 2.0
> > > >> >>>>> cluster
> > > >> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x
> Master
> > > and
> > > >> >>>>> 1.x
> > > >> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> > > >> >>>>>
> > > >> >>>>> * HBASE-17442 Move most of the replication related classes to
> > > >> >>>>> hbase-server package
> > > >> >>>>> The above would be good to do generally but it may make for
> > > ripples
> > > >> in
> > > >> >>>>> API so would be good to do now.
> > > >> >>>>>
> > > >> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> > > >> >>>>> Balazs is working on this. The idea is that we avoid adding
> two
> > > new
> > > >> >>>>> types
> > > >> >>>>> to our API, two types that are nought but curtailed, read-only
> > > >> views on
> > > >> >>>>> internals. Input if you have time appreciated.
> > > >> >>>>>
> > > >> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to
> a
> > > >> hbase2
> > > >> >>>>> cluster; verify
> > > >> >>>>> Esteban is looking at this one
> > > >> >>>>>
> > > >> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> > > >> >>>>> * HBASE-17143 Scan improvement
> > > >> >>>>>
> > > >> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
> > > >> >>>>> implementations
> > > >> >>>>> marked audience Private and returns implementations rather
> than
> > > >> >>>>> Interfaces.
> > > >> >>>>> In a few locations, we allow returning an alternate
> > implementation
> > > >> >>>>> altogether which is probably something we don't want a CP
> doing.
> > > To
> > > >> >>>>> that
> > > >> >>>>> end, the following issues started by Duo and Anoop need to be
> > > taken
> > > >> to
> > > >> >>>>> the
> > > >> >>>>> finish line; ideally they'd have an owner:
> > > >> >>>>>
> > > >> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release
> > <=
> > > >> The
> > > >> >>>>> umbrella issue.
> > > >> >>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
> > expose
> > > >> >>>>> * HBASE-16769 Deprecate/remove PB references from
> MasterObserver
> > > and
> > > >> >>>>> RegionServerObserver
> > > >> >>>>>
> > > >> >>>>>
> > > >> >>>>> Nice-to-haves:
> > > >> >>>>>
> > > >> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private and
> remove
> > > >> unused
> > > >> >>>>> TimeRange constructors
> > > >> >>>>>
> > > >> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
> > references
> > > >> >>>>> existing in the code
> > > >> >>>>> This is the end of an old long-running project moving up on to
> > > Cell
> > > >> >>>>> Interface. We think it is done but for a few little items
> > > >> (deprecate KV
> > > >> >>>>> methods in MR and provide Cell versions instead...)
> > > >> >>>>>
> > > >> >>>>> * HBASE-13271 Table#puts(List<Put>) operation is
> indeterminate;
> > > >> needs
> > > >> >>>>> fixing
> > > >> >>>>>
> > > >> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
> > > >> >>>>>
> > > >> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
> > > >> >>>>> * HBASE-14997
> > > >> >>>>> Move compareOp and Comparators out of filter to client package
> > > >> >>>>>
> > > >> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
> > > >> >>>>>
> > > >> >>>>> What about:
> > > >> >>>>>
> > > >> >>>>> * HBASE-18601 Remove Htrace 3.2
> > > >> >>>>> As has been noted, the HTrace API is our 'trace' API.
> > > >> >>>>>
> > > >> >>>>> If interested in any of the above and you need a legup, just
> ask
> > > in
> > > >> the
> > > >> >>>>> issue and I'll be by....
> > > >> >>>>>
> > > >> >>>>> Thanks,
> > > >> >>>>> St.Ack
> > > >> >>>>>
> > > >> >>>>>
> > > >> >>>>>
> > > >> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
> > wrote:
> > > >> >>>>>
> > > >> >>>>> Heads-up:
> > > >> >>>>>>
> > > >> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
> > > Theme
> > > >> is
> > > >> >>>>>> updated dependencies, reliance on relocated popular libs
> > (guava,
> > > >> >>>>>> netty,
> > > >> >>>>>> protobuf), purge of checked-in generated src, and
> > > >> >>>>>> master-carries-no-regions
> > > >> >>>>>> by default.
> > > >> >>>>>>
> > > >> >>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
> > theme
> > > >> will
> > > >> >>>>>> be
> > > >> >>>>>> settling the APIs and compatibility (At first blush, we are
> not
> > > >> >>>>>> looking too
> > > >> >>>>>> bad; our Sean ran some tests over weekend that have hbase-1
> > > client
> > > >> >>>>>> running
> > > >> >>>>>> against an hbase-2 cluster....). The Coprocessor Interface
> > revamp
> > > >> >>>>>> should be
> > > >> >>>>>> done by alpha3 (i.e. returning Interfaces rather than
> > > >> >>>>>> Implementations, and
> > > >> >>>>>> our shutdown of CPs accessing classes in hbase marked
> > > >> >>>>>> InterfaceAudience).
> > > >> >>>>>> We'll also have purged thirdparty classes from our API; e.g.
> > > guava
> > > >> >>>>>> 0.12
> > > >> >>>>>> Service showing through in our replication API and protobufs
> in
> > > >> Admin
> > > >> >>>>>> Interface. On alpha3, we will have to do a bunch of outreach
> to
> > > >> make
> > > >> >>>>>> sure
> > > >> >>>>>> our downstreamers are up on what is coming down the pipe.
> > > >> >>>>>>
> > > >> >>>>>> Beta1 in mid-September?
> > > >> >>>>>>
> > > >> >>>>>> I encourage you to check out the items marked for hbase2:
> > > >> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> > 12327188
> > > >> Edit
> > > >> >>>>>> as
> > > >> >>>>>> you see appropriate. Punt if you know the JIRA will not get
> any
> > > >> >>>>>> attention
> > > >> >>>>>> in next month or so.
> > > >> >>>>>>
> > > >> >>>>>> A bunch of issues marked blocker are unassigned. I'll leave
> > them
> > > >> as is
> > > >> >>>>>> another while but I'll boot them soon.
> > > >> >>>>>>
> > > >> >>>>>> While I have your attention:
> > > >> >>>>>>
> > > >> >>>>>> + I think we should leave thrift version at 0.9.3. Moving
> hbase
> > > >> thrift
> > > >> >>>>>> to 0.10.0 will break existing clients. The change is easy
> > enough
> > > if
> > > >> >>>>>> folks
> > > >> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> > > >> >>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0
> first
> > > >> >>>>>> (0.98?).
> > > >> >>>>>>
> > > >> >>>>>> St.Ack
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
> > wrote:
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <
> elserj@apache.org
> > >
> > > >> >>>>>>> wrote:
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> > > >> >>>>>>>>
> > > >> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
> > elserj@apache.org
> > > >
> > > >> >>>>>>>>> wrote:
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> ...
> > > >> >>>>>>>>>
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> I like the idea of this also hitting 2.0 as it would make
> > the
> > > >> >>>>>>>>>> feature a
> > > >> >>>>>>>>>> bit more "real", but am obviously a little nervous (I
> have
> > no
> > > >> >>>>>>>>>> reason
> > > >> >>>>>>>>>> to be
> > > >> >>>>>>>>>> nervous though). I am pretty happy with the feature in
> > terms
> > > of
> > > >> >>>>>>>>>> how
> > > >> >>>>>>>>>> much it
> > > >> >>>>>>>>>> is covered via testing.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
> > Josh.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> Last think on space quota says that need doc too. See
> 'Space
> > > >> >>>>>>>>> Quota' in
> > > >> >>>>>>>>> here:
> > > >> >>>>>>>>> https://docs.google.com/document/d/
> > 1WCsVlnHjJeKUcl7wHwqb4z9i
> > > >> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > > >> >>>>>>>>> Does this little section need an update Josh?
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> Thanks,
> > > >> >>>>>>>>> S
> > > >> >>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and
> > some
> > > >> docs
> > > >> >>>>>>>> already included for 17748.  Happy to put that on my plate
> if
> > > >> >>>>>>>> you're good
> > > >> >>>>>>>> with it. I can reasonably assume that no one is against it
> :)
> > > >> >>>>>>>>
> > > >> >>>>>>>> I think I had knocked out docs for the "phase 1" stuff
> before
> > > we
> > > >> >>>>>>>> merged it in from the original feature branch. I'll double
> > > check
> > > >> >>>>>>>> and update
> > > >> >>>>>>>> the gdoc. Perhaps this was just a timing thing.
> > > >> >>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>>>> Thanks Josh,
> > > >> >>>>>>> S
> > > >> >>>>>>>
> > > >> >>>>>>>
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Thu, Oct 26, 2017 at 2:27 AM, Guanghao Zhang <zg...@gmail.com> wrote:

> I saw beta == no new features, no API changes, just fixes. And I am working
> on HBASE-18805 to unify Admin and AsyncAdmin methods. The fix version was
> 2.0-beta-1. But I thought this will introduce API change(deprecate some API
> and introduce new one). So should I change the fix versions to 2.0-alpha-4
> and finish it before we release 2.0-alpha-4?
>
> Issue HBASE-18978 (Align the methods in Table and AsyncTable) may have same
> problem. Thanks.
>
>
Yeah, we were supposed to be done w/ public API changes when we shipped
alpha-3 (smile).

For Admin Interface, semantic versioning applies as per Duo comment. I see
some polish still being done on the Admin/Table APIs (removal of deprecated
methods, alignment of Admin and AsyncAdmin APIs). Hard to say no to this
clean up on our most public facing Interfaces.

When you think you'd be done w/ the sync of Admin and AsyncAdmin
Interfaces? Are the changes to Admin or to AsyncAdmin?

Thank you Guanghao Zhang,

St.Ack


> 2017-10-26 9:51 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
>
> > OK, skimmed,  we are in trouble! The in memory compaction just use the
> same
> > constructor with normal compaction to construct a StoreScanner, and use
> it
> > to do compaction...
> >
> > We have to provide several preXXX and postXXX for it, at least we should
> > allow user reset TTL and max versions, and also do filtering on the
> > scanner.
> >
> > Thanks.
> >
> >
> >
> > 2017-10-26 9:41 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
> >
> > > When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
> > > HBASE-19033, I found that there maybe a hole in our CP hools. It is the
> > in
> > > memory compaction.
> > >
> > > As Anoop suggested in HBASE-19001, we still need to give user the
> ability
> > > to extend the max versions and TTL config, so I plan to add back the
> > hooks
> > > above to let CP users can change the max versions and TTL of a ScanInfo
> > > object. But I'm not sure whether in memory compaction will also discard
> > > expired cells, if so then we are in trouble...
> > >
> > > Thanks.
> > >
> > > 2017-10-25 6:03 GMT+08:00 Stack <st...@duboce.net>:
> > >
> > >> Chatting with my coworker Mr. Mike Drob, we were batting back and
> forth
> > >> what remains to be done. Surfacing our thoughts here so you all clued
> > >> in....Or if you think otherwise, please speak up.
> > >>
> > >> We have ~13 issues to land:
> > >> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About
> > two
> > >> are meta-issues that are about process which leaves 11.
> > >>
> > >> Duo and Zheng Hu are to merge the FilterList fixes improvements
> > >> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
> > changed
> > >> API/semantic that we need to get out earlier rather than later.
> > >>
> > >> Once the above is merged, HBASE-13346, change of Filter method names
> to
> > >> mention Cell instead of KeyValue can land.
> > >>
> > >> HBASE-199048 needs a review (Anoop will probably do it), removing
> > >> IA.Private objects as params to MasterObserver... Hopefully this goes
> in
> > >> soon.
> > >>
> > >> Duo is hard at work on trackers for flush and compaction for CPs
> > >> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
> > >>
> > >> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after
> Duo
> > >> is
> > >> done w/ his work above.
> > >>
> > >> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after
> all
> > >> the
> > >> purges allowing CPs do direct calls against Regions in same Host.
> > >>
> > >> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> > >>
> > >> Another day or two?
> > >>
> > >> St.Ack
> > >>
> > >>
> > >>
> > >> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
> > >>
> > >> >
> > >> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
> > wrote:
> > >> >
> > >> >> +1
> > >> >>
> > >> >> I was trying to work on helping out on the outstanding alpha-4
> stuff
> > >> last
> > >> >> week -- will be continuing to try to do the same this week.
> > >> >>
> > >> >> If you need any help, Stack, or if others need reviews where I
> > haven't
> > >> >> noticed on my own: feel free to @mention me.
> > >> >>
> > >> >>
> > >> > Thanks for the offer Josh. All items seem assigned and are being
> > >> actively
> > >> > worked on. If you get a moment, reviews by you (or anyone else)
> helps
> > >> move
> > >> > the process along.
> > >> >
> > >> > We need to merge in HBASE-18410 branch to pick up Filter
> improvements.
> > >> > Then HBASE-13346 can go in.
> > >> >
> > >> > You are already helping out on HBASE-18906, thanks. Looks like that
> > will
> > >> > be addressed by other alpha-4s about to land.
> > >> >
> > >> > St.Ack
> > >> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/
> > 12341594
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >> On 10/23/17 12:53 PM, Stack wrote:
> > >> >>
> > >> >>> (Reviving this thread)
> > >> >>>
> > >> >>> Lets push out alpha-4 this week. Alpha-4 is the release that has
> the
> > >> >>> refactor of the Coprocessor API shutting down access to internals
> > >> marked
> > >> >>> InterfaceAudience.Private.
> > >> >>>
> > >> >>> The outstanding list is here:
> > >> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > >> >>>
> > >> >>> Please push in anything marked alpha-4 that belongs to you.
> > >> >>>
> > >> >>> If issue, talk out loud on this thread. If you need a review to
> land
> > >> an
> > >> >>> item, shout on the issue and here; we'll help you out.
> > >> >>>
> > >> >>> As is, items are coming along nicely I'd say. We need to merge the
> > >> filter
> > >> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> > >> >>>
> > >> >>> Post alpha-4, we'll have to hunt down our downstreamers and help
> > them
> > >> >>> test
> > >> >>> on top of alpha-4 so rolling into beta-1, we have confidence our
> > >> >>> downstreamers know what to expect (or we discover what we missed
> > >> BEFORE
> > >> >>> we
> > >> >>> beta-1).
> > >> >>>
> > >> >>> Thanks for time,
> > >> >>> S
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> > >> >>>
> > >> >>> I'll put up an alpha3 RC Monday, probably Monday night. That
> should
> > be
> > >> >>>> time, if we all sprint, for the public-facing API fixes to be
> done.
> > >> >>>>
> > >> >>>> I had a bunch of Coprocessor refactor and fixup scheduled for
> > alpha3
> > >> but
> > >> >>>> it is plain that more time is needed (in spite of valiant effort
> so
> > >> far
> > >> >>>> by
> > >> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
> whose
> > >> >>>> theme is
> > >> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> > >> following
> > >> >>>> week.
> > >> >>>>
> > >> >>>> We should then be ready for beta (beta == no new features, no API
> > >> >>>> changes,
> > >> >>>> just fixes).
> > >> >>>>
> > >> >>>> Thanks,
> > >> >>>> St.Ack
> > >> >>>>
> > >> >>>>
> > >> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
> wrote:
> > >> >>>>
> > >> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on
> > it.
> > >> >>>>>
> > >> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
> > get
> > >> a
> > >> >>>>> release out in the next week or so.
> > >> >>>>>
> > >> >>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> > >> >>>>> interested in helping out, below are the items I think we need
> > done
> > >> for
> > >> >>>>> alpha3 (below are at least 'Critical' status, are API possibly
> > >> altering
> > >> >>>>> items, and are absent those JIRAs that are making active
> progress,
> > >> >>>>> i.e. the
> > >> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
> needs
> > >> >>>>> doing is
> > >> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
> > >> >>>>>
> > >> >>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1
> and
> > >> >>>>> branch-2
> > >> >>>>> This is to do what Andrew did between 1.3 and 1.4 branches only
> do
> > >> it
> > >> >>>>> between branch-1 and branch-2.
> > >> >>>>>
> > >> >>>>> * HBASE-10462 Recategorize some of the client facing Public /
> > >> Private
> > >> >>>>> interfaces
> > >> >>>>> This one is almost done. It could do with a finish, attention to
> > the
> > >> >>>>> items in last comment, and then our codebase could do with
> another
> > >> >>>>> sweep
> > >> >>>>> after the spirit of this issue since a bunch has gone in since
> the
> > >> pass
> > >> >>>>> that was the basis of this issue.
> > >> >>>>>
> > >> >>>>> * HBASE-10504 Define Replication Interface
> > >> >>>>> I was going to take a crack at this as part of the revamp forced
> > by
> > >> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> > Service'
> > >> >>>>> but if
> > >> >>>>> anyone else is interested, be my guest.
> > >> >>>>>
> > >> >>>>> * HBASE-14996 Some more API cleanup for 2.0
> > >> >>>>> Has a bunch of subtasks, some of which are being worked on.
> Needs
> > >> >>>>> finishing.
> > >> >>>>>
> > >> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
> Admin
> > >> and
> > >> >>>>> cleanup
> > >> >>>>> Needs a pass. Small issue I think. Could also look at new
> > >> AsyncClient
> > >> >>>>> and
> > >> >>>>> make sure symmetry.
> > >> >>>>>
> > >> >>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> > >> >>>>> Predicated on result of an ongoing DISCUSSION thread but needs
> to
> > be
> > >> >>>>> done.
> > >> >>>>>
> > >> >>>>> Rolling upgrade will have implications for our API. Would be
> good
> > to
> > >> >>>>> try
> > >> >>>>> it and figure what needs fixup (as said above, according to
> trial
> > by
> > >> >>>>> Sean,
> > >> >>>>> we might not be too bad here):
> > >> >>>>> * HBASE-16060 1.x clients cannot access table state talking to
> 2.0
> > >> >>>>> cluster
> > >> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master
> > and
> > >> >>>>> 1.x
> > >> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> > >> >>>>>
> > >> >>>>> * HBASE-17442 Move most of the replication related classes to
> > >> >>>>> hbase-server package
> > >> >>>>> The above would be good to do generally but it may make for
> > ripples
> > >> in
> > >> >>>>> API so would be good to do now.
> > >> >>>>>
> > >> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> > >> >>>>> Balazs is working on this. The idea is that we avoid adding two
> > new
> > >> >>>>> types
> > >> >>>>> to our API, two types that are nought but curtailed, read-only
> > >> views on
> > >> >>>>> internals. Input if you have time appreciated.
> > >> >>>>>
> > >> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
> > >> hbase2
> > >> >>>>> cluster; verify
> > >> >>>>> Esteban is looking at this one
> > >> >>>>>
> > >> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> > >> >>>>> * HBASE-17143 Scan improvement
> > >> >>>>>
> > >> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
> > >> >>>>> implementations
> > >> >>>>> marked audience Private and returns implementations rather than
> > >> >>>>> Interfaces.
> > >> >>>>> In a few locations, we allow returning an alternate
> implementation
> > >> >>>>> altogether which is probably something we don't want a CP doing.
> > To
> > >> >>>>> that
> > >> >>>>> end, the following issues started by Duo and Anoop need to be
> > taken
> > >> to
> > >> >>>>> the
> > >> >>>>> finish line; ideally they'd have an owner:
> > >> >>>>>
> > >> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release
> <=
> > >> The
> > >> >>>>> umbrella issue.
> > >> >>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
> expose
> > >> >>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver
> > and
> > >> >>>>> RegionServerObserver
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> Nice-to-haves:
> > >> >>>>>
> > >> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
> > >> unused
> > >> >>>>> TimeRange constructors
> > >> >>>>>
> > >> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
> references
> > >> >>>>> existing in the code
> > >> >>>>> This is the end of an old long-running project moving up on to
> > Cell
> > >> >>>>> Interface. We think it is done but for a few little items
> > >> (deprecate KV
> > >> >>>>> methods in MR and provide Cell versions instead...)
> > >> >>>>>
> > >> >>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
> > >> needs
> > >> >>>>> fixing
> > >> >>>>>
> > >> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
> > >> >>>>>
> > >> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
> > >> >>>>> * HBASE-14997
> > >> >>>>> Move compareOp and Comparators out of filter to client package
> > >> >>>>>
> > >> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
> > >> >>>>>
> > >> >>>>> What about:
> > >> >>>>>
> > >> >>>>> * HBASE-18601 Remove Htrace 3.2
> > >> >>>>> As has been noted, the HTrace API is our 'trace' API.
> > >> >>>>>
> > >> >>>>> If interested in any of the above and you need a legup, just ask
> > in
> > >> the
> > >> >>>>> issue and I'll be by....
> > >> >>>>>
> > >> >>>>> Thanks,
> > >> >>>>> St.Ack
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
> wrote:
> > >> >>>>>
> > >> >>>>> Heads-up:
> > >> >>>>>>
> > >> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
> > Theme
> > >> is
> > >> >>>>>> updated dependencies, reliance on relocated popular libs
> (guava,
> > >> >>>>>> netty,
> > >> >>>>>> protobuf), purge of checked-in generated src, and
> > >> >>>>>> master-carries-no-regions
> > >> >>>>>> by default.
> > >> >>>>>>
> > >> >>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
> theme
> > >> will
> > >> >>>>>> be
> > >> >>>>>> settling the APIs and compatibility (At first blush, we are not
> > >> >>>>>> looking too
> > >> >>>>>> bad; our Sean ran some tests over weekend that have hbase-1
> > client
> > >> >>>>>> running
> > >> >>>>>> against an hbase-2 cluster....). The Coprocessor Interface
> revamp
> > >> >>>>>> should be
> > >> >>>>>> done by alpha3 (i.e. returning Interfaces rather than
> > >> >>>>>> Implementations, and
> > >> >>>>>> our shutdown of CPs accessing classes in hbase marked
> > >> >>>>>> InterfaceAudience).
> > >> >>>>>> We'll also have purged thirdparty classes from our API; e.g.
> > guava
> > >> >>>>>> 0.12
> > >> >>>>>> Service showing through in our replication API and protobufs in
> > >> Admin
> > >> >>>>>> Interface. On alpha3, we will have to do a bunch of outreach to
> > >> make
> > >> >>>>>> sure
> > >> >>>>>> our downstreamers are up on what is coming down the pipe.
> > >> >>>>>>
> > >> >>>>>> Beta1 in mid-September?
> > >> >>>>>>
> > >> >>>>>> I encourage you to check out the items marked for hbase2:
> > >> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> 12327188
> > >> Edit
> > >> >>>>>> as
> > >> >>>>>> you see appropriate. Punt if you know the JIRA will not get any
> > >> >>>>>> attention
> > >> >>>>>> in next month or so.
> > >> >>>>>>
> > >> >>>>>> A bunch of issues marked blocker are unassigned. I'll leave
> them
> > >> as is
> > >> >>>>>> another while but I'll boot them soon.
> > >> >>>>>>
> > >> >>>>>> While I have your attention:
> > >> >>>>>>
> > >> >>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
> > >> thrift
> > >> >>>>>> to 0.10.0 will break existing clients. The change is easy
> enough
> > if
> > >> >>>>>> folks
> > >> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> > >> >>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> > >> >>>>>> (0.98?).
> > >> >>>>>>
> > >> >>>>>> St.Ack
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
> wrote:
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>>
> > >> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <elserj@apache.org
> >
> > >> >>>>>>> wrote:
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> > >> >>>>>>>>
> > >> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
> elserj@apache.org
> > >
> > >> >>>>>>>>> wrote:
> > >> >>>>>>>>>
> > >> >>>>>>>>> ...
> > >> >>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> I like the idea of this also hitting 2.0 as it would make
> the
> > >> >>>>>>>>>> feature a
> > >> >>>>>>>>>> bit more "real", but am obviously a little nervous (I have
> no
> > >> >>>>>>>>>> reason
> > >> >>>>>>>>>> to be
> > >> >>>>>>>>>> nervous though). I am pretty happy with the feature in
> terms
> > of
> > >> >>>>>>>>>> how
> > >> >>>>>>>>>> much it
> > >> >>>>>>>>>> is covered via testing.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
> Josh.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>> Last think on space quota says that need doc too. See 'Space
> > >> >>>>>>>>> Quota' in
> > >> >>>>>>>>> here:
> > >> >>>>>>>>> https://docs.google.com/document/d/
> 1WCsVlnHjJeKUcl7wHwqb4z9i
> > >> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > >> >>>>>>>>> Does this little section need an update Josh?
> > >> >>>>>>>>>
> > >> >>>>>>>>> Thanks,
> > >> >>>>>>>>> S
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and
> some
> > >> docs
> > >> >>>>>>>> already included for 17748.  Happy to put that on my plate if
> > >> >>>>>>>> you're good
> > >> >>>>>>>> with it. I can reasonably assume that no one is against it :)
> > >> >>>>>>>>
> > >> >>>>>>>> I think I had knocked out docs for the "phase 1" stuff before
> > we
> > >> >>>>>>>> merged it in from the original feature branch. I'll double
> > check
> > >> >>>>>>>> and update
> > >> >>>>>>>> the gdoc. Perhaps this was just a timing thing.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>> Thanks Josh,
> > >> >>>>>>> S
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >
> > >>
> > >
> > >
> >
>

Re: Moving 2.0 forward

["张铎 (Duo Zhang)" <pa...@gmail.com>]

Alpha4 aims to freeze the CP related API, not the public API. We break
everything for CP so we need to push out a stable(I hope) version to let CP
users learn how to implement their project on top of the new APIs.

For public API we have rules on how to keep compatible so I think it is
less hurt for users, beta1 is fine.

2017-10-26 17:27 GMT+08:00 Guanghao Zhang <zg...@gmail.com>:

> I saw beta == no new features, no API changes, just fixes. And I am working
> on HBASE-18805 to unify Admin and AsyncAdmin methods. The fix version was
> 2.0-beta-1. But I thought this will introduce API change(deprecate some API
> and introduce new one). So should I change the fix versions to 2.0-alpha-4
> and finish it before we release 2.0-alpha-4?
>
> Issue HBASE-18978 (Align the methods in Table and AsyncTable) may have same
> problem. Thanks.
>
> 2017-10-26 9:51 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
>
> > OK, skimmed,  we are in trouble! The in memory compaction just use the
> same
> > constructor with normal compaction to construct a StoreScanner, and use
> it
> > to do compaction...
> >
> > We have to provide several preXXX and postXXX for it, at least we should
> > allow user reset TTL and max versions, and also do filtering on the
> > scanner.
> >
> > Thanks.
> >
> >
> >
> > 2017-10-26 9:41 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
> >
> > > When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
> > > HBASE-19033, I found that there maybe a hole in our CP hools. It is the
> > in
> > > memory compaction.
> > >
> > > As Anoop suggested in HBASE-19001, we still need to give user the
> ability
> > > to extend the max versions and TTL config, so I plan to add back the
> > hooks
> > > above to let CP users can change the max versions and TTL of a ScanInfo
> > > object. But I'm not sure whether in memory compaction will also discard
> > > expired cells, if so then we are in trouble...
> > >
> > > Thanks.
> > >
> > > 2017-10-25 6:03 GMT+08:00 Stack <st...@duboce.net>:
> > >
> > >> Chatting with my coworker Mr. Mike Drob, we were batting back and
> forth
> > >> what remains to be done. Surfacing our thoughts here so you all clued
> > >> in....Or if you think otherwise, please speak up.
> > >>
> > >> We have ~13 issues to land:
> > >> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About
> > two
> > >> are meta-issues that are about process which leaves 11.
> > >>
> > >> Duo and Zheng Hu are to merge the FilterList fixes improvements
> > >> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
> > changed
> > >> API/semantic that we need to get out earlier rather than later.
> > >>
> > >> Once the above is merged, HBASE-13346, change of Filter method names
> to
> > >> mention Cell instead of KeyValue can land.
> > >>
> > >> HBASE-199048 needs a review (Anoop will probably do it), removing
> > >> IA.Private objects as params to MasterObserver... Hopefully this goes
> in
> > >> soon.
> > >>
> > >> Duo is hard at work on trackers for flush and compaction for CPs
> > >> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
> > >>
> > >> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after
> Duo
> > >> is
> > >> done w/ his work above.
> > >>
> > >> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after
> all
> > >> the
> > >> purges allowing CPs do direct calls against Regions in same Host.
> > >>
> > >> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> > >>
> > >> Another day or two?
> > >>
> > >> St.Ack
> > >>
> > >>
> > >>
> > >> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
> > >>
> > >> >
> > >> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
> > wrote:
> > >> >
> > >> >> +1
> > >> >>
> > >> >> I was trying to work on helping out on the outstanding alpha-4
> stuff
> > >> last
> > >> >> week -- will be continuing to try to do the same this week.
> > >> >>
> > >> >> If you need any help, Stack, or if others need reviews where I
> > haven't
> > >> >> noticed on my own: feel free to @mention me.
> > >> >>
> > >> >>
> > >> > Thanks for the offer Josh. All items seem assigned and are being
> > >> actively
> > >> > worked on. If you get a moment, reviews by you (or anyone else)
> helps
> > >> move
> > >> > the process along.
> > >> >
> > >> > We need to merge in HBASE-18410 branch to pick up Filter
> improvements.
> > >> > Then HBASE-13346 can go in.
> > >> >
> > >> > You are already helping out on HBASE-18906, thanks. Looks like that
> > will
> > >> > be addressed by other alpha-4s about to land.
> > >> >
> > >> > St.Ack
> > >> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/
> > 12341594
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >> On 10/23/17 12:53 PM, Stack wrote:
> > >> >>
> > >> >>> (Reviving this thread)
> > >> >>>
> > >> >>> Lets push out alpha-4 this week. Alpha-4 is the release that has
> the
> > >> >>> refactor of the Coprocessor API shutting down access to internals
> > >> marked
> > >> >>> InterfaceAudience.Private.
> > >> >>>
> > >> >>> The outstanding list is here:
> > >> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > >> >>>
> > >> >>> Please push in anything marked alpha-4 that belongs to you.
> > >> >>>
> > >> >>> If issue, talk out loud on this thread. If you need a review to
> land
> > >> an
> > >> >>> item, shout on the issue and here; we'll help you out.
> > >> >>>
> > >> >>> As is, items are coming along nicely I'd say. We need to merge the
> > >> filter
> > >> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> > >> >>>
> > >> >>> Post alpha-4, we'll have to hunt down our downstreamers and help
> > them
> > >> >>> test
> > >> >>> on top of alpha-4 so rolling into beta-1, we have confidence our
> > >> >>> downstreamers know what to expect (or we discover what we missed
> > >> BEFORE
> > >> >>> we
> > >> >>> beta-1).
> > >> >>>
> > >> >>> Thanks for time,
> > >> >>> S
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> > >> >>>
> > >> >>> I'll put up an alpha3 RC Monday, probably Monday night. That
> should
> > be
> > >> >>>> time, if we all sprint, for the public-facing API fixes to be
> done.
> > >> >>>>
> > >> >>>> I had a bunch of Coprocessor refactor and fixup scheduled for
> > alpha3
> > >> but
> > >> >>>> it is plain that more time is needed (in spite of valiant effort
> so
> > >> far
> > >> >>>> by
> > >> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4
> whose
> > >> >>>> theme is
> > >> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> > >> following
> > >> >>>> week.
> > >> >>>>
> > >> >>>> We should then be ready for beta (beta == no new features, no API
> > >> >>>> changes,
> > >> >>>> just fixes).
> > >> >>>>
> > >> >>>> Thanks,
> > >> >>>> St.Ack
> > >> >>>>
> > >> >>>>
> > >> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net>
> wrote:
> > >> >>>>
> > >> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on
> > it.
> > >> >>>>>
> > >> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
> > get
> > >> a
> > >> >>>>> release out in the next week or so.
> > >> >>>>>
> > >> >>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> > >> >>>>> interested in helping out, below are the items I think we need
> > done
> > >> for
> > >> >>>>> alpha3 (below are at least 'Critical' status, are API possibly
> > >> altering
> > >> >>>>> items, and are absent those JIRAs that are making active
> progress,
> > >> >>>>> i.e. the
> > >> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that
> needs
> > >> >>>>> doing is
> > >> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
> > >> >>>>>
> > >> >>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1
> and
> > >> >>>>> branch-2
> > >> >>>>> This is to do what Andrew did between 1.3 and 1.4 branches only
> do
> > >> it
> > >> >>>>> between branch-1 and branch-2.
> > >> >>>>>
> > >> >>>>> * HBASE-10462 Recategorize some of the client facing Public /
> > >> Private
> > >> >>>>> interfaces
> > >> >>>>> This one is almost done. It could do with a finish, attention to
> > the
> > >> >>>>> items in last comment, and then our codebase could do with
> another
> > >> >>>>> sweep
> > >> >>>>> after the spirit of this issue since a bunch has gone in since
> the
> > >> pass
> > >> >>>>> that was the basis of this issue.
> > >> >>>>>
> > >> >>>>> * HBASE-10504 Define Replication Interface
> > >> >>>>> I was going to take a crack at this as part of the revamp forced
> > by
> > >> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> > Service'
> > >> >>>>> but if
> > >> >>>>> anyone else is interested, be my guest.
> > >> >>>>>
> > >> >>>>> * HBASE-14996 Some more API cleanup for 2.0
> > >> >>>>> Has a bunch of subtasks, some of which are being worked on.
> Needs
> > >> >>>>> finishing.
> > >> >>>>>
> > >> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods in
> Admin
> > >> and
> > >> >>>>> cleanup
> > >> >>>>> Needs a pass. Small issue I think. Could also look at new
> > >> AsyncClient
> > >> >>>>> and
> > >> >>>>> make sure symmetry.
> > >> >>>>>
> > >> >>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> > >> >>>>> Predicated on result of an ongoing DISCUSSION thread but needs
> to
> > be
> > >> >>>>> done.
> > >> >>>>>
> > >> >>>>> Rolling upgrade will have implications for our API. Would be
> good
> > to
> > >> >>>>> try
> > >> >>>>> it and figure what needs fixup (as said above, according to
> trial
> > by
> > >> >>>>> Sean,
> > >> >>>>> we might not be too bad here):
> > >> >>>>> * HBASE-16060 1.x clients cannot access table state talking to
> 2.0
> > >> >>>>> cluster
> > >> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master
> > and
> > >> >>>>> 1.x
> > >> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> > >> >>>>>
> > >> >>>>> * HBASE-17442 Move most of the replication related classes to
> > >> >>>>> hbase-server package
> > >> >>>>> The above would be good to do generally but it may make for
> > ripples
> > >> in
> > >> >>>>> API so would be good to do now.
> > >> >>>>>
> > >> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> > >> >>>>> Balazs is working on this. The idea is that we avoid adding two
> > new
> > >> >>>>> types
> > >> >>>>> to our API, two types that are nought but curtailed, read-only
> > >> views on
> > >> >>>>> internals. Input if you have time appreciated.
> > >> >>>>>
> > >> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
> > >> hbase2
> > >> >>>>> cluster; verify
> > >> >>>>> Esteban is looking at this one
> > >> >>>>>
> > >> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> > >> >>>>> * HBASE-17143 Scan improvement
> > >> >>>>>
> > >> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
> > >> >>>>> implementations
> > >> >>>>> marked audience Private and returns implementations rather than
> > >> >>>>> Interfaces.
> > >> >>>>> In a few locations, we allow returning an alternate
> implementation
> > >> >>>>> altogether which is probably something we don't want a CP doing.
> > To
> > >> >>>>> that
> > >> >>>>> end, the following issues started by Duo and Anoop need to be
> > taken
> > >> to
> > >> >>>>> the
> > >> >>>>> finish line; ideally they'd have an owner:
> > >> >>>>>
> > >> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release
> <=
> > >> The
> > >> >>>>> umbrella issue.
> > >> >>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP
> expose
> > >> >>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver
> > and
> > >> >>>>> RegionServerObserver
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> Nice-to-haves:
> > >> >>>>>
> > >> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
> > >> unused
> > >> >>>>> TimeRange constructors
> > >> >>>>>
> > >> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow()
> references
> > >> >>>>> existing in the code
> > >> >>>>> This is the end of an old long-running project moving up on to
> > Cell
> > >> >>>>> Interface. We think it is done but for a few little items
> > >> (deprecate KV
> > >> >>>>> methods in MR and provide Cell versions instead...)
> > >> >>>>>
> > >> >>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
> > >> needs
> > >> >>>>> fixing
> > >> >>>>>
> > >> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
> > >> >>>>>
> > >> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
> > >> >>>>> * HBASE-14997
> > >> >>>>> Move compareOp and Comparators out of filter to client package
> > >> >>>>>
> > >> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
> > >> >>>>>
> > >> >>>>> What about:
> > >> >>>>>
> > >> >>>>> * HBASE-18601 Remove Htrace 3.2
> > >> >>>>> As has been noted, the HTrace API is our 'trace' API.
> > >> >>>>>
> > >> >>>>> If interested in any of the above and you need a legup, just ask
> > in
> > >> the
> > >> >>>>> issue and I'll be by....
> > >> >>>>>
> > >> >>>>> Thanks,
> > >> >>>>> St.Ack
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net>
> wrote:
> > >> >>>>>
> > >> >>>>> Heads-up:
> > >> >>>>>>
> > >> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
> > Theme
> > >> is
> > >> >>>>>> updated dependencies, reliance on relocated popular libs
> (guava,
> > >> >>>>>> netty,
> > >> >>>>>> protobuf), purge of checked-in generated src, and
> > >> >>>>>> master-carries-no-regions
> > >> >>>>>> by default.
> > >> >>>>>>
> > >> >>>>>> alpha3 I hope will follow soon after (end-of-August?). Its
> theme
> > >> will
> > >> >>>>>> be
> > >> >>>>>> settling the APIs and compatibility (At first blush, we are not
> > >> >>>>>> looking too
> > >> >>>>>> bad; our Sean ran some tests over weekend that have hbase-1
> > client
> > >> >>>>>> running
> > >> >>>>>> against an hbase-2 cluster....). The Coprocessor Interface
> revamp
> > >> >>>>>> should be
> > >> >>>>>> done by alpha3 (i.e. returning Interfaces rather than
> > >> >>>>>> Implementations, and
> > >> >>>>>> our shutdown of CPs accessing classes in hbase marked
> > >> >>>>>> InterfaceAudience).
> > >> >>>>>> We'll also have purged thirdparty classes from our API; e.g.
> > guava
> > >> >>>>>> 0.12
> > >> >>>>>> Service showing through in our replication API and protobufs in
> > >> Admin
> > >> >>>>>> Interface. On alpha3, we will have to do a bunch of outreach to
> > >> make
> > >> >>>>>> sure
> > >> >>>>>> our downstreamers are up on what is coming down the pipe.
> > >> >>>>>>
> > >> >>>>>> Beta1 in mid-September?
> > >> >>>>>>
> > >> >>>>>> I encourage you to check out the items marked for hbase2:
> > >> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/
> 12327188
> > >> Edit
> > >> >>>>>> as
> > >> >>>>>> you see appropriate. Punt if you know the JIRA will not get any
> > >> >>>>>> attention
> > >> >>>>>> in next month or so.
> > >> >>>>>>
> > >> >>>>>> A bunch of issues marked blocker are unassigned. I'll leave
> them
> > >> as is
> > >> >>>>>> another while but I'll boot them soon.
> > >> >>>>>>
> > >> >>>>>> While I have your attention:
> > >> >>>>>>
> > >> >>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
> > >> thrift
> > >> >>>>>> to 0.10.0 will break existing clients. The change is easy
> enough
> > if
> > >> >>>>>> folks
> > >> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> > >> >>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> > >> >>>>>> (0.98?).
> > >> >>>>>>
> > >> >>>>>> St.Ack
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net>
> wrote:
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>>
> > >> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <elserj@apache.org
> >
> > >> >>>>>>> wrote:
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> > >> >>>>>>>>
> > >> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<
> elserj@apache.org
> > >
> > >> >>>>>>>>> wrote:
> > >> >>>>>>>>>
> > >> >>>>>>>>> ...
> > >> >>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> I like the idea of this also hitting 2.0 as it would make
> the
> > >> >>>>>>>>>> feature a
> > >> >>>>>>>>>> bit more "real", but am obviously a little nervous (I have
> no
> > >> >>>>>>>>>> reason
> > >> >>>>>>>>>> to be
> > >> >>>>>>>>>> nervous though). I am pretty happy with the feature in
> terms
> > of
> > >> >>>>>>>>>> how
> > >> >>>>>>>>>> much it
> > >> >>>>>>>>>> is covered via testing.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1
> Josh.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>> Last think on space quota says that need doc too. See 'Space
> > >> >>>>>>>>> Quota' in
> > >> >>>>>>>>> here:
> > >> >>>>>>>>> https://docs.google.com/document/d/
> 1WCsVlnHjJeKUcl7wHwqb4z9i
> > >> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > >> >>>>>>>>> Does this little section need an update Josh?
> > >> >>>>>>>>>
> > >> >>>>>>>>> Thanks,
> > >> >>>>>>>>> S
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and
> some
> > >> docs
> > >> >>>>>>>> already included for 17748.  Happy to put that on my plate if
> > >> >>>>>>>> you're good
> > >> >>>>>>>> with it. I can reasonably assume that no one is against it :)
> > >> >>>>>>>>
> > >> >>>>>>>> I think I had knocked out docs for the "phase 1" stuff before
> > we
> > >> >>>>>>>> merged it in from the original feature branch. I'll double
> > check
> > >> >>>>>>>> and update
> > >> >>>>>>>> the gdoc. Perhaps this was just a timing thing.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>> Thanks Josh,
> > >> >>>>>>> S
> > >> >>>>>>>
> > >> >>>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >
> > >>
> > >
> > >
> >
>

Re: Moving 2.0 forward

[Guanghao Zhang <zg...@gmail.com>]

I saw beta == no new features, no API changes, just fixes. And I am working
on HBASE-18805 to unify Admin and AsyncAdmin methods. The fix version was
2.0-beta-1. But I thought this will introduce API change(deprecate some API
and introduce new one). So should I change the fix versions to 2.0-alpha-4
and finish it before we release 2.0-alpha-4?

Issue HBASE-18978 (Align the methods in Table and AsyncTable) may have same
problem. Thanks.

2017-10-26 9:51 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:

> OK, skimmed,  we are in trouble! The in memory compaction just use the same
> constructor with normal compaction to construct a StoreScanner, and use it
> to do compaction...
>
> We have to provide several preXXX and postXXX for it, at least we should
> allow user reset TTL and max versions, and also do filtering on the
> scanner.
>
> Thanks.
>
>
>
> 2017-10-26 9:41 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:
>
> > When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
> > HBASE-19033, I found that there maybe a hole in our CP hools. It is the
> in
> > memory compaction.
> >
> > As Anoop suggested in HBASE-19001, we still need to give user the ability
> > to extend the max versions and TTL config, so I plan to add back the
> hooks
> > above to let CP users can change the max versions and TTL of a ScanInfo
> > object. But I'm not sure whether in memory compaction will also discard
> > expired cells, if so then we are in trouble...
> >
> > Thanks.
> >
> > 2017-10-25 6:03 GMT+08:00 Stack <st...@duboce.net>:
> >
> >> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
> >> what remains to be done. Surfacing our thoughts here so you all clued
> >> in....Or if you think otherwise, please speak up.
> >>
> >> We have ~13 issues to land:
> >> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About
> two
> >> are meta-issues that are about process which leaves 11.
> >>
> >> Duo and Zheng Hu are to merge the FilterList fixes improvements
> >> (HBASE-19057, HBASE-18410 et al.). These are blocker because some
> changed
> >> API/semantic that we need to get out earlier rather than later.
> >>
> >> Once the above is merged, HBASE-13346, change of Filter method names to
> >> mention Cell instead of KeyValue can land.
> >>
> >> HBASE-199048 needs a review (Anoop will probably do it), removing
> >> IA.Private objects as params to MasterObserver... Hopefully this goes in
> >> soon.
> >>
> >> Duo is hard at work on trackers for flush and compaction for CPs
> >> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
> >>
> >> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo
> >> is
> >> done w/ his work above.
> >>
> >> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all
> >> the
> >> purges allowing CPs do direct calls against Regions in same Host.
> >>
> >> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
> >>
> >> Another day or two?
> >>
> >> St.Ack
> >>
> >>
> >>
> >> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
> >>
> >> >
> >> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
> wrote:
> >> >
> >> >> +1
> >> >>
> >> >> I was trying to work on helping out on the outstanding alpha-4 stuff
> >> last
> >> >> week -- will be continuing to try to do the same this week.
> >> >>
> >> >> If you need any help, Stack, or if others need reviews where I
> haven't
> >> >> noticed on my own: feel free to @mention me.
> >> >>
> >> >>
> >> > Thanks for the offer Josh. All items seem assigned and are being
> >> actively
> >> > worked on. If you get a moment, reviews by you (or anyone else) helps
> >> move
> >> > the process along.
> >> >
> >> > We need to merge in HBASE-18410 branch to pick up Filter improvements.
> >> > Then HBASE-13346 can go in.
> >> >
> >> > You are already helping out on HBASE-18906, thanks. Looks like that
> will
> >> > be addressed by other alpha-4s about to land.
> >> >
> >> > St.Ack
> >> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/
> 12341594
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >> On 10/23/17 12:53 PM, Stack wrote:
> >> >>
> >> >>> (Reviving this thread)
> >> >>>
> >> >>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
> >> >>> refactor of the Coprocessor API shutting down access to internals
> >> marked
> >> >>> InterfaceAudience.Private.
> >> >>>
> >> >>> The outstanding list is here:
> >> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >> >>>
> >> >>> Please push in anything marked alpha-4 that belongs to you.
> >> >>>
> >> >>> If issue, talk out loud on this thread. If you need a review to land
> >> an
> >> >>> item, shout on the issue and here; we'll help you out.
> >> >>>
> >> >>> As is, items are coming along nicely I'd say. We need to merge the
> >> filter
> >> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> >> >>>
> >> >>> Post alpha-4, we'll have to hunt down our downstreamers and help
> them
> >> >>> test
> >> >>> on top of alpha-4 so rolling into beta-1, we have confidence our
> >> >>> downstreamers know what to expect (or we discover what we missed
> >> BEFORE
> >> >>> we
> >> >>> beta-1).
> >> >>>
> >> >>> Thanks for time,
> >> >>> S
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> >> >>>
> >> >>> I'll put up an alpha3 RC Monday, probably Monday night. That should
> be
> >> >>>> time, if we all sprint, for the public-facing API fixes to be done.
> >> >>>>
> >> >>>> I had a bunch of Coprocessor refactor and fixup scheduled for
> alpha3
> >> but
> >> >>>> it is plain that more time is needed (in spite of valiant effort so
> >> far
> >> >>>> by
> >> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
> >> >>>> theme is
> >> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> >> following
> >> >>>> week.
> >> >>>>
> >> >>>> We should then be ready for beta (beta == no new features, no API
> >> >>>> changes,
> >> >>>> just fixes).
> >> >>>>
> >> >>>> Thanks,
> >> >>>> St.Ack
> >> >>>>
> >> >>>>
> >> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
> >> >>>>
> >> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on
> it.
> >> >>>>>
> >> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
> get
> >> a
> >> >>>>> release out in the next week or so.
> >> >>>>>
> >> >>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> >> >>>>> interested in helping out, below are the items I think we need
> done
> >> for
> >> >>>>> alpha3 (below are at least 'Critical' status, are API possibly
> >> altering
> >> >>>>> items, and are absent those JIRAs that are making active progress,
> >> >>>>> i.e. the
> >> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
> >> >>>>> doing is
> >> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
> >> >>>>>
> >> >>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
> >> >>>>> branch-2
> >> >>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do
> >> it
> >> >>>>> between branch-1 and branch-2.
> >> >>>>>
> >> >>>>> * HBASE-10462 Recategorize some of the client facing Public /
> >> Private
> >> >>>>> interfaces
> >> >>>>> This one is almost done. It could do with a finish, attention to
> the
> >> >>>>> items in last comment, and then our codebase could do with another
> >> >>>>> sweep
> >> >>>>> after the spirit of this issue since a bunch has gone in since the
> >> pass
> >> >>>>> that was the basis of this issue.
> >> >>>>>
> >> >>>>> * HBASE-10504 Define Replication Interface
> >> >>>>> I was going to take a crack at this as part of the revamp forced
> by
> >> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> Service'
> >> >>>>> but if
> >> >>>>> anyone else is interested, be my guest.
> >> >>>>>
> >> >>>>> * HBASE-14996 Some more API cleanup for 2.0
> >> >>>>> Has a bunch of subtasks, some of which are being worked on. Needs
> >> >>>>> finishing.
> >> >>>>>
> >> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
> >> and
> >> >>>>> cleanup
> >> >>>>> Needs a pass. Small issue I think. Could also look at new
> >> AsyncClient
> >> >>>>> and
> >> >>>>> make sure symmetry.
> >> >>>>>
> >> >>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> >> >>>>> Predicated on result of an ongoing DISCUSSION thread but needs to
> be
> >> >>>>> done.
> >> >>>>>
> >> >>>>> Rolling upgrade will have implications for our API. Would be good
> to
> >> >>>>> try
> >> >>>>> it and figure what needs fixup (as said above, according to trial
> by
> >> >>>>> Sean,
> >> >>>>> we might not be too bad here):
> >> >>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
> >> >>>>> cluster
> >> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master
> and
> >> >>>>> 1.x
> >> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> >> >>>>>
> >> >>>>> * HBASE-17442 Move most of the replication related classes to
> >> >>>>> hbase-server package
> >> >>>>> The above would be good to do generally but it may make for
> ripples
> >> in
> >> >>>>> API so would be good to do now.
> >> >>>>>
> >> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> >> >>>>> Balazs is working on this. The idea is that we avoid adding two
> new
> >> >>>>> types
> >> >>>>> to our API, two types that are nought but curtailed, read-only
> >> views on
> >> >>>>> internals. Input if you have time appreciated.
> >> >>>>>
> >> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
> >> hbase2
> >> >>>>> cluster; verify
> >> >>>>> Esteban is looking at this one
> >> >>>>>
> >> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> >> >>>>> * HBASE-17143 Scan improvement
> >> >>>>>
> >> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
> >> >>>>> implementations
> >> >>>>> marked audience Private and returns implementations rather than
> >> >>>>> Interfaces.
> >> >>>>> In a few locations, we allow returning an alternate implementation
> >> >>>>> altogether which is probably something we don't want a CP doing.
> To
> >> >>>>> that
> >> >>>>> end, the following issues started by Duo and Anoop need to be
> taken
> >> to
> >> >>>>> the
> >> >>>>> finish line; ideally they'd have an owner:
> >> >>>>>
> >> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
> >> The
> >> >>>>> umbrella issue.
> >> >>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> >> >>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver
> and
> >> >>>>> RegionServerObserver
> >> >>>>>
> >> >>>>>
> >> >>>>> Nice-to-haves:
> >> >>>>>
> >> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
> >> unused
> >> >>>>> TimeRange constructors
> >> >>>>>
> >> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
> >> >>>>> existing in the code
> >> >>>>> This is the end of an old long-running project moving up on to
> Cell
> >> >>>>> Interface. We think it is done but for a few little items
> >> (deprecate KV
> >> >>>>> methods in MR and provide Cell versions instead...)
> >> >>>>>
> >> >>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
> >> needs
> >> >>>>> fixing
> >> >>>>>
> >> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
> >> >>>>>
> >> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
> >> >>>>> * HBASE-14997
> >> >>>>> Move compareOp and Comparators out of filter to client package
> >> >>>>>
> >> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
> >> >>>>>
> >> >>>>> What about:
> >> >>>>>
> >> >>>>> * HBASE-18601 Remove Htrace 3.2
> >> >>>>> As has been noted, the HTrace API is our 'trace' API.
> >> >>>>>
> >> >>>>> If interested in any of the above and you need a legup, just ask
> in
> >> the
> >> >>>>> issue and I'll be by....
> >> >>>>>
> >> >>>>> Thanks,
> >> >>>>> St.Ack
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
> >> >>>>>
> >> >>>>> Heads-up:
> >> >>>>>>
> >> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
> Theme
> >> is
> >> >>>>>> updated dependencies, reliance on relocated popular libs (guava,
> >> >>>>>> netty,
> >> >>>>>> protobuf), purge of checked-in generated src, and
> >> >>>>>> master-carries-no-regions
> >> >>>>>> by default.
> >> >>>>>>
> >> >>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
> >> will
> >> >>>>>> be
> >> >>>>>> settling the APIs and compatibility (At first blush, we are not
> >> >>>>>> looking too
> >> >>>>>> bad; our Sean ran some tests over weekend that have hbase-1
> client
> >> >>>>>> running
> >> >>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
> >> >>>>>> should be
> >> >>>>>> done by alpha3 (i.e. returning Interfaces rather than
> >> >>>>>> Implementations, and
> >> >>>>>> our shutdown of CPs accessing classes in hbase marked
> >> >>>>>> InterfaceAudience).
> >> >>>>>> We'll also have purged thirdparty classes from our API; e.g.
> guava
> >> >>>>>> 0.12
> >> >>>>>> Service showing through in our replication API and protobufs in
> >> Admin
> >> >>>>>> Interface. On alpha3, we will have to do a bunch of outreach to
> >> make
> >> >>>>>> sure
> >> >>>>>> our downstreamers are up on what is coming down the pipe.
> >> >>>>>>
> >> >>>>>> Beta1 in mid-September?
> >> >>>>>>
> >> >>>>>> I encourage you to check out the items marked for hbase2:
> >> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
> >> Edit
> >> >>>>>> as
> >> >>>>>> you see appropriate. Punt if you know the JIRA will not get any
> >> >>>>>> attention
> >> >>>>>> in next month or so.
> >> >>>>>>
> >> >>>>>> A bunch of issues marked blocker are unassigned. I'll leave them
> >> as is
> >> >>>>>> another while but I'll boot them soon.
> >> >>>>>>
> >> >>>>>> While I have your attention:
> >> >>>>>>
> >> >>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
> >> thrift
> >> >>>>>> to 0.10.0 will break existing clients. The change is easy enough
> if
> >> >>>>>> folks
> >> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> >> >>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> >> >>>>>> (0.98?).
> >> >>>>>>
> >> >>>>>> St.Ack
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>>
> >> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
> >> >>>>>>> wrote:
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> >> >>>>>>>>
> >> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<elserj@apache.org
> >
> >> >>>>>>>>> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>> ...
> >> >>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
> >> >>>>>>>>>> feature a
> >> >>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
> >> >>>>>>>>>> reason
> >> >>>>>>>>>> to be
> >> >>>>>>>>>> nervous though). I am pretty happy with the feature in terms
> of
> >> >>>>>>>>>> how
> >> >>>>>>>>>> much it
> >> >>>>>>>>>> is covered via testing.
> >> >>>>>>>>>>
> >> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> >> >>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>> Last think on space quota says that need doc too. See 'Space
> >> >>>>>>>>> Quota' in
> >> >>>>>>>>> here:
> >> >>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> >> >>>>>>>>> Does this little section need an update Josh?
> >> >>>>>>>>>
> >> >>>>>>>>> Thanks,
> >> >>>>>>>>> S
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
> >> docs
> >> >>>>>>>> already included for 17748.  Happy to put that on my plate if
> >> >>>>>>>> you're good
> >> >>>>>>>> with it. I can reasonably assume that no one is against it :)
> >> >>>>>>>>
> >> >>>>>>>> I think I had knocked out docs for the "phase 1" stuff before
> we
> >> >>>>>>>> merged it in from the original feature branch. I'll double
> check
> >> >>>>>>>> and update
> >> >>>>>>>> the gdoc. Perhaps this was just a timing thing.
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>> Thanks Josh,
> >> >>>>>>> S
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >
> >>
> >
> >
>

Re: Moving 2.0 forward

["张铎 (Duo Zhang)" <pa...@gmail.com>]

OK, skimmed,  we are in trouble! The in memory compaction just use the same
constructor with normal compaction to construct a StoreScanner, and use it
to do compaction...

We have to provide several preXXX and postXXX for it, at least we should
allow user reset TTL and max versions, and also do filtering on the scanner.

Thanks.



2017-10-26 9:41 GMT+08:00 张铎(Duo Zhang) <pa...@gmail.com>:

> When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
> HBASE-19033, I found that there maybe a hole in our CP hools. It is the in
> memory compaction.
>
> As Anoop suggested in HBASE-19001, we still need to give user the ability
> to extend the max versions and TTL config, so I plan to add back the hooks
> above to let CP users can change the max versions and TTL of a ScanInfo
> object. But I'm not sure whether in memory compaction will also discard
> expired cells, if so then we are in trouble...
>
> Thanks.
>
> 2017-10-25 6:03 GMT+08:00 Stack <st...@duboce.net>:
>
>> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
>> what remains to be done. Surfacing our thoughts here so you all clued
>> in....Or if you think otherwise, please speak up.
>>
>> We have ~13 issues to land:
>> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About two
>> are meta-issues that are about process which leaves 11.
>>
>> Duo and Zheng Hu are to merge the FilterList fixes improvements
>> (HBASE-19057, HBASE-18410 et al.). These are blocker because some changed
>> API/semantic that we need to get out earlier rather than later.
>>
>> Once the above is merged, HBASE-13346, change of Filter method names to
>> mention Cell instead of KeyValue can land.
>>
>> HBASE-199048 needs a review (Anoop will probably do it), removing
>> IA.Private objects as params to MasterObserver... Hopefully this goes in
>> soon.
>>
>> Duo is hard at work on trackers for flush and compaction for CPs
>> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>>
>> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo
>> is
>> done w/ his work above.
>>
>> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all
>> the
>> purges allowing CPs do direct calls against Regions in same Host.
>>
>> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>>
>> Another day or two?
>>
>> St.Ack
>>
>>
>>
>> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>>
>> >
>> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>> >
>> >> +1
>> >>
>> >> I was trying to work on helping out on the outstanding alpha-4 stuff
>> last
>> >> week -- will be continuing to try to do the same this week.
>> >>
>> >> If you need any help, Stack, or if others need reviews where I haven't
>> >> noticed on my own: feel free to @mention me.
>> >>
>> >>
>> > Thanks for the offer Josh. All items seem assigned and are being
>> actively
>> > worked on. If you get a moment, reviews by you (or anyone else) helps
>> move
>> > the process along.
>> >
>> > We need to merge in HBASE-18410 branch to pick up Filter improvements.
>> > Then HBASE-13346 can go in.
>> >
>> > You are already helping out on HBASE-18906, thanks. Looks like that will
>> > be addressed by other alpha-4s about to land.
>> >
>> > St.Ack
>> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >> On 10/23/17 12:53 PM, Stack wrote:
>> >>
>> >>> (Reviving this thread)
>> >>>
>> >>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>> >>> refactor of the Coprocessor API shutting down access to internals
>> marked
>> >>> InterfaceAudience.Private.
>> >>>
>> >>> The outstanding list is here:
>> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>> >>>
>> >>> Please push in anything marked alpha-4 that belongs to you.
>> >>>
>> >>> If issue, talk out loud on this thread. If you need a review to land
>> an
>> >>> item, shout on the issue and here; we'll help you out.
>> >>>
>> >>> As is, items are coming along nicely I'd say. We need to merge the
>> filter
>> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>> >>>
>> >>> Post alpha-4, we'll have to hunt down our downstreamers and help them
>> >>> test
>> >>> on top of alpha-4 so rolling into beta-1, we have confidence our
>> >>> downstreamers know what to expect (or we discover what we missed
>> BEFORE
>> >>> we
>> >>> beta-1).
>> >>>
>> >>> Thanks for time,
>> >>> S
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>> >>>
>> >>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
>> >>>> time, if we all sprint, for the public-facing API fixes to be done.
>> >>>>
>> >>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
>> but
>> >>>> it is plain that more time is needed (in spite of valiant effort so
>> far
>> >>>> by
>> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>> >>>> theme is
>> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
>> following
>> >>>> week.
>> >>>>
>> >>>> We should then be ready for beta (beta == no new features, no API
>> >>>> changes,
>> >>>> just fixes).
>> >>>>
>> >>>> Thanks,
>> >>>> St.Ack
>> >>>>
>> >>>>
>> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>> >>>>
>> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>> >>>>>
>> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get
>> a
>> >>>>> release out in the next week or so.
>> >>>>>
>> >>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>> >>>>> interested in helping out, below are the items I think we need done
>> for
>> >>>>> alpha3 (below are at least 'Critical' status, are API possibly
>> altering
>> >>>>> items, and are absent those JIRAs that are making active progress,
>> >>>>> i.e. the
>> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>> >>>>> doing is
>> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
>> >>>>>
>> >>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>> >>>>> branch-2
>> >>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do
>> it
>> >>>>> between branch-1 and branch-2.
>> >>>>>
>> >>>>> * HBASE-10462 Recategorize some of the client facing Public /
>> Private
>> >>>>> interfaces
>> >>>>> This one is almost done. It could do with a finish, attention to the
>> >>>>> items in last comment, and then our codebase could do with another
>> >>>>> sweep
>> >>>>> after the spirit of this issue since a bunch has gone in since the
>> pass
>> >>>>> that was the basis of this issue.
>> >>>>>
>> >>>>> * HBASE-10504 Define Replication Interface
>> >>>>> I was going to take a crack at this as part of the revamp forced by
>> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
>> >>>>> but if
>> >>>>> anyone else is interested, be my guest.
>> >>>>>
>> >>>>> * HBASE-14996 Some more API cleanup for 2.0
>> >>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>> >>>>> finishing.
>> >>>>>
>> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
>> and
>> >>>>> cleanup
>> >>>>> Needs a pass. Small issue I think. Could also look at new
>> AsyncClient
>> >>>>> and
>> >>>>> make sure symmetry.
>> >>>>>
>> >>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>> >>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
>> >>>>> done.
>> >>>>>
>> >>>>> Rolling upgrade will have implications for our API. Would be good to
>> >>>>> try
>> >>>>> it and figure what needs fixup (as said above, according to trial by
>> >>>>> Sean,
>> >>>>> we might not be too bad here):
>> >>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>> >>>>> cluster
>> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
>> >>>>> 1.x
>> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>> >>>>>
>> >>>>> * HBASE-17442 Move most of the replication related classes to
>> >>>>> hbase-server package
>> >>>>> The above would be good to do generally but it may make for ripples
>> in
>> >>>>> API so would be good to do now.
>> >>>>>
>> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>> >>>>> Balazs is working on this. The idea is that we avoid adding two new
>> >>>>> types
>> >>>>> to our API, two types that are nought but curtailed, read-only
>> views on
>> >>>>> internals. Input if you have time appreciated.
>> >>>>>
>> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
>> hbase2
>> >>>>> cluster; verify
>> >>>>> Esteban is looking at this one
>> >>>>>
>> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>> >>>>> * HBASE-17143 Scan improvement
>> >>>>>
>> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
>> >>>>> implementations
>> >>>>> marked audience Private and returns implementations rather than
>> >>>>> Interfaces.
>> >>>>> In a few locations, we allow returning an alternate implementation
>> >>>>> altogether which is probably something we don't want a CP doing. To
>> >>>>> that
>> >>>>> end, the following issues started by Duo and Anoop need to be taken
>> to
>> >>>>> the
>> >>>>> finish line; ideally they'd have an owner:
>> >>>>>
>> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
>> The
>> >>>>> umbrella issue.
>> >>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>> >>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>> >>>>> RegionServerObserver
>> >>>>>
>> >>>>>
>> >>>>> Nice-to-haves:
>> >>>>>
>> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
>> unused
>> >>>>> TimeRange constructors
>> >>>>>
>> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>> >>>>> existing in the code
>> >>>>> This is the end of an old long-running project moving up on to Cell
>> >>>>> Interface. We think it is done but for a few little items
>> (deprecate KV
>> >>>>> methods in MR and provide Cell versions instead...)
>> >>>>>
>> >>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
>> needs
>> >>>>> fixing
>> >>>>>
>> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
>> >>>>>
>> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
>> >>>>> * HBASE-14997
>> >>>>> Move compareOp and Comparators out of filter to client package
>> >>>>>
>> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
>> >>>>>
>> >>>>> What about:
>> >>>>>
>> >>>>> * HBASE-18601 Remove Htrace 3.2
>> >>>>> As has been noted, the HTrace API is our 'trace' API.
>> >>>>>
>> >>>>> If interested in any of the above and you need a legup, just ask in
>> the
>> >>>>> issue and I'll be by....
>> >>>>>
>> >>>>> Thanks,
>> >>>>> St.Ack
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>> >>>>>
>> >>>>> Heads-up:
>> >>>>>>
>> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme
>> is
>> >>>>>> updated dependencies, reliance on relocated popular libs (guava,
>> >>>>>> netty,
>> >>>>>> protobuf), purge of checked-in generated src, and
>> >>>>>> master-carries-no-regions
>> >>>>>> by default.
>> >>>>>>
>> >>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
>> will
>> >>>>>> be
>> >>>>>> settling the APIs and compatibility (At first blush, we are not
>> >>>>>> looking too
>> >>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>> >>>>>> running
>> >>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>> >>>>>> should be
>> >>>>>> done by alpha3 (i.e. returning Interfaces rather than
>> >>>>>> Implementations, and
>> >>>>>> our shutdown of CPs accessing classes in hbase marked
>> >>>>>> InterfaceAudience).
>> >>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
>> >>>>>> 0.12
>> >>>>>> Service showing through in our replication API and protobufs in
>> Admin
>> >>>>>> Interface. On alpha3, we will have to do a bunch of outreach to
>> make
>> >>>>>> sure
>> >>>>>> our downstreamers are up on what is coming down the pipe.
>> >>>>>>
>> >>>>>> Beta1 in mid-September?
>> >>>>>>
>> >>>>>> I encourage you to check out the items marked for hbase2:
>> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
>> Edit
>> >>>>>> as
>> >>>>>> you see appropriate. Punt if you know the JIRA will not get any
>> >>>>>> attention
>> >>>>>> in next month or so.
>> >>>>>>
>> >>>>>> A bunch of issues marked blocker are unassigned. I'll leave them
>> as is
>> >>>>>> another while but I'll boot them soon.
>> >>>>>>
>> >>>>>> While I have your attention:
>> >>>>>>
>> >>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
>> thrift
>> >>>>>> to 0.10.0 will break existing clients. The change is easy enough if
>> >>>>>> folks
>> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>> >>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>> >>>>>> (0.98?).
>> >>>>>>
>> >>>>>> St.Ack
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>> >>>>>>
>> >>>>>>
>> >>>>>>>
>> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>> >>>>>>>>
>> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>> >>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>> ...
>> >>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>> >>>>>>>>>> feature a
>> >>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>> >>>>>>>>>> reason
>> >>>>>>>>>> to be
>> >>>>>>>>>> nervous though). I am pretty happy with the feature in terms of
>> >>>>>>>>>> how
>> >>>>>>>>>> much it
>> >>>>>>>>>> is covered via testing.
>> >>>>>>>>>>
>> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> Last think on space quota says that need doc too. See 'Space
>> >>>>>>>>> Quota' in
>> >>>>>>>>> here:
>> >>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>> >>>>>>>>> Does this little section need an update Josh?
>> >>>>>>>>>
>> >>>>>>>>> Thanks,
>> >>>>>>>>> S
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
>> docs
>> >>>>>>>> already included for 17748.  Happy to put that on my plate if
>> >>>>>>>> you're good
>> >>>>>>>> with it. I can reasonably assume that no one is against it :)
>> >>>>>>>>
>> >>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>> >>>>>>>> merged it in from the original feature branch. I'll double check
>> >>>>>>>> and update
>> >>>>>>>> the gdoc. Perhaps this was just a timing thing.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>> Thanks Josh,
>> >>>>>>> S
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >
>>
>
>

Re: Moving 2.0 forward

["张铎 (Duo Zhang)" <pa...@gmail.com>]

When adding back the pre(Flush/Compact/Store)ScannerOpen methods in
HBASE-19033, I found that there maybe a hole in our CP hools. It is the in
memory compaction.

As Anoop suggested in HBASE-19001, we still need to give user the ability
to extend the max versions and TTL config, so I plan to add back the hooks
above to let CP users can change the max versions and TTL of a ScanInfo
object. But I'm not sure whether in memory compaction will also discard
expired cells, if so then we are in trouble...

Thanks.

2017-10-25 6:03 GMT+08:00 Stack <st...@duboce.net>:

> Chatting with my coworker Mr. Mike Drob, we were batting back and forth
> what remains to be done. Surfacing our thoughts here so you all clued
> in....Or if you think otherwise, please speak up.
>
> We have ~13 issues to land:
> https://issues.apache.org/jira/projects/HBASE/versions/12341594 About two
> are meta-issues that are about process which leaves 11.
>
> Duo and Zheng Hu are to merge the FilterList fixes improvements
> (HBASE-19057, HBASE-18410 et al.). These are blocker because some changed
> API/semantic that we need to get out earlier rather than later.
>
> Once the above is merged, HBASE-13346, change of Filter method names to
> mention Cell instead of KeyValue can land.
>
> HBASE-199048 needs a review (Anoop will probably do it), removing
> IA.Private objects as params to MasterObserver... Hopefully this goes in
> soon.
>
> Duo is hard at work on trackers for flush and compaction for CPs
> (HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?
>
> I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo is
> done w/ his work above.
>
> I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all the
> purges allowing CPs do direct calls against Regions in same Host.
>
> Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.
>
> Another day or two?
>
> St.Ack
>
>
>
> On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:
>
> >
> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
> >
> >> +1
> >>
> >> I was trying to work on helping out on the outstanding alpha-4 stuff
> last
> >> week -- will be continuing to try to do the same this week.
> >>
> >> If you need any help, Stack, or if others need reviews where I haven't
> >> noticed on my own: feel free to @mention me.
> >>
> >>
> > Thanks for the offer Josh. All items seem assigned and are being actively
> > worked on. If you get a moment, reviews by you (or anyone else) helps
> move
> > the process along.
> >
> > We need to merge in HBASE-18410 branch to pick up Filter improvements.
> > Then HBASE-13346 can go in.
> >
> > You are already helping out on HBASE-18906, thanks. Looks like that will
> > be addressed by other alpha-4s about to land.
> >
> > St.Ack
> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >> On 10/23/17 12:53 PM, Stack wrote:
> >>
> >>> (Reviving this thread)
> >>>
> >>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
> >>> refactor of the Coprocessor API shutting down access to internals
> marked
> >>> InterfaceAudience.Private.
> >>>
> >>> The outstanding list is here:
> >>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >>>
> >>> Please push in anything marked alpha-4 that belongs to you.
> >>>
> >>> If issue, talk out loud on this thread. If you need a review to land an
> >>> item, shout on the issue and here; we'll help you out.
> >>>
> >>> As is, items are coming along nicely I'd say. We need to merge the
> filter
> >>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> >>>
> >>> Post alpha-4, we'll have to hunt down our downstreamers and help them
> >>> test
> >>> on top of alpha-4 so rolling into beta-1, we have confidence our
> >>> downstreamers know what to expect (or we discover what we missed BEFORE
> >>> we
> >>> beta-1).
> >>>
> >>> Thanks for time,
> >>> S
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> >>>
> >>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
> >>>> time, if we all sprint, for the public-facing API fixes to be done.
> >>>>
> >>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
> but
> >>>> it is plain that more time is needed (in spite of valiant effort so
> far
> >>>> by
> >>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
> >>>> theme is
> >>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> following
> >>>> week.
> >>>>
> >>>> We should then be ready for beta (beta == no new features, no API
> >>>> changes,
> >>>> just fixes).
> >>>>
> >>>> Thanks,
> >>>> St.Ack
> >>>>
> >>>>
> >>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
> >>>>
> >>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
> >>>>>
> >>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
> >>>>> release out in the next week or so.
> >>>>>
> >>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> >>>>> interested in helping out, below are the items I think we need done
> for
> >>>>> alpha3 (below are at least 'Critical' status, are API possibly
> altering
> >>>>> items, and are absent those JIRAs that are making active progress,
> >>>>> i.e. the
> >>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
> >>>>> doing is
> >>>>> what Andrew did comparing 1.3. and 1.4 APIs
> >>>>>
> >>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
> >>>>> branch-2
> >>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
> >>>>> between branch-1 and branch-2.
> >>>>>
> >>>>> * HBASE-10462 Recategorize some of the client facing Public / Private
> >>>>> interfaces
> >>>>> This one is almost done. It could do with a finish, attention to the
> >>>>> items in last comment, and then our codebase could do with another
> >>>>> sweep
> >>>>> after the spirit of this issue since a bunch has gone in since the
> pass
> >>>>> that was the basis of this issue.
> >>>>>
> >>>>> * HBASE-10504 Define Replication Interface
> >>>>> I was going to take a crack at this as part of the revamp forced by
> >>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
> >>>>> but if
> >>>>> anyone else is interested, be my guest.
> >>>>>
> >>>>> * HBASE-14996 Some more API cleanup for 2.0
> >>>>> Has a bunch of subtasks, some of which are being worked on. Needs
> >>>>> finishing.
> >>>>>
> >>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
> >>>>> cleanup
> >>>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
> >>>>> and
> >>>>> make sure symmetry.
> >>>>>
> >>>>> * HBASE-15607 Remove PB references from Admin for 2.0
> >>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
> >>>>> done.
> >>>>>
> >>>>> Rolling upgrade will have implications for our API. Would be good to
> >>>>> try
> >>>>> it and figure what needs fixup (as said above, according to trial by
> >>>>> Sean,
> >>>>> we might not be too bad here):
> >>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
> >>>>> cluster
> >>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
> >>>>> 1.x
> >>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> >>>>>
> >>>>> * HBASE-17442 Move most of the replication related classes to
> >>>>> hbase-server package
> >>>>> The above would be good to do generally but it may make for ripples
> in
> >>>>> API so would be good to do now.
> >>>>>
> >>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> >>>>> Balazs is working on this. The idea is that we avoid adding two new
> >>>>> types
> >>>>> to our API, two types that are nought but curtailed, read-only views
> on
> >>>>> internals. Input if you have time appreciated.
> >>>>>
> >>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
> hbase2
> >>>>> cluster; verify
> >>>>> Esteban is looking at this one
> >>>>>
> >>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> >>>>> * HBASE-17143 Scan improvement
> >>>>>
> >>>>> Our Coprocessor Interface needs a tough edit. It exposes
> >>>>> implementations
> >>>>> marked audience Private and returns implementations rather than
> >>>>> Interfaces.
> >>>>> In a few locations, we allow returning an alternate implementation
> >>>>> altogether which is probably something we don't want a CP doing. To
> >>>>> that
> >>>>> end, the following issues started by Duo and Anoop need to be taken
> to
> >>>>> the
> >>>>> finish line; ideally they'd have an owner:
> >>>>>
> >>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
> >>>>> umbrella issue.
> >>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> >>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
> >>>>> RegionServerObserver
> >>>>>
> >>>>>
> >>>>> Nice-to-haves:
> >>>>>
> >>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
> unused
> >>>>> TimeRange constructors
> >>>>>
> >>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
> >>>>> existing in the code
> >>>>> This is the end of an old long-running project moving up on to Cell
> >>>>> Interface. We think it is done but for a few little items (deprecate
> KV
> >>>>> methods in MR and provide Cell versions instead...)
> >>>>>
> >>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
> >>>>> fixing
> >>>>>
> >>>>> * HBASE-13346 Clean up Filter package for post 1.0
> >>>>>
> >>>>> * HBASE-14255 Simplify Cell creation post 1.0
> >>>>> * HBASE-14997
> >>>>> Move compareOp and Comparators out of filter to client package
> >>>>>
> >>>>> * HBASE-13740 Stop using Hadoop private interfaces
> >>>>>
> >>>>> What about:
> >>>>>
> >>>>> * HBASE-18601 Remove Htrace 3.2
> >>>>> As has been noted, the HTrace API is our 'trace' API.
> >>>>>
> >>>>> If interested in any of the above and you need a legup, just ask in
> the
> >>>>> issue and I'll be by....
> >>>>>
> >>>>> Thanks,
> >>>>> St.Ack
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
> >>>>>
> >>>>> Heads-up:
> >>>>>>
> >>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme
> is
> >>>>>> updated dependencies, reliance on relocated popular libs (guava,
> >>>>>> netty,
> >>>>>> protobuf), purge of checked-in generated src, and
> >>>>>> master-carries-no-regions
> >>>>>> by default.
> >>>>>>
> >>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
> will
> >>>>>> be
> >>>>>> settling the APIs and compatibility (At first blush, we are not
> >>>>>> looking too
> >>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
> >>>>>> running
> >>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
> >>>>>> should be
> >>>>>> done by alpha3 (i.e. returning Interfaces rather than
> >>>>>> Implementations, and
> >>>>>> our shutdown of CPs accessing classes in hbase marked
> >>>>>> InterfaceAudience).
> >>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
> >>>>>> 0.12
> >>>>>> Service showing through in our replication API and protobufs in
> Admin
> >>>>>> Interface. On alpha3, we will have to do a bunch of outreach to make
> >>>>>> sure
> >>>>>> our downstreamers are up on what is coming down the pipe.
> >>>>>>
> >>>>>> Beta1 in mid-September?
> >>>>>>
> >>>>>> I encourage you to check out the items marked for hbase2:
> >>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
> Edit
> >>>>>> as
> >>>>>> you see appropriate. Punt if you know the JIRA will not get any
> >>>>>> attention
> >>>>>> in next month or so.
> >>>>>>
> >>>>>> A bunch of issues marked blocker are unassigned. I'll leave them as
> is
> >>>>>> another while but I'll boot them soon.
> >>>>>>
> >>>>>> While I have your attention:
> >>>>>>
> >>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
> thrift
> >>>>>> to 0.10.0 will break existing clients. The change is easy enough if
> >>>>>> folks
> >>>>>> need to upgrade their hbase thrift. See HBASE-18591.
> >>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> >>>>>> (0.98?).
> >>>>>>
> >>>>>> St.Ack
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> >>>>>>>>
> >>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> ...
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
> >>>>>>>>>> feature a
> >>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
> >>>>>>>>>> reason
> >>>>>>>>>> to be
> >>>>>>>>>> nervous though). I am pretty happy with the feature in terms of
> >>>>>>>>>> how
> >>>>>>>>>> much it
> >>>>>>>>>> is covered via testing.
> >>>>>>>>>>
> >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Last think on space quota says that need doc too. See 'Space
> >>>>>>>>> Quota' in
> >>>>>>>>> here:
> >>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> >>>>>>>>> Does this little section need an update Josh?
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> S
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
> docs
> >>>>>>>> already included for 17748.  Happy to put that on my plate if
> >>>>>>>> you're good
> >>>>>>>> with it. I can reasonably assume that no one is against it :)
> >>>>>>>>
> >>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
> >>>>>>>> merged it in from the original feature branch. I'll double check
> >>>>>>>> and update
> >>>>>>>> the gdoc. Perhaps this was just a timing thing.
> >>>>>>>>
> >>>>>>>>
> >>>>>>> Thanks Josh,
> >>>>>>> S
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Chatting with my coworker Mr. Mike Drob, we were batting back and forth
what remains to be done. Surfacing our thoughts here so you all clued
in....Or if you think otherwise, please speak up.

We have ~13 issues to land:
https://issues.apache.org/jira/projects/HBASE/versions/12341594 About two
are meta-issues that are about process which leaves 11.

Duo and Zheng Hu are to merge the FilterList fixes improvements
(HBASE-19057, HBASE-18410 et al.). These are blocker because some changed
API/semantic that we need to get out earlier rather than later.

Once the above is merged, HBASE-13346, change of Filter method names to
mention Cell instead of KeyValue can land.

HBASE-199048 needs a review (Anoop will probably do it), removing
IA.Private objects as params to MasterObserver... Hopefully this goes in
soon.

Duo is hard at work on trackers for flush and compaction for CPs
(HBASE-18905). How is HBASE-19033 looking Duo (facility for Tephra)?

I think HBASE-18906 (Phoenix Region#waitFor...) will evaporate after Duo is
done w/ his work above.

I'm on HBASE-18770 bypass and HBASE-19077 restore some parity after all the
purges allowing CPs do direct calls against Regions in same Host.

Anoop is on HBASE-19047 (Fixes) and Ram on cleanup of CellUtil.

Another day or two?

St.Ack



On Mon, Oct 23, 2017 at 2:52 PM, Stack <st...@duboce.net> wrote:

>
> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>
>> +1
>>
>> I was trying to work on helping out on the outstanding alpha-4 stuff last
>> week -- will be continuing to try to do the same this week.
>>
>> If you need any help, Stack, or if others need reviews where I haven't
>> noticed on my own: feel free to @mention me.
>>
>>
> Thanks for the offer Josh. All items seem assigned and are being actively
> worked on. If you get a moment, reviews by you (or anyone else) helps move
> the process along.
>
> We need to merge in HBASE-18410 branch to pick up Filter improvements.
> Then HBASE-13346 can go in.
>
> You are already helping out on HBASE-18906, thanks. Looks like that will
> be addressed by other alpha-4s about to land.
>
> St.Ack
> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>
>
>
>
>
>
>
>
>
>> On 10/23/17 12:53 PM, Stack wrote:
>>
>>> (Reviving this thread)
>>>
>>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>>> refactor of the Coprocessor API shutting down access to internals marked
>>> InterfaceAudience.Private.
>>>
>>> The outstanding list is here:
>>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>>
>>> Please push in anything marked alpha-4 that belongs to you.
>>>
>>> If issue, talk out loud on this thread. If you need a review to land an
>>> item, shout on the issue and here; we'll help you out.
>>>
>>> As is, items are coming along nicely I'd say. We need to merge the filter
>>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>>
>>> Post alpha-4, we'll have to hunt down our downstreamers and help them
>>> test
>>> on top of alpha-4 so rolling into beta-1, we have confidence our
>>> downstreamers know what to expect (or we discover what we missed BEFORE
>>> we
>>> beta-1).
>>>
>>> Thanks for time,
>>> S
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>>
>>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
>>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>>
>>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3 but
>>>> it is plain that more time is needed (in spite of valiant effort so far
>>>> by
>>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
>>>> theme is
>>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the following
>>>> week.
>>>>
>>>> We should then be ready for beta (beta == no new features, no API
>>>> changes,
>>>> just fixes).
>>>>
>>>> Thanks,
>>>> St.Ack
>>>>
>>>>
>>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>>
>>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>>>>>
>>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>>>>> release out in the next week or so.
>>>>>
>>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>>> interested in helping out, below are the items I think we need done for
>>>>> alpha3 (below are at least 'Critical' status, are API possibly altering
>>>>> items, and are absent those JIRAs that are making active progress,
>>>>> i.e. the
>>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>>>>> doing is
>>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>>
>>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>>>> branch-2
>>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
>>>>> between branch-1 and branch-2.
>>>>>
>>>>> * HBASE-10462 Recategorize some of the client facing Public / Private
>>>>> interfaces
>>>>> This one is almost done. It could do with a finish, attention to the
>>>>> items in last comment, and then our codebase could do with another
>>>>> sweep
>>>>> after the spirit of this issue since a bunch has gone in since the pass
>>>>> that was the basis of this issue.
>>>>>
>>>>> * HBASE-10504 Define Replication Interface
>>>>> I was going to take a crack at this as part of the revamp forced by
>>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
>>>>> but if
>>>>> anyone else is interested, be my guest.
>>>>>
>>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>>> finishing.
>>>>>
>>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>>>>> cleanup
>>>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
>>>>> and
>>>>> make sure symmetry.
>>>>>
>>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
>>>>> done.
>>>>>
>>>>> Rolling upgrade will have implications for our API. Would be good to
>>>>> try
>>>>> it and figure what needs fixup (as said above, according to trial by
>>>>> Sean,
>>>>> we might not be too bad here):
>>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>>>>> cluster
>>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
>>>>> 1.x
>>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>>
>>>>> * HBASE-17442 Move most of the replication related classes to
>>>>> hbase-server package
>>>>> The above would be good to do generally but it may make for ripples in
>>>>> API so would be good to do now.
>>>>>
>>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>>> Balazs is working on this. The idea is that we avoid adding two new
>>>>> types
>>>>> to our API, two types that are nought but curtailed, read-only views on
>>>>> internals. Input if you have time appreciated.
>>>>>
>>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
>>>>> cluster; verify
>>>>> Esteban is looking at this one
>>>>>
>>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>>> * HBASE-17143 Scan improvement
>>>>>
>>>>> Our Coprocessor Interface needs a tough edit. It exposes
>>>>> implementations
>>>>> marked audience Private and returns implementations rather than
>>>>> Interfaces.
>>>>> In a few locations, we allow returning an alternate implementation
>>>>> altogether which is probably something we don't want a CP doing. To
>>>>> that
>>>>> end, the following issues started by Duo and Anoop need to be taken to
>>>>> the
>>>>> finish line; ideally they'd have an owner:
>>>>>
>>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
>>>>> umbrella issue.
>>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>>>>> RegionServerObserver
>>>>>
>>>>>
>>>>> Nice-to-haves:
>>>>>
>>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
>>>>> TimeRange constructors
>>>>>
>>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>>>> existing in the code
>>>>> This is the end of an old long-running project moving up on to Cell
>>>>> Interface. We think it is done but for a few little items (deprecate KV
>>>>> methods in MR and provide Cell versions instead...)
>>>>>
>>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
>>>>> fixing
>>>>>
>>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>>
>>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>>> * HBASE-14997
>>>>> Move compareOp and Comparators out of filter to client package
>>>>>
>>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>>
>>>>> What about:
>>>>>
>>>>> * HBASE-18601 Remove Htrace 3.2
>>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>>
>>>>> If interested in any of the above and you need a legup, just ask in the
>>>>> issue and I'll be by....
>>>>>
>>>>> Thanks,
>>>>> St.Ack
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>>>
>>>>> Heads-up:
>>>>>>
>>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
>>>>>> updated dependencies, reliance on relocated popular libs (guava,
>>>>>> netty,
>>>>>> protobuf), purge of checked-in generated src, and
>>>>>> master-carries-no-regions
>>>>>> by default.
>>>>>>
>>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme will
>>>>>> be
>>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>>> looking too
>>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>>>>>> running
>>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>>>>>> should be
>>>>>> done by alpha3 (i.e. returning Interfaces rather than
>>>>>> Implementations, and
>>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>>> InterfaceAudience).
>>>>>> We'll also have purged thirdparty classes from our API; e.g. guava
>>>>>> 0.12
>>>>>> Service showing through in our replication API and protobufs in Admin
>>>>>> Interface. On alpha3, we will have to do a bunch of outreach to make
>>>>>> sure
>>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>>
>>>>>> Beta1 in mid-September?
>>>>>>
>>>>>> I encourage you to check out the items marked for hbase2:
>>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit
>>>>>> as
>>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>>> attention
>>>>>> in next month or so.
>>>>>>
>>>>>> A bunch of issues marked blocker are unassigned. I'll leave them as is
>>>>>> another while but I'll boot them soon.
>>>>>>
>>>>>> While I have your attention:
>>>>>>
>>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase thrift
>>>>>> to 0.10.0 will break existing clients. The change is easy enough if
>>>>>> folks
>>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>>> (0.98?).
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>>
>>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>>> feature a
>>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>>>>>>>>>> reason
>>>>>>>>>> to be
>>>>>>>>>> nervous though). I am pretty happy with the feature in terms of
>>>>>>>>>> how
>>>>>>>>>> much it
>>>>>>>>>> is covered via testing.
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Last think on space quota says that need doc too. See 'Space
>>>>>>>>> Quota' in
>>>>>>>>> here:
>>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>>> Does this little section need an update Josh?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> S
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>>>>>>>> already included for 17748.  Happy to put that on my plate if
>>>>>>>> you're good
>>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>>
>>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>>>> merged it in from the original feature branch. I'll double check
>>>>>>>> and update
>>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>>
>>>>>>>>
>>>>>>> Thanks Josh,
>>>>>>> S
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>

Re: Moving 2.0 forward

[ramkrishna vasudevan <ra...@gmail.com>]

Thanks Stack. Started working on it. Will post a patch ASAP.

On Tue, Oct 24, 2017 at 11:38 AM, Stack <st...@duboce.net> wrote:

> That'd be a good one to get in Ram.
> S
>
> On Mon, Oct 23, 2017 at 9:42 PM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
> > Hi Stack
> >
> > Do you want HBASE-18995 before the alpha-4 (REmoving exposed internal
> APIs
> > from CellUtil)? Because you had mentioned no more API changes. If so I
> will
> > start making changes and put up a patch ASAP.
> >
> > Regards
> > Ram
> >
> > On Tue, Oct 24, 2017 at 3:22 AM, Stack <st...@duboce.net> wrote:
> >
> > > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org>
> wrote:
> > >
> > > > +1
> > > >
> > > > I was trying to work on helping out on the outstanding alpha-4 stuff
> > last
> > > > week -- will be continuing to try to do the same this week.
> > > >
> > > > If you need any help, Stack, or if others need reviews where I
> haven't
> > > > noticed on my own: feel free to @mention me.
> > > >
> > > >
> > > Thanks for the offer Josh. All items seem assigned and are being
> actively
> > > worked on. If you get a moment, reviews by you (or anyone else) helps
> > move
> > > the process along.
> > >
> > > We need to merge in HBASE-18410 branch to pick up Filter improvements.
> > Then
> > > HBASE-13346 can go in.
> > >
> > > You are already helping out on HBASE-18906, thanks. Looks like that
> will
> > be
> > > addressed by other alpha-4s about to land.
> > >
> > > St.Ack
> > > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > > On 10/23/17 12:53 PM, Stack wrote:
> > > >
> > > >> (Reviving this thread)
> > > >>
> > > >> Lets push out alpha-4 this week. Alpha-4 is the release that has the
> > > >> refactor of the Coprocessor API shutting down access to internals
> > marked
> > > >> InterfaceAudience.Private.
> > > >>
> > > >> The outstanding list is here:
> > > >> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > > >>
> > > >> Please push in anything marked alpha-4 that belongs to you.
> > > >>
> > > >> If issue, talk out loud on this thread. If you need a review to land
> > an
> > > >> item, shout on the issue and here; we'll help you out.
> > > >>
> > > >> As is, items are coming along nicely I'd say. We need to merge the
> > > filter
> > > >> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> > > >>
> > > >> Post alpha-4, we'll have to hunt down our downstreamers and help
> them
> > > test
> > > >> on top of alpha-4 so rolling into beta-1, we have confidence our
> > > >> downstreamers know what to expect (or we discover what we missed
> > BEFORE
> > > we
> > > >> beta-1).
> > > >>
> > > >> Thanks for time,
> > > >> S
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> > > >>
> > > >> I'll put up an alpha3 RC Monday, probably Monday night. That should
> be
> > > >>> time, if we all sprint, for the public-facing API fixes to be done.
> > > >>>
> > > >>> I had a bunch of Coprocessor refactor and fixup scheduled for
> alpha3
> > > but
> > > >>> it is plain that more time is needed (in spite of valiant effort so
> > far
> > > >>> by
> > > >>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
> > > theme
> > > >>> is
> > > >>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> > > following
> > > >>> week.
> > > >>>
> > > >>> We should then be ready for beta (beta == no new features, no API
> > > >>> changes,
> > > >>> just fixes).
> > > >>>
> > > >>> Thanks,
> > > >>> St.Ack
> > > >>>
> > > >>>
> > > >>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
> > > >>>
> > > >>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on
> it.
> > > >>>>
> > > >>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to
> get
> > a
> > > >>>> release out in the next week or so.
> > > >>>>
> > > >>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> > > >>>> interested in helping out, below are the items I think we need
> done
> > > for
> > > >>>> alpha3 (below are at least 'Critical' status, are API possibly
> > > altering
> > > >>>> items, and are absent those JIRAs that are making active progress,
> > > i.e.
> > > >>>> the
> > > >>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
> > > >>>> doing is
> > > >>>> what Andrew did comparing 1.3. and 1.4 APIs
> > > >>>>
> > > >>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
> > > >>>> branch-2
> > > >>>> This is to do what Andrew did between 1.3 and 1.4 branches only do
> > it
> > > >>>> between branch-1 and branch-2.
> > > >>>>
> > > >>>> * HBASE-10462 Recategorize some of the client facing Public /
> > Private
> > > >>>> interfaces
> > > >>>> This one is almost done. It could do with a finish, attention to
> the
> > > >>>> items in last comment, and then our codebase could do with another
> > > sweep
> > > >>>> after the spirit of this issue since a bunch has gone in since the
> > > pass
> > > >>>> that was the basis of this issue.
> > > >>>>
> > > >>>> * HBASE-10504 Define Replication Interface
> > > >>>> I was going to take a crack at this as part of the revamp forced
> by
> > > >>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's
> Service'
> > > but
> > > >>>> if
> > > >>>> anyone else is interested, be my guest.
> > > >>>>
> > > >>>> * HBASE-14996 Some more API cleanup for 2.0
> > > >>>> Has a bunch of subtasks, some of which are being worked on. Needs
> > > >>>> finishing.
> > > >>>>
> > > >>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
> > and
> > > >>>> cleanup
> > > >>>> Needs a pass. Small issue I think. Could also look at new
> > AsyncClient
> > > >>>> and
> > > >>>> make sure symmetry.
> > > >>>>
> > > >>>> * HBASE-15607 Remove PB references from Admin for 2.0
> > > >>>> Predicated on result of an ongoing DISCUSSION thread but needs to
> be
> > > >>>> done.
> > > >>>>
> > > >>>> Rolling upgrade will have implications for our API. Would be good
> to
> > > try
> > > >>>> it and figure what needs fixup (as said above, according to trial
> by
> > > >>>> Sean,
> > > >>>> we might not be too bad here):
> > > >>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
> > > >>>> cluster
> > > >>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master
> and
> > > 1.x
> > > >>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> > > >>>>
> > > >>>> * HBASE-17442 Move most of the replication related classes to
> > > >>>> hbase-server package
> > > >>>> The above would be good to do generally but it may make for
> ripples
> > in
> > > >>>> API so would be good to do now.
> > > >>>>
> > > >>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> > > >>>> Balazs is working on this. The idea is that we avoid adding two
> new
> > > >>>> types
> > > >>>> to our API, two types that are nought but curtailed, read-only
> views
> > > on
> > > >>>> internals. Input if you have time appreciated.
> > > >>>>
> > > >>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
> > hbase2
> > > >>>> cluster; verify
> > > >>>> Esteban is looking at this one
> > > >>>>
> > > >>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> > > >>>> * HBASE-17143 Scan improvement
> > > >>>>
> > > >>>> Our Coprocessor Interface needs a tough edit. It exposes
> > > implementations
> > > >>>> marked audience Private and returns implementations rather than
> > > >>>> Interfaces.
> > > >>>> In a few locations, we allow returning an alternate implementation
> > > >>>> altogether which is probably something we don't want a CP doing.
> To
> > > that
> > > >>>> end, the following issues started by Duo and Anoop need to be
> taken
> > to
> > > >>>> the
> > > >>>> finish line; ideally they'd have an owner:
> > > >>>>
> > > >>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
> > The
> > > >>>> umbrella issue.
> > > >>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> > > >>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver
> and
> > > >>>> RegionServerObserver
> > > >>>>
> > > >>>>
> > > >>>> Nice-to-haves:
> > > >>>>
> > > >>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
> > unused
> > > >>>> TimeRange constructors
> > > >>>>
> > > >>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
> > > >>>> existing in the code
> > > >>>> This is the end of an old long-running project moving up on to
> Cell
> > > >>>> Interface. We think it is done but for a few little items
> (deprecate
> > > KV
> > > >>>> methods in MR and provide Cell versions instead...)
> > > >>>>
> > > >>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
> > needs
> > > >>>> fixing
> > > >>>>
> > > >>>> * HBASE-13346 Clean up Filter package for post 1.0
> > > >>>>
> > > >>>> * HBASE-14255 Simplify Cell creation post 1.0
> > > >>>> * HBASE-14997
> > > >>>> Move compareOp and Comparators out of filter to client package
> > > >>>>
> > > >>>> * HBASE-13740 Stop using Hadoop private interfaces
> > > >>>>
> > > >>>> What about:
> > > >>>>
> > > >>>> * HBASE-18601 Remove Htrace 3.2
> > > >>>> As has been noted, the HTrace API is our 'trace' API.
> > > >>>>
> > > >>>> If interested in any of the above and you need a legup, just ask
> in
> > > the
> > > >>>> issue and I'll be by....
> > > >>>>
> > > >>>> Thanks,
> > > >>>> St.Ack
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
> > > >>>>
> > > >>>> Heads-up:
> > > >>>>>
> > > >>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate.
> Theme
> > is
> > > >>>>> updated dependencies, reliance on relocated popular libs (guava,
> > > netty,
> > > >>>>> protobuf), purge of checked-in generated src, and
> > > >>>>> master-carries-no-regions
> > > >>>>> by default.
> > > >>>>>
> > > >>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
> > will
> > > >>>>> be
> > > >>>>> settling the APIs and compatibility (At first blush, we are not
> > > >>>>> looking too
> > > >>>>> bad; our Sean ran some tests over weekend that have hbase-1
> client
> > > >>>>> running
> > > >>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
> > > >>>>> should be
> > > >>>>> done by alpha3 (i.e. returning Interfaces rather than
> > > Implementations,
> > > >>>>> and
> > > >>>>> our shutdown of CPs accessing classes in hbase marked
> > > >>>>> InterfaceAudience).
> > > >>>>> We'll also have purged thirdparty classes from our API; e.g.
> guava
> > > 0.12
> > > >>>>> Service showing through in our replication API and protobufs in
> > Admin
> > > >>>>> Interface. On alpha3, we will have to do a bunch of outreach to
> > make
> > > >>>>> sure
> > > >>>>> our downstreamers are up on what is coming down the pipe.
> > > >>>>>
> > > >>>>> Beta1 in mid-September?
> > > >>>>>
> > > >>>>> I encourage you to check out the items marked for hbase2:
> > > >>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
> > Edit
> > > >>>>> as
> > > >>>>> you see appropriate. Punt if you know the JIRA will not get any
> > > >>>>> attention
> > > >>>>> in next month or so.
> > > >>>>>
> > > >>>>> A bunch of issues marked blocker are unassigned. I'll leave them
> as
> > > is
> > > >>>>> another while but I'll boot them soon.
> > > >>>>>
> > > >>>>> While I have your attention:
> > > >>>>>
> > > >>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
> > > thrift
> > > >>>>> to 0.10.0 will break existing clients. The change is easy enough
> if
> > > >>>>> folks
> > > >>>>> need to upgrade their hbase thrift. See HBASE-18591.
> > > >>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> > > >>>>> (0.98?).
> > > >>>>>
> > > >>>>> St.Ack
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> > > >>>>>
> > > >>>>>
> > > >>>>>>
> > > >>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
> > > wrote:
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> > > >>>>>>>
> > > >>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<elserj@apache.org
> >
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> ...
> > > >>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
> > > >>>>>>>>> feature a
> > > >>>>>>>>> bit more "real", but am obviously a little nervous (I have no
> > > >>>>>>>>> reason
> > > >>>>>>>>> to be
> > > >>>>>>>>> nervous though). I am pretty happy with the feature in terms
> of
> > > how
> > > >>>>>>>>> much it
> > > >>>>>>>>> is covered via testing.
> > > >>>>>>>>>
> > > >>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Last think on space quota says that need doc too. See 'Space
> > > Quota'
> > > >>>>>>>> in
> > > >>>>>>>> here:
> > > >>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> > > >>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > > >>>>>>>> Does this little section need an update Josh?
> > > >>>>>>>>
> > > >>>>>>>> Thanks,
> > > >>>>>>>> S
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
> > > docs
> > > >>>>>>> already included for 17748.  Happy to put that on my plate if
> > > you're
> > > >>>>>>> good
> > > >>>>>>> with it. I can reasonably assume that no one is against it :)
> > > >>>>>>>
> > > >>>>>>> I think I had knocked out docs for the "phase 1" stuff before
> we
> > > >>>>>>> merged it in from the original feature branch. I'll double
> check
> > > and
> > > >>>>>>> update
> > > >>>>>>> the gdoc. Perhaps this was just a timing thing.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>> Thanks Josh,
> > > >>>>>> S
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

That'd be a good one to get in Ram.
S

On Mon, Oct 23, 2017 at 9:42 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:

> Hi Stack
>
> Do you want HBASE-18995 before the alpha-4 (REmoving exposed internal APIs
> from CellUtil)? Because you had mentioned no more API changes. If so I will
> start making changes and put up a patch ASAP.
>
> Regards
> Ram
>
> On Tue, Oct 24, 2017 at 3:22 AM, Stack <st...@duboce.net> wrote:
>
> > On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
> >
> > > +1
> > >
> > > I was trying to work on helping out on the outstanding alpha-4 stuff
> last
> > > week -- will be continuing to try to do the same this week.
> > >
> > > If you need any help, Stack, or if others need reviews where I haven't
> > > noticed on my own: feel free to @mention me.
> > >
> > >
> > Thanks for the offer Josh. All items seem assigned and are being actively
> > worked on. If you get a moment, reviews by you (or anyone else) helps
> move
> > the process along.
> >
> > We need to merge in HBASE-18410 branch to pick up Filter improvements.
> Then
> > HBASE-13346 can go in.
> >
> > You are already helping out on HBASE-18906, thanks. Looks like that will
> be
> > addressed by other alpha-4s about to land.
> >
> > St.Ack
> > TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > > On 10/23/17 12:53 PM, Stack wrote:
> > >
> > >> (Reviving this thread)
> > >>
> > >> Lets push out alpha-4 this week. Alpha-4 is the release that has the
> > >> refactor of the Coprocessor API shutting down access to internals
> marked
> > >> InterfaceAudience.Private.
> > >>
> > >> The outstanding list is here:
> > >> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> > >>
> > >> Please push in anything marked alpha-4 that belongs to you.
> > >>
> > >> If issue, talk out loud on this thread. If you need a review to land
> an
> > >> item, shout on the issue and here; we'll help you out.
> > >>
> > >> As is, items are coming along nicely I'd say. We need to merge the
> > filter
> > >> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> > >>
> > >> Post alpha-4, we'll have to hunt down our downstreamers and help them
> > test
> > >> on top of alpha-4 so rolling into beta-1, we have confidence our
> > >> downstreamers know what to expect (or we discover what we missed
> BEFORE
> > we
> > >> beta-1).
> > >>
> > >> Thanks for time,
> > >> S
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> > >>
> > >> I'll put up an alpha3 RC Monday, probably Monday night. That should be
> > >>> time, if we all sprint, for the public-facing API fixes to be done.
> > >>>
> > >>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
> > but
> > >>> it is plain that more time is needed (in spite of valiant effort so
> far
> > >>> by
> > >>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
> > theme
> > >>> is
> > >>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> > following
> > >>> week.
> > >>>
> > >>> We should then be ready for beta (beta == no new features, no API
> > >>> changes,
> > >>> just fixes).
> > >>>
> > >>> Thanks,
> > >>> St.Ack
> > >>>
> > >>>
> > >>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
> > >>>
> > >>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
> > >>>>
> > >>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get
> a
> > >>>> release out in the next week or so.
> > >>>>
> > >>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> > >>>> interested in helping out, below are the items I think we need done
> > for
> > >>>> alpha3 (below are at least 'Critical' status, are API possibly
> > altering
> > >>>> items, and are absent those JIRAs that are making active progress,
> > i.e.
> > >>>> the
> > >>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
> > >>>> doing is
> > >>>> what Andrew did comparing 1.3. and 1.4 APIs
> > >>>>
> > >>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
> > >>>> branch-2
> > >>>> This is to do what Andrew did between 1.3 and 1.4 branches only do
> it
> > >>>> between branch-1 and branch-2.
> > >>>>
> > >>>> * HBASE-10462 Recategorize some of the client facing Public /
> Private
> > >>>> interfaces
> > >>>> This one is almost done. It could do with a finish, attention to the
> > >>>> items in last comment, and then our codebase could do with another
> > sweep
> > >>>> after the spirit of this issue since a bunch has gone in since the
> > pass
> > >>>> that was the basis of this issue.
> > >>>>
> > >>>> * HBASE-10504 Define Replication Interface
> > >>>> I was going to take a crack at this as part of the revamp forced by
> > >>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
> > but
> > >>>> if
> > >>>> anyone else is interested, be my guest.
> > >>>>
> > >>>> * HBASE-14996 Some more API cleanup for 2.0
> > >>>> Has a bunch of subtasks, some of which are being worked on. Needs
> > >>>> finishing.
> > >>>>
> > >>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin
> and
> > >>>> cleanup
> > >>>> Needs a pass. Small issue I think. Could also look at new
> AsyncClient
> > >>>> and
> > >>>> make sure symmetry.
> > >>>>
> > >>>> * HBASE-15607 Remove PB references from Admin for 2.0
> > >>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
> > >>>> done.
> > >>>>
> > >>>> Rolling upgrade will have implications for our API. Would be good to
> > try
> > >>>> it and figure what needs fixup (as said above, according to trial by
> > >>>> Sean,
> > >>>> we might not be too bad here):
> > >>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
> > >>>> cluster
> > >>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
> > 1.x
> > >>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> > >>>>
> > >>>> * HBASE-17442 Move most of the replication related classes to
> > >>>> hbase-server package
> > >>>> The above would be good to do generally but it may make for ripples
> in
> > >>>> API so would be good to do now.
> > >>>>
> > >>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> > >>>> Balazs is working on this. The idea is that we avoid adding two new
> > >>>> types
> > >>>> to our API, two types that are nought but curtailed, read-only views
> > on
> > >>>> internals. Input if you have time appreciated.
> > >>>>
> > >>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a
> hbase2
> > >>>> cluster; verify
> > >>>> Esteban is looking at this one
> > >>>>
> > >>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> > >>>> * HBASE-17143 Scan improvement
> > >>>>
> > >>>> Our Coprocessor Interface needs a tough edit. It exposes
> > implementations
> > >>>> marked audience Private and returns implementations rather than
> > >>>> Interfaces.
> > >>>> In a few locations, we allow returning an alternate implementation
> > >>>> altogether which is probably something we don't want a CP doing. To
> > that
> > >>>> end, the following issues started by Duo and Anoop need to be taken
> to
> > >>>> the
> > >>>> finish line; ideally they'd have an owner:
> > >>>>
> > >>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <=
> The
> > >>>> umbrella issue.
> > >>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> > >>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
> > >>>> RegionServerObserver
> > >>>>
> > >>>>
> > >>>> Nice-to-haves:
> > >>>>
> > >>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove
> unused
> > >>>> TimeRange constructors
> > >>>>
> > >>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
> > >>>> existing in the code
> > >>>> This is the end of an old long-running project moving up on to Cell
> > >>>> Interface. We think it is done but for a few little items (deprecate
> > KV
> > >>>> methods in MR and provide Cell versions instead...)
> > >>>>
> > >>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate;
> needs
> > >>>> fixing
> > >>>>
> > >>>> * HBASE-13346 Clean up Filter package for post 1.0
> > >>>>
> > >>>> * HBASE-14255 Simplify Cell creation post 1.0
> > >>>> * HBASE-14997
> > >>>> Move compareOp and Comparators out of filter to client package
> > >>>>
> > >>>> * HBASE-13740 Stop using Hadoop private interfaces
> > >>>>
> > >>>> What about:
> > >>>>
> > >>>> * HBASE-18601 Remove Htrace 3.2
> > >>>> As has been noted, the HTrace API is our 'trace' API.
> > >>>>
> > >>>> If interested in any of the above and you need a legup, just ask in
> > the
> > >>>> issue and I'll be by....
> > >>>>
> > >>>> Thanks,
> > >>>> St.Ack
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
> > >>>>
> > >>>> Heads-up:
> > >>>>>
> > >>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme
> is
> > >>>>> updated dependencies, reliance on relocated popular libs (guava,
> > netty,
> > >>>>> protobuf), purge of checked-in generated src, and
> > >>>>> master-carries-no-regions
> > >>>>> by default.
> > >>>>>
> > >>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme
> will
> > >>>>> be
> > >>>>> settling the APIs and compatibility (At first blush, we are not
> > >>>>> looking too
> > >>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
> > >>>>> running
> > >>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
> > >>>>> should be
> > >>>>> done by alpha3 (i.e. returning Interfaces rather than
> > Implementations,
> > >>>>> and
> > >>>>> our shutdown of CPs accessing classes in hbase marked
> > >>>>> InterfaceAudience).
> > >>>>> We'll also have purged thirdparty classes from our API; e.g. guava
> > 0.12
> > >>>>> Service showing through in our replication API and protobufs in
> Admin
> > >>>>> Interface. On alpha3, we will have to do a bunch of outreach to
> make
> > >>>>> sure
> > >>>>> our downstreamers are up on what is coming down the pipe.
> > >>>>>
> > >>>>> Beta1 in mid-September?
> > >>>>>
> > >>>>> I encourage you to check out the items marked for hbase2:
> > >>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188
> Edit
> > >>>>> as
> > >>>>> you see appropriate. Punt if you know the JIRA will not get any
> > >>>>> attention
> > >>>>> in next month or so.
> > >>>>>
> > >>>>> A bunch of issues marked blocker are unassigned. I'll leave them as
> > is
> > >>>>> another while but I'll boot them soon.
> > >>>>>
> > >>>>> While I have your attention:
> > >>>>>
> > >>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
> > thrift
> > >>>>> to 0.10.0 will break existing clients. The change is easy enough if
> > >>>>> folks
> > >>>>> need to upgrade their hbase thrift. See HBASE-18591.
> > >>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> > >>>>> (0.98?).
> > >>>>>
> > >>>>> St.Ack
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> > >>>>>
> > >>>>>
> > >>>>>>
> > >>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
> > wrote:
> > >>>>>>
> > >>>>>>
> > >>>>>>>
> > >>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> > >>>>>>>
> > >>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> ...
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
> > >>>>>>>>> feature a
> > >>>>>>>>> bit more "real", but am obviously a little nervous (I have no
> > >>>>>>>>> reason
> > >>>>>>>>> to be
> > >>>>>>>>> nervous though). I am pretty happy with the feature in terms of
> > how
> > >>>>>>>>> much it
> > >>>>>>>>> is covered via testing.
> > >>>>>>>>>
> > >>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Last think on space quota says that need doc too. See 'Space
> > Quota'
> > >>>>>>>> in
> > >>>>>>>> here:
> > >>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> > >>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > >>>>>>>> Does this little section need an update Josh?
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> S
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
> > docs
> > >>>>>>> already included for 17748.  Happy to put that on my plate if
> > you're
> > >>>>>>> good
> > >>>>>>> with it. I can reasonably assume that no one is against it :)
> > >>>>>>>
> > >>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
> > >>>>>>> merged it in from the original feature branch. I'll double check
> > and
> > >>>>>>> update
> > >>>>>>> the gdoc. Perhaps this was just a timing thing.
> > >>>>>>>
> > >>>>>>>
> > >>>>>> Thanks Josh,
> > >>>>>> S
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
>

Re: Moving 2.0 forward

[ramkrishna vasudevan <ra...@gmail.com>]

Hi Stack

Do you want HBASE-18995 before the alpha-4 (REmoving exposed internal APIs
from CellUtil)? Because you had mentioned no more API changes. If so I will
start making changes and put up a patch ASAP.

Regards
Ram

On Tue, Oct 24, 2017 at 3:22 AM, Stack <st...@duboce.net> wrote:

> On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:
>
> > +1
> >
> > I was trying to work on helping out on the outstanding alpha-4 stuff last
> > week -- will be continuing to try to do the same this week.
> >
> > If you need any help, Stack, or if others need reviews where I haven't
> > noticed on my own: feel free to @mention me.
> >
> >
> Thanks for the offer Josh. All items seem assigned and are being actively
> worked on. If you get a moment, reviews by you (or anyone else) helps move
> the process along.
>
> We need to merge in HBASE-18410 branch to pick up Filter improvements. Then
> HBASE-13346 can go in.
>
> You are already helping out on HBASE-18906, thanks. Looks like that will be
> addressed by other alpha-4s about to land.
>
> St.Ack
> TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594
>
>
>
>
>
>
>
>
>
> > On 10/23/17 12:53 PM, Stack wrote:
> >
> >> (Reviving this thread)
> >>
> >> Lets push out alpha-4 this week. Alpha-4 is the release that has the
> >> refactor of the Coprocessor API shutting down access to internals marked
> >> InterfaceAudience.Private.
> >>
> >> The outstanding list is here:
> >> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> >>
> >> Please push in anything marked alpha-4 that belongs to you.
> >>
> >> If issue, talk out loud on this thread. If you need a review to land an
> >> item, shout on the issue and here; we'll help you out.
> >>
> >> As is, items are coming along nicely I'd say. We need to merge the
> filter
> >> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> >>
> >> Post alpha-4, we'll have to hunt down our downstreamers and help them
> test
> >> on top of alpha-4 so rolling into beta-1, we have confidence our
> >> downstreamers know what to expect (or we discover what we missed BEFORE
> we
> >> beta-1).
> >>
> >> Thanks for time,
> >> S
> >>
> >>
> >>
> >>
> >>
> >> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> >>
> >> I'll put up an alpha3 RC Monday, probably Monday night. That should be
> >>> time, if we all sprint, for the public-facing API fixes to be done.
> >>>
> >>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3
> but
> >>> it is plain that more time is needed (in spite of valiant effort so far
> >>> by
> >>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose
> theme
> >>> is
> >>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the
> following
> >>> week.
> >>>
> >>> We should then be ready for beta (beta == no new features, no API
> >>> changes,
> >>> just fixes).
> >>>
> >>> Thanks,
> >>> St.Ack
> >>>
> >>>
> >>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
> >>>
> >>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
> >>>>
> >>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
> >>>> release out in the next week or so.
> >>>>
> >>>> I did a weeding of 2.0.0 issues over the last day. If folks are
> >>>> interested in helping out, below are the items I think we need done
> for
> >>>> alpha3 (below are at least 'Critical' status, are API possibly
> altering
> >>>> items, and are absent those JIRAs that are making active progress,
> i.e.
> >>>> the
> >>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
> >>>> doing is
> >>>> what Andrew did comparing 1.3. and 1.4 APIs
> >>>>
> >>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
> >>>> branch-2
> >>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
> >>>> between branch-1 and branch-2.
> >>>>
> >>>> * HBASE-10462 Recategorize some of the client facing Public / Private
> >>>> interfaces
> >>>> This one is almost done. It could do with a finish, attention to the
> >>>> items in last comment, and then our codebase could do with another
> sweep
> >>>> after the spirit of this issue since a bunch has gone in since the
> pass
> >>>> that was the basis of this issue.
> >>>>
> >>>> * HBASE-10504 Define Replication Interface
> >>>> I was going to take a crack at this as part of the revamp forced by
> >>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service'
> but
> >>>> if
> >>>> anyone else is interested, be my guest.
> >>>>
> >>>> * HBASE-14996 Some more API cleanup for 2.0
> >>>> Has a bunch of subtasks, some of which are being worked on. Needs
> >>>> finishing.
> >>>>
> >>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
> >>>> cleanup
> >>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
> >>>> and
> >>>> make sure symmetry.
> >>>>
> >>>> * HBASE-15607 Remove PB references from Admin for 2.0
> >>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
> >>>> done.
> >>>>
> >>>> Rolling upgrade will have implications for our API. Would be good to
> try
> >>>> it and figure what needs fixup (as said above, according to trial by
> >>>> Sean,
> >>>> we might not be too bad here):
> >>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
> >>>> cluster
> >>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and
> 1.x
> >>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> >>>>
> >>>> * HBASE-17442 Move most of the replication related classes to
> >>>> hbase-server package
> >>>> The above would be good to do generally but it may make for ripples in
> >>>> API so would be good to do now.
> >>>>
> >>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
> >>>> Balazs is working on this. The idea is that we avoid adding two new
> >>>> types
> >>>> to our API, two types that are nought but curtailed, read-only views
> on
> >>>> internals. Input if you have time appreciated.
> >>>>
> >>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
> >>>> cluster; verify
> >>>> Esteban is looking at this one
> >>>>
> >>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> >>>> * HBASE-17143 Scan improvement
> >>>>
> >>>> Our Coprocessor Interface needs a tough edit. It exposes
> implementations
> >>>> marked audience Private and returns implementations rather than
> >>>> Interfaces.
> >>>> In a few locations, we allow returning an alternate implementation
> >>>> altogether which is probably something we don't want a CP doing. To
> that
> >>>> end, the following issues started by Duo and Anoop need to be taken to
> >>>> the
> >>>> finish line; ideally they'd have an owner:
> >>>>
> >>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
> >>>> umbrella issue.
> >>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> >>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
> >>>> RegionServerObserver
> >>>>
> >>>>
> >>>> Nice-to-haves:
> >>>>
> >>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
> >>>> TimeRange constructors
> >>>>
> >>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
> >>>> existing in the code
> >>>> This is the end of an old long-running project moving up on to Cell
> >>>> Interface. We think it is done but for a few little items (deprecate
> KV
> >>>> methods in MR and provide Cell versions instead...)
> >>>>
> >>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
> >>>> fixing
> >>>>
> >>>> * HBASE-13346 Clean up Filter package for post 1.0
> >>>>
> >>>> * HBASE-14255 Simplify Cell creation post 1.0
> >>>> * HBASE-14997
> >>>> Move compareOp and Comparators out of filter to client package
> >>>>
> >>>> * HBASE-13740 Stop using Hadoop private interfaces
> >>>>
> >>>> What about:
> >>>>
> >>>> * HBASE-18601 Remove Htrace 3.2
> >>>> As has been noted, the HTrace API is our 'trace' API.
> >>>>
> >>>> If interested in any of the above and you need a legup, just ask in
> the
> >>>> issue and I'll be by....
> >>>>
> >>>> Thanks,
> >>>> St.Ack
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
> >>>>
> >>>> Heads-up:
> >>>>>
> >>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
> >>>>> updated dependencies, reliance on relocated popular libs (guava,
> netty,
> >>>>> protobuf), purge of checked-in generated src, and
> >>>>> master-carries-no-regions
> >>>>> by default.
> >>>>>
> >>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme will
> >>>>> be
> >>>>> settling the APIs and compatibility (At first blush, we are not
> >>>>> looking too
> >>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
> >>>>> running
> >>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
> >>>>> should be
> >>>>> done by alpha3 (i.e. returning Interfaces rather than
> Implementations,
> >>>>> and
> >>>>> our shutdown of CPs accessing classes in hbase marked
> >>>>> InterfaceAudience).
> >>>>> We'll also have purged thirdparty classes from our API; e.g. guava
> 0.12
> >>>>> Service showing through in our replication API and protobufs in Admin
> >>>>> Interface. On alpha3, we will have to do a bunch of outreach to make
> >>>>> sure
> >>>>> our downstreamers are up on what is coming down the pipe.
> >>>>>
> >>>>> Beta1 in mid-September?
> >>>>>
> >>>>> I encourage you to check out the items marked for hbase2:
> >>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit
> >>>>> as
> >>>>> you see appropriate. Punt if you know the JIRA will not get any
> >>>>> attention
> >>>>> in next month or so.
> >>>>>
> >>>>> A bunch of issues marked blocker are unassigned. I'll leave them as
> is
> >>>>> another while but I'll boot them soon.
> >>>>>
> >>>>> While I have your attention:
> >>>>>
> >>>>> + I think we should leave thrift version at 0.9.3. Moving hbase
> thrift
> >>>>> to 0.10.0 will break existing clients. The change is easy enough if
> >>>>> folks
> >>>>> need to upgrade their hbase thrift. See HBASE-18591.
> >>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> >>>>> (0.98?).
> >>>>>
> >>>>> St.Ack
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org>
> wrote:
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> On 7/31/17 9:00 AM, Stack wrote:
> >>>>>>>
> >>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> ...
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
> >>>>>>>>> feature a
> >>>>>>>>> bit more "real", but am obviously a little nervous (I have no
> >>>>>>>>> reason
> >>>>>>>>> to be
> >>>>>>>>> nervous though). I am pretty happy with the feature in terms of
> how
> >>>>>>>>> much it
> >>>>>>>>> is covered via testing.
> >>>>>>>>>
> >>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> Last think on space quota says that need doc too. See 'Space
> Quota'
> >>>>>>>> in
> >>>>>>>> here:
> >>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> >>>>>>>> Does this little section need an update Josh?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> S
> >>>>>>>>
> >>>>>>>>
> >>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some
> docs
> >>>>>>> already included for 17748.  Happy to put that on my plate if
> you're
> >>>>>>> good
> >>>>>>> with it. I can reasonably assume that no one is against it :)
> >>>>>>>
> >>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
> >>>>>>> merged it in from the original feature branch. I'll double check
> and
> >>>>>>> update
> >>>>>>> the gdoc. Perhaps this was just a timing thing.
> >>>>>>>
> >>>>>>>
> >>>>>> Thanks Josh,
> >>>>>> S
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Mon, Oct 23, 2017 at 11:59 AM, Josh Elser <el...@apache.org> wrote:

> +1
>
> I was trying to work on helping out on the outstanding alpha-4 stuff last
> week -- will be continuing to try to do the same this week.
>
> If you need any help, Stack, or if others need reviews where I haven't
> noticed on my own: feel free to @mention me.
>
>
Thanks for the offer Josh. All items seem assigned and are being actively
worked on. If you get a moment, reviews by you (or anyone else) helps move
the process along.

We need to merge in HBASE-18410 branch to pick up Filter improvements. Then
HBASE-13346 can go in.

You are already helping out on HBASE-18906, thanks. Looks like that will be
addressed by other alpha-4s about to land.

St.Ack
TODOs: https://issues.apache.org/jira/projects/HBASE/versions/12341594









> On 10/23/17 12:53 PM, Stack wrote:
>
>> (Reviving this thread)
>>
>> Lets push out alpha-4 this week. Alpha-4 is the release that has the
>> refactor of the Coprocessor API shutting down access to internals marked
>> InterfaceAudience.Private.
>>
>> The outstanding list is here:
>> https://issues.apache.org/jira/projects/HBASE/versions/12341594
>>
>> Please push in anything marked alpha-4 that belongs to you.
>>
>> If issue, talk out loud on this thread. If you need a review to land an
>> item, shout on the issue and here; we'll help you out.
>>
>> As is, items are coming along nicely I'd say. We need to merge the filter
>> branch -- HBASE-18410 -- so APIs are finished for hbase2.
>>
>> Post alpha-4, we'll have to hunt down our downstreamers and help them test
>> on top of alpha-4 so rolling into beta-1, we have confidence our
>> downstreamers know what to expect (or we discover what we missed BEFORE we
>> beta-1).
>>
>> Thanks for time,
>> S
>>
>>
>>
>>
>>
>> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
>>
>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
>>> time, if we all sprint, for the public-facing API fixes to be done.
>>>
>>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3 but
>>> it is plain that more time is needed (in spite of valiant effort so far
>>> by
>>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose theme
>>> is
>>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the following
>>> week.
>>>
>>> We should then be ready for beta (beta == no new features, no API
>>> changes,
>>> just fixes).
>>>
>>> Thanks,
>>> St.Ack
>>>
>>>
>>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>>
>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>>>>
>>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>>>> release out in the next week or so.
>>>>
>>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>>> interested in helping out, below are the items I think we need done for
>>>> alpha3 (below are at least 'Critical' status, are API possibly altering
>>>> items, and are absent those JIRAs that are making active progress, i.e.
>>>> the
>>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs
>>>> doing is
>>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>>
>>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>>> branch-2
>>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
>>>> between branch-1 and branch-2.
>>>>
>>>> * HBASE-10462 Recategorize some of the client facing Public / Private
>>>> interfaces
>>>> This one is almost done. It could do with a finish, attention to the
>>>> items in last comment, and then our codebase could do with another sweep
>>>> after the spirit of this issue since a bunch has gone in since the pass
>>>> that was the basis of this issue.
>>>>
>>>> * HBASE-10504 Define Replication Interface
>>>> I was going to take a crack at this as part of the revamp forced by
>>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service' but
>>>> if
>>>> anyone else is interested, be my guest.
>>>>
>>>> * HBASE-14996 Some more API cleanup for 2.0
>>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>>> finishing.
>>>>
>>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>>>> cleanup
>>>> Needs a pass. Small issue I think. Could also look at new AsyncClient
>>>> and
>>>> make sure symmetry.
>>>>
>>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>>> Predicated on result of an ongoing DISCUSSION thread but needs to be
>>>> done.
>>>>
>>>> Rolling upgrade will have implications for our API. Would be good to try
>>>> it and figure what needs fixup (as said above, according to trial by
>>>> Sean,
>>>> we might not be too bad here):
>>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0
>>>> cluster
>>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and 1.x
>>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>>
>>>> * HBASE-17442 Move most of the replication related classes to
>>>> hbase-server package
>>>> The above would be good to do generally but it may make for ripples in
>>>> API so would be good to do now.
>>>>
>>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>>> Balazs is working on this. The idea is that we avoid adding two new
>>>> types
>>>> to our API, two types that are nought but curtailed, read-only views on
>>>> internals. Input if you have time appreciated.
>>>>
>>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
>>>> cluster; verify
>>>> Esteban is looking at this one
>>>>
>>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>>> * HBASE-17143 Scan improvement
>>>>
>>>> Our Coprocessor Interface needs a tough edit. It exposes implementations
>>>> marked audience Private and returns implementations rather than
>>>> Interfaces.
>>>> In a few locations, we allow returning an alternate implementation
>>>> altogether which is probably something we don't want a CP doing. To that
>>>> end, the following issues started by Duo and Anoop need to be taken to
>>>> the
>>>> finish line; ideally they'd have an owner:
>>>>
>>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
>>>> umbrella issue.
>>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>>>> RegionServerObserver
>>>>
>>>>
>>>> Nice-to-haves:
>>>>
>>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
>>>> TimeRange constructors
>>>>
>>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>>> existing in the code
>>>> This is the end of an old long-running project moving up on to Cell
>>>> Interface. We think it is done but for a few little items (deprecate KV
>>>> methods in MR and provide Cell versions instead...)
>>>>
>>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
>>>> fixing
>>>>
>>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>>
>>>> * HBASE-14255 Simplify Cell creation post 1.0
>>>> * HBASE-14997
>>>> Move compareOp and Comparators out of filter to client package
>>>>
>>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>>
>>>> What about:
>>>>
>>>> * HBASE-18601 Remove Htrace 3.2
>>>> As has been noted, the HTrace API is our 'trace' API.
>>>>
>>>> If interested in any of the above and you need a legup, just ask in the
>>>> issue and I'll be by....
>>>>
>>>> Thanks,
>>>> St.Ack
>>>>
>>>>
>>>>
>>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>>
>>>> Heads-up:
>>>>>
>>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
>>>>> updated dependencies, reliance on relocated popular libs (guava, netty,
>>>>> protobuf), purge of checked-in generated src, and
>>>>> master-carries-no-regions
>>>>> by default.
>>>>>
>>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme will
>>>>> be
>>>>> settling the APIs and compatibility (At first blush, we are not
>>>>> looking too
>>>>> bad; our Sean ran some tests over weekend that have hbase-1 client
>>>>> running
>>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp
>>>>> should be
>>>>> done by alpha3 (i.e. returning Interfaces rather than Implementations,
>>>>> and
>>>>> our shutdown of CPs accessing classes in hbase marked
>>>>> InterfaceAudience).
>>>>> We'll also have purged thirdparty classes from our API; e.g. guava 0.12
>>>>> Service showing through in our replication API and protobufs in Admin
>>>>> Interface. On alpha3, we will have to do a bunch of outreach to make
>>>>> sure
>>>>> our downstreamers are up on what is coming down the pipe.
>>>>>
>>>>> Beta1 in mid-September?
>>>>>
>>>>> I encourage you to check out the items marked for hbase2:
>>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit
>>>>> as
>>>>> you see appropriate. Punt if you know the JIRA will not get any
>>>>> attention
>>>>> in next month or so.
>>>>>
>>>>> A bunch of issues marked blocker are unassigned. I'll leave them as is
>>>>> another while but I'll boot them soon.
>>>>>
>>>>> While I have your attention:
>>>>>
>>>>> + I think we should leave thrift version at 0.9.3. Moving hbase thrift
>>>>> to 0.10.0 will break existing clients. The change is easy enough if
>>>>> folks
>>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
>>>>> (0.98?).
>>>>>
>>>>> St.Ack
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>>
>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> ...
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>>> feature a
>>>>>>>>> bit more "real", but am obviously a little nervous (I have no
>>>>>>>>> reason
>>>>>>>>> to be
>>>>>>>>> nervous though). I am pretty happy with the feature in terms of how
>>>>>>>>> much it
>>>>>>>>> is covered via testing.
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Last think on space quota says that need doc too. See 'Space Quota'
>>>>>>>> in
>>>>>>>> here:
>>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>>> Does this little section need an update Josh?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> S
>>>>>>>>
>>>>>>>>
>>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>>>>>>> already included for 17748.  Happy to put that on my plate if you're
>>>>>>> good
>>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>>
>>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>>> merged it in from the original feature branch. I'll double check and
>>>>>>> update
>>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>>
>>>>>>>
>>>>>> Thanks Josh,
>>>>>> S
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>

Re: Moving 2.0 forward

[Josh Elser <el...@apache.org>]

+1

I was trying to work on helping out on the outstanding alpha-4 stuff 
last week -- will be continuing to try to do the same this week.

If you need any help, Stack, or if others need reviews where I haven't 
noticed on my own: feel free to @mention me.

On 10/23/17 12:53 PM, Stack wrote:
> (Reviving this thread)
> 
> Lets push out alpha-4 this week. Alpha-4 is the release that has the
> refactor of the Coprocessor API shutting down access to internals marked
> InterfaceAudience.Private.
> 
> The outstanding list is here:
> https://issues.apache.org/jira/projects/HBASE/versions/12341594
> 
> Please push in anything marked alpha-4 that belongs to you.
> 
> If issue, talk out loud on this thread. If you need a review to land an
> item, shout on the issue and here; we'll help you out.
> 
> As is, items are coming along nicely I'd say. We need to merge the filter
> branch -- HBASE-18410 -- so APIs are finished for hbase2.
> 
> Post alpha-4, we'll have to hunt down our downstreamers and help them test
> on top of alpha-4 so rolling into beta-1, we have confidence our
> downstreamers know what to expect (or we discover what we missed BEFORE we
> beta-1).
> 
> Thanks for time,
> S
> 
> 
> 
> 
> 
> On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:
> 
>> I'll put up an alpha3 RC Monday, probably Monday night. That should be
>> time, if we all sprint, for the public-facing API fixes to be done.
>>
>> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3 but
>> it is plain that more time is needed (in spite of valiant effort so far by
>> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose theme is
>> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the following
>> week.
>>
>> We should then be ready for beta (beta == no new features, no API changes,
>> just fixes).
>>
>> Thanks,
>> St.Ack
>>
>>
>> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>>
>>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>>>
>>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>>> release out in the next week or so.
>>>
>>> I did a weeding of 2.0.0 issues over the last day. If folks are
>>> interested in helping out, below are the items I think we need done for
>>> alpha3 (below are at least 'Critical' status, are API possibly altering
>>> items, and are absent those JIRAs that are making active progress, i.e. the
>>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs doing is
>>> what Andrew did comparing 1.3. and 1.4 APIs
>>>
>>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>>> branch-2
>>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
>>> between branch-1 and branch-2.
>>>
>>> * HBASE-10462 Recategorize some of the client facing Public / Private
>>> interfaces
>>> This one is almost done. It could do with a finish, attention to the
>>> items in last comment, and then our codebase could do with another sweep
>>> after the spirit of this issue since a bunch has gone in since the pass
>>> that was the basis of this issue.
>>>
>>> * HBASE-10504 Define Replication Interface
>>> I was going to take a crack at this as part of the revamp forced by
>>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service' but if
>>> anyone else is interested, be my guest.
>>>
>>> * HBASE-14996 Some more API cleanup for 2.0
>>> Has a bunch of subtasks, some of which are being worked on. Needs
>>> finishing.
>>>
>>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>>> cleanup
>>> Needs a pass. Small issue I think. Could also look at new AsyncClient and
>>> make sure symmetry.
>>>
>>> * HBASE-15607 Remove PB references from Admin for 2.0
>>> Predicated on result of an ongoing DISCUSSION thread but needs to be done.
>>>
>>> Rolling upgrade will have implications for our API. Would be good to try
>>> it and figure what needs fixup (as said above, according to trial by Sean,
>>> we might not be too bad here):
>>> * HBASE-16060 1.x clients cannot access table state talking to 2.0 cluster
>>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and 1.x
>>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>>
>>> * HBASE-17442 Move most of the replication related classes to
>>> hbase-server package
>>> The above would be good to do generally but it may make for ripples in
>>> API so would be good to do now.
>>>
>>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>>> Balazs is working on this. The idea is that we avoid adding two new types
>>> to our API, two types that are nought but curtailed, read-only views on
>>> internals. Input if you have time appreciated.
>>>
>>> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
>>> cluster; verify
>>> Esteban is looking at this one
>>>
>>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>>> * HBASE-17143 Scan improvement
>>>
>>> Our Coprocessor Interface needs a tough edit. It exposes implementations
>>> marked audience Private and returns implementations rather than Interfaces.
>>> In a few locations, we allow returning an alternate implementation
>>> altogether which is probably something we don't want a CP doing. To that
>>> end, the following issues started by Duo and Anoop need to be taken to the
>>> finish line; ideally they'd have an owner:
>>>
>>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
>>> umbrella issue.
>>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>>> RegionServerObserver
>>>
>>>
>>> Nice-to-haves:
>>>
>>> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
>>> TimeRange constructors
>>>
>>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>>> existing in the code
>>> This is the end of an old long-running project moving up on to Cell
>>> Interface. We think it is done but for a few little items (deprecate KV
>>> methods in MR and provide Cell versions instead...)
>>>
>>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
>>> fixing
>>>
>>> * HBASE-13346 Clean up Filter package for post 1.0
>>>
>>> * HBASE-14255 Simplify Cell creation post 1.0
>>> * HBASE-14997
>>> Move compareOp and Comparators out of filter to client package
>>>
>>> * HBASE-13740 Stop using Hadoop private interfaces
>>>
>>> What about:
>>>
>>> * HBASE-18601 Remove Htrace 3.2
>>> As has been noted, the HTrace API is our 'trace' API.
>>>
>>> If interested in any of the above and you need a legup, just ask in the
>>> issue and I'll be by....
>>>
>>> Thanks,
>>> St.Ack
>>>
>>>
>>>
>>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>>
>>>> Heads-up:
>>>>
>>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
>>>> updated dependencies, reliance on relocated popular libs (guava, netty,
>>>> protobuf), purge of checked-in generated src, and master-carries-no-regions
>>>> by default.
>>>>
>>>> alpha3 I hope will follow soon after (end-of-August?). Its theme will be
>>>> settling the APIs and compatibility (At first blush, we are not looking too
>>>> bad; our Sean ran some tests over weekend that have hbase-1 client running
>>>> against an hbase-2 cluster....). The Coprocessor Interface revamp should be
>>>> done by alpha3 (i.e. returning Interfaces rather than Implementations, and
>>>> our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
>>>> We'll also have purged thirdparty classes from our API; e.g. guava 0.12
>>>> Service showing through in our replication API and protobufs in Admin
>>>> Interface. On alpha3, we will have to do a bunch of outreach to make sure
>>>> our downstreamers are up on what is coming down the pipe.
>>>>
>>>> Beta1 in mid-September?
>>>>
>>>> I encourage you to check out the items marked for hbase2:
>>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as
>>>> you see appropriate. Punt if you know the JIRA will not get any attention
>>>> in next month or so.
>>>>
>>>> A bunch of issues marked blocker are unassigned. I'll leave them as is
>>>> another while but I'll boot them soon.
>>>>
>>>> While I have your attention:
>>>>
>>>> + I think we should leave thrift version at 0.9.3. Moving hbase thrift
>>>> to 0.10.0 will break existing clients. The change is easy enough if folks
>>>> need to upgrade their hbase thrift. See HBASE-18591.
>>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).
>>>>
>>>> St.Ack
>>>>
>>>>
>>>>
>>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>>
>>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> ...
>>>>>>>>
>>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>>> feature a
>>>>>>>> bit more "real", but am obviously a little nervous (I have no reason
>>>>>>>> to be
>>>>>>>> nervous though). I am pretty happy with the feature in terms of how
>>>>>>>> much it
>>>>>>>> is covered via testing.
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>>
>>>>>>> Last think on space quota says that need doc too. See 'Space Quota' in
>>>>>>> here:
>>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>>> Does this little section need an update Josh?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> S
>>>>>>>
>>>>>>
>>>>>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>>>>>> already included for 17748.  Happy to put that on my plate if you're good
>>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>>
>>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>>> merged it in from the original feature branch. I'll double check and update
>>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>>
>>>>>
>>>>> Thanks Josh,
>>>>> S
>>>>>
>>>>
>>>>
>>>
>>
> 

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Thu, Aug 17, 2017 at 12:45 PM, Mike Drob <md...@apache.org> wrote:

> I went ahead and tagged those as tentatively 2.0.0-alpha3 in jira so that
> we can have them all in one place later. Folks can move additional issues
> in and out as they see appropriate.
>
> Thanks M,
S



> On Thu, Aug 17, 2017 at 2:35 PM, Stack <st...@duboce.net> wrote:
>
> > I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
> >
> > For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
> > release out in the next week or so.
> >
> > I did a weeding of 2.0.0 issues over the last day. If folks are
> interested
> > in helping out, below are the items I think we need done for alpha3
> (below
> > are at least 'Critical' status, are API possibly altering items, and are
> > absent those JIRAs that are making active progress, i.e. the HTD/HCD
> revamp
> > by Chia-Ping Tsai). A project NOT listed that needs doing is what Andrew
> > did comparing 1.3. and 1.4 APIs
> >
> > * HBASE-18622 Mitigate compatibility concerns between branch-1 and
> branch-2
> > This is to do what Andrew did between 1.3 and 1.4 branches only do it
> > between branch-1 and branch-2.
> >
> > * HBASE-10462 Recategorize some of the client facing Public / Private
> > interfaces
> > This one is almost done. It could do with a finish, attention to the
> items
> > in last comment, and then our codebase could do with another sweep after
> > the spirit of this issue since a bunch has gone in since the pass that
> was
> > the basis of this issue.
> >
> > * HBASE-10504 Define Replication Interface
> > I was going to take a crack at this as part of the revamp forced by
> > 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service' but
> if
> > anyone else is interested, be my guest.
> >
> > * HBASE-14996 Some more API cleanup for 2.0
> > Has a bunch of subtasks, some of which are being worked on. Needs
> > finishing.
> >
> > * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
> > cleanup
> > Needs a pass. Small issue I think. Could also look at new AsyncClient and
> > make sure symmetry.
> >
> > * HBASE-15607 Remove PB references from Admin for 2.0
> > Predicated on result of an ongoing DISCUSSION thread but needs to be
> done.
> >
> > Rolling upgrade will have implications for our API. Would be good to try
> it
> > and figure what needs fixup (as said above, according to trial by Sean,
> we
> > might not be too bad here):
> > * HBASE-16060 1.x clients cannot access table state talking to 2.0
> cluster
> > * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and 1.x
> > RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
> >
> > * HBASE-17442 Move most of the replication related classes to
> hbase-server
> > package
> > The above would be good to do generally but it may make for ripples in
> API
> > so would be good to do now.
> >
> > * HBASE-18106 Redo ProcedureInfo and LockInfo
> > Balazs is working on this. The idea is that we avoid adding two new types
> > to our API, two types that are nought but curtailed, read-only views on
> > internals. Input if you have time appreciated.
> >
> > * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
> > cluster; verify
> > Esteban is looking at this one
> >
> > * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> > * HBASE-17143 Scan improvement
> >
> > Our Coprocessor Interface needs a tough edit. It exposes implementations
> > marked audience Private and returns implementations rather than
> Interfaces.
> > In a few locations, we allow returning an alternate implementation
> > altogether which is probably something we don't want a CP doing. To that
> > end, the following issues started by Duo and Anoop need to be taken to
> the
> > finish line; ideally they'd have an owner:
> >
> > * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
> > umbrella issue.
> > * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> > * HBASE-16769 Deprecate/remove PB references from MasterObserver and
> > RegionServerObserver
> >
> >
> > Nice-to-haves:
> >
> > * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
> > TimeRange constructors
> >
> > * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
> existing
> > in the code
> > This is the end of an old long-running project moving up on to Cell
> > Interface. We think it is done but for a few little items (deprecate KV
> > methods in MR and provide Cell versions instead...)
> >
> > * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
> > fixing
> >
> > * HBASE-13346 Clean up Filter package for post 1.0
> >
> > * HBASE-14255 Simplify Cell creation post 1.0
> > * HBASE-14997
> > Move compareOp and Comparators out of filter to client package
> >
> > * HBASE-13740 Stop using Hadoop private interfaces
> >
> > What about:
> >
> > * HBASE-18601 Remove Htrace 3.2
> > As has been noted, the HTrace API is our 'trace' API.
> >
> > If interested in any of the above and you need a legup, just ask in the
> > issue and I'll be by....
> >
> > Thanks,
> > St.Ack
> >
> >
> >
> > On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
> >
> > > Heads-up:
> > >
> > > I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
> > > updated dependencies, reliance on relocated popular libs (guava, netty,
> > > protobuf), purge of checked-in generated src, and
> > master-carries-no-regions
> > > by default.
> > >
> > > alpha3 I hope will follow soon after (end-of-August?). Its theme will
> be
> > > settling the APIs and compatibility (At first blush, we are not looking
> > too
> > > bad; our Sean ran some tests over weekend that have hbase-1 client
> > running
> > > against an hbase-2 cluster....). The Coprocessor Interface revamp
> should
> > be
> > > done by alpha3 (i.e. returning Interfaces rather than Implementations,
> > and
> > > our shutdown of CPs accessing classes in hbase marked
> InterfaceAudience).
> > > We'll also have purged thirdparty classes from our API; e.g. guava 0.12
> > > Service showing through in our replication API and protobufs in Admin
> > > Interface. On alpha3, we will have to do a bunch of outreach to make
> sure
> > > our downstreamers are up on what is coming down the pipe.
> > >
> > > Beta1 in mid-September?
> > >
> > > I encourage you to check out the items marked for hbase2:
> > > https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit
> as
> > > you see appropriate. Punt if you know the JIRA will not get any
> attention
> > > in next month or so.
> > >
> > > A bunch of issues marked blocker are unassigned. I'll leave them as is
> > > another while but I'll boot them soon.
> > >
> > > While I have your attention:
> > >
> > > + I think we should leave thrift version at 0.9.3. Moving hbase thrift
> to
> > > 0.10.0 will break existing clients. The change is easy enough if folks
> > need
> > > to upgrade their hbase thrift. See HBASE-18591.
> > > + Upgrade from 0.94 is disallowed. You have to get to 1.0 first
> (0.98?).
> > >
> > > St.Ack
> > >
> > >
> > >
> > > On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> > >
> > >>
> > >>
> > >> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
> > >>
> > >>>
> > >>>
> > >>> On 7/31/17 9:00 AM, Stack wrote:
> > >>>
> > >>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
> > wrote:
> > >>>>
> > >>>> ...
> > >>>>>
> > >>>>> I like the idea of this also hitting 2.0 as it would make the
> > feature a
> > >>>>> bit more "real", but am obviously a little nervous (I have no
> reason
> > >>>>> to be
> > >>>>> nervous though). I am pretty happy with the feature in terms of how
> > >>>>> much it
> > >>>>> is covered via testing.
> > >>>>>
> > >>>>> https://issues.apache.org/jira/browse/HBASE-17748
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> > >>>>
> > >>>> Last think on space quota says that need doc too. See 'Space Quota'
> in
> > >>>> here:
> > >>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> > >>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > >>>> Does this little section need an update Josh?
> > >>>>
> > >>>> Thanks,
> > >>>> S
> > >>>>
> > >>>
> > >>> Yep, just a couple of cherry-picks. Good test coverage and some docs
> > >>> already included for 17748.  Happy to put that on my plate if you're
> > good
> > >>> with it. I can reasonably assume that no one is against it :)
> > >>>
> > >>> I think I had knocked out docs for the "phase 1" stuff before we
> merged
> > >>> it in from the original feature branch. I'll double check and update
> > the
> > >>> gdoc. Perhaps this was just a timing thing.
> > >>>
> > >>
> > >> Thanks Josh,
> > >> S
> > >>
> > >
> > >
> >
>

Re: Moving 2.0 forward

[Mike Drob <md...@apache.org>]

I went ahead and tagged those as tentatively 2.0.0-alpha3 in jira so that
we can have them all in one place later. Folks can move additional issues
in and out as they see appropriate.

On Thu, Aug 17, 2017 at 2:35 PM, Stack <st...@duboce.net> wrote:

> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>
> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
> release out in the next week or so.
>
> I did a weeding of 2.0.0 issues over the last day. If folks are interested
> in helping out, below are the items I think we need done for alpha3 (below
> are at least 'Critical' status, are API possibly altering items, and are
> absent those JIRAs that are making active progress, i.e. the HTD/HCD revamp
> by Chia-Ping Tsai). A project NOT listed that needs doing is what Andrew
> did comparing 1.3. and 1.4 APIs
>
> * HBASE-18622 Mitigate compatibility concerns between branch-1 and branch-2
> This is to do what Andrew did between 1.3 and 1.4 branches only do it
> between branch-1 and branch-2.
>
> * HBASE-10462 Recategorize some of the client facing Public / Private
> interfaces
> This one is almost done. It could do with a finish, attention to the items
> in last comment, and then our codebase could do with another sweep after
> the spirit of this issue since a bunch has gone in since the pass that was
> the basis of this issue.
>
> * HBASE-10504 Define Replication Interface
> I was going to take a crack at this as part of the revamp forced by
> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service' but if
> anyone else is interested, be my guest.
>
> * HBASE-14996 Some more API cleanup for 2.0
> Has a bunch of subtasks, some of which are being worked on. Needs
> finishing.
>
> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
> cleanup
> Needs a pass. Small issue I think. Could also look at new AsyncClient and
> make sure symmetry.
>
> * HBASE-15607 Remove PB references from Admin for 2.0
> Predicated on result of an ongoing DISCUSSION thread but needs to be done.
>
> Rolling upgrade will have implications for our API. Would be good to try it
> and figure what needs fixup (as said above, according to trial by Sean, we
> might not be too bad here):
> * HBASE-16060 1.x clients cannot access table state talking to 2.0 cluster
> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and 1.x
> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>
> * HBASE-17442 Move most of the replication related classes to hbase-server
> package
> The above would be good to do generally but it may make for ripples in API
> so would be good to do now.
>
> * HBASE-18106 Redo ProcedureInfo and LockInfo
> Balazs is working on this. The idea is that we avoid adding two new types
> to our API, two types that are nought but curtailed, read-only views on
> internals. Input if you have time appreciated.
>
> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
> cluster; verify
> Esteban is looking at this one
>
> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> * HBASE-17143 Scan improvement
>
> Our Coprocessor Interface needs a tough edit. It exposes implementations
> marked audience Private and returns implementations rather than Interfaces.
> In a few locations, we allow returning an alternate implementation
> altogether which is probably something we don't want a CP doing. To that
> end, the following issues started by Duo and Anoop need to be taken to the
> finish line; ideally they'd have an owner:
>
> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
> umbrella issue.
> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
> RegionServerObserver
>
>
> Nice-to-haves:
>
> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
> TimeRange constructors
>
> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references existing
> in the code
> This is the end of an old long-running project moving up on to Cell
> Interface. We think it is done but for a few little items (deprecate KV
> methods in MR and provide Cell versions instead...)
>
> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
> fixing
>
> * HBASE-13346 Clean up Filter package for post 1.0
>
> * HBASE-14255 Simplify Cell creation post 1.0
> * HBASE-14997
> Move compareOp and Comparators out of filter to client package
>
> * HBASE-13740 Stop using Hadoop private interfaces
>
> What about:
>
> * HBASE-18601 Remove Htrace 3.2
> As has been noted, the HTrace API is our 'trace' API.
>
> If interested in any of the above and you need a legup, just ask in the
> issue and I'll be by....
>
> Thanks,
> St.Ack
>
>
>
> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>
> > Heads-up:
> >
> > I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
> > updated dependencies, reliance on relocated popular libs (guava, netty,
> > protobuf), purge of checked-in generated src, and
> master-carries-no-regions
> > by default.
> >
> > alpha3 I hope will follow soon after (end-of-August?). Its theme will be
> > settling the APIs and compatibility (At first blush, we are not looking
> too
> > bad; our Sean ran some tests over weekend that have hbase-1 client
> running
> > against an hbase-2 cluster....). The Coprocessor Interface revamp should
> be
> > done by alpha3 (i.e. returning Interfaces rather than Implementations,
> and
> > our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
> > We'll also have purged thirdparty classes from our API; e.g. guava 0.12
> > Service showing through in our replication API and protobufs in Admin
> > Interface. On alpha3, we will have to do a bunch of outreach to make sure
> > our downstreamers are up on what is coming down the pipe.
> >
> > Beta1 in mid-September?
> >
> > I encourage you to check out the items marked for hbase2:
> > https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as
> > you see appropriate. Punt if you know the JIRA will not get any attention
> > in next month or so.
> >
> > A bunch of issues marked blocker are unassigned. I'll leave them as is
> > another while but I'll boot them soon.
> >
> > While I have your attention:
> >
> > + I think we should leave thrift version at 0.9.3. Moving hbase thrift to
> > 0.10.0 will break existing clients. The change is easy enough if folks
> need
> > to upgrade their hbase thrift. See HBASE-18591.
> > + Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).
> >
> > St.Ack
> >
> >
> >
> > On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> >
> >>
> >>
> >> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
> >>
> >>>
> >>>
> >>> On 7/31/17 9:00 AM, Stack wrote:
> >>>
> >>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
> wrote:
> >>>>
> >>>> ...
> >>>>>
> >>>>> I like the idea of this also hitting 2.0 as it would make the
> feature a
> >>>>> bit more "real", but am obviously a little nervous (I have no reason
> >>>>> to be
> >>>>> nervous though). I am pretty happy with the feature in terms of how
> >>>>> much it
> >>>>> is covered via testing.
> >>>>>
> >>>>> https://issues.apache.org/jira/browse/HBASE-17748
> >>>>>
> >>>>>
> >>>>>
> >>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> >>>>
> >>>> Last think on space quota says that need doc too. See 'Space Quota' in
> >>>> here:
> >>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> >>>> Does this little section need an update Josh?
> >>>>
> >>>> Thanks,
> >>>> S
> >>>>
> >>>
> >>> Yep, just a couple of cherry-picks. Good test coverage and some docs
> >>> already included for 17748.  Happy to put that on my plate if you're
> good
> >>> with it. I can reasonably assume that no one is against it :)
> >>>
> >>> I think I had knocked out docs for the "phase 1" stuff before we merged
> >>> it in from the original feature branch. I'll double check and update
> the
> >>> gdoc. Perhaps this was just a timing thing.
> >>>
> >>
> >> Thanks Josh,
> >> S
> >>
> >
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

(Reviving this thread)

Lets push out alpha-4 this week. Alpha-4 is the release that has the
refactor of the Coprocessor API shutting down access to internals marked
InterfaceAudience.Private.

The outstanding list is here:
https://issues.apache.org/jira/projects/HBASE/versions/12341594

Please push in anything marked alpha-4 that belongs to you.

If issue, talk out loud on this thread. If you need a review to land an
item, shout on the issue and here; we'll help you out.

As is, items are coming along nicely I'd say. We need to merge the filter
branch -- HBASE-18410 -- so APIs are finished for hbase2.

Post alpha-4, we'll have to hunt down our downstreamers and help them test
on top of alpha-4 so rolling into beta-1, we have confidence our
downstreamers know what to expect (or we discover what we missed BEFORE we
beta-1).

Thanks for time,
S





On Fri, Sep 8, 2017 at 2:04 PM, Stack <st...@duboce.net> wrote:

> I'll put up an alpha3 RC Monday, probably Monday night. That should be
> time, if we all sprint, for the public-facing API fixes to be done.
>
> I had a bunch of Coprocessor refactor and fixup scheduled for alpha3 but
> it is plain that more time is needed (in spite of valiant effort so far by
> Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose theme is
> "Coprocessor Fixup". Hopefully we can put an alpha-4 up by the following
> week.
>
> We should then be ready for beta (beta == no new features, no API changes,
> just fixes).
>
> Thanks,
> St.Ack
>
>
> On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:
>
>> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>>
>> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
>> release out in the next week or so.
>>
>> I did a weeding of 2.0.0 issues over the last day. If folks are
>> interested in helping out, below are the items I think we need done for
>> alpha3 (below are at least 'Critical' status, are API possibly altering
>> items, and are absent those JIRAs that are making active progress, i.e. the
>> HTD/HCD revamp by Chia-Ping Tsai). A project NOT listed that needs doing is
>> what Andrew did comparing 1.3. and 1.4 APIs
>>
>> * HBASE-18622 Mitigate compatibility concerns between branch-1 and
>> branch-2
>> This is to do what Andrew did between 1.3 and 1.4 branches only do it
>> between branch-1 and branch-2.
>>
>> * HBASE-10462 Recategorize some of the client facing Public / Private
>> interfaces
>> This one is almost done. It could do with a finish, attention to the
>> items in last comment, and then our codebase could do with another sweep
>> after the spirit of this issue since a bunch has gone in since the pass
>> that was the basis of this issue.
>>
>> * HBASE-10504 Define Replication Interface
>> I was going to take a crack at this as part of the revamp forced by
>> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service' but if
>> anyone else is interested, be my guest.
>>
>> * HBASE-14996 Some more API cleanup for 2.0
>> Has a bunch of subtasks, some of which are being worked on. Needs
>> finishing.
>>
>> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
>> cleanup
>> Needs a pass. Small issue I think. Could also look at new AsyncClient and
>> make sure symmetry.
>>
>> * HBASE-15607 Remove PB references from Admin for 2.0
>> Predicated on result of an ongoing DISCUSSION thread but needs to be done.
>>
>> Rolling upgrade will have implications for our API. Would be good to try
>> it and figure what needs fixup (as said above, according to trial by Sean,
>> we might not be too bad here):
>> * HBASE-16060 1.x clients cannot access table state talking to 2.0 cluster
>> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and 1.x
>> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>>
>> * HBASE-17442 Move most of the replication related classes to
>> hbase-server package
>> The above would be good to do generally but it may make for ripples in
>> API so would be good to do now.
>>
>> * HBASE-18106 Redo ProcedureInfo and LockInfo
>> Balazs is working on this. The idea is that we avoid adding two new types
>> to our API, two types that are nought but curtailed, read-only views on
>> internals. Input if you have time appreciated.
>>
>> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
>> cluster; verify
>> Esteban is looking at this one
>>
>> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
>> * HBASE-17143 Scan improvement
>>
>> Our Coprocessor Interface needs a tough edit. It exposes implementations
>> marked audience Private and returns implementations rather than Interfaces.
>> In a few locations, we allow returning an alternate implementation
>> altogether which is probably something we don't want a CP doing. To that
>> end, the following issues started by Duo and Anoop need to be taken to the
>> finish line; ideally they'd have an owner:
>>
>> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
>> umbrella issue.
>> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
>> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
>> RegionServerObserver
>>
>>
>> Nice-to-haves:
>>
>> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
>> TimeRange constructors
>>
>> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
>> existing in the code
>> This is the end of an old long-running project moving up on to Cell
>> Interface. We think it is done but for a few little items (deprecate KV
>> methods in MR and provide Cell versions instead...)
>>
>> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
>> fixing
>>
>> * HBASE-13346 Clean up Filter package for post 1.0
>>
>> * HBASE-14255 Simplify Cell creation post 1.0
>> * HBASE-14997
>> Move compareOp and Comparators out of filter to client package
>>
>> * HBASE-13740 Stop using Hadoop private interfaces
>>
>> What about:
>>
>> * HBASE-18601 Remove Htrace 3.2
>> As has been noted, the HTrace API is our 'trace' API.
>>
>> If interested in any of the above and you need a legup, just ask in the
>> issue and I'll be by....
>>
>> Thanks,
>> St.Ack
>>
>>
>>
>> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>>
>>> Heads-up:
>>>
>>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
>>> updated dependencies, reliance on relocated popular libs (guava, netty,
>>> protobuf), purge of checked-in generated src, and master-carries-no-regions
>>> by default.
>>>
>>> alpha3 I hope will follow soon after (end-of-August?). Its theme will be
>>> settling the APIs and compatibility (At first blush, we are not looking too
>>> bad; our Sean ran some tests over weekend that have hbase-1 client running
>>> against an hbase-2 cluster....). The Coprocessor Interface revamp should be
>>> done by alpha3 (i.e. returning Interfaces rather than Implementations, and
>>> our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
>>> We'll also have purged thirdparty classes from our API; e.g. guava 0.12
>>> Service showing through in our replication API and protobufs in Admin
>>> Interface. On alpha3, we will have to do a bunch of outreach to make sure
>>> our downstreamers are up on what is coming down the pipe.
>>>
>>> Beta1 in mid-September?
>>>
>>> I encourage you to check out the items marked for hbase2:
>>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as
>>> you see appropriate. Punt if you know the JIRA will not get any attention
>>> in next month or so.
>>>
>>> A bunch of issues marked blocker are unassigned. I'll leave them as is
>>> another while but I'll boot them soon.
>>>
>>> While I have your attention:
>>>
>>> + I think we should leave thrift version at 0.9.3. Moving hbase thrift
>>> to 0.10.0 will break existing clients. The change is easy enough if folks
>>> need to upgrade their hbase thrift. See HBASE-18591.
>>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).
>>>
>>> St.Ack
>>>
>>>
>>>
>>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
>>>>
>>>>>
>>>>>
>>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>>
>>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> ...
>>>>>>>
>>>>>>> I like the idea of this also hitting 2.0 as it would make the
>>>>>>> feature a
>>>>>>> bit more "real", but am obviously a little nervous (I have no reason
>>>>>>> to be
>>>>>>> nervous though). I am pretty happy with the feature in terms of how
>>>>>>> much it
>>>>>>> is covered via testing.
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>>
>>>>>> Last think on space quota says that need doc too. See 'Space Quota' in
>>>>>> here:
>>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>>> Does this little section need an update Josh?
>>>>>>
>>>>>> Thanks,
>>>>>> S
>>>>>>
>>>>>
>>>>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>>>>> already included for 17748.  Happy to put that on my plate if you're good
>>>>> with it. I can reasonably assume that no one is against it :)
>>>>>
>>>>> I think I had knocked out docs for the "phase 1" stuff before we
>>>>> merged it in from the original feature branch. I'll double check and update
>>>>> the gdoc. Perhaps this was just a timing thing.
>>>>>
>>>>
>>>> Thanks Josh,
>>>> S
>>>>
>>>
>>>
>>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

I'll put up an alpha3 RC Monday, probably Monday night. That should be
time, if we all sprint, for the public-facing API fixes to be done.

I had a bunch of Coprocessor refactor and fixup scheduled for alpha3 but it
is plain that more time is needed (in spite of valiant effort so far by
Anoop, Duo, Appy, etc.). Therefore, lets run a 2.0.0-alpha-4 whose theme is
"Coprocessor Fixup". Hopefully we can put an alpha-4 up by the following
week.

We should then be ready for beta (beta == no new features, no API changes,
just fixes).

Thanks,
St.Ack


On Thu, Aug 17, 2017 at 12:35 PM, Stack <st...@duboce.net> wrote:

> I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.
>
> For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
> release out in the next week or so.
>
> I did a weeding of 2.0.0 issues over the last day. If folks are interested
> in helping out, below are the items I think we need done for alpha3 (below
> are at least 'Critical' status, are API possibly altering items, and are
> absent those JIRAs that are making active progress, i.e. the HTD/HCD revamp
> by Chia-Ping Tsai). A project NOT listed that needs doing is what Andrew
> did comparing 1.3. and 1.4 APIs
>
> * HBASE-18622 Mitigate compatibility concerns between branch-1 and branch-2
> This is to do what Andrew did between 1.3 and 1.4 branches only do it
> between branch-1 and branch-2.
>
> * HBASE-10462 Recategorize some of the client facing Public / Private
> interfaces
> This one is almost done. It could do with a finish, attention to the items
> in last comment, and then our codebase could do with another sweep after
> the spirit of this issue since a bunch has gone in since the pass that was
> the basis of this issue.
>
> * HBASE-10504 Define Replication Interface
> I was going to take a crack at this as part of the revamp forced by
> 'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service' but if
> anyone else is interested, be my guest.
>
> * HBASE-14996 Some more API cleanup for 2.0
> Has a bunch of subtasks, some of which are being worked on. Needs
> finishing.
>
> * HBASE-14998 Unify synchronous and asynchronous methods in Admin and
> cleanup
> Needs a pass. Small issue I think. Could also look at new AsyncClient and
> make sure symmetry.
>
> * HBASE-15607 Remove PB references from Admin for 2.0
> Predicated on result of an ongoing DISCUSSION thread but needs to be done.
>
> Rolling upgrade will have implications for our API. Would be good to try
> it and figure what needs fixup (as said above, according to trial by Sean,
> we might not be too bad here):
> * HBASE-16060 1.x clients cannot access table state talking to 2.0 cluster
> * HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and 1.x
> RSs; i.e. support Rolling Upgrade from hbase-1 to -2.
>
> * HBASE-17442 Move most of the replication related classes to hbase-server
> package
> The above would be good to do generally but it may make for ripples in API
> so would be good to do now.
>
> * HBASE-18106 Redo ProcedureInfo and LockInfo
> Balazs is working on this. The idea is that we avoid adding two new types
> to our API, two types that are nought but curtailed, read-only views on
> internals. Input if you have time appreciated.
>
> * HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
> cluster; verify
> Esteban is looking at this one
>
> * HBASE-9417 SecureBulkLoadEndpoint should be folded in core
> * HBASE-17143 Scan improvement
>
> Our Coprocessor Interface needs a tough edit. It exposes implementations
> marked audience Private and returns implementations rather than Interfaces.
> In a few locations, we allow returning an alternate implementation
> altogether which is probably something we don't want a CP doing. To that
> end, the following issues started by Duo and Anoop need to be taken to the
> finish line; ideally they'd have an owner:
>
> * HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
> umbrella issue.
> * HBASE-18298 RegionServerServices Interface cleanup for CP expose
> * HBASE-16769 Deprecate/remove PB references from MasterObserver and
> RegionServerObserver
>
>
> Nice-to-haves:
>
> * HBASE-15284 Make TimeRange constructors IA.Private and remove unused
> TimeRange constructors
>
> * HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references
> existing in the code
> This is the end of an old long-running project moving up on to Cell
> Interface. We think it is done but for a few little items (deprecate KV
> methods in MR and provide Cell versions instead...)
>
> * HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs
> fixing
>
> * HBASE-13346 Clean up Filter package for post 1.0
>
> * HBASE-14255 Simplify Cell creation post 1.0
> * HBASE-14997
> Move compareOp and Comparators out of filter to client package
>
> * HBASE-13740 Stop using Hadoop private interfaces
>
> What about:
>
> * HBASE-18601 Remove Htrace 3.2
> As has been noted, the HTrace API is our 'trace' API.
>
> If interested in any of the above and you need a legup, just ask in the
> issue and I'll be by....
>
> Thanks,
> St.Ack
>
>
>
> On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:
>
>> Heads-up:
>>
>> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
>> updated dependencies, reliance on relocated popular libs (guava, netty,
>> protobuf), purge of checked-in generated src, and master-carries-no-regions
>> by default.
>>
>> alpha3 I hope will follow soon after (end-of-August?). Its theme will be
>> settling the APIs and compatibility (At first blush, we are not looking too
>> bad; our Sean ran some tests over weekend that have hbase-1 client running
>> against an hbase-2 cluster....). The Coprocessor Interface revamp should be
>> done by alpha3 (i.e. returning Interfaces rather than Implementations, and
>> our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
>> We'll also have purged thirdparty classes from our API; e.g. guava 0.12
>> Service showing through in our replication API and protobufs in Admin
>> Interface. On alpha3, we will have to do a bunch of outreach to make sure
>> our downstreamers are up on what is coming down the pipe.
>>
>> Beta1 in mid-September?
>>
>> I encourage you to check out the items marked for hbase2:
>> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as
>> you see appropriate. Punt if you know the JIRA will not get any attention
>> in next month or so.
>>
>> A bunch of issues marked blocker are unassigned. I'll leave them as is
>> another while but I'll boot them soon.
>>
>> While I have your attention:
>>
>> + I think we should leave thrift version at 0.9.3. Moving hbase thrift to
>> 0.10.0 will break existing clients. The change is easy enough if folks need
>> to upgrade their hbase thrift. See HBASE-18591.
>> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).
>>
>> St.Ack
>>
>>
>>
>> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>>
>>>
>>>
>>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
>>>
>>>>
>>>>
>>>> On 7/31/17 9:00 AM, Stack wrote:
>>>>
>>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
>>>>> wrote:
>>>>>
>>>>> ...
>>>>>>
>>>>>> I like the idea of this also hitting 2.0 as it would make the feature
>>>>>> a
>>>>>> bit more "real", but am obviously a little nervous (I have no reason
>>>>>> to be
>>>>>> nervous though). I am pretty happy with the feature in terms of how
>>>>>> much it
>>>>>> is covered via testing.
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>>
>>>>>>
>>>>>>
>>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>>
>>>>> Last think on space quota says that need doc too. See 'Space Quota' in
>>>>> here:
>>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>>> Does this little section need an update Josh?
>>>>>
>>>>> Thanks,
>>>>> S
>>>>>
>>>>
>>>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>>>> already included for 17748.  Happy to put that on my plate if you're good
>>>> with it. I can reasonably assume that no one is against it :)
>>>>
>>>> I think I had knocked out docs for the "phase 1" stuff before we merged
>>>> it in from the original feature branch. I'll double check and update the
>>>> gdoc. Perhaps this was just a timing thing.
>>>>
>>>
>>> Thanks Josh,
>>> S
>>>
>>
>>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

I put up the hbase-2.0.0-alpha2 release candidate. Please vote on it.

For hbase-2.0.0-alpha3, the theme is solidifying API. I hope to get a
release out in the next week or so.

I did a weeding of 2.0.0 issues over the last day. If folks are interested
in helping out, below are the items I think we need done for alpha3 (below
are at least 'Critical' status, are API possibly altering items, and are
absent those JIRAs that are making active progress, i.e. the HTD/HCD revamp
by Chia-Ping Tsai). A project NOT listed that needs doing is what Andrew
did comparing 1.3. and 1.4 APIs

* HBASE-18622 Mitigate compatibility concerns between branch-1 and branch-2
This is to do what Andrew did between 1.3 and 1.4 branches only do it
between branch-1 and branch-2.

* HBASE-10462 Recategorize some of the client facing Public / Private
interfaces
This one is almost done. It could do with a finish, attention to the items
in last comment, and then our codebase could do with another sweep after
the spirit of this issue since a bunch has gone in since the pass that was
the basis of this issue.

* HBASE-10504 Define Replication Interface
I was going to take a crack at this as part of the revamp forced by
'HBASE-15982 Interface ReplicationEndpoint extends Guava's Service' but if
anyone else is interested, be my guest.

* HBASE-14996 Some more API cleanup for 2.0
Has a bunch of subtasks, some of which are being worked on. Needs finishing.

* HBASE-14998 Unify synchronous and asynchronous methods in Admin and
cleanup
Needs a pass. Small issue I think. Could also look at new AsyncClient and
make sure symmetry.

* HBASE-15607 Remove PB references from Admin for 2.0
Predicated on result of an ongoing DISCUSSION thread but needs to be done.

Rolling upgrade will have implications for our API. Would be good to try it
and figure what needs fixup (as said above, according to trial by Sean, we
might not be too bad here):
* HBASE-16060 1.x clients cannot access table state talking to 2.0 cluster
* HBASE-16550 Procedure v2 - Add AM compatibility for 2.x Master and 1.x
RSs; i.e. support Rolling Upgrade from hbase-1 to -2.

* HBASE-17442 Move most of the replication related classes to hbase-server
package
The above would be good to do generally but it may make for ripples in API
so would be good to do now.

* HBASE-18106 Redo ProcedureInfo and LockInfo
Balazs is working on this. The idea is that we avoid adding two new types
to our API, two types that are nought but curtailed, read-only views on
internals. Input if you have time appreciated.

* HBASE-18596 A hbase1 cluster should be able to replicate to a hbase2
cluster; verify
Esteban is looking at this one

* HBASE-9417 SecureBulkLoadEndpoint should be folded in core
* HBASE-17143 Scan improvement

Our Coprocessor Interface needs a tough edit. It exposes implementations
marked audience Private and returns implementations rather than Interfaces.
In a few locations, we allow returning an alternate implementation
altogether which is probably something we don't want a CP doing. To that
end, the following issues started by Duo and Anoop need to be taken to the
finish line; ideally they'd have an owner:

* HBASE-18169 Coprocessor fix and cleanup before 2.0.0 release <= The
umbrella issue.
* HBASE-18298 RegionServerServices Interface cleanup for CP expose
* HBASE-16769 Deprecate/remove PB references from MasterObserver and
RegionServerObserver


Nice-to-haves:

* HBASE-15284 Make TimeRange constructors IA.Private and remove unused
TimeRange constructors

* HBASE-10944 Remove all kv.getBuffer() and kv.getRow() references existing
in the code
This is the end of an old long-running project moving up on to Cell
Interface. We think it is done but for a few little items (deprecate KV
methods in MR and provide Cell versions instead...)

* HBASE-13271 Table#puts(List<Put>) operation is indeterminate; needs fixing

* HBASE-13346 Clean up Filter package for post 1.0

* HBASE-14255 Simplify Cell creation post 1.0
* HBASE-14997
Move compareOp and Comparators out of filter to client package

* HBASE-13740 Stop using Hadoop private interfaces

What about:

* HBASE-18601 Remove Htrace 3.2
As has been noted, the HTrace API is our 'trace' API.

If interested in any of the above and you need a legup, just ask in the
issue and I'll be by....

Thanks,
St.Ack



On Mon, Aug 14, 2017 at 10:54 AM, Stack <st...@duboce.net> wrote:

> Heads-up:
>
> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
> updated dependencies, reliance on relocated popular libs (guava, netty,
> protobuf), purge of checked-in generated src, and master-carries-no-regions
> by default.
>
> alpha3 I hope will follow soon after (end-of-August?). Its theme will be
> settling the APIs and compatibility (At first blush, we are not looking too
> bad; our Sean ran some tests over weekend that have hbase-1 client running
> against an hbase-2 cluster....). The Coprocessor Interface revamp should be
> done by alpha3 (i.e. returning Interfaces rather than Implementations, and
> our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
> We'll also have purged thirdparty classes from our API; e.g. guava 0.12
> Service showing through in our replication API and protobufs in Admin
> Interface. On alpha3, we will have to do a bunch of outreach to make sure
> our downstreamers are up on what is coming down the pipe.
>
> Beta1 in mid-September?
>
> I encourage you to check out the items marked for hbase2:
> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as
> you see appropriate. Punt if you know the JIRA will not get any attention
> in next month or so.
>
> A bunch of issues marked blocker are unassigned. I'll leave them as is
> another while but I'll boot them soon.
>
> While I have your attention:
>
> + I think we should leave thrift version at 0.9.3. Moving hbase thrift to
> 0.10.0 will break existing clients. The change is easy enough if folks need
> to upgrade their hbase thrift. See HBASE-18591.
> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).
>
> St.Ack
>
>
>
> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>
>>
>>
>> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
>>
>>>
>>>
>>> On 7/31/17 9:00 AM, Stack wrote:
>>>
>>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>  wrote:
>>>>
>>>> ...
>>>>>
>>>>> I like the idea of this also hitting 2.0 as it would make the feature a
>>>>> bit more "real", but am obviously a little nervous (I have no reason
>>>>> to be
>>>>> nervous though). I am pretty happy with the feature in terms of how
>>>>> much it
>>>>> is covered via testing.
>>>>>
>>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>>
>>>>>
>>>>>
>>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>>
>>>> Last think on space quota says that need doc too. See 'Space Quota' in
>>>> here:
>>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>>> Does this little section need an update Josh?
>>>>
>>>> Thanks,
>>>> S
>>>>
>>>
>>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>>> already included for 17748.  Happy to put that on my plate if you're good
>>> with it. I can reasonably assume that no one is against it :)
>>>
>>> I think I had knocked out docs for the "phase 1" stuff before we merged
>>> it in from the original feature branch. I'll double check and update the
>>> gdoc. Perhaps this was just a timing thing.
>>>
>>
>> Thanks Josh,
>> S
>>
>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Wed, Aug 16, 2017 at 10:18 PM, Guanghao Zhang <zg...@gmail.com> wrote:

> About compatibility, there is one incompatible change about the replication
> TableCFs' config. The old config is a string and it concatenate the list of
> tables and column families in format "table1:cf1,cf2;table2:cfA,cfB" in
> zookeeper for table-cf to replication peer mapping. When parse the config,
> it use ":" to split the string. If table name includes namespace, it will
> be wrong (See HBASE-11386
> <https://issues.apache.org/jira/browse/HBASE-11386>). It is a problem
> since
> we support namespace (0.98). So HBASE-11393
> <https://issues.apache.org/jira/browse/HBASE-11393> (and HBASE-16653
> <https://issues.apache.org/jira/browse/HBASE-16653>) changed it to a PB
> object. When rolling update cluster, you need rolling master first. And the
> master will try to translate the string config to a PB object. But there
> are two problems.
> 1. Permission problem. The replication client can write the zookeeper
> directly. So the znode may have different owner. And master may don't have
> the write permission for the znode. It maybe failed to translate old
> table-cfs string to new PB Object. See HBASE-16938
> <https://issues.apache.org/jira/browse/HBASE-16938>
> 2. We usually keep compatibility between old client and new server. But the
> old replication client may write a string config to znode directly. Then
> the new server can't parse them.
>
> PS: HBASE-11386 is a problem since 0.98. But there are not too many users
> report this problem. So it maybe not a big incompatible issue......
>
>
Excellent writeup Guanghao. File a blocker with the above. We need to have
some facility to get folks over the above hump.
Thanks,
St.Ack



> 2017-08-15 1:54 GMT+08:00 Stack <st...@duboce.net>:
>
> > Heads-up:
> >
> > I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
> > updated dependencies, reliance on relocated popular libs (guava, netty,
> > protobuf), purge of checked-in generated src, and
> master-carries-no-regions
> > by default.
> >
> > alpha3 I hope will follow soon after (end-of-August?). Its theme will be
> > settling the APIs and compatibility (At first blush, we are not looking
> too
> > bad; our Sean ran some tests over weekend that have hbase-1 client
> running
> > against an hbase-2 cluster....). The Coprocessor Interface revamp should
> be
> > done by alpha3 (i.e. returning Interfaces rather than Implementations,
> and
> > our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
> > We'll also have purged thirdparty classes from our API; e.g. guava 0.12
> > Service showing through in our replication API and protobufs in Admin
> > Interface. On alpha3, we will have to do a bunch of outreach to make sure
> > our downstreamers are up on what is coming down the pipe.
> >
> > Beta1 in mid-September?
> >
> > I encourage you to check out the items marked for hbase2:
> > https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as
> > you
> > see appropriate. Punt if you know the JIRA will not get any attention in
> > next month or so.
> >
> > A bunch of issues marked blocker are unassigned. I'll leave them as is
> > another while but I'll boot them soon.
> >
> > While I have your attention:
> >
> > + I think we should leave thrift version at 0.9.3. Moving hbase thrift to
> > 0.10.0 will break existing clients. The change is easy enough if folks
> need
> > to upgrade their hbase thrift. See HBASE-18591.
> > + Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).
> >
> > St.Ack
> >
> >
> >
> > On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
> >
> > >
> > >
> > > On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
> > >
> > >>
> > >>
> > >> On 7/31/17 9:00 AM, Stack wrote:
> > >>
> > >>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
> > wrote:
> > >>>
> > >>> ...
> > >>>>
> > >>>> I like the idea of this also hitting 2.0 as it would make the
> feature
> > a
> > >>>> bit more "real", but am obviously a little nervous (I have no reason
> > to
> > >>>> be
> > >>>> nervous though). I am pretty happy with the feature in terms of how
> > >>>> much it
> > >>>> is covered via testing.
> > >>>>
> > >>>> https://issues.apache.org/jira/browse/HBASE-17748
> > >>>>
> > >>>>
> > >>>>
> > >>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> > >>>
> > >>> Last think on space quota says that need doc too. See 'Space Quota'
> in
> > >>> here:
> > >>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> > >>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> > >>> Does this little section need an update Josh?
> > >>>
> > >>> Thanks,
> > >>> S
> > >>>
> > >>
> > >> Yep, just a couple of cherry-picks. Good test coverage and some docs
> > >> already included for 17748.  Happy to put that on my plate if you're
> > good
> > >> with it. I can reasonably assume that no one is against it :)
> > >>
> > >> I think I had knocked out docs for the "phase 1" stuff before we
> merged
> > >> it in from the original feature branch. I'll double check and update
> the
> > >> gdoc. Perhaps this was just a timing thing.
> > >>
> > >
> > > Thanks Josh,
> > > S
> > >
> >
>

Re: Moving 2.0 forward

[Guanghao Zhang <zg...@gmail.com>]

About compatibility, there is one incompatible change about the replication
TableCFs' config. The old config is a string and it concatenate the list of
tables and column families in format "table1:cf1,cf2;table2:cfA,cfB" in
zookeeper for table-cf to replication peer mapping. When parse the config,
it use ":" to split the string. If table name includes namespace, it will
be wrong (See HBASE-11386
<https://issues.apache.org/jira/browse/HBASE-11386>). It is a problem since
we support namespace (0.98). So HBASE-11393
<https://issues.apache.org/jira/browse/HBASE-11393> (and HBASE-16653
<https://issues.apache.org/jira/browse/HBASE-16653>) changed it to a PB
object. When rolling update cluster, you need rolling master first. And the
master will try to translate the string config to a PB object. But there
are two problems.
1. Permission problem. The replication client can write the zookeeper
directly. So the znode may have different owner. And master may don't have
the write permission for the znode. It maybe failed to translate old
table-cfs string to new PB Object. See HBASE-16938
<https://issues.apache.org/jira/browse/HBASE-16938>
2. We usually keep compatibility between old client and new server. But the
old replication client may write a string config to znode directly. Then
the new server can't parse them.

PS: HBASE-11386 is a problem since 0.98. But there are not too many users
report this problem. So it maybe not a big incompatible issue......

2017-08-15 1:54 GMT+08:00 Stack <st...@duboce.net>:

> Heads-up:
>
> I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
> updated dependencies, reliance on relocated popular libs (guava, netty,
> protobuf), purge of checked-in generated src, and master-carries-no-regions
> by default.
>
> alpha3 I hope will follow soon after (end-of-August?). Its theme will be
> settling the APIs and compatibility (At first blush, we are not looking too
> bad; our Sean ran some tests over weekend that have hbase-1 client running
> against an hbase-2 cluster....). The Coprocessor Interface revamp should be
> done by alpha3 (i.e. returning Interfaces rather than Implementations, and
> our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
> We'll also have purged thirdparty classes from our API; e.g. guava 0.12
> Service showing through in our replication API and protobufs in Admin
> Interface. On alpha3, we will have to do a bunch of outreach to make sure
> our downstreamers are up on what is coming down the pipe.
>
> Beta1 in mid-September?
>
> I encourage you to check out the items marked for hbase2:
> https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as
> you
> see appropriate. Punt if you know the JIRA will not get any attention in
> next month or so.
>
> A bunch of issues marked blocker are unassigned. I'll leave them as is
> another while but I'll boot them soon.
>
> While I have your attention:
>
> + I think we should leave thrift version at 0.9.3. Moving hbase thrift to
> 0.10.0 will break existing clients. The change is easy enough if folks need
> to upgrade their hbase thrift. See HBASE-18591.
> + Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).
>
> St.Ack
>
>
>
> On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:
>
> >
> >
> > On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
> >
> >>
> >>
> >> On 7/31/17 9:00 AM, Stack wrote:
> >>
> >>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>
> wrote:
> >>>
> >>> ...
> >>>>
> >>>> I like the idea of this also hitting 2.0 as it would make the feature
> a
> >>>> bit more "real", but am obviously a little nervous (I have no reason
> to
> >>>> be
> >>>> nervous though). I am pretty happy with the feature in terms of how
> >>>> much it
> >>>> is covered via testing.
> >>>>
> >>>> https://issues.apache.org/jira/browse/HBASE-17748
> >>>>
> >>>>
> >>>>
> >>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> >>>
> >>> Last think on space quota says that need doc too. See 'Space Quota' in
> >>> here:
> >>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> >>> Does this little section need an update Josh?
> >>>
> >>> Thanks,
> >>> S
> >>>
> >>
> >> Yep, just a couple of cherry-picks. Good test coverage and some docs
> >> already included for 17748.  Happy to put that on my plate if you're
> good
> >> with it. I can reasonably assume that no one is against it :)
> >>
> >> I think I had knocked out docs for the "phase 1" stuff before we merged
> >> it in from the original feature branch. I'll double check and update the
> >> gdoc. Perhaps this was just a timing thing.
> >>
> >
> > Thanks Josh,
> > S
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Heads-up:

I'm about to put up an hbase-2.0.0-alpha2 Release Candidate. Theme is
updated dependencies, reliance on relocated popular libs (guava, netty,
protobuf), purge of checked-in generated src, and master-carries-no-regions
by default.

alpha3 I hope will follow soon after (end-of-August?). Its theme will be
settling the APIs and compatibility (At first blush, we are not looking too
bad; our Sean ran some tests over weekend that have hbase-1 client running
against an hbase-2 cluster....). The Coprocessor Interface revamp should be
done by alpha3 (i.e. returning Interfaces rather than Implementations, and
our shutdown of CPs accessing classes in hbase marked InterfaceAudience).
We'll also have purged thirdparty classes from our API; e.g. guava 0.12
Service showing through in our replication API and protobufs in Admin
Interface. On alpha3, we will have to do a bunch of outreach to make sure
our downstreamers are up on what is coming down the pipe.

Beta1 in mid-September?

I encourage you to check out the items marked for hbase2:
https://issues.apache.org/jira/projects/HBASE/versions/12327188 Edit as you
see appropriate. Punt if you know the JIRA will not get any attention in
next month or so.

A bunch of issues marked blocker are unassigned. I'll leave them as is
another while but I'll boot them soon.

While I have your attention:

+ I think we should leave thrift version at 0.9.3. Moving hbase thrift to
0.10.0 will break existing clients. The change is easy enough if folks need
to upgrade their hbase thrift. See HBASE-18591.
+ Upgrade from 0.94 is disallowed. You have to get to 1.0 first (0.98?).

St.Ack



On Wed, Aug 2, 2017 at 9:43 AM, Stack <st...@duboce.net> wrote:

>
>
> On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:
>
>>
>>
>> On 7/31/17 9:00 AM, Stack wrote:
>>
>>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>  wrote:
>>>
>>> ...
>>>>
>>>> I like the idea of this also hitting 2.0 as it would make the feature a
>>>> bit more "real", but am obviously a little nervous (I have no reason to
>>>> be
>>>> nervous though). I am pretty happy with the feature in terms of how
>>>> much it
>>>> is covered via testing.
>>>>
>>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>>
>>>>
>>>>
>>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>>
>>> Last think on space quota says that need doc too. See 'Space Quota' in
>>> here:
>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>>> Does this little section need an update Josh?
>>>
>>> Thanks,
>>> S
>>>
>>
>> Yep, just a couple of cherry-picks. Good test coverage and some docs
>> already included for 17748.  Happy to put that on my plate if you're good
>> with it. I can reasonably assume that no one is against it :)
>>
>> I think I had knocked out docs for the "phase 1" stuff before we merged
>> it in from the original feature branch. I'll double check and update the
>> gdoc. Perhaps this was just a timing thing.
>>
>
> Thanks Josh,
> S
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Tue, Aug 1, 2017 at 2:06 PM, Josh Elser <el...@apache.org> wrote:

>
>
> On 7/31/17 9:00 AM, Stack wrote:
>
>> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>  wrote:
>>
>> ...
>>>
>>> I like the idea of this also hitting 2.0 as it would make the feature a
>>> bit more "real", but am obviously a little nervous (I have no reason to
>>> be
>>> nervous though). I am pretty happy with the feature in terms of how much
>>> it
>>> is covered via testing.
>>>
>>> https://issues.apache.org/jira/browse/HBASE-17748
>>>
>>>
>>>
>>> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
>>
>> Last think on space quota says that need doc too. See 'Space Quota' in
>> here:
>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> Eu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
>> Does this little section need an update Josh?
>>
>> Thanks,
>> S
>>
>
> Yep, just a couple of cherry-picks. Good test coverage and some docs
> already included for 17748.  Happy to put that on my plate if you're good
> with it. I can reasonably assume that no one is against it :)
>
> I think I had knocked out docs for the "phase 1" stuff before we merged it
> in from the original feature branch. I'll double check and update the gdoc.
> Perhaps this was just a timing thing.
>

Thanks Josh,
S

Re: Moving 2.0 forward

[Josh Elser <el...@apache.org>]

On 7/31/17 9:00 AM, Stack wrote:
> On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser<el...@apache.org>  wrote:
> 
>> ...
>>
>> I like the idea of this also hitting 2.0 as it would make the feature a
>> bit more "real", but am obviously a little nervous (I have no reason to be
>> nervous though). I am pretty happy with the feature in terms of how much it
>> is covered via testing.
>>
>> https://issues.apache.org/jira/browse/HBASE-17748
>>
>>
>>
> Sounds good to me. Whats involved? Backport? If so, +1 Josh.
> 
> Last think on space quota says that need doc too. See 'Space Quota' in
> here:
> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
> Does this little section need an update Josh?
> 
> Thanks,
> S

Yep, just a couple of cherry-picks. Good test coverage and some docs 
already included for 17748.  Happy to put that on my plate if you're 
good with it. I can reasonably assume that no one is against it :)

I think I had knocked out docs for the "phase 1" stuff before we merged 
it in from the original feature branch. I'll double check and update the 
gdoc. Perhaps this was just a timing thing.

Re: Moving 2.0 forward

[karthikeyan thirunavukkarasu <me...@yahoo.com.INVALID>]

Hi All,
May be wrong forum, but how to unsubscribe my email id from this email list?
ThanksKarthik

      From: ashish singhi <as...@huawei.com>
 To: "dev@hbase.apache.org" <de...@hbase.apache.org> 
 Sent: Wednesday, July 26, 2017 8:57 PM
 Subject: RE: Moving 2.0 forward
   
One question regarding upgrade: 
Do we expect the user to move to latest branch-1 release and then upgrade to 2.0 version ?

I think that should not be imposed on the user. 

Regards,
Ashish

-----Original Message-----
From: 张铎(Duo Zhang) [mailto:palomino219@gmail.com] 
Sent: 26 July 2017 12:10
To: dev@hbase.apache.org
Subject: Re: Moving 2.0 forward

I set HBASE-18169 as a Blocker because I found some critial problems on our current CPs. The semantics are broken. Although we are allowed to break CP in a major release, I think we need to provide the same ability(in another way).

2017-07-25 1:25 GMT+08:00 Josh Elser <el...@apache.org>:

>
>
> On 7/21/17 12:03 PM, Stack wrote:
>
>> Status update girls and boys!
>>
>> hbase-2.0.0-alpha1 went out June 22nd.
>>
>> alpha2 has been a bit slow to follow (holidays) though there has been 
>> steady progress closing out blockers and criticals by a bunch of you all.
>> The plan is for a release in the first week or so of August. It 
>> should be fully up on hbase-thirdparty using updated (and relocated) 
>> versions of netty, guava, and protobuf as well as a default deploy 
>> that has master-carrying-no-regions.
>>
>> alpha3 will follow soon after and will focus on making sure our 
>> user-facing APIs are clean (branch-1 compatible, no illicit 
>> removals/mods, and so on) and that basic upgrade 'works'.
>>
>> betas start in September?
>>
>> I've been keeping a rough general state here [1] (please update any 
>> section that is lagging actuality) but for details on what blockers 
>> and criticals remain, see the JIRA 2.0 view [2]. Recent 
>> issue-gardening has brought 2.0 into better focus. Feel free to 
>> review and punt items you think can wait till 3.0 or 2.1. If you want 
>> to pull in more stuff, please ask first.
>>
>
> Chia-Ping (I think? -- JIRA is being a pain) had asked on the 
> space-quota
> phase2 work (include size of hbase snapshots in a table's "quota 
> usage") if we should try to also include that work in 2.0.
>
> I like the idea of this also hitting 2.0 as it would make the feature 
> a bit more "real", but am obviously a little nervous (I have no reason 
> to be nervous though). I am pretty happy with the feature in terms of 
> how much it is covered via testing.
>
> https://issues.apache.org/jira/browse/HBASE-17748
>
>
> Thanks,
>> St.Ack
>>
>> 1.
>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> Eu_ktczrlKHK8N4SZzs/edit#
>> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>
>
> - Josh
>


   

Re: Moving 2.0 forward

[Sean Busbey <se...@gmail.com>]

+1 on a dedicated thread for upgrade expectations.

On Mon, Jul 31, 2017 at 8:04 AM, Stack <st...@duboce.net> wrote:
> On Wed, Jul 26, 2017 at 10:57 PM, ashish singhi <as...@huawei.com>
> wrote:
>
>> One question regarding upgrade:
>> Do we expect the user to move to latest branch-1 release and then upgrade
>> to 2.0 version ?
>>
>>
> I think, as you seem to, that this is too much to ask Ashish.
>
> I think we should support at the low end going from 1.0.0 to 2.0.0; i.e.
> you'd have to upgrade to 1.x if you were running 0.98 before you could go
> to 2.0.0.
>
> What do folks think of this? I should start a separate thread on upgrade
> expectations?
>
> St.Ack
>
>
>
>> I think that should not be imposed on the user.
>>
>> Regards,
>> Ashish
>>
>> -----Original Message-----
>> From: 张铎(Duo Zhang) [mailto:palomino219@gmail.com]
>> Sent: 26 July 2017 12:10
>> To: dev@hbase.apache.org
>> Subject: Re: Moving 2.0 forward
>>
>> I set HBASE-18169 as a Blocker because I found some critial problems on
>> our current CPs. The semantics are broken. Although we are allowed to break
>> CP in a major release, I think we need to provide the same ability(in
>> another way).
>>
>> 2017-07-25 1:25 GMT+08:00 Josh Elser <el...@apache.org>:
>>
>> >
>> >
>> > On 7/21/17 12:03 PM, Stack wrote:
>> >
>> >> Status update girls and boys!
>> >>
>> >> hbase-2.0.0-alpha1 went out June 22nd.
>> >>
>> >> alpha2 has been a bit slow to follow (holidays) though there has been
>> >> steady progress closing out blockers and criticals by a bunch of you
>> all.
>> >> The plan is for a release in the first week or so of August. It
>> >> should be fully up on hbase-thirdparty using updated (and relocated)
>> >> versions of netty, guava, and protobuf as well as a default deploy
>> >> that has master-carrying-no-regions.
>> >>
>> >> alpha3 will follow soon after and will focus on making sure our
>> >> user-facing APIs are clean (branch-1 compatible, no illicit
>> >> removals/mods, and so on) and that basic upgrade 'works'.
>> >>
>> >> betas start in September?
>> >>
>> >> I've been keeping a rough general state here [1] (please update any
>> >> section that is lagging actuality) but for details on what blockers
>> >> and criticals remain, see the JIRA 2.0 view [2]. Recent
>> >> issue-gardening has brought 2.0 into better focus. Feel free to
>> >> review and punt items you think can wait till 3.0 or 2.1. If you want
>> >> to pull in more stuff, please ask first.
>> >>
>> >
>> > Chia-Ping (I think? -- JIRA is being a pain) had asked on the
>> > space-quota
>> > phase2 work (include size of hbase snapshots in a table's "quota
>> > usage") if we should try to also include that work in 2.0.
>> >
>> > I like the idea of this also hitting 2.0 as it would make the feature
>> > a bit more "real", but am obviously a little nervous (I have no reason
>> > to be nervous though). I am pretty happy with the feature in terms of
>> > how much it is covered via testing.
>> >
>> > https://issues.apache.org/jira/browse/HBASE-17748
>> >
>> >
>> > Thanks,
>> >> St.Ack
>> >>
>> >> 1.
>> >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> >> Eu_ktczrlKHK8N4SZzs/edit#
>> >> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
>> >>
>> >
>> > - Josh
>> >
>>



-- 
Sean

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Wed, Jul 26, 2017 at 10:57 PM, ashish singhi <as...@huawei.com>
wrote:

> One question regarding upgrade:
> Do we expect the user to move to latest branch-1 release and then upgrade
> to 2.0 version ?
>
>
I think, as you seem to, that this is too much to ask Ashish.

I think we should support at the low end going from 1.0.0 to 2.0.0; i.e.
you'd have to upgrade to 1.x if you were running 0.98 before you could go
to 2.0.0.

What do folks think of this? I should start a separate thread on upgrade
expectations?

St.Ack



> I think that should not be imposed on the user.
>
> Regards,
> Ashish
>
> -----Original Message-----
> From: 张铎(Duo Zhang) [mailto:palomino219@gmail.com]
> Sent: 26 July 2017 12:10
> To: dev@hbase.apache.org
> Subject: Re: Moving 2.0 forward
>
> I set HBASE-18169 as a Blocker because I found some critial problems on
> our current CPs. The semantics are broken. Although we are allowed to break
> CP in a major release, I think we need to provide the same ability(in
> another way).
>
> 2017-07-25 1:25 GMT+08:00 Josh Elser <el...@apache.org>:
>
> >
> >
> > On 7/21/17 12:03 PM, Stack wrote:
> >
> >> Status update girls and boys!
> >>
> >> hbase-2.0.0-alpha1 went out June 22nd.
> >>
> >> alpha2 has been a bit slow to follow (holidays) though there has been
> >> steady progress closing out blockers and criticals by a bunch of you
> all.
> >> The plan is for a release in the first week or so of August. It
> >> should be fully up on hbase-thirdparty using updated (and relocated)
> >> versions of netty, guava, and protobuf as well as a default deploy
> >> that has master-carrying-no-regions.
> >>
> >> alpha3 will follow soon after and will focus on making sure our
> >> user-facing APIs are clean (branch-1 compatible, no illicit
> >> removals/mods, and so on) and that basic upgrade 'works'.
> >>
> >> betas start in September?
> >>
> >> I've been keeping a rough general state here [1] (please update any
> >> section that is lagging actuality) but for details on what blockers
> >> and criticals remain, see the JIRA 2.0 view [2]. Recent
> >> issue-gardening has brought 2.0 into better focus. Feel free to
> >> review and punt items you think can wait till 3.0 or 2.1. If you want
> >> to pull in more stuff, please ask first.
> >>
> >
> > Chia-Ping (I think? -- JIRA is being a pain) had asked on the
> > space-quota
> > phase2 work (include size of hbase snapshots in a table's "quota
> > usage") if we should try to also include that work in 2.0.
> >
> > I like the idea of this also hitting 2.0 as it would make the feature
> > a bit more "real", but am obviously a little nervous (I have no reason
> > to be nervous though). I am pretty happy with the feature in terms of
> > how much it is covered via testing.
> >
> > https://issues.apache.org/jira/browse/HBASE-17748
> >
> >
> > Thanks,
> >> St.Ack
> >>
> >> 1.
> >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >> Eu_ktczrlKHK8N4SZzs/edit#
> >> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
> >>
> >
> > - Josh
> >
>

RE: Moving 2.0 forward

[ashish singhi <as...@huawei.com>]

One question regarding upgrade: 
Do we expect the user to move to latest branch-1 release and then upgrade to 2.0 version ?

I think that should not be imposed on the user. 

Regards,
Ashish

-----Original Message-----
From: 张铎(Duo Zhang) [mailto:palomino219@gmail.com] 
Sent: 26 July 2017 12:10
To: dev@hbase.apache.org
Subject: Re: Moving 2.0 forward

I set HBASE-18169 as a Blocker because I found some critial problems on our current CPs. The semantics are broken. Although we are allowed to break CP in a major release, I think we need to provide the same ability(in another way).

2017-07-25 1:25 GMT+08:00 Josh Elser <el...@apache.org>:

>
>
> On 7/21/17 12:03 PM, Stack wrote:
>
>> Status update girls and boys!
>>
>> hbase-2.0.0-alpha1 went out June 22nd.
>>
>> alpha2 has been a bit slow to follow (holidays) though there has been 
>> steady progress closing out blockers and criticals by a bunch of you all.
>> The plan is for a release in the first week or so of August. It 
>> should be fully up on hbase-thirdparty using updated (and relocated) 
>> versions of netty, guava, and protobuf as well as a default deploy 
>> that has master-carrying-no-regions.
>>
>> alpha3 will follow soon after and will focus on making sure our 
>> user-facing APIs are clean (branch-1 compatible, no illicit 
>> removals/mods, and so on) and that basic upgrade 'works'.
>>
>> betas start in September?
>>
>> I've been keeping a rough general state here [1] (please update any 
>> section that is lagging actuality) but for details on what blockers 
>> and criticals remain, see the JIRA 2.0 view [2]. Recent 
>> issue-gardening has brought 2.0 into better focus. Feel free to 
>> review and punt items you think can wait till 3.0 or 2.1. If you want 
>> to pull in more stuff, please ask first.
>>
>
> Chia-Ping (I think? -- JIRA is being a pain) had asked on the 
> space-quota
> phase2 work (include size of hbase snapshots in a table's "quota 
> usage") if we should try to also include that work in 2.0.
>
> I like the idea of this also hitting 2.0 as it would make the feature 
> a bit more "real", but am obviously a little nervous (I have no reason 
> to be nervous though). I am pretty happy with the feature in terms of 
> how much it is covered via testing.
>
> https://issues.apache.org/jira/browse/HBASE-17748
>
>
> Thanks,
>> St.Ack
>>
>> 1.
>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> Eu_ktczrlKHK8N4SZzs/edit#
>> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>
>
> - Josh
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Wed, Jul 26, 2017 at 1:39 AM, 张铎(Duo Zhang) <pa...@gmail.com>
wrote:

> I set HBASE-18169 as a Blocker because I found some critial problems on our
> current CPs. The semantics are broken. Although we are allowed to break CP
> in a major release, I think we need to provide the same ability(in another
> way).
>
> Thank you. Agree. Chatting w/ Anoop this morning (offline), perhaps we can
huddle at hbasecon for an hour to clean this all up.

Thanks Duo,

S



> 2017-07-25 1:25 GMT+08:00 Josh Elser <el...@apache.org>:
>
> >
> >
> > On 7/21/17 12:03 PM, Stack wrote:
> >
> >> Status update girls and boys!
> >>
> >> hbase-2.0.0-alpha1 went out June 22nd.
> >>
> >> alpha2 has been a bit slow to follow (holidays) though there has been
> >> steady progress closing out blockers and criticals by a bunch of you
> all.
> >> The plan is for a release in the first week or so of August. It should
> be
> >> fully up on hbase-thirdparty using updated (and relocated) versions of
> >> netty, guava, and protobuf as well as a default deploy that has
> >> master-carrying-no-regions.
> >>
> >> alpha3 will follow soon after and will focus on making sure our
> >> user-facing
> >> APIs are clean (branch-1 compatible, no illicit removals/mods, and so
> on)
> >> and that basic upgrade 'works'.
> >>
> >> betas start in September?
> >>
> >> I've been keeping a rough general state here [1] (please update any
> >> section
> >> that is lagging actuality) but for details on what blockers and
> criticals
> >> remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought
> 2.0
> >> into better focus. Feel free to review and punt items you think can wait
> >> till 3.0 or 2.1. If you want to pull in more stuff, please ask first.
> >>
> >
> > Chia-Ping (I think? -- JIRA is being a pain) had asked on the space-quota
> > phase2 work (include size of hbase snapshots in a table's "quota usage")
> if
> > we should try to also include that work in 2.0.
> >
> > I like the idea of this also hitting 2.0 as it would make the feature a
> > bit more "real", but am obviously a little nervous (I have no reason to
> be
> > nervous though). I am pretty happy with the feature in terms of how much
> it
> > is covered via testing.
> >
> > https://issues.apache.org/jira/browse/HBASE-17748
> >
> >
> > Thanks,
> >> St.Ack
> >>
> >> 1.
> >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >> Eu_ktczrlKHK8N4SZzs/edit#
> >> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
> >>
> >
> > - Josh
> >
>

Re: Moving 2.0 forward

["张铎 (Duo Zhang)" <pa...@gmail.com>]

I set HBASE-18169 as a Blocker because I found some critial problems on our
current CPs. The semantics are broken. Although we are allowed to break CP
in a major release, I think we need to provide the same ability(in another
way).

2017-07-25 1:25 GMT+08:00 Josh Elser <el...@apache.org>:

>
>
> On 7/21/17 12:03 PM, Stack wrote:
>
>> Status update girls and boys!
>>
>> hbase-2.0.0-alpha1 went out June 22nd.
>>
>> alpha2 has been a bit slow to follow (holidays) though there has been
>> steady progress closing out blockers and criticals by a bunch of you all.
>> The plan is for a release in the first week or so of August. It should be
>> fully up on hbase-thirdparty using updated (and relocated) versions of
>> netty, guava, and protobuf as well as a default deploy that has
>> master-carrying-no-regions.
>>
>> alpha3 will follow soon after and will focus on making sure our
>> user-facing
>> APIs are clean (branch-1 compatible, no illicit removals/mods, and so on)
>> and that basic upgrade 'works'.
>>
>> betas start in September?
>>
>> I've been keeping a rough general state here [1] (please update any
>> section
>> that is lagging actuality) but for details on what blockers and criticals
>> remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought 2.0
>> into better focus. Feel free to review and punt items you think can wait
>> till 3.0 or 2.1. If you want to pull in more stuff, please ask first.
>>
>
> Chia-Ping (I think? -- JIRA is being a pain) had asked on the space-quota
> phase2 work (include size of hbase snapshots in a table's "quota usage") if
> we should try to also include that work in 2.0.
>
> I like the idea of this also hitting 2.0 as it would make the feature a
> bit more "real", but am obviously a little nervous (I have no reason to be
> nervous though). I am pretty happy with the feature in terms of how much it
> is covered via testing.
>
> https://issues.apache.org/jira/browse/HBASE-17748
>
>
> Thanks,
>> St.Ack
>>
>> 1.
>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> Eu_ktczrlKHK8N4SZzs/edit#
>> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>
>
> - Josh
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Mon, Jul 24, 2017 at 12:25 PM, Josh Elser <el...@apache.org> wrote:

> ...
>
> I like the idea of this also hitting 2.0 as it would make the feature a
> bit more "real", but am obviously a little nervous (I have no reason to be
> nervous though). I am pretty happy with the feature in terms of how much it
> is covered via testing.
>
> https://issues.apache.org/jira/browse/HBASE-17748
>
>
>
Sounds good to me. Whats involved? Backport? If so, +1 Josh.

Last think on space quota says that need doc too. See 'Space Quota' in
here:
https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.wuw3a6jukzo5
Does this little section need an update Josh?

Thanks,
S




> Thanks,
>> St.Ack
>>
>> 1.
>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> Eu_ktczrlKHK8N4SZzs/edit#
>> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
>>
>
> - Josh
>

Re: Moving 2.0 forward

[Josh Elser <el...@apache.org>]

On 7/21/17 12:03 PM, Stack wrote:
> Status update girls and boys!
> 
> hbase-2.0.0-alpha1 went out June 22nd.
> 
> alpha2 has been a bit slow to follow (holidays) though there has been
> steady progress closing out blockers and criticals by a bunch of you all.
> The plan is for a release in the first week or so of August. It should be
> fully up on hbase-thirdparty using updated (and relocated) versions of
> netty, guava, and protobuf as well as a default deploy that has
> master-carrying-no-regions.
> 
> alpha3 will follow soon after and will focus on making sure our user-facing
> APIs are clean (branch-1 compatible, no illicit removals/mods, and so on)
> and that basic upgrade 'works'.
> 
> betas start in September?
> 
> I've been keeping a rough general state here [1] (please update any section
> that is lagging actuality) but for details on what blockers and criticals
> remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought 2.0
> into better focus. Feel free to review and punt items you think can wait
> till 3.0 or 2.1. If you want to pull in more stuff, please ask first.

Chia-Ping (I think? -- JIRA is being a pain) had asked on the 
space-quota phase2 work (include size of hbase snapshots in a table's 
"quota usage") if we should try to also include that work in 2.0.

I like the idea of this also hitting 2.0 as it would make the feature a 
bit more "real", but am obviously a little nervous (I have no reason to 
be nervous though). I am pretty happy with the feature in terms of how 
much it is covered via testing.

https://issues.apache.org/jira/browse/HBASE-17748

> Thanks,
> St.Ack
> 
> 1.
> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#
> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188

- Josh

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Thu, Mar 30, 2017 at 6:22 PM, Enis Söztutar <en...@apache.org> wrote:

> Thanks Stack for the update. +1 on branching as soon as possible. For
> getting aforementioned stability, we need to start rejecting patches/
> features from 2.0.0. Branching early gives us the option of gradually
> working towards that, but also does not block new development to happen on
> master. I think the most important job for the RM is to say NO to
> improvement jiras going into 2.0, if they have nothing to do with the
> agreed upon goals of the release.
>
>
Agreed (I've been practicing with a mirror).

And thanks Yu Li for volunteering to help with stability (a few others have
expressed interest too).

St.Ack



> Enis
>
> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> wrote:
>
> > Some notes on progress toward hbase2.
> >
> > Given that stability and performance are NOT emergent behaviors but
> rather
> > projects unto themselves, my thought is that we commit all that we've
> > agreed as core for hbase2 (see [1]), branch, and then work on stabilizing
> > and perf rather than do stabilize, commit, and then branch. What this
> means
> > in practice is that for features like Inmemory Compaction, we commit it
> > defaulted 'on' ("BASIC" mode) which is what we want in hbase2. Should it
> > prove problematic under test, we disable it before release.
> >
> > Are folks good w/ this mode? I ask because, in a few issues there are
> > requests for proof that a master feature is 'stable' before commit. This
> is
> > normally a healthy request only in master's case, it is hard to
> demonstrate
> > stability given its current state.
> >
> > Other outstanding issues such as decisions about whether master hosts
> > system tables only (by default), I'm thinking, we can work out post
> branch
> > in alpha/betas before release.
> >
> > The awkward item is the long-pole Assignment Manager. This is an
> > all-or-nothing affair. Here we are switching in a new Master core. While
> I
> > think it fine that AMv2 is incomplete come branch time, those of us
> working
> > on the new AM still need to demonstrate to you all that it basically
> > viable.
> >
> > The point-of-no-return is commit of the patch in HBASE-14614. HBASE-14614
> > (AMv2) is coming close to passing all unit tests. We'll spend some time
> > running it on a cluster to make sure it fundamentally sound and will
> report
> > back on our experience. There has been an ask for some dev doc and
> > low-levels on how it works (in progress). Let satisfaction of these
> > requests be blockers on commit. We'll put the HBASE-14614 commit up for a
> > vote on dev list given its import.
> >
> > Branch will happen after HBASE-14614 goes in (or its rejection) with our
> > first alpha soon after. Its looking like a week or two at least given how
> > things have been going up to this.
> >
> > I intend to start in on hbase2 stability/perf projects after we branch.
> >
> > Interested in any thoughts you all might have on the above (Would also
> > appreciate updates on state in [1] if you are a feature owner).
> >
> > Thanks,
> > St.Ack
> >
> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> > z9iEu_ktczrlKHK8N4SZzs/edit#
> >
> >
> >
> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <el...@apache.org> wrote:
> >
> > >
> > > Stack wrote:
> > >
> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<el...@apache.org>  wrote:
> > >>
> > >> Thanks for pulling in the FS Quotas work, Stack. I'm trying to cross
> the
> > >>> last T's and dot the last I's.
> > >>>
> > >>> The biggest thing I know I need to do still is to write a new chapter
> > to
> > >>> the book. After that, I'd start entertaining larger
> reviews/discussions
> > >>> to
> > >>> merge the feature into master. Anyone with free time (giggles) is
> more
> > >>> than
> > >>> welcome to start perusing :)
> > >>>
> > >>>
> > >>> Out of interest, this could come in after 2.0 Josh? Any 2.0 specific
> > >> needs
> > >> to make this work?
> > >>
> > >> Meantime, updated the 2.0 doc 1.
> > >>
> > >> Thanks Josh,
> > >> St.Ack
> > >>
> > >> 1.
> > >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> > >> Eu_ktczrlKHK8N4SZzs/edit#
> > >>
> > >>
> > > Nope, no need to block 2.0 on this one (given the other, related
> > chatter).
> > > Would be nice to get it in, but I completely understand if it slips :)
> > >
> > > Thanks for updating the doc for me!
> > >
> >
>

Re: Moving 2.0 forward

[Yu Li <ca...@gmail.com>]

+1 on early branch, and count me in for the coming stability/perf projects
(smile).

Best Regards,
Yu

On 31 March 2017 at 09:22, Enis Söztutar <en...@apache.org> wrote:

> Thanks Stack for the update. +1 on branching as soon as possible. For
> getting aforementioned stability, we need to start rejecting patches/
> features from 2.0.0. Branching early gives us the option of gradually
> working towards that, but also does not block new development to happen on
> master. I think the most important job for the RM is to say NO to
> improvement jiras going into 2.0, if they have nothing to do with the
> agreed upon goals of the release.
>
> Enis
>
> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> wrote:
>
> > Some notes on progress toward hbase2.
> >
> > Given that stability and performance are NOT emergent behaviors but
> rather
> > projects unto themselves, my thought is that we commit all that we've
> > agreed as core for hbase2 (see [1]), branch, and then work on stabilizing
> > and perf rather than do stabilize, commit, and then branch. What this
> means
> > in practice is that for features like Inmemory Compaction, we commit it
> > defaulted 'on' ("BASIC" mode) which is what we want in hbase2. Should it
> > prove problematic under test, we disable it before release.
> >
> > Are folks good w/ this mode? I ask because, in a few issues there are
> > requests for proof that a master feature is 'stable' before commit. This
> is
> > normally a healthy request only in master's case, it is hard to
> demonstrate
> > stability given its current state.
> >
> > Other outstanding issues such as decisions about whether master hosts
> > system tables only (by default), I'm thinking, we can work out post
> branch
> > in alpha/betas before release.
> >
> > The awkward item is the long-pole Assignment Manager. This is an
> > all-or-nothing affair. Here we are switching in a new Master core. While
> I
> > think it fine that AMv2 is incomplete come branch time, those of us
> working
> > on the new AM still need to demonstrate to you all that it basically
> > viable.
> >
> > The point-of-no-return is commit of the patch in HBASE-14614. HBASE-14614
> > (AMv2) is coming close to passing all unit tests. We'll spend some time
> > running it on a cluster to make sure it fundamentally sound and will
> report
> > back on our experience. There has been an ask for some dev doc and
> > low-levels on how it works (in progress). Let satisfaction of these
> > requests be blockers on commit. We'll put the HBASE-14614 commit up for a
> > vote on dev list given its import.
> >
> > Branch will happen after HBASE-14614 goes in (or its rejection) with our
> > first alpha soon after. Its looking like a week or two at least given how
> > things have been going up to this.
> >
> > I intend to start in on hbase2 stability/perf projects after we branch.
> >
> > Interested in any thoughts you all might have on the above (Would also
> > appreciate updates on state in [1] if you are a feature owner).
> >
> > Thanks,
> > St.Ack
> >
> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> > z9iEu_ktczrlKHK8N4SZzs/edit#
> >
> >
> >
> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <el...@apache.org> wrote:
> >
> > >
> > > Stack wrote:
> > >
> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<el...@apache.org>  wrote:
> > >>
> > >> Thanks for pulling in the FS Quotas work, Stack. I'm trying to cross
> the
> > >>> last T's and dot the last I's.
> > >>>
> > >>> The biggest thing I know I need to do still is to write a new chapter
> > to
> > >>> the book. After that, I'd start entertaining larger
> reviews/discussions
> > >>> to
> > >>> merge the feature into master. Anyone with free time (giggles) is
> more
> > >>> than
> > >>> welcome to start perusing :)
> > >>>
> > >>>
> > >>> Out of interest, this could come in after 2.0 Josh? Any 2.0 specific
> > >> needs
> > >> to make this work?
> > >>
> > >> Meantime, updated the 2.0 doc 1.
> > >>
> > >> Thanks Josh,
> > >> St.Ack
> > >>
> > >> 1.
> > >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> > >> Eu_ktczrlKHK8N4SZzs/edit#
> > >>
> > >>
> > > Nope, no need to block 2.0 on this one (given the other, related
> > chatter).
> > > Would be nice to get it in, but I completely understand if it slips :)
> > >
> > > Thanks for updating the doc for me!
> > >
> >
>

Re: Moving 2.0 forward

[Enis Söztutar <en...@apache.org>]

Thanks Stack for the update. +1 on branching as soon as possible. For
getting aforementioned stability, we need to start rejecting patches/
features from 2.0.0. Branching early gives us the option of gradually
working towards that, but also does not block new development to happen on
master. I think the most important job for the RM is to say NO to
improvement jiras going into 2.0, if they have nothing to do with the
agreed upon goals of the release.

Enis

On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> wrote:

> Some notes on progress toward hbase2.
>
> Given that stability and performance are NOT emergent behaviors but rather
> projects unto themselves, my thought is that we commit all that we've
> agreed as core for hbase2 (see [1]), branch, and then work on stabilizing
> and perf rather than do stabilize, commit, and then branch. What this means
> in practice is that for features like Inmemory Compaction, we commit it
> defaulted 'on' ("BASIC" mode) which is what we want in hbase2. Should it
> prove problematic under test, we disable it before release.
>
> Are folks good w/ this mode? I ask because, in a few issues there are
> requests for proof that a master feature is 'stable' before commit. This is
> normally a healthy request only in master's case, it is hard to demonstrate
> stability given its current state.
>
> Other outstanding issues such as decisions about whether master hosts
> system tables only (by default), I'm thinking, we can work out post branch
> in alpha/betas before release.
>
> The awkward item is the long-pole Assignment Manager. This is an
> all-or-nothing affair. Here we are switching in a new Master core. While I
> think it fine that AMv2 is incomplete come branch time, those of us working
> on the new AM still need to demonstrate to you all that it basically
> viable.
>
> The point-of-no-return is commit of the patch in HBASE-14614. HBASE-14614
> (AMv2) is coming close to passing all unit tests. We'll spend some time
> running it on a cluster to make sure it fundamentally sound and will report
> back on our experience. There has been an ask for some dev doc and
> low-levels on how it works (in progress). Let satisfaction of these
> requests be blockers on commit. We'll put the HBASE-14614 commit up for a
> vote on dev list given its import.
>
> Branch will happen after HBASE-14614 goes in (or its rejection) with our
> first alpha soon after. Its looking like a week or two at least given how
> things have been going up to this.
>
> I intend to start in on hbase2 stability/perf projects after we branch.
>
> Interested in any thoughts you all might have on the above (Would also
> appreciate updates on state in [1] if you are a feature owner).
>
> Thanks,
> St.Ack
>
> 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> z9iEu_ktczrlKHK8N4SZzs/edit#
>
>
>
> On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <el...@apache.org> wrote:
>
> >
> > Stack wrote:
> >
> >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<el...@apache.org>  wrote:
> >>
> >> Thanks for pulling in the FS Quotas work, Stack. I'm trying to cross the
> >>> last T's and dot the last I's.
> >>>
> >>> The biggest thing I know I need to do still is to write a new chapter
> to
> >>> the book. After that, I'd start entertaining larger reviews/discussions
> >>> to
> >>> merge the feature into master. Anyone with free time (giggles) is more
> >>> than
> >>> welcome to start perusing :)
> >>>
> >>>
> >>> Out of interest, this could come in after 2.0 Josh? Any 2.0 specific
> >> needs
> >> to make this work?
> >>
> >> Meantime, updated the 2.0 doc 1.
> >>
> >> Thanks Josh,
> >> St.Ack
> >>
> >> 1.
> >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >> Eu_ktczrlKHK8N4SZzs/edit#
> >>
> >>
> > Nope, no need to block 2.0 on this one (given the other, related
> chatter).
> > Would be nice to get it in, but I completely understand if it slips :)
> >
> > Thanks for updating the doc for me!
> >
>

Re: Moving 2.0 forward

[Mike Drob <md...@apache.org>]

Yea, beta sounds good to me. Probably we'll find more things once we start
running ITBLL and friends for long periods, but that can maybe come later.
Existing tests should be cleaned up "soon".

On Fri, Jul 21, 2017 at 3:45 PM, Andrew Purtell <ap...@apache.org> wrote:

> Beta?
>
>
> On Fri, Jul 21, 2017 at 1:16 PM, Stack <st...@duboce.net> wrote:
>
> > On Fri, Jul 21, 2017 at 5:30 PM, Mike Drob <md...@apache.org> wrote:
> >
> > >
> > > One thing that is not clear to me from your summary and the linked
> > document
> > > is which release we can block on tests? Unit tests are in bad shape
> (but
> > > getting better) and I recently started looking at ITs which also need
> > some
> > > care.
> > >
> > >
> > Dunno. What you think boss? I'd say we can't release if tests are a mess.
> > Should we block on RC (as in alpha, beta, then RC)?  Or before then? i.e.
> > no beta unless all tests (unit and IT) are passing.
> >
> > For sure testing -- as in exercising new features, changed configs, and
> > just straight longevity under stress -- is badly wanting. I should write
> up
> > a matrix I suppose? Let me work on it.
> >
> > Thanks Mike,
> > S
> >
> >
> >
> >
> > > Mike
> > >
> > > On Fri, Jul 21, 2017 at 11:03 AM, Stack <st...@duboce.net> wrote:
> > >
> > > > Status update girls and boys!
> > > >
> > > > hbase-2.0.0-alpha1 went out June 22nd.
> > > >
> > > > alpha2 has been a bit slow to follow (holidays) though there has been
> > > > steady progress closing out blockers and criticals by a bunch of you
> > all.
> > > > The plan is for a release in the first week or so of August. It
> should
> > be
> > > > fully up on hbase-thirdparty using updated (and relocated) versions
> of
> > > > netty, guava, and protobuf as well as a default deploy that has
> > > > master-carrying-no-regions.
> > > >
> > > > alpha3 will follow soon after and will focus on making sure our
> > > user-facing
> > > > APIs are clean (branch-1 compatible, no illicit removals/mods, and so
> > on)
> > > > and that basic upgrade 'works'.
> > > >
> > > > betas start in September?
> > > >
> > > > I've been keeping a rough general state here [1] (please update any
> > > section
> > > > that is lagging actuality) but for details on what blockers and
> > criticals
> > > > remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought
> > 2.0
> > > > into better focus. Feel free to review and punt items you think can
> > wait
> > > > till 3.0 or 2.1. If you want to pull in more stuff, please ask first.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > >
> > > > 1.
> > > > https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > > > ktczrlKHK8N4SZzs/edit#
> > > > 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
> > > >
> > > >
> > > > On Mon, May 15, 2017 at 4:05 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > >
> > > > > On Mon, May 15, 2017 at 1:46 AM, 张铎(Duo Zhang) <
> > palomino219@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> HBASE-18037 is a new blocker. I'm currently working on it, will be
> > > > >> finished
> > > > >> soon I think.
> > > > >>
> > > > >> I made it a blocker then and added it to our hbase2 release doc
> [1]
> > > list
> > > > > as a blocker.
> > > > >
> > > > > Thanks,
> > > > > St.Ack
> > > > >
> > > > > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > > > > ktczrlKHK8N4SZzs/edit#
> > > > >
> > > > >
> > > > >> 2017-05-15 14:12 GMT+08:00 Stack <st...@duboce.net>:
> > > > >>
> > > > >>
> > > > >> > A month on. Status.
> > > > >> >
> > > > >> > I've been working on the HBASE-14614 branch cluster testing.
> > After a
> > > > >> load
> > > > >> > of fixing, the branch passes smaller test runs (an hour or so of
> > > ITBLL
> > > > >> up
> > > > >> > to 2B rows w/ killing monkeys). When I go larger, to a scale
> I've
> > > not
> > > > >> done
> > > > >> > in a while, I start to run into other interesting issues -- some
> > of
> > > > >> which
> > > > >> > are related to AMv2 (I'm fixing), but others are not (100G WALs
> > that
> > > > >> take
> > > > >> > ten minutes to split makes for interesting cascades when monkeys
> > > kill
> > > > >> > inside the ten minutes...). I intend to keep on with this larger
> > > scale
> > > > >> > testing since it is uncovering good stuff (especially when HDFS
> is
> > > dog
> > > > >> slow
> > > > >> > because of background replications) but my thinking is that I
> > should
> > > > be
> > > > >> > large scale testing branch-2, not just HBASE-14614. I think
> > > > HBASE-14614,
> > > > >> > the new AMv2, is good enough to merge to master these times.
> Given
> > > it
> > > > is
> > > > >> > the last blocker, once in, I'll cut the hbase2 branch.
> > > > >> >
> > > > >> > I'll start up a 'Merge HBASE-14614' DISCUSSION thread in the
> next
> > > day
> > > > >> or so
> > > > >> > (I need to fix some unit tests...).
> > > > >> >
> > > > >> > The AMv2 doc is still a work in progress but should give a gist
> on
> > > > >> where we
> > > > >> > are currently[1].  There is a bunch of todo still but seems
> > > tractable;
> > > > >> e.g.
> > > > >> > rolling upgrade, finish doc., and we don't have an HBCK since it
> > > needs
> > > > >> to
> > > > >> > be recast in light of how stuff now works but a redo on HBCK is
> > > > >> premature
> > > > >> > given we don't know failure types as yet (we just fix the
> problems
> > > as
> > > > >> they
> > > > >> > come up).
> > > > >> >
> > > > >> > St.Ack
> > > > >> > 1.
> > > > >> > https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > > > >> > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Thu, Apr 13, 2017 at 1:43 PM, Stack <st...@duboce.net>
> wrote:
> > > > >> >
> > > > >> > > Some status:
> > > > >> > >
> > > > >> > > AMv2 (HBASE-14614) is near to passing all tests caveat my
> > > disabling
> > > > of
> > > > >> > all
> > > > >> > > to-do w/ fsck (fsck needs revamp) and tests that expect that
> > they
> > > > can
> > > > >> > move
> > > > >> > > hbase;meta off master (AMv2 enforces this constraint; it is
> > > supposed
> > > > >> to
> > > > >> > be
> > > > >> > > enforced on AMv1 but meta-on-master is incompletely realized
> in
> > > AMv1
> > > > >> and
> > > > >> > > AMv2). A few other tests have been disabled for various
> reasons.
> > > See
> > > > >> [1]
> > > > >> > > for full list.
> > > > >> > >
> > > > >> > > There is a hefty list of TODOs still (Again see the messy doc
> > [1])
> > > > but
> > > > >> > the
> > > > >> > > only 'blocker', IMO, is community confidence in AMv2.
> Currently,
> > > > >> cluster
> > > > >> > > tests with chaos fail (new form of 'stuck' regions). Takes
> time
> > > > >> > > investigating.
> > > > >> > >
> > > > >> > > Will keep you all posted.
> > > > >> > > St.Ack
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > > > >> > > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell <
> > > > apurtell@apache.org>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > >> +1 on branching (yay!)
> > > > >> > >>
> > > > >> > >> I have EC2 resources for running ITBLL etc.
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net>
> > wrote:
> > > > >> > >>
> > > > >> > >> > Some notes on progress toward hbase2.
> > > > >> > >> >
> > > > >> > >> > Given that stability and performance are NOT emergent
> > behaviors
> > > > but
> > > > >> > >> rather
> > > > >> > >> > projects unto themselves, my thought is that we commit all
> > that
> > > > >> we've
> > > > >> > >> > agreed as core for hbase2 (see [1]), branch, and then work
> on
> > > > >> > >> stabilizing
> > > > >> > >> > and perf rather than do stabilize, commit, and then branch.
> > > What
> > > > >> this
> > > > >> > >> means
> > > > >> > >> > in practice is that for features like Inmemory Compaction,
> we
> > > > >> commit
> > > > >> > it
> > > > >> > >> > defaulted 'on' ("BASIC" mode) which is what we want in
> > hbase2.
> > > > >> Should
> > > > >> > it
> > > > >> > >> > prove problematic under test, we disable it before release.
> > > > >> > >> >
> > > > >> > >> > Are folks good w/ this mode? I ask because, in a few issues
> > > there
> > > > >> are
> > > > >> > >> > requests for proof that a master feature is 'stable' before
> > > > commit.
> > > > >> > >> This is
> > > > >> > >> > normally a healthy request only in master's case, it is
> hard
> > to
> > > > >> > >> demonstrate
> > > > >> > >> > stability given its current state.
> > > > >> > >> >
> > > > >> > >> > Other outstanding issues such as decisions about whether
> > master
> > > > >> hosts
> > > > >> > >> > system tables only (by default), I'm thinking, we can work
> > out
> > > > post
> > > > >> > >> branch
> > > > >> > >> > in alpha/betas before release.
> > > > >> > >> >
> > > > >> > >> > The awkward item is the long-pole Assignment Manager. This
> is
> > > an
> > > > >> > >> > all-or-nothing affair. Here we are switching in a new
> Master
> > > > core.
> > > > >> > >> While I
> > > > >> > >> > think it fine that AMv2 is incomplete come branch time,
> those
> > > of
> > > > us
> > > > >> > >> working
> > > > >> > >> > on the new AM still need to demonstrate to you all that it
> > > > >> basically
> > > > >> > >> > viable.
> > > > >> > >> >
> > > > >> > >> > The point-of-no-return is commit of the patch in
> HBASE-14614.
> > > > >> > >> HBASE-14614
> > > > >> > >> > (AMv2) is coming close to passing all unit tests. We'll
> spend
> > > > some
> > > > >> > time
> > > > >> > >> > running it on a cluster to make sure it fundamentally sound
> > and
> > > > >> will
> > > > >> > >> report
> > > > >> > >> > back on our experience. There has been an ask for some dev
> > doc
> > > > and
> > > > >> > >> > low-levels on how it works (in progress). Let satisfaction
> of
> > > > these
> > > > >> > >> > requests be blockers on commit. We'll put the HBASE-14614
> > > commit
> > > > up
> > > > >> > for
> > > > >> > >> a
> > > > >> > >> > vote on dev list given its import.
> > > > >> > >> >
> > > > >> > >> > Branch will happen after HBASE-14614 goes in (or its
> > rejection)
> > > > >> with
> > > > >> > our
> > > > >> > >> > first alpha soon after. Its looking like a week or two at
> > least
> > > > >> given
> > > > >> > >> how
> > > > >> > >> > things have been going up to this.
> > > > >> > >> >
> > > > >> > >> > I intend to start in on hbase2 stability/perf projects
> after
> > we
> > > > >> > branch.
> > > > >> > >> >
> > > > >> > >> > Interested in any thoughts you all might have on the above
> > > (Would
> > > > >> also
> > > > >> > >> > appreciate updates on state in [1] if you are a feature
> > owner).
> > > > >> > >> >
> > > > >> > >> > Thanks,
> > > > >> > >> > St.Ack
> > > > >> > >> >
> > > > >> > >> > 1. https://docs.google.com/document/d/
> 1WCsVlnHjJeKUcl7wHwqb4
> > > > >> > >> > z9iEu_ktczrlKHK8N4SZzs/edit#
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <
> > elserj@apache.org
> > > >
> > > > >> > wrote:
> > > > >> > >> >
> > > > >> > >> > >
> > > > >> > >> > > Stack wrote:
> > > > >> > >> > >
> > > > >> > >> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<
> > > elserj@apache.org>
> > > > >> > >> wrote:
> > > > >> > >> > >>
> > > > >> > >> > >> Thanks for pulling in the FS Quotas work, Stack. I'm
> > trying
> > > to
> > > > >> > cross
> > > > >> > >> the
> > > > >> > >> > >>> last T's and dot the last I's.
> > > > >> > >> > >>>
> > > > >> > >> > >>> The biggest thing I know I need to do still is to
> write a
> > > new
> > > > >> > >> chapter
> > > > >> > >> > to
> > > > >> > >> > >>> the book. After that, I'd start entertaining larger
> > > > >> > >> reviews/discussions
> > > > >> > >> > >>> to
> > > > >> > >> > >>> merge the feature into master. Anyone with free time
> > > > (giggles)
> > > > >> is
> > > > >> > >> more
> > > > >> > >> > >>> than
> > > > >> > >> > >>> welcome to start perusing :)
> > > > >> > >> > >>>
> > > > >> > >> > >>>
> > > > >> > >> > >>> Out of interest, this could come in after 2.0 Josh? Any
> > 2.0
> > > > >> > specific
> > > > >> > >> > >> needs
> > > > >> > >> > >> to make this work?
> > > > >> > >> > >>
> > > > >> > >> > >> Meantime, updated the 2.0 doc 1.
> > > > >> > >> > >>
> > > > >> > >> > >> Thanks Josh,
> > > > >> > >> > >> St.Ack
> > > > >> > >> > >>
> > > > >> > >> > >> 1.
> > > > >> > >> > >> https://docs.google.com/document/d/
> > > 1WCsVlnHjJeKUcl7wHwqb4z9i
> > > > >> > >> > >> Eu_ktczrlKHK8N4SZzs/edit#
> > > > >> > >> > >>
> > > > >> > >> > >>
> > > > >> > >> > > Nope, no need to block 2.0 on this one (given the other,
> > > > related
> > > > >> > >> > chatter).
> > > > >> > >> > > Would be nice to get it in, but I completely understand
> if
> > it
> > > > >> slips
> > > > >> > :)
> > > > >> > >> > >
> > > > >> > >> > > Thanks for updating the doc for me!
> > > > >> > >> > >
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> --
> > > > >> > >> Best regards,
> > > > >> > >>
> > > > >> > >>    - Andy
> > > > >> > >>
> > > > >> > >> If you are given a choice, you believe you have acted
> freely. -
> > > > >> Raymond
> > > > >> > >> Teller (via Peter Watts)
> > > > >> > >>
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>    - A23, Crosstalk
>

Re: Moving 2.0 forward

[Andrew Purtell <ap...@apache.org>]

Beta?


On Fri, Jul 21, 2017 at 1:16 PM, Stack <st...@duboce.net> wrote:

> On Fri, Jul 21, 2017 at 5:30 PM, Mike Drob <md...@apache.org> wrote:
>
> >
> > One thing that is not clear to me from your summary and the linked
> document
> > is which release we can block on tests? Unit tests are in bad shape (but
> > getting better) and I recently started looking at ITs which also need
> some
> > care.
> >
> >
> Dunno. What you think boss? I'd say we can't release if tests are a mess.
> Should we block on RC (as in alpha, beta, then RC)?  Or before then? i.e.
> no beta unless all tests (unit and IT) are passing.
>
> For sure testing -- as in exercising new features, changed configs, and
> just straight longevity under stress -- is badly wanting. I should write up
> a matrix I suppose? Let me work on it.
>
> Thanks Mike,
> S
>
>
>
>
> > Mike
> >
> > On Fri, Jul 21, 2017 at 11:03 AM, Stack <st...@duboce.net> wrote:
> >
> > > Status update girls and boys!
> > >
> > > hbase-2.0.0-alpha1 went out June 22nd.
> > >
> > > alpha2 has been a bit slow to follow (holidays) though there has been
> > > steady progress closing out blockers and criticals by a bunch of you
> all.
> > > The plan is for a release in the first week or so of August. It should
> be
> > > fully up on hbase-thirdparty using updated (and relocated) versions of
> > > netty, guava, and protobuf as well as a default deploy that has
> > > master-carrying-no-regions.
> > >
> > > alpha3 will follow soon after and will focus on making sure our
> > user-facing
> > > APIs are clean (branch-1 compatible, no illicit removals/mods, and so
> on)
> > > and that basic upgrade 'works'.
> > >
> > > betas start in September?
> > >
> > > I've been keeping a rough general state here [1] (please update any
> > section
> > > that is lagging actuality) but for details on what blockers and
> criticals
> > > remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought
> 2.0
> > > into better focus. Feel free to review and punt items you think can
> wait
> > > till 3.0 or 2.1. If you want to pull in more stuff, please ask first.
> > >
> > > Thanks,
> > > St.Ack
> > >
> > > 1.
> > > https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > > ktczrlKHK8N4SZzs/edit#
> > > 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
> > >
> > >
> > > On Mon, May 15, 2017 at 4:05 PM, Stack <st...@duboce.net> wrote:
> > >
> > > >
> > > > On Mon, May 15, 2017 at 1:46 AM, 张铎(Duo Zhang) <
> palomino219@gmail.com>
> > > > wrote:
> > > >
> > > >> HBASE-18037 is a new blocker. I'm currently working on it, will be
> > > >> finished
> > > >> soon I think.
> > > >>
> > > >> I made it a blocker then and added it to our hbase2 release doc [1]
> > list
> > > > as a blocker.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > >
> > > > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > > > ktczrlKHK8N4SZzs/edit#
> > > >
> > > >
> > > >> 2017-05-15 14:12 GMT+08:00 Stack <st...@duboce.net>:
> > > >>
> > > >>
> > > >> > A month on. Status.
> > > >> >
> > > >> > I've been working on the HBASE-14614 branch cluster testing.
> After a
> > > >> load
> > > >> > of fixing, the branch passes smaller test runs (an hour or so of
> > ITBLL
> > > >> up
> > > >> > to 2B rows w/ killing monkeys). When I go larger, to a scale I've
> > not
> > > >> done
> > > >> > in a while, I start to run into other interesting issues -- some
> of
> > > >> which
> > > >> > are related to AMv2 (I'm fixing), but others are not (100G WALs
> that
> > > >> take
> > > >> > ten minutes to split makes for interesting cascades when monkeys
> > kill
> > > >> > inside the ten minutes...). I intend to keep on with this larger
> > scale
> > > >> > testing since it is uncovering good stuff (especially when HDFS is
> > dog
> > > >> slow
> > > >> > because of background replications) but my thinking is that I
> should
> > > be
> > > >> > large scale testing branch-2, not just HBASE-14614. I think
> > > HBASE-14614,
> > > >> > the new AMv2, is good enough to merge to master these times. Given
> > it
> > > is
> > > >> > the last blocker, once in, I'll cut the hbase2 branch.
> > > >> >
> > > >> > I'll start up a 'Merge HBASE-14614' DISCUSSION thread in the next
> > day
> > > >> or so
> > > >> > (I need to fix some unit tests...).
> > > >> >
> > > >> > The AMv2 doc is still a work in progress but should give a gist on
> > > >> where we
> > > >> > are currently[1].  There is a bunch of todo still but seems
> > tractable;
> > > >> e.g.
> > > >> > rolling upgrade, finish doc., and we don't have an HBCK since it
> > needs
> > > >> to
> > > >> > be recast in light of how stuff now works but a redo on HBCK is
> > > >> premature
> > > >> > given we don't know failure types as yet (we just fix the problems
> > as
> > > >> they
> > > >> > come up).
> > > >> >
> > > >> > St.Ack
> > > >> > 1.
> > > >> > https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > > >> > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Thu, Apr 13, 2017 at 1:43 PM, Stack <st...@duboce.net> wrote:
> > > >> >
> > > >> > > Some status:
> > > >> > >
> > > >> > > AMv2 (HBASE-14614) is near to passing all tests caveat my
> > disabling
> > > of
> > > >> > all
> > > >> > > to-do w/ fsck (fsck needs revamp) and tests that expect that
> they
> > > can
> > > >> > move
> > > >> > > hbase;meta off master (AMv2 enforces this constraint; it is
> > supposed
> > > >> to
> > > >> > be
> > > >> > > enforced on AMv1 but meta-on-master is incompletely realized in
> > AMv1
> > > >> and
> > > >> > > AMv2). A few other tests have been disabled for various reasons.
> > See
> > > >> [1]
> > > >> > > for full list.
> > > >> > >
> > > >> > > There is a hefty list of TODOs still (Again see the messy doc
> [1])
> > > but
> > > >> > the
> > > >> > > only 'blocker', IMO, is community confidence in AMv2. Currently,
> > > >> cluster
> > > >> > > tests with chaos fail (new form of 'stuck' regions). Takes time
> > > >> > > investigating.
> > > >> > >
> > > >> > > Will keep you all posted.
> > > >> > > St.Ack
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > > >> > > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell <
> > > apurtell@apache.org>
> > > >> > > wrote:
> > > >> > >
> > > >> > >> +1 on branching (yay!)
> > > >> > >>
> > > >> > >> I have EC2 resources for running ITBLL etc.
> > > >> > >>
> > > >> > >>
> > > >> > >> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net>
> wrote:
> > > >> > >>
> > > >> > >> > Some notes on progress toward hbase2.
> > > >> > >> >
> > > >> > >> > Given that stability and performance are NOT emergent
> behaviors
> > > but
> > > >> > >> rather
> > > >> > >> > projects unto themselves, my thought is that we commit all
> that
> > > >> we've
> > > >> > >> > agreed as core for hbase2 (see [1]), branch, and then work on
> > > >> > >> stabilizing
> > > >> > >> > and perf rather than do stabilize, commit, and then branch.
> > What
> > > >> this
> > > >> > >> means
> > > >> > >> > in practice is that for features like Inmemory Compaction, we
> > > >> commit
> > > >> > it
> > > >> > >> > defaulted 'on' ("BASIC" mode) which is what we want in
> hbase2.
> > > >> Should
> > > >> > it
> > > >> > >> > prove problematic under test, we disable it before release.
> > > >> > >> >
> > > >> > >> > Are folks good w/ this mode? I ask because, in a few issues
> > there
> > > >> are
> > > >> > >> > requests for proof that a master feature is 'stable' before
> > > commit.
> > > >> > >> This is
> > > >> > >> > normally a healthy request only in master's case, it is hard
> to
> > > >> > >> demonstrate
> > > >> > >> > stability given its current state.
> > > >> > >> >
> > > >> > >> > Other outstanding issues such as decisions about whether
> master
> > > >> hosts
> > > >> > >> > system tables only (by default), I'm thinking, we can work
> out
> > > post
> > > >> > >> branch
> > > >> > >> > in alpha/betas before release.
> > > >> > >> >
> > > >> > >> > The awkward item is the long-pole Assignment Manager. This is
> > an
> > > >> > >> > all-or-nothing affair. Here we are switching in a new Master
> > > core.
> > > >> > >> While I
> > > >> > >> > think it fine that AMv2 is incomplete come branch time, those
> > of
> > > us
> > > >> > >> working
> > > >> > >> > on the new AM still need to demonstrate to you all that it
> > > >> basically
> > > >> > >> > viable.
> > > >> > >> >
> > > >> > >> > The point-of-no-return is commit of the patch in HBASE-14614.
> > > >> > >> HBASE-14614
> > > >> > >> > (AMv2) is coming close to passing all unit tests. We'll spend
> > > some
> > > >> > time
> > > >> > >> > running it on a cluster to make sure it fundamentally sound
> and
> > > >> will
> > > >> > >> report
> > > >> > >> > back on our experience. There has been an ask for some dev
> doc
> > > and
> > > >> > >> > low-levels on how it works (in progress). Let satisfaction of
> > > these
> > > >> > >> > requests be blockers on commit. We'll put the HBASE-14614
> > commit
> > > up
> > > >> > for
> > > >> > >> a
> > > >> > >> > vote on dev list given its import.
> > > >> > >> >
> > > >> > >> > Branch will happen after HBASE-14614 goes in (or its
> rejection)
> > > >> with
> > > >> > our
> > > >> > >> > first alpha soon after. Its looking like a week or two at
> least
> > > >> given
> > > >> > >> how
> > > >> > >> > things have been going up to this.
> > > >> > >> >
> > > >> > >> > I intend to start in on hbase2 stability/perf projects after
> we
> > > >> > branch.
> > > >> > >> >
> > > >> > >> > Interested in any thoughts you all might have on the above
> > (Would
> > > >> also
> > > >> > >> > appreciate updates on state in [1] if you are a feature
> owner).
> > > >> > >> >
> > > >> > >> > Thanks,
> > > >> > >> > St.Ack
> > > >> > >> >
> > > >> > >> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> > > >> > >> > z9iEu_ktczrlKHK8N4SZzs/edit#
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <
> elserj@apache.org
> > >
> > > >> > wrote:
> > > >> > >> >
> > > >> > >> > >
> > > >> > >> > > Stack wrote:
> > > >> > >> > >
> > > >> > >> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<
> > elserj@apache.org>
> > > >> > >> wrote:
> > > >> > >> > >>
> > > >> > >> > >> Thanks for pulling in the FS Quotas work, Stack. I'm
> trying
> > to
> > > >> > cross
> > > >> > >> the
> > > >> > >> > >>> last T's and dot the last I's.
> > > >> > >> > >>>
> > > >> > >> > >>> The biggest thing I know I need to do still is to write a
> > new
> > > >> > >> chapter
> > > >> > >> > to
> > > >> > >> > >>> the book. After that, I'd start entertaining larger
> > > >> > >> reviews/discussions
> > > >> > >> > >>> to
> > > >> > >> > >>> merge the feature into master. Anyone with free time
> > > (giggles)
> > > >> is
> > > >> > >> more
> > > >> > >> > >>> than
> > > >> > >> > >>> welcome to start perusing :)
> > > >> > >> > >>>
> > > >> > >> > >>>
> > > >> > >> > >>> Out of interest, this could come in after 2.0 Josh? Any
> 2.0
> > > >> > specific
> > > >> > >> > >> needs
> > > >> > >> > >> to make this work?
> > > >> > >> > >>
> > > >> > >> > >> Meantime, updated the 2.0 doc 1.
> > > >> > >> > >>
> > > >> > >> > >> Thanks Josh,
> > > >> > >> > >> St.Ack
> > > >> > >> > >>
> > > >> > >> > >> 1.
> > > >> > >> > >> https://docs.google.com/document/d/
> > 1WCsVlnHjJeKUcl7wHwqb4z9i
> > > >> > >> > >> Eu_ktczrlKHK8N4SZzs/edit#
> > > >> > >> > >>
> > > >> > >> > >>
> > > >> > >> > > Nope, no need to block 2.0 on this one (given the other,
> > > related
> > > >> > >> > chatter).
> > > >> > >> > > Would be nice to get it in, but I completely understand if
> it
> > > >> slips
> > > >> > :)
> > > >> > >> > >
> > > >> > >> > > Thanks for updating the doc for me!
> > > >> > >> > >
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >> --
> > > >> > >> Best regards,
> > > >> > >>
> > > >> > >>    - Andy
> > > >> > >>
> > > >> > >> If you are given a choice, you believe you have acted freely. -
> > > >> Raymond
> > > >> > >> Teller (via Peter Watts)
> > > >> > >>
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>



-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Fri, Jul 21, 2017 at 5:30 PM, Mike Drob <md...@apache.org> wrote:

>
> One thing that is not clear to me from your summary and the linked document
> is which release we can block on tests? Unit tests are in bad shape (but
> getting better) and I recently started looking at ITs which also need some
> care.
>
>
Dunno. What you think boss? I'd say we can't release if tests are a mess.
Should we block on RC (as in alpha, beta, then RC)?  Or before then? i.e.
no beta unless all tests (unit and IT) are passing.

For sure testing -- as in exercising new features, changed configs, and
just straight longevity under stress -- is badly wanting. I should write up
a matrix I suppose? Let me work on it.

Thanks Mike,
S




> Mike
>
> On Fri, Jul 21, 2017 at 11:03 AM, Stack <st...@duboce.net> wrote:
>
> > Status update girls and boys!
> >
> > hbase-2.0.0-alpha1 went out June 22nd.
> >
> > alpha2 has been a bit slow to follow (holidays) though there has been
> > steady progress closing out blockers and criticals by a bunch of you all.
> > The plan is for a release in the first week or so of August. It should be
> > fully up on hbase-thirdparty using updated (and relocated) versions of
> > netty, guava, and protobuf as well as a default deploy that has
> > master-carrying-no-regions.
> >
> > alpha3 will follow soon after and will focus on making sure our
> user-facing
> > APIs are clean (branch-1 compatible, no illicit removals/mods, and so on)
> > and that basic upgrade 'works'.
> >
> > betas start in September?
> >
> > I've been keeping a rough general state here [1] (please update any
> section
> > that is lagging actuality) but for details on what blockers and criticals
> > remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought 2.0
> > into better focus. Feel free to review and punt items you think can wait
> > till 3.0 or 2.1. If you want to pull in more stuff, please ask first.
> >
> > Thanks,
> > St.Ack
> >
> > 1.
> > https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > ktczrlKHK8N4SZzs/edit#
> > 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
> >
> >
> > On Mon, May 15, 2017 at 4:05 PM, Stack <st...@duboce.net> wrote:
> >
> > >
> > > On Mon, May 15, 2017 at 1:46 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> > > wrote:
> > >
> > >> HBASE-18037 is a new blocker. I'm currently working on it, will be
> > >> finished
> > >> soon I think.
> > >>
> > >> I made it a blocker then and added it to our hbase2 release doc [1]
> list
> > > as a blocker.
> > >
> > > Thanks,
> > > St.Ack
> > >
> > > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > > ktczrlKHK8N4SZzs/edit#
> > >
> > >
> > >> 2017-05-15 14:12 GMT+08:00 Stack <st...@duboce.net>:
> > >>
> > >>
> > >> > A month on. Status.
> > >> >
> > >> > I've been working on the HBASE-14614 branch cluster testing. After a
> > >> load
> > >> > of fixing, the branch passes smaller test runs (an hour or so of
> ITBLL
> > >> up
> > >> > to 2B rows w/ killing monkeys). When I go larger, to a scale I've
> not
> > >> done
> > >> > in a while, I start to run into other interesting issues -- some of
> > >> which
> > >> > are related to AMv2 (I'm fixing), but others are not (100G WALs that
> > >> take
> > >> > ten minutes to split makes for interesting cascades when monkeys
> kill
> > >> > inside the ten minutes...). I intend to keep on with this larger
> scale
> > >> > testing since it is uncovering good stuff (especially when HDFS is
> dog
> > >> slow
> > >> > because of background replications) but my thinking is that I should
> > be
> > >> > large scale testing branch-2, not just HBASE-14614. I think
> > HBASE-14614,
> > >> > the new AMv2, is good enough to merge to master these times. Given
> it
> > is
> > >> > the last blocker, once in, I'll cut the hbase2 branch.
> > >> >
> > >> > I'll start up a 'Merge HBASE-14614' DISCUSSION thread in the next
> day
> > >> or so
> > >> > (I need to fix some unit tests...).
> > >> >
> > >> > The AMv2 doc is still a work in progress but should give a gist on
> > >> where we
> > >> > are currently[1].  There is a bunch of todo still but seems
> tractable;
> > >> e.g.
> > >> > rolling upgrade, finish doc., and we don't have an HBCK since it
> needs
> > >> to
> > >> > be recast in light of how stuff now works but a redo on HBCK is
> > >> premature
> > >> > given we don't know failure types as yet (we just fix the problems
> as
> > >> they
> > >> > come up).
> > >> >
> > >> > St.Ack
> > >> > 1.
> > >> > https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > >> > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
> > >> >
> > >> >
> > >> >
> > >> > On Thu, Apr 13, 2017 at 1:43 PM, Stack <st...@duboce.net> wrote:
> > >> >
> > >> > > Some status:
> > >> > >
> > >> > > AMv2 (HBASE-14614) is near to passing all tests caveat my
> disabling
> > of
> > >> > all
> > >> > > to-do w/ fsck (fsck needs revamp) and tests that expect that they
> > can
> > >> > move
> > >> > > hbase;meta off master (AMv2 enforces this constraint; it is
> supposed
> > >> to
> > >> > be
> > >> > > enforced on AMv1 but meta-on-master is incompletely realized in
> AMv1
> > >> and
> > >> > > AMv2). A few other tests have been disabled for various reasons.
> See
> > >> [1]
> > >> > > for full list.
> > >> > >
> > >> > > There is a hefty list of TODOs still (Again see the messy doc [1])
> > but
> > >> > the
> > >> > > only 'blocker', IMO, is community confidence in AMv2. Currently,
> > >> cluster
> > >> > > tests with chaos fail (new form of 'stuck' regions). Takes time
> > >> > > investigating.
> > >> > >
> > >> > > Will keep you all posted.
> > >> > > St.Ack
> > >> > >
> > >> > >
> > >> > >
> > >> > > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > >> > > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell <
> > apurtell@apache.org>
> > >> > > wrote:
> > >> > >
> > >> > >> +1 on branching (yay!)
> > >> > >>
> > >> > >> I have EC2 resources for running ITBLL etc.
> > >> > >>
> > >> > >>
> > >> > >> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> wrote:
> > >> > >>
> > >> > >> > Some notes on progress toward hbase2.
> > >> > >> >
> > >> > >> > Given that stability and performance are NOT emergent behaviors
> > but
> > >> > >> rather
> > >> > >> > projects unto themselves, my thought is that we commit all that
> > >> we've
> > >> > >> > agreed as core for hbase2 (see [1]), branch, and then work on
> > >> > >> stabilizing
> > >> > >> > and perf rather than do stabilize, commit, and then branch.
> What
> > >> this
> > >> > >> means
> > >> > >> > in practice is that for features like Inmemory Compaction, we
> > >> commit
> > >> > it
> > >> > >> > defaulted 'on' ("BASIC" mode) which is what we want in hbase2.
> > >> Should
> > >> > it
> > >> > >> > prove problematic under test, we disable it before release.
> > >> > >> >
> > >> > >> > Are folks good w/ this mode? I ask because, in a few issues
> there
> > >> are
> > >> > >> > requests for proof that a master feature is 'stable' before
> > commit.
> > >> > >> This is
> > >> > >> > normally a healthy request only in master's case, it is hard to
> > >> > >> demonstrate
> > >> > >> > stability given its current state.
> > >> > >> >
> > >> > >> > Other outstanding issues such as decisions about whether master
> > >> hosts
> > >> > >> > system tables only (by default), I'm thinking, we can work out
> > post
> > >> > >> branch
> > >> > >> > in alpha/betas before release.
> > >> > >> >
> > >> > >> > The awkward item is the long-pole Assignment Manager. This is
> an
> > >> > >> > all-or-nothing affair. Here we are switching in a new Master
> > core.
> > >> > >> While I
> > >> > >> > think it fine that AMv2 is incomplete come branch time, those
> of
> > us
> > >> > >> working
> > >> > >> > on the new AM still need to demonstrate to you all that it
> > >> basically
> > >> > >> > viable.
> > >> > >> >
> > >> > >> > The point-of-no-return is commit of the patch in HBASE-14614.
> > >> > >> HBASE-14614
> > >> > >> > (AMv2) is coming close to passing all unit tests. We'll spend
> > some
> > >> > time
> > >> > >> > running it on a cluster to make sure it fundamentally sound and
> > >> will
> > >> > >> report
> > >> > >> > back on our experience. There has been an ask for some dev doc
> > and
> > >> > >> > low-levels on how it works (in progress). Let satisfaction of
> > these
> > >> > >> > requests be blockers on commit. We'll put the HBASE-14614
> commit
> > up
> > >> > for
> > >> > >> a
> > >> > >> > vote on dev list given its import.
> > >> > >> >
> > >> > >> > Branch will happen after HBASE-14614 goes in (or its rejection)
> > >> with
> > >> > our
> > >> > >> > first alpha soon after. Its looking like a week or two at least
> > >> given
> > >> > >> how
> > >> > >> > things have been going up to this.
> > >> > >> >
> > >> > >> > I intend to start in on hbase2 stability/perf projects after we
> > >> > branch.
> > >> > >> >
> > >> > >> > Interested in any thoughts you all might have on the above
> (Would
> > >> also
> > >> > >> > appreciate updates on state in [1] if you are a feature owner).
> > >> > >> >
> > >> > >> > Thanks,
> > >> > >> > St.Ack
> > >> > >> >
> > >> > >> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> > >> > >> > z9iEu_ktczrlKHK8N4SZzs/edit#
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <elserj@apache.org
> >
> > >> > wrote:
> > >> > >> >
> > >> > >> > >
> > >> > >> > > Stack wrote:
> > >> > >> > >
> > >> > >> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<
> elserj@apache.org>
> > >> > >> wrote:
> > >> > >> > >>
> > >> > >> > >> Thanks for pulling in the FS Quotas work, Stack. I'm trying
> to
> > >> > cross
> > >> > >> the
> > >> > >> > >>> last T's and dot the last I's.
> > >> > >> > >>>
> > >> > >> > >>> The biggest thing I know I need to do still is to write a
> new
> > >> > >> chapter
> > >> > >> > to
> > >> > >> > >>> the book. After that, I'd start entertaining larger
> > >> > >> reviews/discussions
> > >> > >> > >>> to
> > >> > >> > >>> merge the feature into master. Anyone with free time
> > (giggles)
> > >> is
> > >> > >> more
> > >> > >> > >>> than
> > >> > >> > >>> welcome to start perusing :)
> > >> > >> > >>>
> > >> > >> > >>>
> > >> > >> > >>> Out of interest, this could come in after 2.0 Josh? Any 2.0
> > >> > specific
> > >> > >> > >> needs
> > >> > >> > >> to make this work?
> > >> > >> > >>
> > >> > >> > >> Meantime, updated the 2.0 doc 1.
> > >> > >> > >>
> > >> > >> > >> Thanks Josh,
> > >> > >> > >> St.Ack
> > >> > >> > >>
> > >> > >> > >> 1.
> > >> > >> > >> https://docs.google.com/document/d/
> 1WCsVlnHjJeKUcl7wHwqb4z9i
> > >> > >> > >> Eu_ktczrlKHK8N4SZzs/edit#
> > >> > >> > >>
> > >> > >> > >>
> > >> > >> > > Nope, no need to block 2.0 on this one (given the other,
> > related
> > >> > >> > chatter).
> > >> > >> > > Would be nice to get it in, but I completely understand if it
> > >> slips
> > >> > :)
> > >> > >> > >
> > >> > >> > > Thanks for updating the doc for me!
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >> --
> > >> > >> Best regards,
> > >> > >>
> > >> > >>    - Andy
> > >> > >>
> > >> > >> If you are given a choice, you believe you have acted freely. -
> > >> Raymond
> > >> > >> Teller (via Peter Watts)
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: Moving 2.0 forward

[Mike Drob <md...@apache.org>]

Thanks for tracking this, Stack!

One thing that is not clear to me from your summary and the linked document
is which release we can block on tests? Unit tests are in bad shape (but
getting better) and I recently started looking at ITs which also need some
care.

Mike

On Fri, Jul 21, 2017 at 11:03 AM, Stack <st...@duboce.net> wrote:

> Status update girls and boys!
>
> hbase-2.0.0-alpha1 went out June 22nd.
>
> alpha2 has been a bit slow to follow (holidays) though there has been
> steady progress closing out blockers and criticals by a bunch of you all.
> The plan is for a release in the first week or so of August. It should be
> fully up on hbase-thirdparty using updated (and relocated) versions of
> netty, guava, and protobuf as well as a default deploy that has
> master-carrying-no-regions.
>
> alpha3 will follow soon after and will focus on making sure our user-facing
> APIs are clean (branch-1 compatible, no illicit removals/mods, and so on)
> and that basic upgrade 'works'.
>
> betas start in September?
>
> I've been keeping a rough general state here [1] (please update any section
> that is lagging actuality) but for details on what blockers and criticals
> remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought 2.0
> into better focus. Feel free to review and punt items you think can wait
> till 3.0 or 2.1. If you want to pull in more stuff, please ask first.
>
> Thanks,
> St.Ack
>
> 1.
> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> ktczrlKHK8N4SZzs/edit#
> 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188
>
>
> On Mon, May 15, 2017 at 4:05 PM, Stack <st...@duboce.net> wrote:
>
> >
> > On Mon, May 15, 2017 at 1:46 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> > wrote:
> >
> >> HBASE-18037 is a new blocker. I'm currently working on it, will be
> >> finished
> >> soon I think.
> >>
> >> I made it a blocker then and added it to our hbase2 release doc [1] list
> > as a blocker.
> >
> > Thanks,
> > St.Ack
> >
> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > ktczrlKHK8N4SZzs/edit#
> >
> >
> >> 2017-05-15 14:12 GMT+08:00 Stack <st...@duboce.net>:
> >>
> >>
> >> > A month on. Status.
> >> >
> >> > I've been working on the HBASE-14614 branch cluster testing. After a
> >> load
> >> > of fixing, the branch passes smaller test runs (an hour or so of ITBLL
> >> up
> >> > to 2B rows w/ killing monkeys). When I go larger, to a scale I've not
> >> done
> >> > in a while, I start to run into other interesting issues -- some of
> >> which
> >> > are related to AMv2 (I'm fixing), but others are not (100G WALs that
> >> take
> >> > ten minutes to split makes for interesting cascades when monkeys kill
> >> > inside the ten minutes...). I intend to keep on with this larger scale
> >> > testing since it is uncovering good stuff (especially when HDFS is dog
> >> slow
> >> > because of background replications) but my thinking is that I should
> be
> >> > large scale testing branch-2, not just HBASE-14614. I think
> HBASE-14614,
> >> > the new AMv2, is good enough to merge to master these times. Given it
> is
> >> > the last blocker, once in, I'll cut the hbase2 branch.
> >> >
> >> > I'll start up a 'Merge HBASE-14614' DISCUSSION thread in the next day
> >> or so
> >> > (I need to fix some unit tests...).
> >> >
> >> > The AMv2 doc is still a work in progress but should give a gist on
> >> where we
> >> > are currently[1].  There is a bunch of todo still but seems tractable;
> >> e.g.
> >> > rolling upgrade, finish doc., and we don't have an HBCK since it needs
> >> to
> >> > be recast in light of how stuff now works but a redo on HBCK is
> >> premature
> >> > given we don't know failure types as yet (we just fix the problems as
> >> they
> >> > come up).
> >> >
> >> > St.Ack
> >> > 1.
> >> > https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> >> > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
> >> >
> >> >
> >> >
> >> > On Thu, Apr 13, 2017 at 1:43 PM, Stack <st...@duboce.net> wrote:
> >> >
> >> > > Some status:
> >> > >
> >> > > AMv2 (HBASE-14614) is near to passing all tests caveat my disabling
> of
> >> > all
> >> > > to-do w/ fsck (fsck needs revamp) and tests that expect that they
> can
> >> > move
> >> > > hbase;meta off master (AMv2 enforces this constraint; it is supposed
> >> to
> >> > be
> >> > > enforced on AMv1 but meta-on-master is incompletely realized in AMv1
> >> and
> >> > > AMv2). A few other tests have been disabled for various reasons. See
> >> [1]
> >> > > for full list.
> >> > >
> >> > > There is a hefty list of TODOs still (Again see the messy doc [1])
> but
> >> > the
> >> > > only 'blocker', IMO, is community confidence in AMv2. Currently,
> >> cluster
> >> > > tests with chaos fail (new form of 'stuck' regions). Takes time
> >> > > investigating.
> >> > >
> >> > > Will keep you all posted.
> >> > > St.Ack
> >> > >
> >> > >
> >> > >
> >> > > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> >> > > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod
> >> > >
> >> > >
> >> > >
> >> > > On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell <
> apurtell@apache.org>
> >> > > wrote:
> >> > >
> >> > >> +1 on branching (yay!)
> >> > >>
> >> > >> I have EC2 resources for running ITBLL etc.
> >> > >>
> >> > >>
> >> > >> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> wrote:
> >> > >>
> >> > >> > Some notes on progress toward hbase2.
> >> > >> >
> >> > >> > Given that stability and performance are NOT emergent behaviors
> but
> >> > >> rather
> >> > >> > projects unto themselves, my thought is that we commit all that
> >> we've
> >> > >> > agreed as core for hbase2 (see [1]), branch, and then work on
> >> > >> stabilizing
> >> > >> > and perf rather than do stabilize, commit, and then branch. What
> >> this
> >> > >> means
> >> > >> > in practice is that for features like Inmemory Compaction, we
> >> commit
> >> > it
> >> > >> > defaulted 'on' ("BASIC" mode) which is what we want in hbase2.
> >> Should
> >> > it
> >> > >> > prove problematic under test, we disable it before release.
> >> > >> >
> >> > >> > Are folks good w/ this mode? I ask because, in a few issues there
> >> are
> >> > >> > requests for proof that a master feature is 'stable' before
> commit.
> >> > >> This is
> >> > >> > normally a healthy request only in master's case, it is hard to
> >> > >> demonstrate
> >> > >> > stability given its current state.
> >> > >> >
> >> > >> > Other outstanding issues such as decisions about whether master
> >> hosts
> >> > >> > system tables only (by default), I'm thinking, we can work out
> post
> >> > >> branch
> >> > >> > in alpha/betas before release.
> >> > >> >
> >> > >> > The awkward item is the long-pole Assignment Manager. This is an
> >> > >> > all-or-nothing affair. Here we are switching in a new Master
> core.
> >> > >> While I
> >> > >> > think it fine that AMv2 is incomplete come branch time, those of
> us
> >> > >> working
> >> > >> > on the new AM still need to demonstrate to you all that it
> >> basically
> >> > >> > viable.
> >> > >> >
> >> > >> > The point-of-no-return is commit of the patch in HBASE-14614.
> >> > >> HBASE-14614
> >> > >> > (AMv2) is coming close to passing all unit tests. We'll spend
> some
> >> > time
> >> > >> > running it on a cluster to make sure it fundamentally sound and
> >> will
> >> > >> report
> >> > >> > back on our experience. There has been an ask for some dev doc
> and
> >> > >> > low-levels on how it works (in progress). Let satisfaction of
> these
> >> > >> > requests be blockers on commit. We'll put the HBASE-14614 commit
> up
> >> > for
> >> > >> a
> >> > >> > vote on dev list given its import.
> >> > >> >
> >> > >> > Branch will happen after HBASE-14614 goes in (or its rejection)
> >> with
> >> > our
> >> > >> > first alpha soon after. Its looking like a week or two at least
> >> given
> >> > >> how
> >> > >> > things have been going up to this.
> >> > >> >
> >> > >> > I intend to start in on hbase2 stability/perf projects after we
> >> > branch.
> >> > >> >
> >> > >> > Interested in any thoughts you all might have on the above (Would
> >> also
> >> > >> > appreciate updates on state in [1] if you are a feature owner).
> >> > >> >
> >> > >> > Thanks,
> >> > >> > St.Ack
> >> > >> >
> >> > >> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> >> > >> > z9iEu_ktczrlKHK8N4SZzs/edit#
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <el...@apache.org>
> >> > wrote:
> >> > >> >
> >> > >> > >
> >> > >> > > Stack wrote:
> >> > >> > >
> >> > >> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<el...@apache.org>
> >> > >> wrote:
> >> > >> > >>
> >> > >> > >> Thanks for pulling in the FS Quotas work, Stack. I'm trying to
> >> > cross
> >> > >> the
> >> > >> > >>> last T's and dot the last I's.
> >> > >> > >>>
> >> > >> > >>> The biggest thing I know I need to do still is to write a new
> >> > >> chapter
> >> > >> > to
> >> > >> > >>> the book. After that, I'd start entertaining larger
> >> > >> reviews/discussions
> >> > >> > >>> to
> >> > >> > >>> merge the feature into master. Anyone with free time
> (giggles)
> >> is
> >> > >> more
> >> > >> > >>> than
> >> > >> > >>> welcome to start perusing :)
> >> > >> > >>>
> >> > >> > >>>
> >> > >> > >>> Out of interest, this could come in after 2.0 Josh? Any 2.0
> >> > specific
> >> > >> > >> needs
> >> > >> > >> to make this work?
> >> > >> > >>
> >> > >> > >> Meantime, updated the 2.0 doc 1.
> >> > >> > >>
> >> > >> > >> Thanks Josh,
> >> > >> > >> St.Ack
> >> > >> > >>
> >> > >> > >> 1.
> >> > >> > >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >> > >> > >> Eu_ktczrlKHK8N4SZzs/edit#
> >> > >> > >>
> >> > >> > >>
> >> > >> > > Nope, no need to block 2.0 on this one (given the other,
> related
> >> > >> > chatter).
> >> > >> > > Would be nice to get it in, but I completely understand if it
> >> slips
> >> > :)
> >> > >> > >
> >> > >> > > Thanks for updating the doc for me!
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Best regards,
> >> > >>
> >> > >>    - Andy
> >> > >>
> >> > >> If you are given a choice, you believe you have acted freely. -
> >> Raymond
> >> > >> Teller (via Peter Watts)
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Status update girls and boys!

hbase-2.0.0-alpha1 went out June 22nd.

alpha2 has been a bit slow to follow (holidays) though there has been
steady progress closing out blockers and criticals by a bunch of you all.
The plan is for a release in the first week or so of August. It should be
fully up on hbase-thirdparty using updated (and relocated) versions of
netty, guava, and protobuf as well as a default deploy that has
master-carrying-no-regions.

alpha3 will follow soon after and will focus on making sure our user-facing
APIs are clean (branch-1 compatible, no illicit removals/mods, and so on)
and that basic upgrade 'works'.

betas start in September?

I've been keeping a rough general state here [1] (please update any section
that is lagging actuality) but for details on what blockers and criticals
remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought 2.0
into better focus. Feel free to review and punt items you think can wait
till 3.0 or 2.1. If you want to pull in more stuff, please ask first.

Thanks,
St.Ack

1.
https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#
2. https://issues.apache.org/jira/projects/HBASE/versions/12327188


On Mon, May 15, 2017 at 4:05 PM, Stack <st...@duboce.net> wrote:

>
> On Mon, May 15, 2017 at 1:46 AM, 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
>> HBASE-18037 is a new blocker. I'm currently working on it, will be
>> finished
>> soon I think.
>>
>> I made it a blocker then and added it to our hbase2 release doc [1] list
> as a blocker.
>
> Thanks,
> St.Ack
>
> 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> ktczrlKHK8N4SZzs/edit#
>
>
>> 2017-05-15 14:12 GMT+08:00 Stack <st...@duboce.net>:
>>
>>
>> > A month on. Status.
>> >
>> > I've been working on the HBASE-14614 branch cluster testing. After a
>> load
>> > of fixing, the branch passes smaller test runs (an hour or so of ITBLL
>> up
>> > to 2B rows w/ killing monkeys). When I go larger, to a scale I've not
>> done
>> > in a while, I start to run into other interesting issues -- some of
>> which
>> > are related to AMv2 (I'm fixing), but others are not (100G WALs that
>> take
>> > ten minutes to split makes for interesting cascades when monkeys kill
>> > inside the ten minutes...). I intend to keep on with this larger scale
>> > testing since it is uncovering good stuff (especially when HDFS is dog
>> slow
>> > because of background replications) but my thinking is that I should be
>> > large scale testing branch-2, not just HBASE-14614. I think HBASE-14614,
>> > the new AMv2, is good enough to merge to master these times. Given it is
>> > the last blocker, once in, I'll cut the hbase2 branch.
>> >
>> > I'll start up a 'Merge HBASE-14614' DISCUSSION thread in the next day
>> or so
>> > (I need to fix some unit tests...).
>> >
>> > The AMv2 doc is still a work in progress but should give a gist on
>> where we
>> > are currently[1].  There is a bunch of todo still but seems tractable;
>> e.g.
>> > rolling upgrade, finish doc., and we don't have an HBCK since it needs
>> to
>> > be recast in light of how stuff now works but a redo on HBCK is
>> premature
>> > given we don't know failure types as yet (we just fix the problems as
>> they
>> > come up).
>> >
>> > St.Ack
>> > 1.
>> > https://docs.google.com/document/d/1eVKa7FHdeoJ1-
>> > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
>> >
>> >
>> >
>> > On Thu, Apr 13, 2017 at 1:43 PM, Stack <st...@duboce.net> wrote:
>> >
>> > > Some status:
>> > >
>> > > AMv2 (HBASE-14614) is near to passing all tests caveat my disabling of
>> > all
>> > > to-do w/ fsck (fsck needs revamp) and tests that expect that they can
>> > move
>> > > hbase;meta off master (AMv2 enforces this constraint; it is supposed
>> to
>> > be
>> > > enforced on AMv1 but meta-on-master is incompletely realized in AMv1
>> and
>> > > AMv2). A few other tests have been disabled for various reasons. See
>> [1]
>> > > for full list.
>> > >
>> > > There is a hefty list of TODOs still (Again see the messy doc [1]) but
>> > the
>> > > only 'blocker', IMO, is community confidence in AMv2. Currently,
>> cluster
>> > > tests with chaos fail (new form of 'stuck' regions). Takes time
>> > > investigating.
>> > >
>> > > Will keep you all posted.
>> > > St.Ack
>> > >
>> > >
>> > >
>> > > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1-
>> > > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod
>> > >
>> > >
>> > >
>> > > On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell <ap...@apache.org>
>> > > wrote:
>> > >
>> > >> +1 on branching (yay!)
>> > >>
>> > >> I have EC2 resources for running ITBLL etc.
>> > >>
>> > >>
>> > >> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> wrote:
>> > >>
>> > >> > Some notes on progress toward hbase2.
>> > >> >
>> > >> > Given that stability and performance are NOT emergent behaviors but
>> > >> rather
>> > >> > projects unto themselves, my thought is that we commit all that
>> we've
>> > >> > agreed as core for hbase2 (see [1]), branch, and then work on
>> > >> stabilizing
>> > >> > and perf rather than do stabilize, commit, and then branch. What
>> this
>> > >> means
>> > >> > in practice is that for features like Inmemory Compaction, we
>> commit
>> > it
>> > >> > defaulted 'on' ("BASIC" mode) which is what we want in hbase2.
>> Should
>> > it
>> > >> > prove problematic under test, we disable it before release.
>> > >> >
>> > >> > Are folks good w/ this mode? I ask because, in a few issues there
>> are
>> > >> > requests for proof that a master feature is 'stable' before commit.
>> > >> This is
>> > >> > normally a healthy request only in master's case, it is hard to
>> > >> demonstrate
>> > >> > stability given its current state.
>> > >> >
>> > >> > Other outstanding issues such as decisions about whether master
>> hosts
>> > >> > system tables only (by default), I'm thinking, we can work out post
>> > >> branch
>> > >> > in alpha/betas before release.
>> > >> >
>> > >> > The awkward item is the long-pole Assignment Manager. This is an
>> > >> > all-or-nothing affair. Here we are switching in a new Master core.
>> > >> While I
>> > >> > think it fine that AMv2 is incomplete come branch time, those of us
>> > >> working
>> > >> > on the new AM still need to demonstrate to you all that it
>> basically
>> > >> > viable.
>> > >> >
>> > >> > The point-of-no-return is commit of the patch in HBASE-14614.
>> > >> HBASE-14614
>> > >> > (AMv2) is coming close to passing all unit tests. We'll spend some
>> > time
>> > >> > running it on a cluster to make sure it fundamentally sound and
>> will
>> > >> report
>> > >> > back on our experience. There has been an ask for some dev doc and
>> > >> > low-levels on how it works (in progress). Let satisfaction of these
>> > >> > requests be blockers on commit. We'll put the HBASE-14614 commit up
>> > for
>> > >> a
>> > >> > vote on dev list given its import.
>> > >> >
>> > >> > Branch will happen after HBASE-14614 goes in (or its rejection)
>> with
>> > our
>> > >> > first alpha soon after. Its looking like a week or two at least
>> given
>> > >> how
>> > >> > things have been going up to this.
>> > >> >
>> > >> > I intend to start in on hbase2 stability/perf projects after we
>> > branch.
>> > >> >
>> > >> > Interested in any thoughts you all might have on the above (Would
>> also
>> > >> > appreciate updates on state in [1] if you are a feature owner).
>> > >> >
>> > >> > Thanks,
>> > >> > St.Ack
>> > >> >
>> > >> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
>> > >> > z9iEu_ktczrlKHK8N4SZzs/edit#
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <el...@apache.org>
>> > wrote:
>> > >> >
>> > >> > >
>> > >> > > Stack wrote:
>> > >> > >
>> > >> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<el...@apache.org>
>> > >> wrote:
>> > >> > >>
>> > >> > >> Thanks for pulling in the FS Quotas work, Stack. I'm trying to
>> > cross
>> > >> the
>> > >> > >>> last T's and dot the last I's.
>> > >> > >>>
>> > >> > >>> The biggest thing I know I need to do still is to write a new
>> > >> chapter
>> > >> > to
>> > >> > >>> the book. After that, I'd start entertaining larger
>> > >> reviews/discussions
>> > >> > >>> to
>> > >> > >>> merge the feature into master. Anyone with free time (giggles)
>> is
>> > >> more
>> > >> > >>> than
>> > >> > >>> welcome to start perusing :)
>> > >> > >>>
>> > >> > >>>
>> > >> > >>> Out of interest, this could come in after 2.0 Josh? Any 2.0
>> > specific
>> > >> > >> needs
>> > >> > >> to make this work?
>> > >> > >>
>> > >> > >> Meantime, updated the 2.0 doc 1.
>> > >> > >>
>> > >> > >> Thanks Josh,
>> > >> > >> St.Ack
>> > >> > >>
>> > >> > >> 1.
>> > >> > >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> > >> > >> Eu_ktczrlKHK8N4SZzs/edit#
>> > >> > >>
>> > >> > >>
>> > >> > > Nope, no need to block 2.0 on this one (given the other, related
>> > >> > chatter).
>> > >> > > Would be nice to get it in, but I completely understand if it
>> slips
>> > :)
>> > >> > >
>> > >> > > Thanks for updating the doc for me!
>> > >> > >
>> > >> >
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Best regards,
>> > >>
>> > >>    - Andy
>> > >>
>> > >> If you are given a choice, you believe you have acted freely. -
>> Raymond
>> > >> Teller (via Peter Watts)
>> > >>
>> > >
>> > >
>> >
>>
>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Mon, May 15, 2017 at 1:46 AM, 张铎(Duo Zhang) <pa...@gmail.com>
wrote:

> HBASE-18037 is a new blocker. I'm currently working on it, will be finished
> soon I think.
>
> I made it a blocker then and added it to our hbase2 release doc [1] list
as a blocker.

Thanks,
St.Ack

1.
https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#


> 2017-05-15 14:12 GMT+08:00 Stack <st...@duboce.net>:
>
> > A month on. Status.
> >
> > I've been working on the HBASE-14614 branch cluster testing. After a load
> > of fixing, the branch passes smaller test runs (an hour or so of ITBLL up
> > to 2B rows w/ killing monkeys). When I go larger, to a scale I've not
> done
> > in a while, I start to run into other interesting issues -- some of which
> > are related to AMv2 (I'm fixing), but others are not (100G WALs that take
> > ten minutes to split makes for interesting cascades when monkeys kill
> > inside the ten minutes...). I intend to keep on with this larger scale
> > testing since it is uncovering good stuff (especially when HDFS is dog
> slow
> > because of background replications) but my thinking is that I should be
> > large scale testing branch-2, not just HBASE-14614. I think HBASE-14614,
> > the new AMv2, is good enough to merge to master these times. Given it is
> > the last blocker, once in, I'll cut the hbase2 branch.
> >
> > I'll start up a 'Merge HBASE-14614' DISCUSSION thread in the next day or
> so
> > (I need to fix some unit tests...).
> >
> > The AMv2 doc is still a work in progress but should give a gist on where
> we
> > are currently[1].  There is a bunch of todo still but seems tractable;
> e.g.
> > rolling upgrade, finish doc., and we don't have an HBCK since it needs to
> > be recast in light of how stuff now works but a redo on HBCK is premature
> > given we don't know failure types as yet (we just fix the problems as
> they
> > come up).
> >
> > St.Ack
> > 1.
> > https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
> >
> >
> >
> > On Thu, Apr 13, 2017 at 1:43 PM, Stack <st...@duboce.net> wrote:
> >
> > > Some status:
> > >
> > > AMv2 (HBASE-14614) is near to passing all tests caveat my disabling of
> > all
> > > to-do w/ fsck (fsck needs revamp) and tests that expect that they can
> > move
> > > hbase;meta off master (AMv2 enforces this constraint; it is supposed to
> > be
> > > enforced on AMv1 but meta-on-master is incompletely realized in AMv1
> and
> > > AMv2). A few other tests have been disabled for various reasons. See
> [1]
> > > for full list.
> > >
> > > There is a hefty list of TODOs still (Again see the messy doc [1]) but
> > the
> > > only 'blocker', IMO, is community confidence in AMv2. Currently,
> cluster
> > > tests with chaos fail (new form of 'stuck' regions). Takes time
> > > investigating.
> > >
> > > Will keep you all posted.
> > > St.Ack
> > >
> > >
> > >
> > > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod
> > >
> > >
> > >
> > > On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell <ap...@apache.org>
> > > wrote:
> > >
> > >> +1 on branching (yay!)
> > >>
> > >> I have EC2 resources for running ITBLL etc.
> > >>
> > >>
> > >> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> wrote:
> > >>
> > >> > Some notes on progress toward hbase2.
> > >> >
> > >> > Given that stability and performance are NOT emergent behaviors but
> > >> rather
> > >> > projects unto themselves, my thought is that we commit all that
> we've
> > >> > agreed as core for hbase2 (see [1]), branch, and then work on
> > >> stabilizing
> > >> > and perf rather than do stabilize, commit, and then branch. What
> this
> > >> means
> > >> > in practice is that for features like Inmemory Compaction, we commit
> > it
> > >> > defaulted 'on' ("BASIC" mode) which is what we want in hbase2.
> Should
> > it
> > >> > prove problematic under test, we disable it before release.
> > >> >
> > >> > Are folks good w/ this mode? I ask because, in a few issues there
> are
> > >> > requests for proof that a master feature is 'stable' before commit.
> > >> This is
> > >> > normally a healthy request only in master's case, it is hard to
> > >> demonstrate
> > >> > stability given its current state.
> > >> >
> > >> > Other outstanding issues such as decisions about whether master
> hosts
> > >> > system tables only (by default), I'm thinking, we can work out post
> > >> branch
> > >> > in alpha/betas before release.
> > >> >
> > >> > The awkward item is the long-pole Assignment Manager. This is an
> > >> > all-or-nothing affair. Here we are switching in a new Master core.
> > >> While I
> > >> > think it fine that AMv2 is incomplete come branch time, those of us
> > >> working
> > >> > on the new AM still need to demonstrate to you all that it basically
> > >> > viable.
> > >> >
> > >> > The point-of-no-return is commit of the patch in HBASE-14614.
> > >> HBASE-14614
> > >> > (AMv2) is coming close to passing all unit tests. We'll spend some
> > time
> > >> > running it on a cluster to make sure it fundamentally sound and will
> > >> report
> > >> > back on our experience. There has been an ask for some dev doc and
> > >> > low-levels on how it works (in progress). Let satisfaction of these
> > >> > requests be blockers on commit. We'll put the HBASE-14614 commit up
> > for
> > >> a
> > >> > vote on dev list given its import.
> > >> >
> > >> > Branch will happen after HBASE-14614 goes in (or its rejection) with
> > our
> > >> > first alpha soon after. Its looking like a week or two at least
> given
> > >> how
> > >> > things have been going up to this.
> > >> >
> > >> > I intend to start in on hbase2 stability/perf projects after we
> > branch.
> > >> >
> > >> > Interested in any thoughts you all might have on the above (Would
> also
> > >> > appreciate updates on state in [1] if you are a feature owner).
> > >> >
> > >> > Thanks,
> > >> > St.Ack
> > >> >
> > >> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> > >> > z9iEu_ktczrlKHK8N4SZzs/edit#
> > >> >
> > >> >
> > >> >
> > >> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <el...@apache.org>
> > wrote:
> > >> >
> > >> > >
> > >> > > Stack wrote:
> > >> > >
> > >> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<el...@apache.org>
> > >> wrote:
> > >> > >>
> > >> > >> Thanks for pulling in the FS Quotas work, Stack. I'm trying to
> > cross
> > >> the
> > >> > >>> last T's and dot the last I's.
> > >> > >>>
> > >> > >>> The biggest thing I know I need to do still is to write a new
> > >> chapter
> > >> > to
> > >> > >>> the book. After that, I'd start entertaining larger
> > >> reviews/discussions
> > >> > >>> to
> > >> > >>> merge the feature into master. Anyone with free time (giggles)
> is
> > >> more
> > >> > >>> than
> > >> > >>> welcome to start perusing :)
> > >> > >>>
> > >> > >>>
> > >> > >>> Out of interest, this could come in after 2.0 Josh? Any 2.0
> > specific
> > >> > >> needs
> > >> > >> to make this work?
> > >> > >>
> > >> > >> Meantime, updated the 2.0 doc 1.
> > >> > >>
> > >> > >> Thanks Josh,
> > >> > >> St.Ack
> > >> > >>
> > >> > >> 1.
> > >> > >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> > >> > >> Eu_ktczrlKHK8N4SZzs/edit#
> > >> > >>
> > >> > >>
> > >> > > Nope, no need to block 2.0 on this one (given the other, related
> > >> > chatter).
> > >> > > Would be nice to get it in, but I completely understand if it
> slips
> > :)
> > >> > >
> > >> > > Thanks for updating the doc for me!
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Best regards,
> > >>
> > >>    - Andy
> > >>
> > >> If you are given a choice, you believe you have acted freely. -
> Raymond
> > >> Teller (via Peter Watts)
> > >>
> > >
> > >
> >
>

Re: Moving 2.0 forward

["张铎 (Duo Zhang)" <pa...@gmail.com>]

HBASE-18037 is a new blocker. I'm currently working on it, will be finished
soon I think.

2017-05-15 14:12 GMT+08:00 Stack <st...@duboce.net>:

> A month on. Status.
>
> I've been working on the HBASE-14614 branch cluster testing. After a load
> of fixing, the branch passes smaller test runs (an hour or so of ITBLL up
> to 2B rows w/ killing monkeys). When I go larger, to a scale I've not done
> in a while, I start to run into other interesting issues -- some of which
> are related to AMv2 (I'm fixing), but others are not (100G WALs that take
> ten minutes to split makes for interesting cascades when monkeys kill
> inside the ten minutes...). I intend to keep on with this larger scale
> testing since it is uncovering good stuff (especially when HDFS is dog slow
> because of background replications) but my thinking is that I should be
> large scale testing branch-2, not just HBASE-14614. I think HBASE-14614,
> the new AMv2, is good enough to merge to master these times. Given it is
> the last blocker, once in, I'll cut the hbase2 branch.
>
> I'll start up a 'Merge HBASE-14614' DISCUSSION thread in the next day or so
> (I need to fix some unit tests...).
>
> The AMv2 doc is still a work in progress but should give a gist on where we
> are currently[1].  There is a bunch of todo still but seems tractable; e.g.
> rolling upgrade, finish doc., and we don't have an HBCK since it needs to
> be recast in light of how stuff now works but a redo on HBCK is premature
> given we don't know failure types as yet (we just fix the problems as they
> come up).
>
> St.Ack
> 1.
> https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#
>
>
>
> On Thu, Apr 13, 2017 at 1:43 PM, Stack <st...@duboce.net> wrote:
>
> > Some status:
> >
> > AMv2 (HBASE-14614) is near to passing all tests caveat my disabling of
> all
> > to-do w/ fsck (fsck needs revamp) and tests that expect that they can
> move
> > hbase;meta off master (AMv2 enforces this constraint; it is supposed to
> be
> > enforced on AMv1 but meta-on-master is incompletely realized in AMv1 and
> > AMv2). A few other tests have been disabled for various reasons. See [1]
> > for full list.
> >
> > There is a hefty list of TODOs still (Again see the messy doc [1]) but
> the
> > only 'blocker', IMO, is community confidence in AMv2. Currently, cluster
> > tests with chaos fail (new form of 'stuck' regions). Takes time
> > investigating.
> >
> > Will keep you all posted.
> > St.Ack
> >
> >
> >
> > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod
> >
> >
> >
> > On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> >> +1 on branching (yay!)
> >>
> >> I have EC2 resources for running ITBLL etc.
> >>
> >>
> >> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> wrote:
> >>
> >> > Some notes on progress toward hbase2.
> >> >
> >> > Given that stability and performance are NOT emergent behaviors but
> >> rather
> >> > projects unto themselves, my thought is that we commit all that we've
> >> > agreed as core for hbase2 (see [1]), branch, and then work on
> >> stabilizing
> >> > and perf rather than do stabilize, commit, and then branch. What this
> >> means
> >> > in practice is that for features like Inmemory Compaction, we commit
> it
> >> > defaulted 'on' ("BASIC" mode) which is what we want in hbase2. Should
> it
> >> > prove problematic under test, we disable it before release.
> >> >
> >> > Are folks good w/ this mode? I ask because, in a few issues there are
> >> > requests for proof that a master feature is 'stable' before commit.
> >> This is
> >> > normally a healthy request only in master's case, it is hard to
> >> demonstrate
> >> > stability given its current state.
> >> >
> >> > Other outstanding issues such as decisions about whether master hosts
> >> > system tables only (by default), I'm thinking, we can work out post
> >> branch
> >> > in alpha/betas before release.
> >> >
> >> > The awkward item is the long-pole Assignment Manager. This is an
> >> > all-or-nothing affair. Here we are switching in a new Master core.
> >> While I
> >> > think it fine that AMv2 is incomplete come branch time, those of us
> >> working
> >> > on the new AM still need to demonstrate to you all that it basically
> >> > viable.
> >> >
> >> > The point-of-no-return is commit of the patch in HBASE-14614.
> >> HBASE-14614
> >> > (AMv2) is coming close to passing all unit tests. We'll spend some
> time
> >> > running it on a cluster to make sure it fundamentally sound and will
> >> report
> >> > back on our experience. There has been an ask for some dev doc and
> >> > low-levels on how it works (in progress). Let satisfaction of these
> >> > requests be blockers on commit. We'll put the HBASE-14614 commit up
> for
> >> a
> >> > vote on dev list given its import.
> >> >
> >> > Branch will happen after HBASE-14614 goes in (or its rejection) with
> our
> >> > first alpha soon after. Its looking like a week or two at least given
> >> how
> >> > things have been going up to this.
> >> >
> >> > I intend to start in on hbase2 stability/perf projects after we
> branch.
> >> >
> >> > Interested in any thoughts you all might have on the above (Would also
> >> > appreciate updates on state in [1] if you are a feature owner).
> >> >
> >> > Thanks,
> >> > St.Ack
> >> >
> >> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> >> > z9iEu_ktczrlKHK8N4SZzs/edit#
> >> >
> >> >
> >> >
> >> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <el...@apache.org>
> wrote:
> >> >
> >> > >
> >> > > Stack wrote:
> >> > >
> >> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<el...@apache.org>
> >> wrote:
> >> > >>
> >> > >> Thanks for pulling in the FS Quotas work, Stack. I'm trying to
> cross
> >> the
> >> > >>> last T's and dot the last I's.
> >> > >>>
> >> > >>> The biggest thing I know I need to do still is to write a new
> >> chapter
> >> > to
> >> > >>> the book. After that, I'd start entertaining larger
> >> reviews/discussions
> >> > >>> to
> >> > >>> merge the feature into master. Anyone with free time (giggles) is
> >> more
> >> > >>> than
> >> > >>> welcome to start perusing :)
> >> > >>>
> >> > >>>
> >> > >>> Out of interest, this could come in after 2.0 Josh? Any 2.0
> specific
> >> > >> needs
> >> > >> to make this work?
> >> > >>
> >> > >> Meantime, updated the 2.0 doc 1.
> >> > >>
> >> > >> Thanks Josh,
> >> > >> St.Ack
> >> > >>
> >> > >> 1.
> >> > >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >> > >> Eu_ktczrlKHK8N4SZzs/edit#
> >> > >>
> >> > >>
> >> > > Nope, no need to block 2.0 on this one (given the other, related
> >> > chatter).
> >> > > Would be nice to get it in, but I completely understand if it slips
> :)
> >> > >
> >> > > Thanks for updating the doc for me!
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >>    - Andy
> >>
> >> If you are given a choice, you believe you have acted freely. - Raymond
> >> Teller (via Peter Watts)
> >>
> >
> >
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

A month on. Status.

I've been working on the HBASE-14614 branch cluster testing. After a load
of fixing, the branch passes smaller test runs (an hour or so of ITBLL up
to 2B rows w/ killing monkeys). When I go larger, to a scale I've not done
in a while, I start to run into other interesting issues -- some of which
are related to AMv2 (I'm fixing), but others are not (100G WALs that take
ten minutes to split makes for interesting cascades when monkeys kill
inside the ten minutes...). I intend to keep on with this larger scale
testing since it is uncovering good stuff (especially when HDFS is dog slow
because of background replications) but my thinking is that I should be
large scale testing branch-2, not just HBASE-14614. I think HBASE-14614,
the new AMv2, is good enough to merge to master these times. Given it is
the last blocker, once in, I'll cut the hbase2 branch.

I'll start up a 'Merge HBASE-14614' DISCUSSION thread in the next day or so
(I need to fix some unit tests...).

The AMv2 doc is still a work in progress but should give a gist on where we
are currently[1].  There is a bunch of todo still but seems tractable; e.g.
rolling upgrade, finish doc., and we don't have an HBCK since it needs to
be recast in light of how stuff now works but a redo on HBCK is premature
given we don't know failure types as yet (we just fix the problems as they
come up).

St.Ack
1.
https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#



On Thu, Apr 13, 2017 at 1:43 PM, Stack <st...@duboce.net> wrote:

> Some status:
>
> AMv2 (HBASE-14614) is near to passing all tests caveat my disabling of all
> to-do w/ fsck (fsck needs revamp) and tests that expect that they can move
> hbase;meta off master (AMv2 enforces this constraint; it is supposed to be
> enforced on AMv1 but meta-on-master is incompletely realized in AMv1 and
> AMv2). A few other tests have been disabled for various reasons. See [1]
> for full list.
>
> There is a hefty list of TODOs still (Again see the messy doc [1]) but the
> only 'blocker', IMO, is community confidence in AMv2. Currently, cluster
> tests with chaos fail (new form of 'stuck' regions). Takes time
> investigating.
>
> Will keep you all posted.
> St.Ack
>
>
>
> 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1-
> 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod
>
>
>
> On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
>> +1 on branching (yay!)
>>
>> I have EC2 resources for running ITBLL etc.
>>
>>
>> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> wrote:
>>
>> > Some notes on progress toward hbase2.
>> >
>> > Given that stability and performance are NOT emergent behaviors but
>> rather
>> > projects unto themselves, my thought is that we commit all that we've
>> > agreed as core for hbase2 (see [1]), branch, and then work on
>> stabilizing
>> > and perf rather than do stabilize, commit, and then branch. What this
>> means
>> > in practice is that for features like Inmemory Compaction, we commit it
>> > defaulted 'on' ("BASIC" mode) which is what we want in hbase2. Should it
>> > prove problematic under test, we disable it before release.
>> >
>> > Are folks good w/ this mode? I ask because, in a few issues there are
>> > requests for proof that a master feature is 'stable' before commit.
>> This is
>> > normally a healthy request only in master's case, it is hard to
>> demonstrate
>> > stability given its current state.
>> >
>> > Other outstanding issues such as decisions about whether master hosts
>> > system tables only (by default), I'm thinking, we can work out post
>> branch
>> > in alpha/betas before release.
>> >
>> > The awkward item is the long-pole Assignment Manager. This is an
>> > all-or-nothing affair. Here we are switching in a new Master core.
>> While I
>> > think it fine that AMv2 is incomplete come branch time, those of us
>> working
>> > on the new AM still need to demonstrate to you all that it basically
>> > viable.
>> >
>> > The point-of-no-return is commit of the patch in HBASE-14614.
>> HBASE-14614
>> > (AMv2) is coming close to passing all unit tests. We'll spend some time
>> > running it on a cluster to make sure it fundamentally sound and will
>> report
>> > back on our experience. There has been an ask for some dev doc and
>> > low-levels on how it works (in progress). Let satisfaction of these
>> > requests be blockers on commit. We'll put the HBASE-14614 commit up for
>> a
>> > vote on dev list given its import.
>> >
>> > Branch will happen after HBASE-14614 goes in (or its rejection) with our
>> > first alpha soon after. Its looking like a week or two at least given
>> how
>> > things have been going up to this.
>> >
>> > I intend to start in on hbase2 stability/perf projects after we branch.
>> >
>> > Interested in any thoughts you all might have on the above (Would also
>> > appreciate updates on state in [1] if you are a feature owner).
>> >
>> > Thanks,
>> > St.Ack
>> >
>> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
>> > z9iEu_ktczrlKHK8N4SZzs/edit#
>> >
>> >
>> >
>> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <el...@apache.org> wrote:
>> >
>> > >
>> > > Stack wrote:
>> > >
>> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<el...@apache.org>
>> wrote:
>> > >>
>> > >> Thanks for pulling in the FS Quotas work, Stack. I'm trying to cross
>> the
>> > >>> last T's and dot the last I's.
>> > >>>
>> > >>> The biggest thing I know I need to do still is to write a new
>> chapter
>> > to
>> > >>> the book. After that, I'd start entertaining larger
>> reviews/discussions
>> > >>> to
>> > >>> merge the feature into master. Anyone with free time (giggles) is
>> more
>> > >>> than
>> > >>> welcome to start perusing :)
>> > >>>
>> > >>>
>> > >>> Out of interest, this could come in after 2.0 Josh? Any 2.0 specific
>> > >> needs
>> > >> to make this work?
>> > >>
>> > >> Meantime, updated the 2.0 doc 1.
>> > >>
>> > >> Thanks Josh,
>> > >> St.Ack
>> > >>
>> > >> 1.
>> > >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> > >> Eu_ktczrlKHK8N4SZzs/edit#
>> > >>
>> > >>
>> > > Nope, no need to block 2.0 on this one (given the other, related
>> > chatter).
>> > > Would be nice to get it in, but I completely understand if it slips :)
>> > >
>> > > Thanks for updating the doc for me!
>> > >
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>>    - Andy
>>
>> If you are given a choice, you believe you have acted freely. - Raymond
>> Teller (via Peter Watts)
>>
>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Some status:

AMv2 (HBASE-14614) is near to passing all tests caveat my disabling of all
to-do w/ fsck (fsck needs revamp) and tests that expect that they can move
hbase;meta off master (AMv2 enforces this constraint; it is supposed to be
enforced on AMv1 but meta-on-master is incompletely realized in AMv1 and
AMv2). A few other tests have been disabled for various reasons. See [1]
for full list.

There is a hefty list of TODOs still (Again see the messy doc [1]) but the
only 'blocker', IMO, is community confidence in AMv2. Currently, cluster
tests with chaos fail (new form of 'stuck' regions). Takes time
investigating.

Will keep you all posted.
St.Ack



1.
https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod



On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell <ap...@apache.org> wrote:

> +1 on branching (yay!)
>
> I have EC2 resources for running ITBLL etc.
>
>
> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> wrote:
>
> > Some notes on progress toward hbase2.
> >
> > Given that stability and performance are NOT emergent behaviors but
> rather
> > projects unto themselves, my thought is that we commit all that we've
> > agreed as core for hbase2 (see [1]), branch, and then work on stabilizing
> > and perf rather than do stabilize, commit, and then branch. What this
> means
> > in practice is that for features like Inmemory Compaction, we commit it
> > defaulted 'on' ("BASIC" mode) which is what we want in hbase2. Should it
> > prove problematic under test, we disable it before release.
> >
> > Are folks good w/ this mode? I ask because, in a few issues there are
> > requests for proof that a master feature is 'stable' before commit. This
> is
> > normally a healthy request only in master's case, it is hard to
> demonstrate
> > stability given its current state.
> >
> > Other outstanding issues such as decisions about whether master hosts
> > system tables only (by default), I'm thinking, we can work out post
> branch
> > in alpha/betas before release.
> >
> > The awkward item is the long-pole Assignment Manager. This is an
> > all-or-nothing affair. Here we are switching in a new Master core. While
> I
> > think it fine that AMv2 is incomplete come branch time, those of us
> working
> > on the new AM still need to demonstrate to you all that it basically
> > viable.
> >
> > The point-of-no-return is commit of the patch in HBASE-14614. HBASE-14614
> > (AMv2) is coming close to passing all unit tests. We'll spend some time
> > running it on a cluster to make sure it fundamentally sound and will
> report
> > back on our experience. There has been an ask for some dev doc and
> > low-levels on how it works (in progress). Let satisfaction of these
> > requests be blockers on commit. We'll put the HBASE-14614 commit up for a
> > vote on dev list given its import.
> >
> > Branch will happen after HBASE-14614 goes in (or its rejection) with our
> > first alpha soon after. Its looking like a week or two at least given how
> > things have been going up to this.
> >
> > I intend to start in on hbase2 stability/perf projects after we branch.
> >
> > Interested in any thoughts you all might have on the above (Would also
> > appreciate updates on state in [1] if you are a feature owner).
> >
> > Thanks,
> > St.Ack
> >
> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> > z9iEu_ktczrlKHK8N4SZzs/edit#
> >
> >
> >
> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <el...@apache.org> wrote:
> >
> > >
> > > Stack wrote:
> > >
> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<el...@apache.org>  wrote:
> > >>
> > >> Thanks for pulling in the FS Quotas work, Stack. I'm trying to cross
> the
> > >>> last T's and dot the last I's.
> > >>>
> > >>> The biggest thing I know I need to do still is to write a new chapter
> > to
> > >>> the book. After that, I'd start entertaining larger
> reviews/discussions
> > >>> to
> > >>> merge the feature into master. Anyone with free time (giggles) is
> more
> > >>> than
> > >>> welcome to start perusing :)
> > >>>
> > >>>
> > >>> Out of interest, this could come in after 2.0 Josh? Any 2.0 specific
> > >> needs
> > >> to make this work?
> > >>
> > >> Meantime, updated the 2.0 doc 1.
> > >>
> > >> Thanks Josh,
> > >> St.Ack
> > >>
> > >> 1.
> > >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> > >> Eu_ktczrlKHK8N4SZzs/edit#
> > >>
> > >>
> > > Nope, no need to block 2.0 on this one (given the other, related
> > chatter).
> > > Would be nice to get it in, but I completely understand if it slips :)
> > >
> > > Thanks for updating the doc for me!
> > >
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> If you are given a choice, you believe you have acted freely. - Raymond
> Teller (via Peter Watts)
>

Re: Moving 2.0 forward

[Andrew Purtell <ap...@apache.org>]

+1 on branching (yay!)

I have EC2 resources for running ITBLL etc.


On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> wrote:

> Some notes on progress toward hbase2.
>
> Given that stability and performance are NOT emergent behaviors but rather
> projects unto themselves, my thought is that we commit all that we've
> agreed as core for hbase2 (see [1]), branch, and then work on stabilizing
> and perf rather than do stabilize, commit, and then branch. What this means
> in practice is that for features like Inmemory Compaction, we commit it
> defaulted 'on' ("BASIC" mode) which is what we want in hbase2. Should it
> prove problematic under test, we disable it before release.
>
> Are folks good w/ this mode? I ask because, in a few issues there are
> requests for proof that a master feature is 'stable' before commit. This is
> normally a healthy request only in master's case, it is hard to demonstrate
> stability given its current state.
>
> Other outstanding issues such as decisions about whether master hosts
> system tables only (by default), I'm thinking, we can work out post branch
> in alpha/betas before release.
>
> The awkward item is the long-pole Assignment Manager. This is an
> all-or-nothing affair. Here we are switching in a new Master core. While I
> think it fine that AMv2 is incomplete come branch time, those of us working
> on the new AM still need to demonstrate to you all that it basically
> viable.
>
> The point-of-no-return is commit of the patch in HBASE-14614. HBASE-14614
> (AMv2) is coming close to passing all unit tests. We'll spend some time
> running it on a cluster to make sure it fundamentally sound and will report
> back on our experience. There has been an ask for some dev doc and
> low-levels on how it works (in progress). Let satisfaction of these
> requests be blockers on commit. We'll put the HBASE-14614 commit up for a
> vote on dev list given its import.
>
> Branch will happen after HBASE-14614 goes in (or its rejection) with our
> first alpha soon after. Its looking like a week or two at least given how
> things have been going up to this.
>
> I intend to start in on hbase2 stability/perf projects after we branch.
>
> Interested in any thoughts you all might have on the above (Would also
> appreciate updates on state in [1] if you are a feature owner).
>
> Thanks,
> St.Ack
>
> 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
> z9iEu_ktczrlKHK8N4SZzs/edit#
>
>
>
> On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <el...@apache.org> wrote:
>
> >
> > Stack wrote:
> >
> >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<el...@apache.org>  wrote:
> >>
> >> Thanks for pulling in the FS Quotas work, Stack. I'm trying to cross the
> >>> last T's and dot the last I's.
> >>>
> >>> The biggest thing I know I need to do still is to write a new chapter
> to
> >>> the book. After that, I'd start entertaining larger reviews/discussions
> >>> to
> >>> merge the feature into master. Anyone with free time (giggles) is more
> >>> than
> >>> welcome to start perusing :)
> >>>
> >>>
> >>> Out of interest, this could come in after 2.0 Josh? Any 2.0 specific
> >> needs
> >> to make this work?
> >>
> >> Meantime, updated the 2.0 doc 1.
> >>
> >> Thanks Josh,
> >> St.Ack
> >>
> >> 1.
> >> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
> >> Eu_ktczrlKHK8N4SZzs/edit#
> >>
> >>
> > Nope, no need to block 2.0 on this one (given the other, related
> chatter).
> > Would be nice to get it in, but I completely understand if it slips :)
> >
> > Thanks for updating the doc for me!
> >
>



-- 
Best regards,

   - Andy

If you are given a choice, you believe you have acted freely. - Raymond
Teller (via Peter Watts)

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Some notes on progress toward hbase2.

Given that stability and performance are NOT emergent behaviors but rather
projects unto themselves, my thought is that we commit all that we've
agreed as core for hbase2 (see [1]), branch, and then work on stabilizing
and perf rather than do stabilize, commit, and then branch. What this means
in practice is that for features like Inmemory Compaction, we commit it
defaulted 'on' ("BASIC" mode) which is what we want in hbase2. Should it
prove problematic under test, we disable it before release.

Are folks good w/ this mode? I ask because, in a few issues there are
requests for proof that a master feature is 'stable' before commit. This is
normally a healthy request only in master's case, it is hard to demonstrate
stability given its current state.

Other outstanding issues such as decisions about whether master hosts
system tables only (by default), I'm thinking, we can work out post branch
in alpha/betas before release.

The awkward item is the long-pole Assignment Manager. This is an
all-or-nothing affair. Here we are switching in a new Master core. While I
think it fine that AMv2 is incomplete come branch time, those of us working
on the new AM still need to demonstrate to you all that it basically viable.

The point-of-no-return is commit of the patch in HBASE-14614. HBASE-14614
(AMv2) is coming close to passing all unit tests. We'll spend some time
running it on a cluster to make sure it fundamentally sound and will report
back on our experience. There has been an ask for some dev doc and
low-levels on how it works (in progress). Let satisfaction of these
requests be blockers on commit. We'll put the HBASE-14614 commit up for a
vote on dev list given its import.

Branch will happen after HBASE-14614 goes in (or its rejection) with our
first alpha soon after. Its looking like a week or two at least given how
things have been going up to this.

I intend to start in on hbase2 stability/perf projects after we branch.

Interested in any thoughts you all might have on the above (Would also
appreciate updates on state in [1] if you are a feature owner).

Thanks,
St.Ack

1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4
z9iEu_ktczrlKHK8N4SZzs/edit#



On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <el...@apache.org> wrote:

>
> Stack wrote:
>
>> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<el...@apache.org>  wrote:
>>
>> Thanks for pulling in the FS Quotas work, Stack. I'm trying to cross the
>>> last T's and dot the last I's.
>>>
>>> The biggest thing I know I need to do still is to write a new chapter to
>>> the book. After that, I'd start entertaining larger reviews/discussions
>>> to
>>> merge the feature into master. Anyone with free time (giggles) is more
>>> than
>>> welcome to start perusing :)
>>>
>>>
>>> Out of interest, this could come in after 2.0 Josh? Any 2.0 specific
>> needs
>> to make this work?
>>
>> Meantime, updated the 2.0 doc 1.
>>
>> Thanks Josh,
>> St.Ack
>>
>> 1.
>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>> Eu_ktczrlKHK8N4SZzs/edit#
>>
>>
> Nope, no need to block 2.0 on this one (given the other, related chatter).
> Would be nice to get it in, but I completely understand if it slips :)
>
> Thanks for updating the doc for me!
>

Re: Moving 2.0 forward

[Josh Elser <el...@apache.org>]

Stack wrote:
> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser<el...@apache.org>  wrote:
>
>> Thanks for pulling in the FS Quotas work, Stack. I'm trying to cross the
>> last T's and dot the last I's.
>>
>> The biggest thing I know I need to do still is to write a new chapter to
>> the book. After that, I'd start entertaining larger reviews/discussions to
>> merge the feature into master. Anyone with free time (giggles) is more than
>> welcome to start perusing :)
>>
>>
> Out of interest, this could come in after 2.0 Josh? Any 2.0 specific needs
> to make this work?
>
> Meantime, updated the 2.0 doc 1.
>
> Thanks Josh,
> St.Ack
>
> 1.
> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#
>

Nope, no need to block 2.0 on this one (given the other, related 
chatter). Would be nice to get it in, but I completely understand if it 
slips :)

Thanks for updating the doc for me!

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Thu, Mar 9, 2017 at 10:25 PM, Stack <st...@duboce.net> wrote:

> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser <el...@apache.org> wrote:
>
>> Thanks for pulling in the FS Quotas work, Stack. I'm trying to cross the
>> last T's and dot the last I's.
>>
>> The biggest thing I know I need to do still is to write a new chapter to
>> the book. After that, I'd start entertaining larger reviews/discussions to
>> merge the feature into master. Anyone with free time (giggles) is more than
>> welcome to start perusing :)
>>
>>
> Out of interest, this could come in after 2.0 Josh? Any 2.0 specific needs
> to make this work?
>
> nvm Josh. I have it in the Ancillaries section so its not blocker. Fix it
if I have it wrong.
St.Ack




> Meantime, updated the 2.0 doc 1.
>
> Thanks Josh,
> St.Ack
>
> 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> ktczrlKHK8N4SZzs/edit#
>
>
>
>>
>> Stack wrote:
>>
>>> Just a quick update (Please feel free to amend your own status in our
>>> hbase-2.0 doc [1]).
>>>
>>> On the criticals:
>>>
>>> + Little progress on the blocker AMv2. A month or more necessary still
>>> before code complete (Estimate)
>>> + Accordion (inmemory compaction) progressing nicely as is the offheap
>>> read/write path. Both are almost done.
>>> + Rolling upgrade testing. No work done.
>>> + No recent work on core decision tasks (clean-up narrative around RPC
>>> timeout, hbase:meta on master or not, batch vs partial semantic, etc.)
>>>
>>> Non-criticals/Ancillaries
>>>
>>> + Async client and C++ client are both making good progress. Not done.
>>> + Backup/Restore is making good progress
>>> + RegionServer-based assignment got a bunch of scrutiny lately and is now
>>> 'done'.
>>> + FileSystem Quotas making good progress.
>>>
>>> I'm seeing another month or two at least before branch and probably
>>> three.
>>> See doc [1] for more detail.
>>>
>>> Yours,
>>> St.Ack
>>>
>>> 1.  https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
>>> ktczrlKHK8N4SZzs/edit#
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Jan 29, 2017 at 9:16 PM, Stack<st...@duboce.net>  wrote:
>>>
>>> On Thu, Jan 26, 2017 at 11:49 PM, ramkrishna vasudevan<
>>>> ramkrishna.s.vasudevan@gmail.com>  wrote:
>>>>
>>>> Hi All
>>>>>
>>>>> Thanks Stack. The doc looks great. The offheap write path/read path- I
>>>>> think from the read path perspective we have some good feedback from
>>>>> Alibaba folks.
>>>>>
>>>>> Agree.
>>>>
>>>>
>>>> The write path subtasks are all done. We are currently working on some
>>>>> perf
>>>>> results that would help us to come up with some docs that suggests best
>>>>> configs and tunings for the offheap write path configurations.
>>>>>
>>>>>
>>>>> Thanks Ram. Would be good to hear what configs you are looking to
>>>> implement as default so those of us also starting to test can enable
>>>> them
>>>> to get you feedback.
>>>>
>>>> Also suggest you fill the above short status into the doc (You are
>>>> keeping
>>>> up full status elsewhere). I've been trying to add status as I see it
>>>> popping up; e.g. Enis did a nice state-of-the-C++ client recently up in
>>>> JIRA and I added pointer to the 2.0 doc. Anyone else working on 2.0
>>>> features, would be good if you kept a short state in this overview doc;
>>>> just ask for edit perms.
>>>>
>>>> Thanks,
>>>> St.Ack
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Regards
>>>>> Ram
>>>>>
>>>>> On Thu, Jan 19, 2017 at 5:37 AM, Andrew Purtell<andrew.purtell@gmail.c
>>>>> om
>>>>> wrote:
>>>>>
>>>>> I'm interested in both split meta and rsgroups. Good news. I'd like to
>>>>>> help test.
>>>>>>
>>>>>>
>>>>>> On Jan 18, 2017, at 2:53 PM, Stack<st...@duboce.net>  wrote:
>>>>>>>
>>>>>>> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu<to...@apache.org>
>>>>>>>>
>>>>>>> wrote:
>>>>>
>>>>>> Hi Stack,
>>>>>>>> I'd like to get split meta (HBASE-112288) in 2.x as well. I can have
>>>>>>>>
>>>>>>> a
>>>>>
>>>>>> 2.x
>>>>>>
>>>>>>> draft up next week. Was working on the 1.x version internally.
>>>>>>>> Also if you'd like I can be the owner for rsgroups as well.
>>>>>>>> Thanks,Francis
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I added splitting meta as a possible and had you and I as owner on
>>>>>>>>
>>>>>>> rsgroups (I was doing to do a bit of testing and doc for this
>>>>>>>
>>>>>> feature).
>>>>>
>>>>>> Would love to see splittable meta show up. Needs to be rolling
>>>>>>>
>>>>>> upgradeable
>>>>>>
>>>>>>> though. Lets chat up on the issue.
>>>>>>> St.Ack
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>     On Wednesday, January 18, 2017 11:29 AM, Stack<stack@duboce.net
>>>>>>>> >
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Done Thiruvel (And thanks Guanghao for adding hbase-replication).
>>>>>>>> St.Ack
>>>>>>>>
>>>>>>>> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan<
>>>>>>>> thiruvel@yahoo-inc.com.invalid>  wrote:
>>>>>>>>
>>>>>>>> Hi Stack,
>>>>>>>>> I would like to add Favored Nodes to the ancillary section.
>>>>>>>>> HBASE-15531: Favored Nodes EnhancementsStatus: Active
>>>>>>>>>
>>>>>>>> development.Owner:
>>>>>>
>>>>>>> Thiruvel Thanks!-Thiruvel
>>>>>>>>>
>>>>>>>>>    On Monday, January 16, 2017 2:10 PM, Stack<st...@duboce.net>
>>>>>>>>>
>>>>>>>> wrote:
>>>>>
>>>>>>
>>>>>>>>> On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang<zghaobac@gmail.com
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> For 6. Significant contirbs in master only, there are some issues
>>>>>>>>>>
>>>>>>>>> about
>>>>>>
>>>>>>> replication operations routed through master. They are sub-task
>>>>>>>>>> of HBASE-10504. And there are other umbrella issue for
>>>>>>>>>> replication,
>>>>>>>>>>
>>>>>>>>> like
>>>>>>>>
>>>>>>>>> HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
>>>>>>>>>> tracking from Zookeeper to HBase. So i thought we can add a new
>>>>>>>>>>
>>>>>>>>> section
>>>>>>
>>>>>>> named
>>>>>>>>>> hbase-replication to possible 2.0.0s. This will help us to track
>>>>>>>>>>
>>>>>>>>> the
>>>>>
>>>>>> state.
>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> Thanks Guanghao Zhang. I agree. I made you an editor. If you want
>>>>>>>>> to
>>>>>>>>>
>>>>>>>> have a
>>>>>>>>
>>>>>>>>> go at a first cut, be my guest. If nothing done in the next day or
>>>>>>>>>
>>>>>>>> so,
>>>>>
>>>>>> I'll
>>>>>>>>
>>>>>>>>> add this section Sir.
>>>>>>>>> Thanks,
>>>>>>>>> M
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>
>>>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser <el...@apache.org> wrote:

> Thanks for pulling in the FS Quotas work, Stack. I'm trying to cross the
> last T's and dot the last I's.
>
> The biggest thing I know I need to do still is to write a new chapter to
> the book. After that, I'd start entertaining larger reviews/discussions to
> merge the feature into master. Anyone with free time (giggles) is more than
> welcome to start perusing :)
>
>
Out of interest, this could come in after 2.0 Josh? Any 2.0 specific needs
to make this work?

Meantime, updated the 2.0 doc 1.

Thanks Josh,
St.Ack

1.
https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#



>
> Stack wrote:
>
>> Just a quick update (Please feel free to amend your own status in our
>> hbase-2.0 doc [1]).
>>
>> On the criticals:
>>
>> + Little progress on the blocker AMv2. A month or more necessary still
>> before code complete (Estimate)
>> + Accordion (inmemory compaction) progressing nicely as is the offheap
>> read/write path. Both are almost done.
>> + Rolling upgrade testing. No work done.
>> + No recent work on core decision tasks (clean-up narrative around RPC
>> timeout, hbase:meta on master or not, batch vs partial semantic, etc.)
>>
>> Non-criticals/Ancillaries
>>
>> + Async client and C++ client are both making good progress. Not done.
>> + Backup/Restore is making good progress
>> + RegionServer-based assignment got a bunch of scrutiny lately and is now
>> 'done'.
>> + FileSystem Quotas making good progress.
>>
>> I'm seeing another month or two at least before branch and probably three.
>> See doc [1] for more detail.
>>
>> Yours,
>> St.Ack
>>
>> 1.  https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
>> ktczrlKHK8N4SZzs/edit#
>>
>>
>>
>>
>>
>> On Sun, Jan 29, 2017 at 9:16 PM, Stack<st...@duboce.net>  wrote:
>>
>> On Thu, Jan 26, 2017 at 11:49 PM, ramkrishna vasudevan<
>>> ramkrishna.s.vasudevan@gmail.com>  wrote:
>>>
>>> Hi All
>>>>
>>>> Thanks Stack. The doc looks great. The offheap write path/read path- I
>>>> think from the read path perspective we have some good feedback from
>>>> Alibaba folks.
>>>>
>>>> Agree.
>>>
>>>
>>> The write path subtasks are all done. We are currently working on some
>>>> perf
>>>> results that would help us to come up with some docs that suggests best
>>>> configs and tunings for the offheap write path configurations.
>>>>
>>>>
>>>> Thanks Ram. Would be good to hear what configs you are looking to
>>> implement as default so those of us also starting to test can enable them
>>> to get you feedback.
>>>
>>> Also suggest you fill the above short status into the doc (You are
>>> keeping
>>> up full status elsewhere). I've been trying to add status as I see it
>>> popping up; e.g. Enis did a nice state-of-the-C++ client recently up in
>>> JIRA and I added pointer to the 2.0 doc. Anyone else working on 2.0
>>> features, would be good if you kept a short state in this overview doc;
>>> just ask for edit perms.
>>>
>>> Thanks,
>>> St.Ack
>>>
>>>
>>>
>>>
>>>
>>> Regards
>>>> Ram
>>>>
>>>> On Thu, Jan 19, 2017 at 5:37 AM, Andrew Purtell<andrew.purtell@gmail.c
>>>> om
>>>> wrote:
>>>>
>>>> I'm interested in both split meta and rsgroups. Good news. I'd like to
>>>>> help test.
>>>>>
>>>>>
>>>>> On Jan 18, 2017, at 2:53 PM, Stack<st...@duboce.net>  wrote:
>>>>>>
>>>>>> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu<to...@apache.org>
>>>>>>>
>>>>>> wrote:
>>>>
>>>>> Hi Stack,
>>>>>>> I'd like to get split meta (HBASE-112288) in 2.x as well. I can have
>>>>>>>
>>>>>> a
>>>>
>>>>> 2.x
>>>>>
>>>>>> draft up next week. Was working on the 1.x version internally.
>>>>>>> Also if you'd like I can be the owner for rsgroups as well.
>>>>>>> Thanks,Francis
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I added splitting meta as a possible and had you and I as owner on
>>>>>>>
>>>>>> rsgroups (I was doing to do a bit of testing and doc for this
>>>>>>
>>>>> feature).
>>>>
>>>>> Would love to see splittable meta show up. Needs to be rolling
>>>>>>
>>>>> upgradeable
>>>>>
>>>>>> though. Lets chat up on the issue.
>>>>>> St.Ack
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>     On Wednesday, January 18, 2017 11:29 AM, Stack<st...@duboce.net>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Done Thiruvel (And thanks Guanghao for adding hbase-replication).
>>>>>>> St.Ack
>>>>>>>
>>>>>>> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan<
>>>>>>> thiruvel@yahoo-inc.com.invalid>  wrote:
>>>>>>>
>>>>>>> Hi Stack,
>>>>>>>> I would like to add Favored Nodes to the ancillary section.
>>>>>>>> HBASE-15531: Favored Nodes EnhancementsStatus: Active
>>>>>>>>
>>>>>>> development.Owner:
>>>>>
>>>>>> Thiruvel Thanks!-Thiruvel
>>>>>>>>
>>>>>>>>    On Monday, January 16, 2017 2:10 PM, Stack<st...@duboce.net>
>>>>>>>>
>>>>>>> wrote:
>>>>
>>>>>
>>>>>>>> On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang<zghaobac@gmail.com
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> For 6. Significant contirbs in master only, there are some issues
>>>>>>>>>
>>>>>>>> about
>>>>>
>>>>>> replication operations routed through master. They are sub-task
>>>>>>>>> of HBASE-10504. And there are other umbrella issue for replication,
>>>>>>>>>
>>>>>>>> like
>>>>>>>
>>>>>>>> HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
>>>>>>>>> tracking from Zookeeper to HBase. So i thought we can add a new
>>>>>>>>>
>>>>>>>> section
>>>>>
>>>>>> named
>>>>>>>>> hbase-replication to possible 2.0.0s. This will help us to track
>>>>>>>>>
>>>>>>>> the
>>>>
>>>>> state.
>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> Thanks Guanghao Zhang. I agree. I made you an editor. If you want
>>>>>>>> to
>>>>>>>>
>>>>>>> have a
>>>>>>>
>>>>>>>> go at a first cut, be my guest. If nothing done in the next day or
>>>>>>>>
>>>>>>> so,
>>>>
>>>>> I'll
>>>>>>>
>>>>>>>> add this section Sir.
>>>>>>>> Thanks,
>>>>>>>> M
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>
>>

Re: Moving 2.0 forward

[Josh Elser <el...@apache.org>]

Thanks for pulling in the FS Quotas work, Stack. I'm trying to cross the 
last T's and dot the last I's.

The biggest thing I know I need to do still is to write a new chapter to 
the book. After that, I'd start entertaining larger reviews/discussions 
to merge the feature into master. Anyone with free time (giggles) is 
more than welcome to start perusing :)

Stack wrote:
> Just a quick update (Please feel free to amend your own status in our
> hbase-2.0 doc [1]).
>
> On the criticals:
>
> + Little progress on the blocker AMv2. A month or more necessary still
> before code complete (Estimate)
> + Accordion (inmemory compaction) progressing nicely as is the offheap
> read/write path. Both are almost done.
> + Rolling upgrade testing. No work done.
> + No recent work on core decision tasks (clean-up narrative around RPC
> timeout, hbase:meta on master or not, batch vs partial semantic, etc.)
>
> Non-criticals/Ancillaries
>
> + Async client and C++ client are both making good progress. Not done.
> + Backup/Restore is making good progress
> + RegionServer-based assignment got a bunch of scrutiny lately and is now
> 'done'.
> + FileSystem Quotas making good progress.
>
> I'm seeing another month or two at least before branch and probably three.
> See doc [1] for more detail.
>
> Yours,
> St.Ack
>
> 1.  https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> ktczrlKHK8N4SZzs/edit#
>
>
>
>
>
> On Sun, Jan 29, 2017 at 9:16 PM, Stack<st...@duboce.net>  wrote:
>
>> On Thu, Jan 26, 2017 at 11:49 PM, ramkrishna vasudevan<
>> ramkrishna.s.vasudevan@gmail.com>  wrote:
>>
>>> Hi All
>>>
>>> Thanks Stack. The doc looks great. The offheap write path/read path- I
>>> think from the read path perspective we have some good feedback from
>>> Alibaba folks.
>>>
>> Agree.
>>
>>
>>> The write path subtasks are all done. We are currently working on some
>>> perf
>>> results that would help us to come up with some docs that suggests best
>>> configs and tunings for the offheap write path configurations.
>>>
>>>
>> Thanks Ram. Would be good to hear what configs you are looking to
>> implement as default so those of us also starting to test can enable them
>> to get you feedback.
>>
>> Also suggest you fill the above short status into the doc (You are keeping
>> up full status elsewhere). I've been trying to add status as I see it
>> popping up; e.g. Enis did a nice state-of-the-C++ client recently up in
>> JIRA and I added pointer to the 2.0 doc. Anyone else working on 2.0
>> features, would be good if you kept a short state in this overview doc;
>> just ask for edit perms.
>>
>> Thanks,
>> St.Ack
>>
>>
>>
>>
>>
>>> Regards
>>> Ram
>>>
>>> On Thu, Jan 19, 2017 at 5:37 AM, Andrew Purtell<andrew.purtell@gmail.com
>>> wrote:
>>>
>>>> I'm interested in both split meta and rsgroups. Good news. I'd like to
>>>> help test.
>>>>
>>>>
>>>>> On Jan 18, 2017, at 2:53 PM, Stack<st...@duboce.net>  wrote:
>>>>>
>>>>>> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu<to...@apache.org>
>>> wrote:
>>>>>> Hi Stack,
>>>>>> I'd like to get split meta (HBASE-112288) in 2.x as well. I can have
>>> a
>>>> 2.x
>>>>>> draft up next week. Was working on the 1.x version internally.
>>>>>> Also if you'd like I can be the owner for rsgroups as well.
>>>>>> Thanks,Francis
>>>>>>
>>>>>>
>>>>>>
>>>>>> I added splitting meta as a possible and had you and I as owner on
>>>>> rsgroups (I was doing to do a bit of testing and doc for this
>>> feature).
>>>>> Would love to see splittable meta show up. Needs to be rolling
>>>> upgradeable
>>>>> though. Lets chat up on the issue.
>>>>> St.Ack
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>     On Wednesday, January 18, 2017 11:29 AM, Stack<st...@duboce.net>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> Done Thiruvel (And thanks Guanghao for adding hbase-replication).
>>>>>> St.Ack
>>>>>>
>>>>>> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan<
>>>>>> thiruvel@yahoo-inc.com.invalid>  wrote:
>>>>>>
>>>>>>> Hi Stack,
>>>>>>> I would like to add Favored Nodes to the ancillary section.
>>>>>>> HBASE-15531: Favored Nodes EnhancementsStatus: Active
>>>> development.Owner:
>>>>>>> Thiruvel Thanks!-Thiruvel
>>>>>>>
>>>>>>>    On Monday, January 16, 2017 2:10 PM, Stack<st...@duboce.net>
>>> wrote:
>>>>>>>
>>>>>>> On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang<zghaobac@gmail.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> For 6. Significant contirbs in master only, there are some issues
>>>> about
>>>>>>>> replication operations routed through master. They are sub-task
>>>>>>>> of HBASE-10504. And there are other umbrella issue for replication,
>>>>>> like
>>>>>>>> HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
>>>>>>>> tracking from Zookeeper to HBase. So i thought we can add a new
>>>> section
>>>>>>>> named
>>>>>>>> hbase-replication to possible 2.0.0s. This will help us to track
>>> the
>>>>>>> state.
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>> Thanks Guanghao Zhang. I agree. I made you an editor. If you want to
>>>>>> have a
>>>>>>> go at a first cut, be my guest. If nothing done in the next day or
>>> so,
>>>>>> I'll
>>>>>>> add this section Sir.
>>>>>>> Thanks,
>>>>>>> M
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Just a quick update (Please feel free to amend your own status in our
hbase-2.0 doc [1]).

On the criticals:

+ Little progress on the blocker AMv2. A month or more necessary still
before code complete (Estimate)
+ Accordion (inmemory compaction) progressing nicely as is the offheap
read/write path. Both are almost done.
+ Rolling upgrade testing. No work done.
+ No recent work on core decision tasks (clean-up narrative around RPC
timeout, hbase:meta on master or not, batch vs partial semantic, etc.)

Non-criticals/Ancillaries

+ Async client and C++ client are both making good progress. Not done.
+ Backup/Restore is making good progress
+ RegionServer-based assignment got a bunch of scrutiny lately and is now
'done'.
+ FileSystem Quotas making good progress.

I'm seeing another month or two at least before branch and probably three.
See doc [1] for more detail.

Yours,
St.Ack

1.  https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
ktczrlKHK8N4SZzs/edit#





On Sun, Jan 29, 2017 at 9:16 PM, Stack <st...@duboce.net> wrote:

> On Thu, Jan 26, 2017 at 11:49 PM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
>> Hi All
>>
>> Thanks Stack. The doc looks great. The offheap write path/read path- I
>> think from the read path perspective we have some good feedback from
>> Alibaba folks.
>>
>
> Agree.
>
>
>> The write path subtasks are all done. We are currently working on some
>> perf
>> results that would help us to come up with some docs that suggests best
>> configs and tunings for the offheap write path configurations.
>>
>>
> Thanks Ram. Would be good to hear what configs you are looking to
> implement as default so those of us also starting to test can enable them
> to get you feedback.
>
> Also suggest you fill the above short status into the doc (You are keeping
> up full status elsewhere). I've been trying to add status as I see it
> popping up; e.g. Enis did a nice state-of-the-C++ client recently up in
> JIRA and I added pointer to the 2.0 doc. Anyone else working on 2.0
> features, would be good if you kept a short state in this overview doc;
> just ask for edit perms.
>
> Thanks,
> St.Ack
>
>
>
>
>
>>
>> Regards
>> Ram
>>
>> On Thu, Jan 19, 2017 at 5:37 AM, Andrew Purtell <andrew.purtell@gmail.com
>> >
>> wrote:
>>
>> > I'm interested in both split meta and rsgroups. Good news. I'd like to
>> > help test.
>> >
>> >
>> > > On Jan 18, 2017, at 2:53 PM, Stack <st...@duboce.net> wrote:
>> > >
>> > >> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu <to...@apache.org>
>> wrote:
>> > >>
>> > >> Hi Stack,
>> > >> I'd like to get split meta (HBASE-112288) in 2.x as well. I can have
>> a
>> > 2.x
>> > >> draft up next week. Was working on the 1.x version internally.
>> > >> Also if you'd like I can be the owner for rsgroups as well.
>> > >> Thanks,Francis
>> > >>
>> > >>
>> > >>
>> > >> I added splitting meta as a possible and had you and I as owner on
>> > > rsgroups (I was doing to do a bit of testing and doc for this
>> feature).
>> > >
>> > > Would love to see splittable meta show up. Needs to be rolling
>> > upgradeable
>> > > though. Lets chat up on the issue.
>> > > St.Ack
>> > >
>> > >
>> > >
>> > >
>> > >>
>> > >>
>> > >>
>> > >>    On Wednesday, January 18, 2017 11:29 AM, Stack <st...@duboce.net>
>> > >> wrote:
>> > >>
>> > >>
>> > >> Done Thiruvel (And thanks Guanghao for adding hbase-replication).
>> > >> St.Ack
>> > >>
>> > >> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
>> > >> thiruvel@yahoo-inc.com.invalid> wrote:
>> > >>
>> > >>> Hi Stack,
>> > >>> I would like to add Favored Nodes to the ancillary section.
>> > >>> HBASE-15531: Favored Nodes EnhancementsStatus: Active
>> > development.Owner:
>> > >>> Thiruvel Thanks!-Thiruvel
>> > >>>
>> > >>>   On Monday, January 16, 2017 2:10 PM, Stack <st...@duboce.net>
>> wrote:
>> > >>>
>> > >>>
>> > >>> On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <zghaobac@gmail.com
>> >
>> > >>> wrote:
>> > >>>
>> > >>>> For 6. Significant contirbs in master only, there are some issues
>> > about
>> > >>>> replication operations routed through master. They are sub-task
>> > >>>> of HBASE-10504. And there are other umbrella issue for replication,
>> > >> like
>> > >>>> HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
>> > >>>> tracking from Zookeeper to HBase. So i thought we can add a new
>> > section
>> > >>>> named
>> > >>>> hbase-replication to possible 2.0.0s. This will help us to track
>> the
>> > >>> state.
>> > >>>> Thanks.
>> > >>>>
>> > >>>
>> > >>> Thanks Guanghao Zhang. I agree. I made you an editor. If you want to
>> > >> have a
>> > >>> go at a first cut, be my guest. If nothing done in the next day or
>> so,
>> > >> I'll
>> > >>> add this section Sir.
>> > >>> Thanks,
>> > >>> M
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>
>> > >>
>> > >>
>> > >>
>> >
>>
>
>

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Thu, Jan 26, 2017 at 11:49 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:

> Hi All
>
> Thanks Stack. The doc looks great. The offheap write path/read path- I
> think from the read path perspective we have some good feedback from
> Alibaba folks.
>

Agree.


> The write path subtasks are all done. We are currently working on some perf
> results that would help us to come up with some docs that suggests best
> configs and tunings for the offheap write path configurations.
>
>
Thanks Ram. Would be good to hear what configs you are looking to implement
as default so those of us also starting to test can enable them to get you
feedback.

Also suggest you fill the above short status into the doc (You are keeping
up full status elsewhere). I've been trying to add status as I see it
popping up; e.g. Enis did a nice state-of-the-C++ client recently up in
JIRA and I added pointer to the 2.0 doc. Anyone else working on 2.0
features, would be good if you kept a short state in this overview doc;
just ask for edit perms.

Thanks,
St.Ack





>
> Regards
> Ram
>
> On Thu, Jan 19, 2017 at 5:37 AM, Andrew Purtell <an...@gmail.com>
> wrote:
>
> > I'm interested in both split meta and rsgroups. Good news. I'd like to
> > help test.
> >
> >
> > > On Jan 18, 2017, at 2:53 PM, Stack <st...@duboce.net> wrote:
> > >
> > >> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu <to...@apache.org>
> wrote:
> > >>
> > >> Hi Stack,
> > >> I'd like to get split meta (HBASE-112288) in 2.x as well. I can have a
> > 2.x
> > >> draft up next week. Was working on the 1.x version internally.
> > >> Also if you'd like I can be the owner for rsgroups as well.
> > >> Thanks,Francis
> > >>
> > >>
> > >>
> > >> I added splitting meta as a possible and had you and I as owner on
> > > rsgroups (I was doing to do a bit of testing and doc for this feature).
> > >
> > > Would love to see splittable meta show up. Needs to be rolling
> > upgradeable
> > > though. Lets chat up on the issue.
> > > St.Ack
> > >
> > >
> > >
> > >
> > >>
> > >>
> > >>
> > >>    On Wednesday, January 18, 2017 11:29 AM, Stack <st...@duboce.net>
> > >> wrote:
> > >>
> > >>
> > >> Done Thiruvel (And thanks Guanghao for adding hbase-replication).
> > >> St.Ack
> > >>
> > >> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
> > >> thiruvel@yahoo-inc.com.invalid> wrote:
> > >>
> > >>> Hi Stack,
> > >>> I would like to add Favored Nodes to the ancillary section.
> > >>> HBASE-15531: Favored Nodes EnhancementsStatus: Active
> > development.Owner:
> > >>> Thiruvel Thanks!-Thiruvel
> > >>>
> > >>>   On Monday, January 16, 2017 2:10 PM, Stack <st...@duboce.net>
> wrote:
> > >>>
> > >>>
> > >>> On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <zg...@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> For 6. Significant contirbs in master only, there are some issues
> > about
> > >>>> replication operations routed through master. They are sub-task
> > >>>> of HBASE-10504. And there are other umbrella issue for replication,
> > >> like
> > >>>> HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
> > >>>> tracking from Zookeeper to HBase. So i thought we can add a new
> > section
> > >>>> named
> > >>>> hbase-replication to possible 2.0.0s. This will help us to track the
> > >>> state.
> > >>>> Thanks.
> > >>>>
> > >>>
> > >>> Thanks Guanghao Zhang. I agree. I made you an editor. If you want to
> > >> have a
> > >>> go at a first cut, be my guest. If nothing done in the next day or
> so,
> > >> I'll
> > >>> add this section Sir.
> > >>> Thanks,
> > >>> M
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> > >>
> > >>
> >
>

Re: Moving 2.0 forward

[ramkrishna vasudevan <ra...@gmail.com>]

Hi All

Thanks Stack. The doc looks great. The offheap write path/read path- I
think from the read path perspective we have some good feedback from
Alibaba folks.
The write path subtasks are all done. We are currently working on some perf
results that would help us to come up with some docs that suggests best
configs and tunings for the offheap write path configurations.


Regards
Ram

On Thu, Jan 19, 2017 at 5:37 AM, Andrew Purtell <an...@gmail.com>
wrote:

> I'm interested in both split meta and rsgroups. Good news. I'd like to
> help test.
>
>
> > On Jan 18, 2017, at 2:53 PM, Stack <st...@duboce.net> wrote:
> >
> >> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu <to...@apache.org> wrote:
> >>
> >> Hi Stack,
> >> I'd like to get split meta (HBASE-112288) in 2.x as well. I can have a
> 2.x
> >> draft up next week. Was working on the 1.x version internally.
> >> Also if you'd like I can be the owner for rsgroups as well.
> >> Thanks,Francis
> >>
> >>
> >>
> >> I added splitting meta as a possible and had you and I as owner on
> > rsgroups (I was doing to do a bit of testing and doc for this feature).
> >
> > Would love to see splittable meta show up. Needs to be rolling
> upgradeable
> > though. Lets chat up on the issue.
> > St.Ack
> >
> >
> >
> >
> >>
> >>
> >>
> >>    On Wednesday, January 18, 2017 11:29 AM, Stack <st...@duboce.net>
> >> wrote:
> >>
> >>
> >> Done Thiruvel (And thanks Guanghao for adding hbase-replication).
> >> St.Ack
> >>
> >> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
> >> thiruvel@yahoo-inc.com.invalid> wrote:
> >>
> >>> Hi Stack,
> >>> I would like to add Favored Nodes to the ancillary section.
> >>> HBASE-15531: Favored Nodes EnhancementsStatus: Active
> development.Owner:
> >>> Thiruvel Thanks!-Thiruvel
> >>>
> >>>   On Monday, January 16, 2017 2:10 PM, Stack <st...@duboce.net> wrote:
> >>>
> >>>
> >>> On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <zg...@gmail.com>
> >>> wrote:
> >>>
> >>>> For 6. Significant contirbs in master only, there are some issues
> about
> >>>> replication operations routed through master. They are sub-task
> >>>> of HBASE-10504. And there are other umbrella issue for replication,
> >> like
> >>>> HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
> >>>> tracking from Zookeeper to HBase. So i thought we can add a new
> section
> >>>> named
> >>>> hbase-replication to possible 2.0.0s. This will help us to track the
> >>> state.
> >>>> Thanks.
> >>>>
> >>>
> >>> Thanks Guanghao Zhang. I agree. I made you an editor. If you want to
> >> have a
> >>> go at a first cut, be my guest. If nothing done in the next day or so,
> >> I'll
> >>> add this section Sir.
> >>> Thanks,
> >>> M
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>
>

Re: Moving 2.0 forward

[Andrew Purtell <an...@gmail.com>]

I have started testing rsgroups. Unfortunately I can't release from master
into our production so have had to backport onto ~1.3.0, including all
subsequent fixes applied in-situ to trunk. Thanks so much for the work
separating the rsgroup code into a separate maven module, that helps
tremendously with the latter. It's close enough so our experience should
translate directly back to master/2.0.

On Wed, Jan 18, 2017 at 4:07 PM, Andrew Purtell <an...@gmail.com>
wrote:

> I'm interested in both split meta and rsgroups. Good news. I'd like to
> help test.
>
>
> > On Jan 18, 2017, at 2:53 PM, Stack <st...@duboce.net> wrote:
> >
> >> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu <to...@apache.org> wrote:
> >>
> >> Hi Stack,
> >> I'd like to get split meta (HBASE-112288) in 2.x as well. I can have a
> 2.x
> >> draft up next week. Was working on the 1.x version internally.
> >> Also if you'd like I can be the owner for rsgroups as well.
> >> Thanks,Francis
> >>
> >>
> >>
> >> I added splitting meta as a possible and had you and I as owner on
> > rsgroups (I was doing to do a bit of testing and doc for this feature).
> >
> > Would love to see splittable meta show up. Needs to be rolling
> upgradeable
> > though. Lets chat up on the issue.
> > St.Ack
> >
> >
> >
> >
> >>
> >>
> >>
> >>    On Wednesday, January 18, 2017 11:29 AM, Stack <st...@duboce.net>
> >> wrote:
> >>
> >>
> >> Done Thiruvel (And thanks Guanghao for adding hbase-replication).
> >> St.Ack
> >>
> >> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
> >> thiruvel@yahoo-inc.com.invalid> wrote:
> >>
> >>> Hi Stack,
> >>> I would like to add Favored Nodes to the ancillary section.
> >>> HBASE-15531: Favored Nodes EnhancementsStatus: Active
> development.Owner:
> >>> Thiruvel Thanks!-Thiruvel
> >>>
> >>>   On Monday, January 16, 2017 2:10 PM, Stack <st...@duboce.net> wrote:
> >>>
> >>>
> >>> On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <zg...@gmail.com>
> >>> wrote:
> >>>
> >>>> For 6. Significant contirbs in master only, there are some issues
> about
> >>>> replication operations routed through master. They are sub-task
> >>>> of HBASE-10504. And there are other umbrella issue for replication,
> >> like
> >>>> HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
> >>>> tracking from Zookeeper to HBase. So i thought we can add a new
> section
> >>>> named
> >>>> hbase-replication to possible 2.0.0s. This will help us to track the
> >>> state.
> >>>> Thanks.
> >>>>
> >>>
> >>> Thanks Guanghao Zhang. I agree. I made you an editor. If you want to
> >> have a
> >>> go at a first cut, be my guest. If nothing done in the next day or so,
> >> I'll
> >>> add this section Sir.
> >>> Thanks,
> >>> M
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>
>

Re: Moving 2.0 forward

[Andrew Purtell <an...@gmail.com>]

I'm interested in both split meta and rsgroups. Good news. I'd like to help test. 


> On Jan 18, 2017, at 2:53 PM, Stack <st...@duboce.net> wrote:
> 
>> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu <to...@apache.org> wrote:
>> 
>> Hi Stack,
>> I'd like to get split meta (HBASE-112288) in 2.x as well. I can have a 2.x
>> draft up next week. Was working on the 1.x version internally.
>> Also if you'd like I can be the owner for rsgroups as well.
>> Thanks,Francis
>> 
>> 
>> 
>> I added splitting meta as a possible and had you and I as owner on
> rsgroups (I was doing to do a bit of testing and doc for this feature).
> 
> Would love to see splittable meta show up. Needs to be rolling upgradeable
> though. Lets chat up on the issue.
> St.Ack
> 
> 
> 
> 
>> 
>> 
>> 
>>    On Wednesday, January 18, 2017 11:29 AM, Stack <st...@duboce.net>
>> wrote:
>> 
>> 
>> Done Thiruvel (And thanks Guanghao for adding hbase-replication).
>> St.Ack
>> 
>> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
>> thiruvel@yahoo-inc.com.invalid> wrote:
>> 
>>> Hi Stack,
>>> I would like to add Favored Nodes to the ancillary section.
>>> HBASE-15531: Favored Nodes EnhancementsStatus: Active development.Owner:
>>> Thiruvel Thanks!-Thiruvel
>>> 
>>>   On Monday, January 16, 2017 2:10 PM, Stack <st...@duboce.net> wrote:
>>> 
>>> 
>>> On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <zg...@gmail.com>
>>> wrote:
>>> 
>>>> For 6. Significant contirbs in master only, there are some issues about
>>>> replication operations routed through master. They are sub-task
>>>> of HBASE-10504. And there are other umbrella issue for replication,
>> like
>>>> HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
>>>> tracking from Zookeeper to HBase. So i thought we can add a new section
>>>> named
>>>> hbase-replication to possible 2.0.0s. This will help us to track the
>>> state.
>>>> Thanks.
>>>> 
>>> 
>>> Thanks Guanghao Zhang. I agree. I made you an editor. If you want to
>> have a
>>> go at a first cut, be my guest. If nothing done in the next day or so,
>> I'll
>>> add this section Sir.
>>> Thanks,
>>> M
>>> 
>>> 
>>> 
>>> 
>> 
>> 
>> 
>> 

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu <to...@apache.org> wrote:

> Hi Stack,
> I'd like to get split meta (HBASE-112288) in 2.x as well. I can have a 2.x
> draft up next week. Was working on the 1.x version internally.
> Also if you'd like I can be the owner for rsgroups as well.
> Thanks,Francis
>
>
>
> I added splitting meta as a possible and had you and I as owner on
rsgroups (I was doing to do a bit of testing and doc for this feature).

Would love to see splittable meta show up. Needs to be rolling upgradeable
though. Lets chat up on the issue.
St.Ack




>
>
>
>     On Wednesday, January 18, 2017 11:29 AM, Stack <st...@duboce.net>
> wrote:
>
>
>  Done Thiruvel (And thanks Guanghao for adding hbase-replication).
> St.Ack
>
> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
> thiruvel@yahoo-inc.com.invalid> wrote:
>
> > Hi Stack,
> > I would like to add Favored Nodes to the ancillary section.
> > HBASE-15531: Favored Nodes EnhancementsStatus: Active development.Owner:
> > Thiruvel Thanks!-Thiruvel
> >
> >    On Monday, January 16, 2017 2:10 PM, Stack <st...@duboce.net> wrote:
> >
> >
> >  On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <zg...@gmail.com>
> > wrote:
> >
> > > For 6. Significant contirbs in master only, there are some issues about
> > > replication operations routed through master. They are sub-task
> > > of HBASE-10504. And there are other umbrella issue for replication,
> like
> > > HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
> > > tracking from Zookeeper to HBase. So i thought we can add a new section
> > > named
> > > hbase-replication to possible 2.0.0s. This will help us to track the
> > state.
> > > Thanks.
> > >
> >
> > Thanks Guanghao Zhang. I agree. I made you an editor. If you want to
> have a
> > go at a first cut, be my guest. If nothing done in the next day or so,
> I'll
> > add this section Sir.
> > Thanks,
> > M
> >
> >
> >
> >
>
>
>
>

Re: Moving 2.0 forward

[Francis Liu <to...@apache.org>]

Hi Stack,
I'd like to get split meta (HBASE-112288) in 2.x as well. I can have a 2.x draft up next week. Was working on the 1.x version internally.
Also if you'd like I can be the owner for rsgroups as well. 
Thanks,Francis






    On Wednesday, January 18, 2017 11:29 AM, Stack <st...@duboce.net> wrote:
 

 Done Thiruvel (And thanks Guanghao for adding hbase-replication).
St.Ack

On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
thiruvel@yahoo-inc.com.invalid> wrote:

> Hi Stack,
> I would like to add Favored Nodes to the ancillary section.
> HBASE-15531: Favored Nodes EnhancementsStatus: Active development.Owner:
> Thiruvel Thanks!-Thiruvel
>
>    On Monday, January 16, 2017 2:10 PM, Stack <st...@duboce.net> wrote:
>
>
>  On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <zg...@gmail.com>
> wrote:
>
> > For 6. Significant contirbs in master only, there are some issues about
> > replication operations routed through master. They are sub-task
> > of HBASE-10504. And there are other umbrella issue for replication, like
> > HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
> > tracking from Zookeeper to HBase. So i thought we can add a new section
> > named
> > hbase-replication to possible 2.0.0s. This will help us to track the
> state.
> > Thanks.
> >
>
> Thanks Guanghao Zhang. I agree. I made you an editor. If you want to have a
> go at a first cut, be my guest. If nothing done in the next day or so, I'll
> add this section Sir.
> Thanks,
> M
>
>
>
>


   

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Done Thiruvel (And thanks Guanghao for adding hbase-replication).
St.Ack

On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan <
thiruvel@yahoo-inc.com.invalid> wrote:

> Hi Stack,
> I would like to add Favored Nodes to the ancillary section.
> HBASE-15531: Favored Nodes EnhancementsStatus: Active development.Owner:
> Thiruvel Thanks!-Thiruvel
>
>     On Monday, January 16, 2017 2:10 PM, Stack <st...@duboce.net> wrote:
>
>
>  On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <zg...@gmail.com>
> wrote:
>
> > For 6. Significant contirbs in master only, there are some issues about
> > replication operations routed through master. They are sub-task
> > of HBASE-10504. And there are other umbrella issue for replication, like
> > HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
> > tracking from Zookeeper to HBase. So i thought we can add a new section
> > named
> > hbase-replication to possible 2.0.0s. This will help us to track the
> state.
> > Thanks.
> >
>
> Thanks Guanghao Zhang. I agree. I made you an editor. If you want to have a
> go at a first cut, be my guest. If nothing done in the next day or so, I'll
> add this section Sir.
> Thanks,
> M
>
>
>
>

Re: Moving 2.0 forward

[Thiruvel Thirumoolan <th...@yahoo-inc.com.INVALID>]

Hi Stack,
I would like to add Favored Nodes to the ancillary section.
HBASE-15531: Favored Nodes EnhancementsStatus: Active development.Owner: Thiruvel Thanks!-Thiruvel 

    On Monday, January 16, 2017 2:10 PM, Stack <st...@duboce.net> wrote:
 

 On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <zg...@gmail.com> wrote:

> For 6. Significant contirbs in master only, there are some issues about
> replication operations routed through master. They are sub-task
> of HBASE-10504. And there are other umbrella issue for replication, like
> HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
> tracking from Zookeeper to HBase. So i thought we can add a new section
> named
> hbase-replication to possible 2.0.0s. This will help us to track the state.
> Thanks.
>

Thanks Guanghao Zhang. I agree. I made you an editor. If you want to have a
go at a first cut, be my guest. If nothing done in the next day or so, I'll
add this section Sir.
Thanks,
M


   

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang <zg...@gmail.com> wrote:

> For 6. Significant contirbs in master only, there are some issues about
> replication operations routed through master. They are sub-task
> of HBASE-10504. And there are other umbrella issue for replication, like
> HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
> tracking from Zookeeper to HBase. So i thought we can add a new section
> named
> hbase-replication to possible 2.0.0s. This will help us to track the state.
> Thanks.
>

Thanks Guanghao Zhang. I agree. I made you an editor. If you want to have a
go at a first cut, be my guest. If nothing done in the next day or so, I'll
add this section Sir.
Thanks,
M

Re: Moving 2.0 forward

[Guanghao Zhang <zg...@gmail.com>]

For 6. Significant contirbs in master only, there are some issues about
replication operations routed through master. They are sub-task
of HBASE-10504. And there are other umbrella issue for replication, like
HBase-14379 Replication V2 and HBASE-15867 Moving HBase Replication
tracking from Zookeeper to HBase. So i thought we can add a new section named
hbase-replication to possible 2.0.0s. This will help us to track the state.
Thanks.

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

Hey Eric!

See this thread where it is suggested we remove the module for want of a
sponsor [1]. The hbase-spark story is also muddled by there being different
options available neither of which is complete instead of just 'the one'
(Do a google search).

St.Ack
1.
http://apache-hbase.679495.n3.nabble.com/DISCUSS-hbase-spark-module-in-branch-1-and-branch-2-td4084343.html



On Sat, Jan 14, 2017 at 12:30 PM, Eric Charles <er...@apache.org> wrote:

> I read "3.3 hbase-spark STATUS: Needs work. No one on it at mo. Doc. is
> just wrong. What is there is dodgy. Could get punted."
>
> Unit tests are working and base functionality is there. Besides the doc
> and compilation against spark-2 (and scala-2.11), what else do you want to
> see?
>
>
>
> On 14/01/17 10:29, Ted Yu wrote:
>
>> For 3.3, hbase-spark module, there is HBASE-16179 which enables support
>> for Spark 2.0
>> It needs some review.
>>
>> Cheers
>>
>> On Jan 13, 2017, at 11:25 PM, Stack <st...@duboce.net> wrote:
>>>
>>> On Sat, Dec 31, 2016 at 12:16 PM, Stephen Jiang <syuanjiangdev@gmail.com
>>> >
>>> wrote:
>>>
>>> Hello, Andrew, I was a helper on Matteo so that we can help each other
>>>> while we are focusing on the new Assignment Manager work.  Now he is not
>>>> available (at least in the next few months).  I have to be more focused
>>>> on
>>>> the new AM work; plus other work in my company; it would be too much
>>>> for me
>>>> to 2.0 RM alone.  I am happy someone would help to take primary 2.0 RM
>>>> role
>>>> while I am still help to make this 2.0 release smooth.
>>>>
>>> (I could help out Stephen. We could co-RM?)
>>>
>>>
>>> For branch-2, I think it is too early to cut it, as we still have a lot
>>>> of
>>>> moving parts and on-going project that needs to be part of 2.0.  For
>>>> example, the mentioned new AM (and other projects, such as HBASE-14414,
>>>> HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531, just
>>>> name
>>>> a few).  Cutting branch now would add burden to complete those projects.
>>>>
>>> Agree with Stephen. A bunch of stuff is half-baked so a '2.0.0' now would
>>> be all loose ends and it'd make for a messy narrative.
>>>
>>> I started a doc listing state of 2.0.0:
>>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9i
>>> Eu_ktczrlKHK8N4SZzs/edit?usp=sharing
>>>
>>> In the doc I made an estimate of what the community considers core 2.0.0
>>> items based in part off old lists and after survey of current state of
>>> JIRA. The doc is open for comment. Please chime in if I am off or if I am
>>> missing something that should be included. I also make a rough estimate
>>> on
>>> state of each core item.
>>>
>>> I intend to keep up this macro-view doc as we progress on 2.0.0 with
>>> reflection where pertinent in JIRA . Suggest we branch only when code
>>> compete on the core set most of which are complete or near-so.
>>> End-of-February should be time enough (First 2.0.0 RC in at the start of
>>> May?).
>>>
>>> Thanks,
>>> St.Ack
>>>
>>>
>>>
>>> thanks
>>>> Stephen
>>>>
>>>> On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <
>>>> andrew.purtell@gmail.com
>>>> wrote:
>>>>
>>>> Hi all,
>>>>>
>>>>> I've heard a rumor the co-RM situation with 2.0 may have changed. Can
>>>>> we
>>>>> get an update from co-RMs Matteo and Steven on their availability and
>>>>> interest in continuing in this role?
>>>>>
>>>>> To assist in moving 2.0 forward I intend to branch branch-2 from master
>>>>> next week. Unless there is an objection I will take this action under
>>>>> assumption of lazy consensus. Master branch will be renumbered to
>>>>> 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin scale
>>>>> tests and stabilization (via bug fixes or reverts of unfinished work)
>>>>> and
>>>>> invite interested collaborators to do the same.
>>>>>
>>>>
>>>>

Re: Moving 2.0 forward

[Eric Charles <er...@apache.org>]

I read "3.3 hbase-spark STATUS: Needs work. No one on it at mo. Doc. is 
just wrong. What is there is dodgy. Could get punted."

Unit tests are working and base functionality is there. Besides the doc 
and compilation against spark-2 (and scala-2.11), what else do you want 
to see?


On 14/01/17 10:29, Ted Yu wrote:
> For 3.3, hbase-spark module, there is HBASE-16179 which enables support for Spark 2.0
> It needs some review.
>
> Cheers
>
>> On Jan 13, 2017, at 11:25 PM, Stack <st...@duboce.net> wrote:
>>
>> On Sat, Dec 31, 2016 at 12:16 PM, Stephen Jiang <sy...@gmail.com>
>> wrote:
>>
>>> Hello, Andrew, I was a helper on Matteo so that we can help each other
>>> while we are focusing on the new Assignment Manager work.  Now he is not
>>> available (at least in the next few months).  I have to be more focused on
>>> the new AM work; plus other work in my company; it would be too much for me
>>> to 2.0 RM alone.  I am happy someone would help to take primary 2.0 RM role
>>> while I am still help to make this 2.0 release smooth.
>> (I could help out Stephen. We could co-RM?)
>>
>>
>>> For branch-2, I think it is too early to cut it, as we still have a lot of
>>> moving parts and on-going project that needs to be part of 2.0.  For
>>> example, the mentioned new AM (and other projects, such as HBASE-14414,
>>> HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531, just name
>>> a few).  Cutting branch now would add burden to complete those projects.
>> Agree with Stephen. A bunch of stuff is half-baked so a '2.0.0' now would
>> be all loose ends and it'd make for a messy narrative.
>>
>> I started a doc listing state of 2.0.0:
>> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit?usp=sharing
>>
>> In the doc I made an estimate of what the community considers core 2.0.0
>> items based in part off old lists and after survey of current state of
>> JIRA. The doc is open for comment. Please chime in if I am off or if I am
>> missing something that should be included. I also make a rough estimate on
>> state of each core item.
>>
>> I intend to keep up this macro-view doc as we progress on 2.0.0 with
>> reflection where pertinent in JIRA . Suggest we branch only when code
>> compete on the core set most of which are complete or near-so.
>> End-of-February should be time enough (First 2.0.0 RC in at the start of
>> May?).
>>
>> Thanks,
>> St.Ack
>>
>>
>>
>>> thanks
>>> Stephen
>>>
>>> On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <andrew.purtell@gmail.com
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I've heard a rumor the co-RM situation with 2.0 may have changed. Can we
>>>> get an update from co-RMs Matteo and Steven on their availability and
>>>> interest in continuing in this role?
>>>>
>>>> To assist in moving 2.0 forward I intend to branch branch-2 from master
>>>> next week. Unless there is an objection I will take this action under
>>>> assumption of lazy consensus. Master branch will be renumbered to
>>>> 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin scale
>>>> tests and stabilization (via bug fixes or reverts of unfinished work) and
>>>> invite interested collaborators to do the same.
>>>

Re: Moving 2.0 forward

[Ted Yu <yu...@gmail.com>]

For 3.3, hbase-spark module, there is HBASE-16179 which enables support for Spark 2.0  
It needs some review. 

Cheers

> On Jan 13, 2017, at 11:25 PM, Stack <st...@duboce.net> wrote:
> 
> On Sat, Dec 31, 2016 at 12:16 PM, Stephen Jiang <sy...@gmail.com>
> wrote:
> 
>> Hello, Andrew, I was a helper on Matteo so that we can help each other
>> while we are focusing on the new Assignment Manager work.  Now he is not
>> available (at least in the next few months).  I have to be more focused on
>> the new AM work; plus other work in my company; it would be too much for me
>> to 2.0 RM alone.  I am happy someone would help to take primary 2.0 RM role
>> while I am still help to make this 2.0 release smooth.
> (I could help out Stephen. We could co-RM?)
> 
> 
>> For branch-2, I think it is too early to cut it, as we still have a lot of
>> moving parts and on-going project that needs to be part of 2.0.  For
>> example, the mentioned new AM (and other projects, such as HBASE-14414,
>> HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531, just name
>> a few).  Cutting branch now would add burden to complete those projects.
> Agree with Stephen. A bunch of stuff is half-baked so a '2.0.0' now would
> be all loose ends and it'd make for a messy narrative.
> 
> I started a doc listing state of 2.0.0:
> https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit?usp=sharing
> 
> In the doc I made an estimate of what the community considers core 2.0.0
> items based in part off old lists and after survey of current state of
> JIRA. The doc is open for comment. Please chime in if I am off or if I am
> missing something that should be included. I also make a rough estimate on
> state of each core item.
> 
> I intend to keep up this macro-view doc as we progress on 2.0.0 with
> reflection where pertinent in JIRA . Suggest we branch only when code
> compete on the core set most of which are complete or near-so.
> End-of-February should be time enough (First 2.0.0 RC in at the start of
> May?).
> 
> Thanks,
> St.Ack
> 
> 
> 
>> thanks
>> Stephen
>> 
>> On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <andrew.purtell@gmail.com
>> wrote:
>> 
>>> Hi all,
>>> 
>>> I've heard a rumor the co-RM situation with 2.0 may have changed. Can we
>>> get an update from co-RMs Matteo and Steven on their availability and
>>> interest in continuing in this role?
>>> 
>>> To assist in moving 2.0 forward I intend to branch branch-2 from master
>>> next week. Unless there is an objection I will take this action under
>>> assumption of lazy consensus. Master branch will be renumbered to
>>> 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin scale
>>> tests and stabilization (via bug fixes or reverts of unfinished work) and
>>> invite interested collaborators to do the same.
>> 

Re: Moving 2.0 forward

[Stack <st...@duboce.net>]

On Sat, Dec 31, 2016 at 12:16 PM, Stephen Jiang <sy...@gmail.com>
wrote:

> Hello, Andrew, I was a helper on Matteo so that we can help each other
> while we are focusing on the new Assignment Manager work.  Now he is not
> available (at least in the next few months).  I have to be more focused on
> the new AM work; plus other work in my company; it would be too much for me
> to 2.0 RM alone.  I am happy someone would help to take primary 2.0 RM role
> while I am still help to make this 2.0 release smooth.
>
>
(I could help out Stephen. We could co-RM?)


> For branch-2, I think it is too early to cut it, as we still have a lot of
> moving parts and on-going project that needs to be part of 2.0.  For
> example, the mentioned new AM (and other projects, such as HBASE-14414,
> HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531, just name
> a few).  Cutting branch now would add burden to complete those projects.
>
>
Agree with Stephen. A bunch of stuff is half-baked so a '2.0.0' now would
be all loose ends and it'd make for a messy narrative.

I started a doc listing state of 2.0.0:
https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit?usp=sharing

In the doc I made an estimate of what the community considers core 2.0.0
items based in part off old lists and after survey of current state of
JIRA. The doc is open for comment. Please chime in if I am off or if I am
missing something that should be included. I also make a rough estimate on
state of each core item.

I intend to keep up this macro-view doc as we progress on 2.0.0 with
reflection where pertinent in JIRA . Suggest we branch only when code
compete on the core set most of which are complete or near-so.
End-of-February should be time enough (First 2.0.0 RC in at the start of
May?).

Thanks,
St.Ack



> thanks
> Stephen
>
> On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <andrew.purtell@gmail.com
> >
> wrote:
>
> > Hi all,
> >
> > I've heard a rumor the co-RM situation with 2.0 may have changed. Can we
> > get an update from co-RMs Matteo and Steven on their availability and
> > interest in continuing in this role?
> >
> > To assist in moving 2.0 forward I intend to branch branch-2 from master
> > next week. Unless there is an objection I will take this action under
> > assumption of lazy consensus. Master branch will be renumbered to
> > 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin scale
> > tests and stabilization (via bug fixes or reverts of unfinished work) and
> > invite interested collaborators to do the same.
> >
> >
> >
>

Re: Moving 2.0 forward

[Stephen Jiang <sy...@gmail.com>]

Hello, Andrew, I was a helper on Matteo so that we can help each other
while we are focusing on the new Assignment Manager work.  Now he is not
available (at least in the next few months).  I have to be more focused on
the new AM work; plus other work in my company; it would be too much for me
to 2.0 RM alone.  I am happy someone would help to take primary 2.0 RM role
while I am still help to make this 2.0 release smooth.

For branch-2, I think it is too early to cut it, as we still have a lot of
moving parts and on-going project that needs to be part of 2.0.  For
example, the mentioned new AM (and other projects, such as HBASE-14414,
HBASE-15179, HBASE-14070, HBASE-14850, HBASE-16833, HBASE-15531, just name
a few).  Cutting branch now would add burden to complete those projects.

thanks
Stephen

On Sat, Dec 31, 2016 at 10:54 AM, Andrew Purtell <an...@gmail.com>
wrote:

> Hi all,
>
> I've heard a rumor the co-RM situation with 2.0 may have changed. Can we
> get an update from co-RMs Matteo and Steven on their availability and
> interest in continuing in this role?
>
> To assist in moving 2.0 forward I intend to branch branch-2 from master
> next week. Unless there is an objection I will take this action under
> assumption of lazy consensus. Master branch will be renumbered to
> 3.0.0-SNAPSHOT. Once we have a branch-2 I will immediately begin scale
> tests and stabilization (via bug fixes or reverts of unfinished work) and
> invite interested collaborators to do the same.
>
>
>