Split Meta Design Reset Status

[Stack <st...@duboce.net>]

FYI, a few of us have been working on the redo/reset of the split meta
design (HBASE-25382). We (think we've) finished the requirements. Are there
any others to consider?

Feedback and contribs welcome. Otherwise, on to the next phase -- design.

Thanks,
S

Re: Split Meta Design Reset Status

[Stack <st...@duboce.net>]

A few of us met this afternoon to talk some split-meta design. It was Zach,
Bharath, Francis, and myself. Duo showed up 'on-time' but we all were on
daylight savings so we were an hour early! (My fault).

Most of this kick-off meeting was Francis and I rehearsing where we (Duo,
Francis, and I) have gotten to-date (see [1]), the current thinking and
implementation ideas.

We went over the ConnectionRegistry API, and how it might change and we
pulled up RegionLocator too...

Talked about possibly moving ConnectionRegistry impl to a set of
'bootstrapping' RegionServers...

Suggestion was that we break up the problem into aspects to make the
problem more tractable.

We'll try another in a week or so but meantime lets try and make some
progress on the design doc...

Thanks,
S

1.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2




On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net> wrote:

> Now the requirements are in [1], we're going to move to the next stage --
> actual design for split-meta -- and have set up a chat for this thursday
> afternoon (4PM California time/8AM Beijing time) to get the ball rolling.
> Please come if interested. Zoom details are below.
>
> Yours,
> S
> 1.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
>
>
> Topic: hbase split-meta design warmup chat
> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
>
> Join Zoom Meeting
> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
>
> Meeting ID: 759 8800 3798
> Passcode: hbase
>
>
> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net> wrote:
>
>> FYI, a few of us have been working on the redo/reset of the split meta
>> design (HBASE-25382). We (think we've) finished the requirements. Are there
>> any others to consider?
>>
>> Feedback and contribs welcome. Otherwise, on to the next phase -- design.
>>
>> Thanks,
>> S
>>
>

Re: Split Meta Design Reset Status

["张铎 (Duo Zhang)" <pa...@gmail.com>]

I checked the google doc history, it can show the difference, so I do
not think we need to move it to git. There are still lots of
unresolved comments on the doc, moving to git will lose them.

And I do not think the online meeting breaks the rule in ASF, we will
send out the meeting notes here and modify the design doc accordingly,
everyone can comment on the design doc or just reply here.

Thanks.

Stack <st...@duboce.net> 于2021年8月14日周六 上午12:33写道：
>
> On Fri, Aug 13, 2021 at 8:39 AM Andrew Purtell <an...@gmail.com>
> wrote:
>
> > Also let me say that your entreaty for more to join your calls is fine and
> > I hope that happens, but this is Apache, and discussions should take place
> > in suitably asynchronous fashion so all can participate - even those who
> > have competing responsibilities and commitments and cannot attend the
> > calls. This is not criticism of the calls! I’m so glad to see you lot are
> > continuing to communicate. But we need to allow and support asynchronous
> > communication and comment either by mailing list or on a document. Perhaps
> > a doc that lists notes from each of your session… Or otherwise randos like
> > me will be posting responses to your emailed notes here.
> >
> >
> Commenting on the notes here is great. Thanks. Otherwise, there is the
> (public) design doc which is what the calls are trying to move forward; all
> are free to comment/trash and improve it.
>
> (We're thinking of moving the doc off gdoc to
> dev-support/design-docs/split-meta to make it easier to see changes).
>
> Thanks Andrew,
> S
>
>
>
>
> > > On Aug 13, 2021, at 8:30 AM, Andrew Purtell <an...@gmail.com>
> > wrote:
> > >
> > > Moving state into zookeeper is the wrong direction. Absolutely not. No,
> > no, no. We can discuss it but it is very unlikely I would even not veto
> > such a change. It doesn’t seem like a problem because you have other MUCH
> > better and more viable options.
> > >
> > >> On Aug 13, 2021, at 7:50 AM, Stack <st...@duboce.net> wrote:
> > >>
> > >> Rough minutes from our meeting of  Wed Aug 11 17:04:44 PDT 2021
> > >>
> > >> Attendees: Francis, Duo, Francis.
> > >>
> > >> We moved to discuss the proposed agenda item, how to implement the
> > >> "ROOT" Region.
> > >>
> > >> Duo had a new proposal that we host meta Regions in zookeeper (either
> > >> all in a single znode or a znode per location).
> > >>
> > >> We talked about it. Mainly, it was noted that zookeeper doesn't scale,
> > >> that a metatable in zk would not be able to satisfy
> > >> "2.0.6 Support an hbase:meta Table of 10k Regions" [1], and that it also
> > >> in violation of the old invariant that we keep no permanent state in
> > >> zookeeper [2].
> > >>
> > >> Much back and forth on the fact that we violate the invariant
> > currently; i.e.
> > >> there is info in zk now that can't be scrubbed; that we need to fix
> > this before
> > >> we can go forward -- that the fact that meta location is currently
> > persistent
> > >> means we could make split meta Region locations also zk persistent.
> > Counter
> > >> arguments note that the original invariant script allowed that we had
> > >> violations,
> > >> that we have worked to undo, that addressing invariant violations was
> > off-topic,
> > >> as well as variations on two wrongs do not make a right, etc.
> > >>
> > >> We tried to reset to talk of previously suggested options of ROOT as a
> > >> new table to host hbase:meta Regions or just use first-Region in
> > >> hbase:meta as special case Region. A new ROOT table was generally
> > >> dismissed as an impediment when small installs in no need of split meta.
> > >>
> > >> The second approach was called 'ugly'  -- I think 'ugly' was the term
> > >> used -- and too much if/else and possibly in the way of future
> > evolutions.
> > >> Counter suggestions thought any uglyness could be hidden from client and
> > >> isolated and that this approach seemed to meet requirements.
> > Counter-arguments
> > >> to the counter-argument where this approach can not hide everything
> > from client,
> > >> as opposed to separate ROOT table or master local region... see [1] for
> > detail.
> > >>
> > >> We ran out of time. Will try again next week (meantime on-going
> > >> developments on design doc [1] and in master).
> > >>
> > >> (Would be good if we could get others on the call, other PMCers
> > >> with in an interest in split meta. Could be of help when we hit
> > >> blockage)
> > >> Thanks,
> > >> S
> > >>
> > >> 1.
> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wv90yxilng75
> > >> 2. http://hbase.apache.org/book.html#design.invariants.zk.data
> > >>
> > >>
> > >>>> On Tue, Aug 10, 2021 at 6:11 PM Stack <st...@duboce.net> wrote:
> > >>>
> > >>> Meeting tomorrow afternoon @ 5pm (to accommodate Beijing time). Lets
> > >>> continue our Split Meta Design Reset discussion.
> > >>>
> > >>> Time: Aug 11, 2021 05:00 PM Pacific Time (US and Canada)
> > >>>
> > >>> Join Zoom Meeting
> > >>>
> > https://us04web.zoom.us/j/73900142989?pwd=b2s1UmJ2a2hiUFRsRERYQzZBV0NHdz09
> > >>>
> > >>> Meeting ID: 739 0014 2989
> > >>> Passcode: hbase
> > >>>
> > >>> Please review the design doc "Design/Brainstorming" Section 4.1 [1]
> > before
> > >>> the meeting.
> > >>>
> > >>> Topics for discussion:
> > >>>
> > >>> * Reports on progress since last meeting.
> > >>> * How will we actually implement the ROOT table (as a distinct ROOT
> > table,
> > >>> as the first region of hbase:meta, etc.)
> > >>>
> > >>> 1.
> > >>>
> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> > >>>
> > >>> Thanks,
> > >>> S
> > >>>
> > >>>
> > >>>> On Tue, Aug 3, 2021 at 4:13 PM Stack <st...@duboce.net> wrote:
> > >>>>
> > >>>> Any time Yu Li!
> > >>>>
> > >>>> No meeting tomorrow... Lets meet next week, the 10th.
> > >>>>
> > >>>> Thanks,
> > >>>> S
> > >>>>
> > >>>>> On Wed, Jul 28, 2021 at 11:25 PM Yu Li <ca...@gmail.com> wrote:
> > >>>>
> > >>>>> Thanks for the notes and efforts Stack, it's pretty helpful to know
> > the
> > >>>>> progress and latest status of the work!
> > >>>>>
> > >>>>> Best Regards,
> > >>>>> Yu
> > >>>>>
> > >>>>>
> > >>>>> On Wed, 28 Jul 2021 at 12:13, Stack <st...@duboce.net> wrote:
> > >>>>>
> > >>>>>> Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of
> > >>>>>> August.
> > >>>>>> Thanks,
> > >>>>>> S
> > >>>>>>
> > >>>>>> On Thu, Jul 22, 2021 at 8:53 PM Stack <st...@duboce.net> wrote:
> > >>>>>>
> > >>>>>>> Notes from yesterday's meeting (attendees, please amend if I
> > >>>>> misrepresent
> > >>>>>>> or if you have anything extra to add!)
> > >>>>>>>
> > >>>>>>> Split Meta Design Reset Status
> > >>>>>>>
> > >>>>>>> Wed Jul 21 21:24:38 PDT 2021
> > >>>>>>>
> > >>>>>>> Attendees: Bharath, Stack, Duo, and Francis
> > >>>>>>>
> > >>>>>>> We went over the new updates to the Brainstorming [1] section under
> > >>>>>>>
> > >>>>>>> Design in the Super Split Meta Design doc [2].
> > >>>>>>>
> > >>>>>>> First was the new addition, 4.1.2 Extend (& Move)
> > ConnectionRegistry;
> > >>>>>> hide
> > >>>>>>>
> > >>>>>>> ROOT from Client [3]. In particular, filling out how "ROOT" might
> > be
> > >>>>>>>
> > >>>>>>> implemented behind the new API in ConnectionRegistry. On option 1.,
> > >>>>>>>
> > >>>>>>> replicating master-local Region to RegionServers, options
> > considered
> > >>>>>>>
> > >>>>>>> included
> > >>>>>>>
> > >>>>>>> * Listener on master-local Region WAL to replicate.
> > >>>>>>>
> > >>>>>>> * Perhaps Read-Replica but master-local is not an actual Region
> > >>>>>>>
> > >>>>>>> * Needs to be incremental edits because ROOT could get too big to
> > >>>>> ship
> > >>>>>>>
> > >>>>>>>  in a lump; need to visit how...
> > >>>>>>>
> > >>>>>>> * Possibly in-memory-only Regions on RS replicated from
> > master-local
> > >>>>>>>
> > >>>>>>>  Region via WAL tailing <= zhangduo@apache.org
> > >>>>>>>
> > >>>>>>> * Which RegionServers? Those hosting ROOT replicas?
> > >>>>>>>
> > >>>>>>> * How to bootstrap? Failure scenarios.
> > >>>>>>>
> > >>>>>>> * This would be a new replication system alongside current; could
> > >>>>>>>
> > >>>>>>>  evolve to replace/improve old?
> > >>>>>>>
> > >>>>>>> Duo offered to look into means of replicating the master-local
> > Region
> > >>>>>>>
> > >>>>>>> out to RegionServers.
> > >>>>>>>
> > >>>>>>> Next up was discussion constrasting ROOT as a standalone table vs
> > >>>>>>>
> > >>>>>>> First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]);
> > i.e.
> > >>>>>>>
> > >>>>>>> options 2 and 3 for how we'd implement a ROOT. One item that came
> > up
> > >>>>>>>
> > >>>>>>> was whether a need to specify one replica count for a ROOT table vs
> > >>>>>>>
> > >>>>>>> another for hbase:meta. If so, then it would be argument for ROOT
> > as
> > >>>>>>>
> > >>>>>>> standalone table (Others of us argued it not a concern of
> > >>>>> consequence).
> > >>>>>>>
> > >>>>>>> If ROOT access is behind a new simple API in ConnectionRegistry,
> > how
> > >>>>>>>
> > >>>>>>> to stop clients reading hbase:meta table if not Master or fronted
> > by
> > >>>>>>>
> > >>>>>>> a ConnectionRegistry request? (Should be able to switch on client
> > >>>>>>>
> > >>>>>>> identity/source). One suggestion for First-meta-Region-as-ROOT was
> > >>>>>>>
> > >>>>>>> NOT returning the first Region to the client post-meta split when
> > >>>>>>>
> > >>>>>>> accessing via the simple API. Some concern this would confuse old
> > >>>>>>>
> > >>>>>>> Clients (Francis was going to take a look).
> > >>>>>>>
> > >>>>>>> Moved to discussion how we'd move ConnectionRegistry from
> > >>>>>>>
> > >>>>>>> hosted-by-Master to hosted-by-RegionServers. How to bootstrap such
> > a
> > >>>>>>>
> > >>>>>>> system came up? Where do clients go? How do they know which
> > >>>>>>>
> > >>>>>>> RegionServers (special regionserver group?  Every RS fields
> > >>>>>>>
> > >>>>>>> ConnectionRegistry requests but only designated core serve the ROOT
> > >>>>>>>
> > >>>>>>> lookup APIs?). This was a TODO.
> > >>>>>>>
> > >>>>>>> This led naturally into 4.1.5 System RS group for client meta
> > >>>>> services
> > >>>>>>>
> > >>>>>>> [5], a new addition under Brainstorming. Discussion. Bharath to
> > look
> > >>>>>>>
> > >>>>>>> into feedback.
> > >>>>>>>
> > >>>>>>> On the end of the discussion, group expressed support for adding
> > >>>>>>>
> > >>>>>>> simple API to the ConnectionRegistry to hide ROOT implementation
> > >>>>>>>
> > >>>>>>> detail from client. Support was expressed for moving
> > >>>>> ConnectionRegistry
> > >>>>>>>
> > >>>>>>> from Master to RegionServers. Intent is to move forward on design
> > of
> > >>>>>>>
> > >>>>>>> these pieces: e.g. how client bootstraps.
> > >>>>>>>
> > >>>>>>> Support was expressed for getting at least the bones of a split
> > meta
> > >>>>>>>
> > >>>>>>> into an hbase3 before the RCs.
> > >>>>>>>
> > >>>>>>> Where we'd actually store hbase:meta Region locations -- i.e. how a
> > >>>>>>>
> > >>>>>>> "ROOT' would be implemented -- was for our next meeting informed by
> > >>>>>>>
> > >>>>>>> research of the various approaches noted mostly above. It was
> > >>>>>>>
> > >>>>>>> also thought that the new ConnectionRegistry should not preclude
> > >>>>>>>
> > >>>>>>> making progress on the "ROOT" implementation.
> > >>>>>>>
> > >>>>>>> Will post notice of next meeting (next Weds or the one
> > >>>>>>>
> > >>>>>>> following).
> > >>>>>>>
> > >>>>>>> 1.
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
> > >>>>>>>
> > >>>>>>> 2.
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq
> > >>>>>>>
> > >>>>>>> 3.
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153
> > >>>>>>>
> > >>>>>>> 4.
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> > >>>>>>>
> > >>>>>>> 5.
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b
> > >>>>>>>
> > >>>>>>> On Tue, Jul 20, 2021 at 11:00 AM Stack <st...@duboce.net> wrote:
> > >>>>>>>
> > >>>>>>>> Lets meet tomorrow. Please review the design doc
> > >>>>> "Design/Brainstorming"
> > >>>>>>>> Section 4.1 [1] before the meeting if you can (No harm if a
> > refresh
> > >>>>> of
> > >>>>>> the
> > >>>>>>>> requirements section while you are at it).
> > >>>>>>>>
> > >>>>>>>> Topic: Split Meta Design Reset Status
> > >>>>>>>> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada)
> > >>>>>>>>
> > >>>>>>>> Join Zoom Meeting
> > >>>>>>>>
> > >>>>>>
> > >>>>>
> > https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09
> > >>>>>>>>
> > >>>>>>>> Meeting ID: 773 1892 0525
> > >>>>>>>> Passcode: hbase
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> S
> > >>>>>>>>
> > >>>>>>>> 1. 1.
> > >>>>>>>>
> > >>>>>>
> > >>>>>
> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
> > >>>>>>>>
> > >>>>>>>> On Thu, Jul 8, 2021 at 1:04 PM Stack <st...@duboce.net> wrote:
> > >>>>>>>>
> > >>>>>>>>> Meeting notes (Meeting happend after I figured the zoom had a
> > >>>>> 'waiting
> > >>>>>>>>> room' -- sorry Duo)
> > >>>>>>>>>
> > >>>>>>>>> Split Meta Status Zoom Meeting
> > >>>>>>>>> Wed Jul  7, 2021 @ 5pm for ~90minutes
> > >>>>>>>>> Attendees: Duo, Francis, Stack, and Clay
> > >>>>>>>>> Agenda: Mainly talk about the one-pager design and PoC proposed
> > in
> > >>>>> [2]
> > >>>>>>>>> and added to the
> > >>>>>>>>> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as
> > >>>>> ROOT'
> > >>>>>>>>> approach)
> > >>>>>>>>>
> > >>>>>>>>> Duo suggested no advantage treating the first meta of hbase:meta
> > >>>>> table
> > >>>>>>>>> special; as a "root"
> > >>>>>>>>> and other comments (see remarks in [2]).
> > >>>>>>>>>
> > >>>>>>>>> Some pushback. When meta is NOT split, all works as it did before
> > >>>>> (big
> > >>>>>>>>> on backward-compatible).
> > >>>>>>>>>
> > >>>>>>>>> Duo suggested just intro a ROOT table altogether rather than
> > treat
> > >>>>> the
> > >>>>>>>>> first Region
> > >>>>>>>>> in the hbase:meta table as a 'root' and then mirror to zk the
> > first
> > >>>>>> meta
> > >>>>>>>>> region; if no
> > >>>>>>>>> split, then old clients should just work even though now hbase
> > >>>>> cluster
> > >>>>>>>>> has a ROOT table.
> > >>>>>>>>> Discussion. If no split, what's to do, etc.?
> > >>>>>>>>>
> > >>>>>>>>> Duo expressed concern that if split-meta is not on always --
> > >>>>> enabled
> > >>>>>>>>> optionally -- then
> > >>>>>>>>> the code path will not see exercise and split-meta will likely
> > >>>>> fail and
> > >>>>>>>>> go the way of
> > >>>>>>>>> prefix tree and distributed log replay -- a feature that failed,
> > >>>>>>>>> cluttered up the
> > >>>>>>>>> code base, and only later was removed. Discussion. Was allowed
> > this
> > >>>>>>>>> could happen.
> > >>>>>>>>> Counter examples: RegionServer Groups. A few of the attendees
> > >>>>>>>>> volunteered they need
> > >>>>>>>>> split meta for their production so would try to see it through.
> > >>>>>>>>>
> > >>>>>>>>> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1
> > >>>>> as
> > >>>>>>>>> ROOT' if we don't
> > >>>>>>>>> split; not much changes.
> > >>>>>>>>>
> > >>>>>>>>> Comment on the PoC that its all
> > >>>>>>>>>
> > >>>>>>>>> if 'first special meta region' do this
> > >>>>>>>>> else do something else...
> > >>>>>>>>>
> > >>>>>>>>> (all over the codebase) but it was suggested that this will be
> > the
> > >>>>> case
> > >>>>>>>>> if ROOT table
> > >>>>>>>>> added also and argued any implementation will have this issue (if
> > >>>>> ROOT
> > >>>>>>>>> then....) and
> > >>>>>>>>> THAT ugly 'root' comparator too whether a ROOT table or the
> > >>>>>>>>> 'hbase:meta,,1 as ROOT'
> > >>>>>>>>> approach.
> > >>>>>>>>>
> > >>>>>>>>> Clay asking if meta is a bottleneck (Clay played the 'operator'
> > and
> > >>>>>>>>> 'new-to-the-topic'
> > >>>>>>>>> roles). Some discussion around how indeed it is and why we want
> > to
> > >>>>>> split
> > >>>>>>>>> meta at all.
> > >>>>>>>>>
> > >>>>>>>>> Clay mentioned how Master is inline for data now (Master-hosted
> > >>>>>>>>> Registry)....
> > >>>>>>>>> Discussion. Hopefully temporary state -- Registry doesn't need to
> > >>>>> be
> > >>>>>>>>> hosted on
> > >>>>>>>>> Master -- and Master will return to its background role -- soon.
> > >>>>>>>>>
> > >>>>>>>>> Clay brought up rollback after meta split. Discussion. Some
> > >>>>> agreement
> > >>>>>> it
> > >>>>>>>>> could be done for
> > >>>>>>>>> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root
> > >>>>> table
> > >>>>>> and
> > >>>>>>>>> client had
> > >>>>>>>>> started to use ROOT, it might be harder...
> > >>>>>>>>>
> > >>>>>>>>> Duo suggested that we not change the client at all; that client
> > >>>>> stays
> > >>>>>>>>> same however split
> > >>>>>>>>> meta is implemented (with addition of a root table or using
> > >>>>>>>>> hbase:meta,,1 as 'root').
> > >>>>>>>>> This sounded attractive. We discussed how this could be done in a
> > >>>>>>>>> backward compatible way;
> > >>>>>>>>> add simple location lookup API to Registry...A write-up on how
> > this
> > >>>>>>>>> might work will be posted
> > >>>>>>>>> in next day or so for others to review (Need to figure getting
> > >>>>> Registry
> > >>>>>>>>> off Master).
> > >>>>>>>>>
> > >>>>>>>>> Clay suggested, as an operator, how he thought the split meta
> > >>>>> should
> > >>>>>>>>> roll out. One of us
> > >>>>>>>>> claimed this described the 'hbase:meta,,1 as ROOT' approach.
> > >>>>>>>>>
> > >>>>>>>>> Duo had to go to work so we called it quits and said we'd meet
> > >>>>> again
> > >>>>>>>>> same time, next week.
> > >>>>>>>>>
> > >>>>>>>>> 1.
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> > >>>>>>>>> 2. https://issues.apache.org/jira/browse/HBASE-25761
> > >>>>>>>>>
> > >>>>>>>>> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <
> > >>>>> palomino219@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Is it only me? I tried to join the meeting but it always tell me
> > >>>>> that
> > >>>>>> I
> > >>>>>>>>>> need to wait for the presenter to invite me to join...
> > >>>>>>>>>>
> > >>>>>>>>>> Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：
> > >>>>>>>>>>
> > >>>>>>>>>>> Short notice but reminder that this meeting is today at 5pm PST
> > >>>>> (We
> > >>>>>>>>>> talked
> > >>>>>>>>>>> of doing it earlier but in the end lets just keep the original
> > >>>>> 5pm
> > >>>>>>>>>> time).
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>> S
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net>
> > wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Lets do a quick chat on current state of hbase split-meta
> > >>>>> project.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Francis has posted a suggested one-pager design in HBASE-25761
> > >>>>>> which
> > >>>>>>>>>>> would
> > >>>>>>>>>>>> be good to review before the meeting if you can. Lets discuss
> > >>>>> this
> > >>>>>>>>>> and
> > >>>>>>>>>>> any
> > >>>>>>>>>>>> other suggestions for moving the project forward.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Meeting details below.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>> S
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Topic Split Meta Design Reset Status
> > >>>>>>>>>>>> Description Review one-pager design attached to
> > >>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-25761
> > >>>>>>>>>>>> Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Meeting ID 736 3907 8852
> > >>>>>>>>>>>> Security
> > >>>>>>>>>>>> Passcode *hbase* Hide
> > >>>>>>>>>>>> Waiting Room
> > >>>>>>>>>>>> Invite Link
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>
> > >>>>>
> > https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
> > >>>>>>>>>>> Copy
> > >>>>>>>>>>>> Invitation
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net>
> > >>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Now the requirements are in [1], we're going to move to the
> > >>>>> next
> > >>>>>>>>>> stage
> > >>>>>>>>>>> --
> > >>>>>>>>>>>>> actual design for split-meta -- and have set up a chat for
> > >>>>> this
> > >>>>>>>>>> thursday
> > >>>>>>>>>>>>> afternoon (4PM California time/8AM Beijing time) to get the
> > >>>>> ball
> > >>>>>>>>>>> rolling.
> > >>>>>>>>>>>>> Please come if interested. Zoom details are below.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Yours,
> > >>>>>>>>>>>>> S
> > >>>>>>>>>>>>> 1.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>
> > >>>>>
> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Topic: hbase split-meta design warmup chat
> > >>>>>>>>>>>>> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Join Zoom Meeting
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>
> > >>>>>
> > https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Meeting ID: 759 8800 3798
> > >>>>>>>>>>>>> Passcode: hbase
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net>
> > >>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> FYI, a few of us have been working on the redo/reset of the
> > >>>>>> split
> > >>>>>>>>>> meta
> > >>>>>>>>>>>>>> design (HBASE-25382). We (think we've) finished the
> > >>>>>> requirements.
> > >>>>>>>>>> Are
> > >>>>>>>>>>> there
> > >>>>>>>>>>>>>> any others to consider?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Feedback and contribs welcome. Otherwise, on to the next
> > >>>>> phase
> > >>>>>> --
> > >>>>>>>>>>> design.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>> S
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> >

Re: Split Meta Design Reset Status

[Stack <st...@duboce.net>]

On Fri, Aug 13, 2021 at 8:39 AM Andrew Purtell <an...@gmail.com>
wrote:

> Also let me say that your entreaty for more to join your calls is fine and
> I hope that happens, but this is Apache, and discussions should take place
> in suitably asynchronous fashion so all can participate - even those who
> have competing responsibilities and commitments and cannot attend the
> calls. This is not criticism of the calls! I’m so glad to see you lot are
> continuing to communicate. But we need to allow and support asynchronous
> communication and comment either by mailing list or on a document. Perhaps
> a doc that lists notes from each of your session… Or otherwise randos like
> me will be posting responses to your emailed notes here.
>
>
Commenting on the notes here is great. Thanks. Otherwise, there is the
(public) design doc which is what the calls are trying to move forward; all
are free to comment/trash and improve it.

(We're thinking of moving the doc off gdoc to
dev-support/design-docs/split-meta to make it easier to see changes).

Thanks Andrew,
S




> > On Aug 13, 2021, at 8:30 AM, Andrew Purtell <an...@gmail.com>
> wrote:
> >
> > Moving state into zookeeper is the wrong direction. Absolutely not. No,
> no, no. We can discuss it but it is very unlikely I would even not veto
> such a change. It doesn’t seem like a problem because you have other MUCH
> better and more viable options.
> >
> >> On Aug 13, 2021, at 7:50 AM, Stack <st...@duboce.net> wrote:
> >>
> >> Rough minutes from our meeting of  Wed Aug 11 17:04:44 PDT 2021
> >>
> >> Attendees: Francis, Duo, Francis.
> >>
> >> We moved to discuss the proposed agenda item, how to implement the
> >> "ROOT" Region.
> >>
> >> Duo had a new proposal that we host meta Regions in zookeeper (either
> >> all in a single znode or a znode per location).
> >>
> >> We talked about it. Mainly, it was noted that zookeeper doesn't scale,
> >> that a metatable in zk would not be able to satisfy
> >> "2.0.6 Support an hbase:meta Table of 10k Regions" [1], and that it also
> >> in violation of the old invariant that we keep no permanent state in
> >> zookeeper [2].
> >>
> >> Much back and forth on the fact that we violate the invariant
> currently; i.e.
> >> there is info in zk now that can't be scrubbed; that we need to fix
> this before
> >> we can go forward -- that the fact that meta location is currently
> persistent
> >> means we could make split meta Region locations also zk persistent.
> Counter
> >> arguments note that the original invariant script allowed that we had
> >> violations,
> >> that we have worked to undo, that addressing invariant violations was
> off-topic,
> >> as well as variations on two wrongs do not make a right, etc.
> >>
> >> We tried to reset to talk of previously suggested options of ROOT as a
> >> new table to host hbase:meta Regions or just use first-Region in
> >> hbase:meta as special case Region. A new ROOT table was generally
> >> dismissed as an impediment when small installs in no need of split meta.
> >>
> >> The second approach was called 'ugly'  -- I think 'ugly' was the term
> >> used -- and too much if/else and possibly in the way of future
> evolutions.
> >> Counter suggestions thought any uglyness could be hidden from client and
> >> isolated and that this approach seemed to meet requirements.
> Counter-arguments
> >> to the counter-argument where this approach can not hide everything
> from client,
> >> as opposed to separate ROOT table or master local region... see [1] for
> detail.
> >>
> >> We ran out of time. Will try again next week (meantime on-going
> >> developments on design doc [1] and in master).
> >>
> >> (Would be good if we could get others on the call, other PMCers
> >> with in an interest in split meta. Could be of help when we hit
> >> blockage)
> >> Thanks,
> >> S
> >>
> >> 1.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wv90yxilng75
> >> 2. http://hbase.apache.org/book.html#design.invariants.zk.data
> >>
> >>
> >>>> On Tue, Aug 10, 2021 at 6:11 PM Stack <st...@duboce.net> wrote:
> >>>
> >>> Meeting tomorrow afternoon @ 5pm (to accommodate Beijing time). Lets
> >>> continue our Split Meta Design Reset discussion.
> >>>
> >>> Time: Aug 11, 2021 05:00 PM Pacific Time (US and Canada)
> >>>
> >>> Join Zoom Meeting
> >>>
> https://us04web.zoom.us/j/73900142989?pwd=b2s1UmJ2a2hiUFRsRERYQzZBV0NHdz09
> >>>
> >>> Meeting ID: 739 0014 2989
> >>> Passcode: hbase
> >>>
> >>> Please review the design doc "Design/Brainstorming" Section 4.1 [1]
> before
> >>> the meeting.
> >>>
> >>> Topics for discussion:
> >>>
> >>> * Reports on progress since last meeting.
> >>> * How will we actually implement the ROOT table (as a distinct ROOT
> table,
> >>> as the first region of hbase:meta, etc.)
> >>>
> >>> 1.
> >>>
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> >>>
> >>> Thanks,
> >>> S
> >>>
> >>>
> >>>> On Tue, Aug 3, 2021 at 4:13 PM Stack <st...@duboce.net> wrote:
> >>>>
> >>>> Any time Yu Li!
> >>>>
> >>>> No meeting tomorrow... Lets meet next week, the 10th.
> >>>>
> >>>> Thanks,
> >>>> S
> >>>>
> >>>>> On Wed, Jul 28, 2021 at 11:25 PM Yu Li <ca...@gmail.com> wrote:
> >>>>
> >>>>> Thanks for the notes and efforts Stack, it's pretty helpful to know
> the
> >>>>> progress and latest status of the work!
> >>>>>
> >>>>> Best Regards,
> >>>>> Yu
> >>>>>
> >>>>>
> >>>>> On Wed, 28 Jul 2021 at 12:13, Stack <st...@duboce.net> wrote:
> >>>>>
> >>>>>> Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of
> >>>>>> August.
> >>>>>> Thanks,
> >>>>>> S
> >>>>>>
> >>>>>> On Thu, Jul 22, 2021 at 8:53 PM Stack <st...@duboce.net> wrote:
> >>>>>>
> >>>>>>> Notes from yesterday's meeting (attendees, please amend if I
> >>>>> misrepresent
> >>>>>>> or if you have anything extra to add!)
> >>>>>>>
> >>>>>>> Split Meta Design Reset Status
> >>>>>>>
> >>>>>>> Wed Jul 21 21:24:38 PDT 2021
> >>>>>>>
> >>>>>>> Attendees: Bharath, Stack, Duo, and Francis
> >>>>>>>
> >>>>>>> We went over the new updates to the Brainstorming [1] section under
> >>>>>>>
> >>>>>>> Design in the Super Split Meta Design doc [2].
> >>>>>>>
> >>>>>>> First was the new addition, 4.1.2 Extend (& Move)
> ConnectionRegistry;
> >>>>>> hide
> >>>>>>>
> >>>>>>> ROOT from Client [3]. In particular, filling out how "ROOT" might
> be
> >>>>>>>
> >>>>>>> implemented behind the new API in ConnectionRegistry. On option 1.,
> >>>>>>>
> >>>>>>> replicating master-local Region to RegionServers, options
> considered
> >>>>>>>
> >>>>>>> included
> >>>>>>>
> >>>>>>> * Listener on master-local Region WAL to replicate.
> >>>>>>>
> >>>>>>> * Perhaps Read-Replica but master-local is not an actual Region
> >>>>>>>
> >>>>>>> * Needs to be incremental edits because ROOT could get too big to
> >>>>> ship
> >>>>>>>
> >>>>>>>  in a lump; need to visit how...
> >>>>>>>
> >>>>>>> * Possibly in-memory-only Regions on RS replicated from
> master-local
> >>>>>>>
> >>>>>>>  Region via WAL tailing <= zhangduo@apache.org
> >>>>>>>
> >>>>>>> * Which RegionServers? Those hosting ROOT replicas?
> >>>>>>>
> >>>>>>> * How to bootstrap? Failure scenarios.
> >>>>>>>
> >>>>>>> * This would be a new replication system alongside current; could
> >>>>>>>
> >>>>>>>  evolve to replace/improve old?
> >>>>>>>
> >>>>>>> Duo offered to look into means of replicating the master-local
> Region
> >>>>>>>
> >>>>>>> out to RegionServers.
> >>>>>>>
> >>>>>>> Next up was discussion constrasting ROOT as a standalone table vs
> >>>>>>>
> >>>>>>> First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]);
> i.e.
> >>>>>>>
> >>>>>>> options 2 and 3 for how we'd implement a ROOT. One item that came
> up
> >>>>>>>
> >>>>>>> was whether a need to specify one replica count for a ROOT table vs
> >>>>>>>
> >>>>>>> another for hbase:meta. If so, then it would be argument for ROOT
> as
> >>>>>>>
> >>>>>>> standalone table (Others of us argued it not a concern of
> >>>>> consequence).
> >>>>>>>
> >>>>>>> If ROOT access is behind a new simple API in ConnectionRegistry,
> how
> >>>>>>>
> >>>>>>> to stop clients reading hbase:meta table if not Master or fronted
> by
> >>>>>>>
> >>>>>>> a ConnectionRegistry request? (Should be able to switch on client
> >>>>>>>
> >>>>>>> identity/source). One suggestion for First-meta-Region-as-ROOT was
> >>>>>>>
> >>>>>>> NOT returning the first Region to the client post-meta split when
> >>>>>>>
> >>>>>>> accessing via the simple API. Some concern this would confuse old
> >>>>>>>
> >>>>>>> Clients (Francis was going to take a look).
> >>>>>>>
> >>>>>>> Moved to discussion how we'd move ConnectionRegistry from
> >>>>>>>
> >>>>>>> hosted-by-Master to hosted-by-RegionServers. How to bootstrap such
> a
> >>>>>>>
> >>>>>>> system came up? Where do clients go? How do they know which
> >>>>>>>
> >>>>>>> RegionServers (special regionserver group?  Every RS fields
> >>>>>>>
> >>>>>>> ConnectionRegistry requests but only designated core serve the ROOT
> >>>>>>>
> >>>>>>> lookup APIs?). This was a TODO.
> >>>>>>>
> >>>>>>> This led naturally into 4.1.5 System RS group for client meta
> >>>>> services
> >>>>>>>
> >>>>>>> [5], a new addition under Brainstorming. Discussion. Bharath to
> look
> >>>>>>>
> >>>>>>> into feedback.
> >>>>>>>
> >>>>>>> On the end of the discussion, group expressed support for adding
> >>>>>>>
> >>>>>>> simple API to the ConnectionRegistry to hide ROOT implementation
> >>>>>>>
> >>>>>>> detail from client. Support was expressed for moving
> >>>>> ConnectionRegistry
> >>>>>>>
> >>>>>>> from Master to RegionServers. Intent is to move forward on design
> of
> >>>>>>>
> >>>>>>> these pieces: e.g. how client bootstraps.
> >>>>>>>
> >>>>>>> Support was expressed for getting at least the bones of a split
> meta
> >>>>>>>
> >>>>>>> into an hbase3 before the RCs.
> >>>>>>>
> >>>>>>> Where we'd actually store hbase:meta Region locations -- i.e. how a
> >>>>>>>
> >>>>>>> "ROOT' would be implemented -- was for our next meeting informed by
> >>>>>>>
> >>>>>>> research of the various approaches noted mostly above. It was
> >>>>>>>
> >>>>>>> also thought that the new ConnectionRegistry should not preclude
> >>>>>>>
> >>>>>>> making progress on the "ROOT" implementation.
> >>>>>>>
> >>>>>>> Will post notice of next meeting (next Weds or the one
> >>>>>>>
> >>>>>>> following).
> >>>>>>>
> >>>>>>> 1.
> >>>>>>>
> >>>>>>
> >>>>>
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
> >>>>>>>
> >>>>>>> 2.
> >>>>>>>
> >>>>>>
> >>>>>
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq
> >>>>>>>
> >>>>>>> 3.
> >>>>>>>
> >>>>>>
> >>>>>
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153
> >>>>>>>
> >>>>>>> 4.
> >>>>>>>
> >>>>>>
> >>>>>
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> >>>>>>>
> >>>>>>> 5.
> >>>>>>>
> >>>>>>
> >>>>>
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b
> >>>>>>>
> >>>>>>> On Tue, Jul 20, 2021 at 11:00 AM Stack <st...@duboce.net> wrote:
> >>>>>>>
> >>>>>>>> Lets meet tomorrow. Please review the design doc
> >>>>> "Design/Brainstorming"
> >>>>>>>> Section 4.1 [1] before the meeting if you can (No harm if a
> refresh
> >>>>> of
> >>>>>> the
> >>>>>>>> requirements section while you are at it).
> >>>>>>>>
> >>>>>>>> Topic: Split Meta Design Reset Status
> >>>>>>>> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada)
> >>>>>>>>
> >>>>>>>> Join Zoom Meeting
> >>>>>>>>
> >>>>>>
> >>>>>
> https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09
> >>>>>>>>
> >>>>>>>> Meeting ID: 773 1892 0525
> >>>>>>>> Passcode: hbase
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> S
> >>>>>>>>
> >>>>>>>> 1. 1.
> >>>>>>>>
> >>>>>>
> >>>>>
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
> >>>>>>>>
> >>>>>>>> On Thu, Jul 8, 2021 at 1:04 PM Stack <st...@duboce.net> wrote:
> >>>>>>>>
> >>>>>>>>> Meeting notes (Meeting happend after I figured the zoom had a
> >>>>> 'waiting
> >>>>>>>>> room' -- sorry Duo)
> >>>>>>>>>
> >>>>>>>>> Split Meta Status Zoom Meeting
> >>>>>>>>> Wed Jul  7, 2021 @ 5pm for ~90minutes
> >>>>>>>>> Attendees: Duo, Francis, Stack, and Clay
> >>>>>>>>> Agenda: Mainly talk about the one-pager design and PoC proposed
> in
> >>>>> [2]
> >>>>>>>>> and added to the
> >>>>>>>>> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as
> >>>>> ROOT'
> >>>>>>>>> approach)
> >>>>>>>>>
> >>>>>>>>> Duo suggested no advantage treating the first meta of hbase:meta
> >>>>> table
> >>>>>>>>> special; as a "root"
> >>>>>>>>> and other comments (see remarks in [2]).
> >>>>>>>>>
> >>>>>>>>> Some pushback. When meta is NOT split, all works as it did before
> >>>>> (big
> >>>>>>>>> on backward-compatible).
> >>>>>>>>>
> >>>>>>>>> Duo suggested just intro a ROOT table altogether rather than
> treat
> >>>>> the
> >>>>>>>>> first Region
> >>>>>>>>> in the hbase:meta table as a 'root' and then mirror to zk the
> first
> >>>>>> meta
> >>>>>>>>> region; if no
> >>>>>>>>> split, then old clients should just work even though now hbase
> >>>>> cluster
> >>>>>>>>> has a ROOT table.
> >>>>>>>>> Discussion. If no split, what's to do, etc.?
> >>>>>>>>>
> >>>>>>>>> Duo expressed concern that if split-meta is not on always --
> >>>>> enabled
> >>>>>>>>> optionally -- then
> >>>>>>>>> the code path will not see exercise and split-meta will likely
> >>>>> fail and
> >>>>>>>>> go the way of
> >>>>>>>>> prefix tree and distributed log replay -- a feature that failed,
> >>>>>>>>> cluttered up the
> >>>>>>>>> code base, and only later was removed. Discussion. Was allowed
> this
> >>>>>>>>> could happen.
> >>>>>>>>> Counter examples: RegionServer Groups. A few of the attendees
> >>>>>>>>> volunteered they need
> >>>>>>>>> split meta for their production so would try to see it through.
> >>>>>>>>>
> >>>>>>>>> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1
> >>>>> as
> >>>>>>>>> ROOT' if we don't
> >>>>>>>>> split; not much changes.
> >>>>>>>>>
> >>>>>>>>> Comment on the PoC that its all
> >>>>>>>>>
> >>>>>>>>> if 'first special meta region' do this
> >>>>>>>>> else do something else...
> >>>>>>>>>
> >>>>>>>>> (all over the codebase) but it was suggested that this will be
> the
> >>>>> case
> >>>>>>>>> if ROOT table
> >>>>>>>>> added also and argued any implementation will have this issue (if
> >>>>> ROOT
> >>>>>>>>> then....) and
> >>>>>>>>> THAT ugly 'root' comparator too whether a ROOT table or the
> >>>>>>>>> 'hbase:meta,,1 as ROOT'
> >>>>>>>>> approach.
> >>>>>>>>>
> >>>>>>>>> Clay asking if meta is a bottleneck (Clay played the 'operator'
> and
> >>>>>>>>> 'new-to-the-topic'
> >>>>>>>>> roles). Some discussion around how indeed it is and why we want
> to
> >>>>>> split
> >>>>>>>>> meta at all.
> >>>>>>>>>
> >>>>>>>>> Clay mentioned how Master is inline for data now (Master-hosted
> >>>>>>>>> Registry)....
> >>>>>>>>> Discussion. Hopefully temporary state -- Registry doesn't need to
> >>>>> be
> >>>>>>>>> hosted on
> >>>>>>>>> Master -- and Master will return to its background role -- soon.
> >>>>>>>>>
> >>>>>>>>> Clay brought up rollback after meta split. Discussion. Some
> >>>>> agreement
> >>>>>> it
> >>>>>>>>> could be done for
> >>>>>>>>> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root
> >>>>> table
> >>>>>> and
> >>>>>>>>> client had
> >>>>>>>>> started to use ROOT, it might be harder...
> >>>>>>>>>
> >>>>>>>>> Duo suggested that we not change the client at all; that client
> >>>>> stays
> >>>>>>>>> same however split
> >>>>>>>>> meta is implemented (with addition of a root table or using
> >>>>>>>>> hbase:meta,,1 as 'root').
> >>>>>>>>> This sounded attractive. We discussed how this could be done in a
> >>>>>>>>> backward compatible way;
> >>>>>>>>> add simple location lookup API to Registry...A write-up on how
> this
> >>>>>>>>> might work will be posted
> >>>>>>>>> in next day or so for others to review (Need to figure getting
> >>>>> Registry
> >>>>>>>>> off Master).
> >>>>>>>>>
> >>>>>>>>> Clay suggested, as an operator, how he thought the split meta
> >>>>> should
> >>>>>>>>> roll out. One of us
> >>>>>>>>> claimed this described the 'hbase:meta,,1 as ROOT' approach.
> >>>>>>>>>
> >>>>>>>>> Duo had to go to work so we called it quits and said we'd meet
> >>>>> again
> >>>>>>>>> same time, next week.
> >>>>>>>>>
> >>>>>>>>> 1.
> >>>>>>>>>
> >>>>>>
> >>>>>
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> >>>>>>>>> 2. https://issues.apache.org/jira/browse/HBASE-25761
> >>>>>>>>>
> >>>>>>>>> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <
> >>>>> palomino219@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Is it only me? I tried to join the meeting but it always tell me
> >>>>> that
> >>>>>> I
> >>>>>>>>>> need to wait for the presenter to invite me to join...
> >>>>>>>>>>
> >>>>>>>>>> Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：
> >>>>>>>>>>
> >>>>>>>>>>> Short notice but reminder that this meeting is today at 5pm PST
> >>>>> (We
> >>>>>>>>>> talked
> >>>>>>>>>>> of doing it earlier but in the end lets just keep the original
> >>>>> 5pm
> >>>>>>>>>> time).
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> S
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net>
> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Lets do a quick chat on current state of hbase split-meta
> >>>>> project.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Francis has posted a suggested one-pager design in HBASE-25761
> >>>>>> which
> >>>>>>>>>>> would
> >>>>>>>>>>>> be good to review before the meeting if you can. Lets discuss
> >>>>> this
> >>>>>>>>>> and
> >>>>>>>>>>> any
> >>>>>>>>>>>> other suggestions for moving the project forward.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Meeting details below.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> S
> >>>>>>>>>>>>
> >>>>>>>>>>>> Topic Split Meta Design Reset Status
> >>>>>>>>>>>> Description Review one-pager design attached to
> >>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-25761
> >>>>>>>>>>>> Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Meeting ID 736 3907 8852
> >>>>>>>>>>>> Security
> >>>>>>>>>>>> Passcode *hbase* Hide
> >>>>>>>>>>>> Waiting Room
> >>>>>>>>>>>> Invite Link
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>>>>
> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
> >>>>>>>>>>> Copy
> >>>>>>>>>>>> Invitation
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net>
> >>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Now the requirements are in [1], we're going to move to the
> >>>>> next
> >>>>>>>>>> stage
> >>>>>>>>>>> --
> >>>>>>>>>>>>> actual design for split-meta -- and have set up a chat for
> >>>>> this
> >>>>>>>>>> thursday
> >>>>>>>>>>>>> afternoon (4PM California time/8AM Beijing time) to get the
> >>>>> ball
> >>>>>>>>>>> rolling.
> >>>>>>>>>>>>> Please come if interested. Zoom details are below.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Yours,
> >>>>>>>>>>>>> S
> >>>>>>>>>>>>> 1.
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>>>>
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Topic: hbase split-meta design warmup chat
> >>>>>>>>>>>>> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Join Zoom Meeting
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>>>>
> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Meeting ID: 759 8800 3798
> >>>>>>>>>>>>> Passcode: hbase
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net>
> >>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> FYI, a few of us have been working on the redo/reset of the
> >>>>>> split
> >>>>>>>>>> meta
> >>>>>>>>>>>>>> design (HBASE-25382). We (think we've) finished the
> >>>>>> requirements.
> >>>>>>>>>> Are
> >>>>>>>>>>> there
> >>>>>>>>>>>>>> any others to consider?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Feedback and contribs welcome. Otherwise, on to the next
> >>>>> phase
> >>>>>> --
> >>>>>>>>>>> design.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> S
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> >>>>
>

Re: Split Meta Design Reset Status

[Andrew Purtell <an...@gmail.com>]

Also let me say that your entreaty for more to join your calls is fine and I hope that happens, but this is Apache, and discussions should take place in suitably asynchronous fashion so all can participate - even those who have competing responsibilities and commitments and cannot attend the calls. This is not criticism of the calls! I’m so glad to see you lot are continuing to communicate. But we need to allow and support asynchronous communication and comment either by mailing list or on a document. Perhaps a doc that lists notes from each of your session… Or otherwise randos like me will be posting responses to your emailed notes here. 

> On Aug 13, 2021, at 8:30 AM, Andrew Purtell <an...@gmail.com> wrote:
> 
> Moving state into zookeeper is the wrong direction. Absolutely not. No, no, no. We can discuss it but it is very unlikely I would even not veto such a change. It doesn’t seem like a problem because you have other MUCH better and more viable options. 
> 
>> On Aug 13, 2021, at 7:50 AM, Stack <st...@duboce.net> wrote:
>> 
>> Rough minutes from our meeting of  Wed Aug 11 17:04:44 PDT 2021
>> 
>> Attendees: Francis, Duo, Francis.
>> 
>> We moved to discuss the proposed agenda item, how to implement the
>> "ROOT" Region.
>> 
>> Duo had a new proposal that we host meta Regions in zookeeper (either
>> all in a single znode or a znode per location).
>> 
>> We talked about it. Mainly, it was noted that zookeeper doesn't scale,
>> that a metatable in zk would not be able to satisfy
>> "2.0.6 Support an hbase:meta Table of 10k Regions" [1], and that it also
>> in violation of the old invariant that we keep no permanent state in
>> zookeeper [2].
>> 
>> Much back and forth on the fact that we violate the invariant currently; i.e.
>> there is info in zk now that can't be scrubbed; that we need to fix this before
>> we can go forward -- that the fact that meta location is currently persistent
>> means we could make split meta Region locations also zk persistent. Counter
>> arguments note that the original invariant script allowed that we had
>> violations,
>> that we have worked to undo, that addressing invariant violations was off-topic,
>> as well as variations on two wrongs do not make a right, etc.
>> 
>> We tried to reset to talk of previously suggested options of ROOT as a
>> new table to host hbase:meta Regions or just use first-Region in
>> hbase:meta as special case Region. A new ROOT table was generally
>> dismissed as an impediment when small installs in no need of split meta.
>> 
>> The second approach was called 'ugly'  -- I think 'ugly' was the term
>> used -- and too much if/else and possibly in the way of future evolutions.
>> Counter suggestions thought any uglyness could be hidden from client and
>> isolated and that this approach seemed to meet requirements.  Counter-arguments
>> to the counter-argument where this approach can not hide everything from client,
>> as opposed to separate ROOT table or master local region... see [1] for detail.
>> 
>> We ran out of time. Will try again next week (meantime on-going
>> developments on design doc [1] and in master).
>> 
>> (Would be good if we could get others on the call, other PMCers
>> with in an interest in split meta. Could be of help when we hit
>> blockage)
>> Thanks,
>> S
>> 
>> 1. https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wv90yxilng75
>> 2. http://hbase.apache.org/book.html#design.invariants.zk.data
>> 
>> 
>>>> On Tue, Aug 10, 2021 at 6:11 PM Stack <st...@duboce.net> wrote:
>>> 
>>> Meeting tomorrow afternoon @ 5pm (to accommodate Beijing time). Lets
>>> continue our Split Meta Design Reset discussion.
>>> 
>>> Time: Aug 11, 2021 05:00 PM Pacific Time (US and Canada)
>>> 
>>> Join Zoom Meeting
>>> https://us04web.zoom.us/j/73900142989?pwd=b2s1UmJ2a2hiUFRsRERYQzZBV0NHdz09
>>> 
>>> Meeting ID: 739 0014 2989
>>> Passcode: hbase
>>> 
>>> Please review the design doc "Design/Brainstorming" Section 4.1 [1] before
>>> the meeting.
>>> 
>>> Topics for discussion:
>>> 
>>> * Reports on progress since last meeting.
>>> * How will we actually implement the ROOT table (as a distinct ROOT table,
>>> as the first region of hbase:meta, etc.)
>>> 
>>> 1.
>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>>> 
>>> Thanks,
>>> S
>>> 
>>> 
>>>> On Tue, Aug 3, 2021 at 4:13 PM Stack <st...@duboce.net> wrote:
>>>> 
>>>> Any time Yu Li!
>>>> 
>>>> No meeting tomorrow... Lets meet next week, the 10th.
>>>> 
>>>> Thanks,
>>>> S
>>>> 
>>>>> On Wed, Jul 28, 2021 at 11:25 PM Yu Li <ca...@gmail.com> wrote:
>>>> 
>>>>> Thanks for the notes and efforts Stack, it's pretty helpful to know the
>>>>> progress and latest status of the work!
>>>>> 
>>>>> Best Regards,
>>>>> Yu
>>>>> 
>>>>> 
>>>>> On Wed, 28 Jul 2021 at 12:13, Stack <st...@duboce.net> wrote:
>>>>> 
>>>>>> Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of
>>>>>> August.
>>>>>> Thanks,
>>>>>> S
>>>>>> 
>>>>>> On Thu, Jul 22, 2021 at 8:53 PM Stack <st...@duboce.net> wrote:
>>>>>> 
>>>>>>> Notes from yesterday's meeting (attendees, please amend if I
>>>>> misrepresent
>>>>>>> or if you have anything extra to add!)
>>>>>>> 
>>>>>>> Split Meta Design Reset Status
>>>>>>> 
>>>>>>> Wed Jul 21 21:24:38 PDT 2021
>>>>>>> 
>>>>>>> Attendees: Bharath, Stack, Duo, and Francis
>>>>>>> 
>>>>>>> We went over the new updates to the Brainstorming [1] section under
>>>>>>> 
>>>>>>> Design in the Super Split Meta Design doc [2].
>>>>>>> 
>>>>>>> First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry;
>>>>>> hide
>>>>>>> 
>>>>>>> ROOT from Client [3]. In particular, filling out how "ROOT" might be
>>>>>>> 
>>>>>>> implemented behind the new API in ConnectionRegistry. On option 1.,
>>>>>>> 
>>>>>>> replicating master-local Region to RegionServers, options considered
>>>>>>> 
>>>>>>> included
>>>>>>> 
>>>>>>> * Listener on master-local Region WAL to replicate.
>>>>>>> 
>>>>>>> * Perhaps Read-Replica but master-local is not an actual Region
>>>>>>> 
>>>>>>> * Needs to be incremental edits because ROOT could get too big to
>>>>> ship
>>>>>>> 
>>>>>>>  in a lump; need to visit how...
>>>>>>> 
>>>>>>> * Possibly in-memory-only Regions on RS replicated from master-local
>>>>>>> 
>>>>>>>  Region via WAL tailing <= zhangduo@apache.org
>>>>>>> 
>>>>>>> * Which RegionServers? Those hosting ROOT replicas?
>>>>>>> 
>>>>>>> * How to bootstrap? Failure scenarios.
>>>>>>> 
>>>>>>> * This would be a new replication system alongside current; could
>>>>>>> 
>>>>>>>  evolve to replace/improve old?
>>>>>>> 
>>>>>>> Duo offered to look into means of replicating the master-local Region
>>>>>>> 
>>>>>>> out to RegionServers.
>>>>>>> 
>>>>>>> Next up was discussion constrasting ROOT as a standalone table vs
>>>>>>> 
>>>>>>> First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e.
>>>>>>> 
>>>>>>> options 2 and 3 for how we'd implement a ROOT. One item that came up
>>>>>>> 
>>>>>>> was whether a need to specify one replica count for a ROOT table vs
>>>>>>> 
>>>>>>> another for hbase:meta. If so, then it would be argument for ROOT as
>>>>>>> 
>>>>>>> standalone table (Others of us argued it not a concern of
>>>>> consequence).
>>>>>>> 
>>>>>>> If ROOT access is behind a new simple API in ConnectionRegistry, how
>>>>>>> 
>>>>>>> to stop clients reading hbase:meta table if not Master or fronted by
>>>>>>> 
>>>>>>> a ConnectionRegistry request? (Should be able to switch on client
>>>>>>> 
>>>>>>> identity/source). One suggestion for First-meta-Region-as-ROOT was
>>>>>>> 
>>>>>>> NOT returning the first Region to the client post-meta split when
>>>>>>> 
>>>>>>> accessing via the simple API. Some concern this would confuse old
>>>>>>> 
>>>>>>> Clients (Francis was going to take a look).
>>>>>>> 
>>>>>>> Moved to discussion how we'd move ConnectionRegistry from
>>>>>>> 
>>>>>>> hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a
>>>>>>> 
>>>>>>> system came up? Where do clients go? How do they know which
>>>>>>> 
>>>>>>> RegionServers (special regionserver group?  Every RS fields
>>>>>>> 
>>>>>>> ConnectionRegistry requests but only designated core serve the ROOT
>>>>>>> 
>>>>>>> lookup APIs?). This was a TODO.
>>>>>>> 
>>>>>>> This led naturally into 4.1.5 System RS group for client meta
>>>>> services
>>>>>>> 
>>>>>>> [5], a new addition under Brainstorming. Discussion. Bharath to look
>>>>>>> 
>>>>>>> into feedback.
>>>>>>> 
>>>>>>> On the end of the discussion, group expressed support for adding
>>>>>>> 
>>>>>>> simple API to the ConnectionRegistry to hide ROOT implementation
>>>>>>> 
>>>>>>> detail from client. Support was expressed for moving
>>>>> ConnectionRegistry
>>>>>>> 
>>>>>>> from Master to RegionServers. Intent is to move forward on design of
>>>>>>> 
>>>>>>> these pieces: e.g. how client bootstraps.
>>>>>>> 
>>>>>>> Support was expressed for getting at least the bones of a split meta
>>>>>>> 
>>>>>>> into an hbase3 before the RCs.
>>>>>>> 
>>>>>>> Where we'd actually store hbase:meta Region locations -- i.e. how a
>>>>>>> 
>>>>>>> "ROOT' would be implemented -- was for our next meeting informed by
>>>>>>> 
>>>>>>> research of the various approaches noted mostly above. It was
>>>>>>> 
>>>>>>> also thought that the new ConnectionRegistry should not preclude
>>>>>>> 
>>>>>>> making progress on the "ROOT" implementation.
>>>>>>> 
>>>>>>> Will post notice of next meeting (next Weds or the one
>>>>>>> 
>>>>>>> following).
>>>>>>> 
>>>>>>> 1.
>>>>>>> 
>>>>>> 
>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
>>>>>>> 
>>>>>>> 2.
>>>>>>> 
>>>>>> 
>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq
>>>>>>> 
>>>>>>> 3.
>>>>>>> 
>>>>>> 
>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153
>>>>>>> 
>>>>>>> 4.
>>>>>>> 
>>>>>> 
>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>>>>>>> 
>>>>>>> 5.
>>>>>>> 
>>>>>> 
>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b
>>>>>>> 
>>>>>>> On Tue, Jul 20, 2021 at 11:00 AM Stack <st...@duboce.net> wrote:
>>>>>>> 
>>>>>>>> Lets meet tomorrow. Please review the design doc
>>>>> "Design/Brainstorming"
>>>>>>>> Section 4.1 [1] before the meeting if you can (No harm if a refresh
>>>>> of
>>>>>> the
>>>>>>>> requirements section while you are at it).
>>>>>>>> 
>>>>>>>> Topic: Split Meta Design Reset Status
>>>>>>>> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada)
>>>>>>>> 
>>>>>>>> Join Zoom Meeting
>>>>>>>> 
>>>>>> 
>>>>> https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09
>>>>>>>> 
>>>>>>>> Meeting ID: 773 1892 0525
>>>>>>>> Passcode: hbase
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> S
>>>>>>>> 
>>>>>>>> 1. 1.
>>>>>>>> 
>>>>>> 
>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
>>>>>>>> 
>>>>>>>> On Thu, Jul 8, 2021 at 1:04 PM Stack <st...@duboce.net> wrote:
>>>>>>>> 
>>>>>>>>> Meeting notes (Meeting happend after I figured the zoom had a
>>>>> 'waiting
>>>>>>>>> room' -- sorry Duo)
>>>>>>>>> 
>>>>>>>>> Split Meta Status Zoom Meeting
>>>>>>>>> Wed Jul  7, 2021 @ 5pm for ~90minutes
>>>>>>>>> Attendees: Duo, Francis, Stack, and Clay
>>>>>>>>> Agenda: Mainly talk about the one-pager design and PoC proposed in
>>>>> [2]
>>>>>>>>> and added to the
>>>>>>>>> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as
>>>>> ROOT'
>>>>>>>>> approach)
>>>>>>>>> 
>>>>>>>>> Duo suggested no advantage treating the first meta of hbase:meta
>>>>> table
>>>>>>>>> special; as a "root"
>>>>>>>>> and other comments (see remarks in [2]).
>>>>>>>>> 
>>>>>>>>> Some pushback. When meta is NOT split, all works as it did before
>>>>> (big
>>>>>>>>> on backward-compatible).
>>>>>>>>> 
>>>>>>>>> Duo suggested just intro a ROOT table altogether rather than treat
>>>>> the
>>>>>>>>> first Region
>>>>>>>>> in the hbase:meta table as a 'root' and then mirror to zk the first
>>>>>> meta
>>>>>>>>> region; if no
>>>>>>>>> split, then old clients should just work even though now hbase
>>>>> cluster
>>>>>>>>> has a ROOT table.
>>>>>>>>> Discussion. If no split, what's to do, etc.?
>>>>>>>>> 
>>>>>>>>> Duo expressed concern that if split-meta is not on always --
>>>>> enabled
>>>>>>>>> optionally -- then
>>>>>>>>> the code path will not see exercise and split-meta will likely
>>>>> fail and
>>>>>>>>> go the way of
>>>>>>>>> prefix tree and distributed log replay -- a feature that failed,
>>>>>>>>> cluttered up the
>>>>>>>>> code base, and only later was removed. Discussion. Was allowed this
>>>>>>>>> could happen.
>>>>>>>>> Counter examples: RegionServer Groups. A few of the attendees
>>>>>>>>> volunteered they need
>>>>>>>>> split meta for their production so would try to see it through.
>>>>>>>>> 
>>>>>>>>> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1
>>>>> as
>>>>>>>>> ROOT' if we don't
>>>>>>>>> split; not much changes.
>>>>>>>>> 
>>>>>>>>> Comment on the PoC that its all
>>>>>>>>> 
>>>>>>>>> if 'first special meta region' do this
>>>>>>>>> else do something else...
>>>>>>>>> 
>>>>>>>>> (all over the codebase) but it was suggested that this will be the
>>>>> case
>>>>>>>>> if ROOT table
>>>>>>>>> added also and argued any implementation will have this issue (if
>>>>> ROOT
>>>>>>>>> then....) and
>>>>>>>>> THAT ugly 'root' comparator too whether a ROOT table or the
>>>>>>>>> 'hbase:meta,,1 as ROOT'
>>>>>>>>> approach.
>>>>>>>>> 
>>>>>>>>> Clay asking if meta is a bottleneck (Clay played the 'operator' and
>>>>>>>>> 'new-to-the-topic'
>>>>>>>>> roles). Some discussion around how indeed it is and why we want to
>>>>>> split
>>>>>>>>> meta at all.
>>>>>>>>> 
>>>>>>>>> Clay mentioned how Master is inline for data now (Master-hosted
>>>>>>>>> Registry)....
>>>>>>>>> Discussion. Hopefully temporary state -- Registry doesn't need to
>>>>> be
>>>>>>>>> hosted on
>>>>>>>>> Master -- and Master will return to its background role -- soon.
>>>>>>>>> 
>>>>>>>>> Clay brought up rollback after meta split. Discussion. Some
>>>>> agreement
>>>>>> it
>>>>>>>>> could be done for
>>>>>>>>> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root
>>>>> table
>>>>>> and
>>>>>>>>> client had
>>>>>>>>> started to use ROOT, it might be harder...
>>>>>>>>> 
>>>>>>>>> Duo suggested that we not change the client at all; that client
>>>>> stays
>>>>>>>>> same however split
>>>>>>>>> meta is implemented (with addition of a root table or using
>>>>>>>>> hbase:meta,,1 as 'root').
>>>>>>>>> This sounded attractive. We discussed how this could be done in a
>>>>>>>>> backward compatible way;
>>>>>>>>> add simple location lookup API to Registry...A write-up on how this
>>>>>>>>> might work will be posted
>>>>>>>>> in next day or so for others to review (Need to figure getting
>>>>> Registry
>>>>>>>>> off Master).
>>>>>>>>> 
>>>>>>>>> Clay suggested, as an operator, how he thought the split meta
>>>>> should
>>>>>>>>> roll out. One of us
>>>>>>>>> claimed this described the 'hbase:meta,,1 as ROOT' approach.
>>>>>>>>> 
>>>>>>>>> Duo had to go to work so we called it quits and said we'd meet
>>>>> again
>>>>>>>>> same time, next week.
>>>>>>>>> 
>>>>>>>>> 1.
>>>>>>>>> 
>>>>>> 
>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>>>>>>>>> 2. https://issues.apache.org/jira/browse/HBASE-25761
>>>>>>>>> 
>>>>>>>>> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <
>>>>> palomino219@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Is it only me? I tried to join the meeting but it always tell me
>>>>> that
>>>>>> I
>>>>>>>>>> need to wait for the presenter to invite me to join...
>>>>>>>>>> 
>>>>>>>>>> Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：
>>>>>>>>>> 
>>>>>>>>>>> Short notice but reminder that this meeting is today at 5pm PST
>>>>> (We
>>>>>>>>>> talked
>>>>>>>>>>> of doing it earlier but in the end lets just keep the original
>>>>> 5pm
>>>>>>>>>> time).
>>>>>>>>>>> Thanks,
>>>>>>>>>>> S
>>>>>>>>>>> 
>>>>>>>>>>> On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Lets do a quick chat on current state of hbase split-meta
>>>>> project.
>>>>>>>>>>>> 
>>>>>>>>>>>> Francis has posted a suggested one-pager design in HBASE-25761
>>>>>> which
>>>>>>>>>>> would
>>>>>>>>>>>> be good to review before the meeting if you can. Lets discuss
>>>>> this
>>>>>>>>>> and
>>>>>>>>>>> any
>>>>>>>>>>>> other suggestions for moving the project forward.
>>>>>>>>>>>> 
>>>>>>>>>>>> Meeting details below.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> S
>>>>>>>>>>>> 
>>>>>>>>>>>> Topic Split Meta Design Reset Status
>>>>>>>>>>>> Description Review one-pager design attached to
>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-25761
>>>>>>>>>>>> Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
>>>>>>>>>>>> 
>>>>>>>>>>>> Meeting ID 736 3907 8852
>>>>>>>>>>>> Security
>>>>>>>>>>>> Passcode *hbase* Hide
>>>>>>>>>>>> Waiting Room
>>>>>>>>>>>> Invite Link
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>> 
>>>>> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
>>>>>>>>>>> Copy
>>>>>>>>>>>> Invitation
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net>
>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Now the requirements are in [1], we're going to move to the
>>>>> next
>>>>>>>>>> stage
>>>>>>>>>>> --
>>>>>>>>>>>>> actual design for split-meta -- and have set up a chat for
>>>>> this
>>>>>>>>>> thursday
>>>>>>>>>>>>> afternoon (4PM California time/8AM Beijing time) to get the
>>>>> ball
>>>>>>>>>>> rolling.
>>>>>>>>>>>>> Please come if interested. Zoom details are below.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Yours,
>>>>>>>>>>>>> S
>>>>>>>>>>>>> 1.
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>> 
>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Topic: hbase split-meta design warmup chat
>>>>>>>>>>>>> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Join Zoom Meeting
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>> 
>>>>> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Meeting ID: 759 8800 3798
>>>>>>>>>>>>> Passcode: hbase
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net>
>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> FYI, a few of us have been working on the redo/reset of the
>>>>>> split
>>>>>>>>>> meta
>>>>>>>>>>>>>> design (HBASE-25382). We (think we've) finished the
>>>>>> requirements.
>>>>>>>>>> Are
>>>>>>>>>>> there
>>>>>>>>>>>>>> any others to consider?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Feedback and contribs welcome. Otherwise, on to the next
>>>>> phase
>>>>>> --
>>>>>>>>>>> design.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> S
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> 
>>>> 

Re: Split Meta Design Reset Status

[Andrew Purtell <an...@gmail.com>]

It should be possible to recreate all necessary state in ZK in the cold boot scenario and recover from a total loss of ZK state without trouble (except some process restarts). We’ve been moving replication state out of zookeeper for this reason, etc. 

We’ve taken shortcuts that still need to be fixed. Will look at HBASE-26193, thanks. 


> On Aug 13, 2021, at 9:24 PM, 张铎 <pa...@gmail.com> wrote:
> 
> This is not moving state to zookeeper, we already have this state in
> zookeeper... The location of meta region is only stored in zookeeper,
> and if it is cleaned, we will be in trouble...
> 
> And I filed HBASE-26193 to fix this problem. PTAL if you have interest.
> 
> Thanks.
> 
> Andrew Purtell <an...@gmail.com> 于2021年8月13日周五 下午11:30写道：
>> 
>> Moving state into zookeeper is the wrong direction. Absolutely not. No, no, no. We can discuss it but it is very unlikely I would even not veto such a change. It doesn’t seem like a problem because you have other MUCH better and more viable options.
>> 
>>>> On Aug 13, 2021, at 7:50 AM, Stack <st...@duboce.net> wrote:
>>> 
>>> Rough minutes from our meeting of  Wed Aug 11 17:04:44 PDT 2021
>>> 
>>> Attendees: Francis, Duo, Francis.
>>> 
>>> We moved to discuss the proposed agenda item, how to implement the
>>> "ROOT" Region.
>>> 
>>> Duo had a new proposal that we host meta Regions in zookeeper (either
>>> all in a single znode or a znode per location).
>>> 
>>> We talked about it. Mainly, it was noted that zookeeper doesn't scale,
>>> that a metatable in zk would not be able to satisfy
>>> "2.0.6 Support an hbase:meta Table of 10k Regions" [1], and that it also
>>> in violation of the old invariant that we keep no permanent state in
>>> zookeeper [2].
>>> 
>>> Much back and forth on the fact that we violate the invariant currently; i.e.
>>> there is info in zk now that can't be scrubbed; that we need to fix this before
>>> we can go forward -- that the fact that meta location is currently persistent
>>> means we could make split meta Region locations also zk persistent. Counter
>>> arguments note that the original invariant script allowed that we had
>>> violations,
>>> that we have worked to undo, that addressing invariant violations was off-topic,
>>> as well as variations on two wrongs do not make a right, etc.
>>> 
>>> We tried to reset to talk of previously suggested options of ROOT as a
>>> new table to host hbase:meta Regions or just use first-Region in
>>> hbase:meta as special case Region. A new ROOT table was generally
>>> dismissed as an impediment when small installs in no need of split meta.
>>> 
>>> The second approach was called 'ugly'  -- I think 'ugly' was the term
>>> used -- and too much if/else and possibly in the way of future evolutions.
>>> Counter suggestions thought any uglyness could be hidden from client and
>>> isolated and that this approach seemed to meet requirements.  Counter-arguments
>>> to the counter-argument where this approach can not hide everything from client,
>>> as opposed to separate ROOT table or master local region... see [1] for detail.
>>> 
>>> We ran out of time. Will try again next week (meantime on-going
>>> developments on design doc [1] and in master).
>>> 
>>> (Would be good if we could get others on the call, other PMCers
>>> with in an interest in split meta. Could be of help when we hit
>>> blockage)
>>> Thanks,
>>> S
>>> 
>>> 1. https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wv90yxilng75
>>> 2. http://hbase.apache.org/book.html#design.invariants.zk.data
>>> 
>>> 
>>>> On Tue, Aug 10, 2021 at 6:11 PM Stack <st...@duboce.net> wrote:
>>>> 
>>>> Meeting tomorrow afternoon @ 5pm (to accommodate Beijing time). Lets
>>>> continue our Split Meta Design Reset discussion.
>>>> 
>>>> Time: Aug 11, 2021 05:00 PM Pacific Time (US and Canada)
>>>> 
>>>> Join Zoom Meeting
>>>> https://us04web.zoom.us/j/73900142989?pwd=b2s1UmJ2a2hiUFRsRERYQzZBV0NHdz09
>>>> 
>>>> Meeting ID: 739 0014 2989
>>>> Passcode: hbase
>>>> 
>>>> Please review the design doc "Design/Brainstorming" Section 4.1 [1] before
>>>> the meeting.
>>>> 
>>>> Topics for discussion:
>>>> 
>>>> * Reports on progress since last meeting.
>>>> * How will we actually implement the ROOT table (as a distinct ROOT table,
>>>> as the first region of hbase:meta, etc.)
>>>> 
>>>> 1.
>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>>>> 
>>>> Thanks,
>>>> S
>>>> 
>>>> 
>>>>> On Tue, Aug 3, 2021 at 4:13 PM Stack <st...@duboce.net> wrote:
>>>>> 
>>>>> Any time Yu Li!
>>>>> 
>>>>> No meeting tomorrow... Lets meet next week, the 10th.
>>>>> 
>>>>> Thanks,
>>>>> S
>>>>> 
>>>>>> On Wed, Jul 28, 2021 at 11:25 PM Yu Li <ca...@gmail.com> wrote:
>>>>> 
>>>>>> Thanks for the notes and efforts Stack, it's pretty helpful to know the
>>>>>> progress and latest status of the work!
>>>>>> 
>>>>>> Best Regards,
>>>>>> Yu
>>>>>> 
>>>>>> 
>>>>>> On Wed, 28 Jul 2021 at 12:13, Stack <st...@duboce.net> wrote:
>>>>>> 
>>>>>>> Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of
>>>>>>> August.
>>>>>>> Thanks,
>>>>>>> S
>>>>>>> 
>>>>>>> On Thu, Jul 22, 2021 at 8:53 PM Stack <st...@duboce.net> wrote:
>>>>>>> 
>>>>>>>> Notes from yesterday's meeting (attendees, please amend if I
>>>>>> misrepresent
>>>>>>>> or if you have anything extra to add!)
>>>>>>>> 
>>>>>>>> Split Meta Design Reset Status
>>>>>>>> 
>>>>>>>> Wed Jul 21 21:24:38 PDT 2021
>>>>>>>> 
>>>>>>>> Attendees: Bharath, Stack, Duo, and Francis
>>>>>>>> 
>>>>>>>> We went over the new updates to the Brainstorming [1] section under
>>>>>>>> 
>>>>>>>> Design in the Super Split Meta Design doc [2].
>>>>>>>> 
>>>>>>>> First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry;
>>>>>>> hide
>>>>>>>> 
>>>>>>>> ROOT from Client [3]. In particular, filling out how "ROOT" might be
>>>>>>>> 
>>>>>>>> implemented behind the new API in ConnectionRegistry. On option 1.,
>>>>>>>> 
>>>>>>>> replicating master-local Region to RegionServers, options considered
>>>>>>>> 
>>>>>>>> included
>>>>>>>> 
>>>>>>>> * Listener on master-local Region WAL to replicate.
>>>>>>>> 
>>>>>>>> * Perhaps Read-Replica but master-local is not an actual Region
>>>>>>>> 
>>>>>>>> * Needs to be incremental edits because ROOT could get too big to
>>>>>> ship
>>>>>>>> 
>>>>>>>>  in a lump; need to visit how...
>>>>>>>> 
>>>>>>>> * Possibly in-memory-only Regions on RS replicated from master-local
>>>>>>>> 
>>>>>>>>  Region via WAL tailing <= zhangduo@apache.org
>>>>>>>> 
>>>>>>>> * Which RegionServers? Those hosting ROOT replicas?
>>>>>>>> 
>>>>>>>> * How to bootstrap? Failure scenarios.
>>>>>>>> 
>>>>>>>> * This would be a new replication system alongside current; could
>>>>>>>> 
>>>>>>>>  evolve to replace/improve old?
>>>>>>>> 
>>>>>>>> Duo offered to look into means of replicating the master-local Region
>>>>>>>> 
>>>>>>>> out to RegionServers.
>>>>>>>> 
>>>>>>>> Next up was discussion constrasting ROOT as a standalone table vs
>>>>>>>> 
>>>>>>>> First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e.
>>>>>>>> 
>>>>>>>> options 2 and 3 for how we'd implement a ROOT. One item that came up
>>>>>>>> 
>>>>>>>> was whether a need to specify one replica count for a ROOT table vs
>>>>>>>> 
>>>>>>>> another for hbase:meta. If so, then it would be argument for ROOT as
>>>>>>>> 
>>>>>>>> standalone table (Others of us argued it not a concern of
>>>>>> consequence).
>>>>>>>> 
>>>>>>>> If ROOT access is behind a new simple API in ConnectionRegistry, how
>>>>>>>> 
>>>>>>>> to stop clients reading hbase:meta table if not Master or fronted by
>>>>>>>> 
>>>>>>>> a ConnectionRegistry request? (Should be able to switch on client
>>>>>>>> 
>>>>>>>> identity/source). One suggestion for First-meta-Region-as-ROOT was
>>>>>>>> 
>>>>>>>> NOT returning the first Region to the client post-meta split when
>>>>>>>> 
>>>>>>>> accessing via the simple API. Some concern this would confuse old
>>>>>>>> 
>>>>>>>> Clients (Francis was going to take a look).
>>>>>>>> 
>>>>>>>> Moved to discussion how we'd move ConnectionRegistry from
>>>>>>>> 
>>>>>>>> hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a
>>>>>>>> 
>>>>>>>> system came up? Where do clients go? How do they know which
>>>>>>>> 
>>>>>>>> RegionServers (special regionserver group?  Every RS fields
>>>>>>>> 
>>>>>>>> ConnectionRegistry requests but only designated core serve the ROOT
>>>>>>>> 
>>>>>>>> lookup APIs?). This was a TODO.
>>>>>>>> 
>>>>>>>> This led naturally into 4.1.5 System RS group for client meta
>>>>>> services
>>>>>>>> 
>>>>>>>> [5], a new addition under Brainstorming. Discussion. Bharath to look
>>>>>>>> 
>>>>>>>> into feedback.
>>>>>>>> 
>>>>>>>> On the end of the discussion, group expressed support for adding
>>>>>>>> 
>>>>>>>> simple API to the ConnectionRegistry to hide ROOT implementation
>>>>>>>> 
>>>>>>>> detail from client. Support was expressed for moving
>>>>>> ConnectionRegistry
>>>>>>>> 
>>>>>>>> from Master to RegionServers. Intent is to move forward on design of
>>>>>>>> 
>>>>>>>> these pieces: e.g. how client bootstraps.
>>>>>>>> 
>>>>>>>> Support was expressed for getting at least the bones of a split meta
>>>>>>>> 
>>>>>>>> into an hbase3 before the RCs.
>>>>>>>> 
>>>>>>>> Where we'd actually store hbase:meta Region locations -- i.e. how a
>>>>>>>> 
>>>>>>>> "ROOT' would be implemented -- was for our next meeting informed by
>>>>>>>> 
>>>>>>>> research of the various approaches noted mostly above. It was
>>>>>>>> 
>>>>>>>> also thought that the new ConnectionRegistry should not preclude
>>>>>>>> 
>>>>>>>> making progress on the "ROOT" implementation.
>>>>>>>> 
>>>>>>>> Will post notice of next meeting (next Weds or the one
>>>>>>>> 
>>>>>>>> following).
>>>>>>>> 
>>>>>>>> 1.
>>>>>>>> 
>>>>>>> 
>>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
>>>>>>>> 
>>>>>>>> 2.
>>>>>>>> 
>>>>>>> 
>>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq
>>>>>>>> 
>>>>>>>> 3.
>>>>>>>> 
>>>>>>> 
>>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153
>>>>>>>> 
>>>>>>>> 4.
>>>>>>>> 
>>>>>>> 
>>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>>>>>>>> 
>>>>>>>> 5.
>>>>>>>> 
>>>>>>> 
>>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b
>>>>>>>> 
>>>>>>>> On Tue, Jul 20, 2021 at 11:00 AM Stack <st...@duboce.net> wrote:
>>>>>>>> 
>>>>>>>>> Lets meet tomorrow. Please review the design doc
>>>>>> "Design/Brainstorming"
>>>>>>>>> Section 4.1 [1] before the meeting if you can (No harm if a refresh
>>>>>> of
>>>>>>> the
>>>>>>>>> requirements section while you are at it).
>>>>>>>>> 
>>>>>>>>> Topic: Split Meta Design Reset Status
>>>>>>>>> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada)
>>>>>>>>> 
>>>>>>>>> Join Zoom Meeting
>>>>>>>>> 
>>>>>>> 
>>>>>> https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09
>>>>>>>>> 
>>>>>>>>> Meeting ID: 773 1892 0525
>>>>>>>>> Passcode: hbase
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> S
>>>>>>>>> 
>>>>>>>>> 1. 1.
>>>>>>>>> 
>>>>>>> 
>>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
>>>>>>>>> 
>>>>>>>>> On Thu, Jul 8, 2021 at 1:04 PM Stack <st...@duboce.net> wrote:
>>>>>>>>> 
>>>>>>>>>> Meeting notes (Meeting happend after I figured the zoom had a
>>>>>> 'waiting
>>>>>>>>>> room' -- sorry Duo)
>>>>>>>>>> 
>>>>>>>>>> Split Meta Status Zoom Meeting
>>>>>>>>>> Wed Jul  7, 2021 @ 5pm for ~90minutes
>>>>>>>>>> Attendees: Duo, Francis, Stack, and Clay
>>>>>>>>>> Agenda: Mainly talk about the one-pager design and PoC proposed in
>>>>>> [2]
>>>>>>>>>> and added to the
>>>>>>>>>> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as
>>>>>> ROOT'
>>>>>>>>>> approach)
>>>>>>>>>> 
>>>>>>>>>> Duo suggested no advantage treating the first meta of hbase:meta
>>>>>> table
>>>>>>>>>> special; as a "root"
>>>>>>>>>> and other comments (see remarks in [2]).
>>>>>>>>>> 
>>>>>>>>>> Some pushback. When meta is NOT split, all works as it did before
>>>>>> (big
>>>>>>>>>> on backward-compatible).
>>>>>>>>>> 
>>>>>>>>>> Duo suggested just intro a ROOT table altogether rather than treat
>>>>>> the
>>>>>>>>>> first Region
>>>>>>>>>> in the hbase:meta table as a 'root' and then mirror to zk the first
>>>>>>> meta
>>>>>>>>>> region; if no
>>>>>>>>>> split, then old clients should just work even though now hbase
>>>>>> cluster
>>>>>>>>>> has a ROOT table.
>>>>>>>>>> Discussion. If no split, what's to do, etc.?
>>>>>>>>>> 
>>>>>>>>>> Duo expressed concern that if split-meta is not on always --
>>>>>> enabled
>>>>>>>>>> optionally -- then
>>>>>>>>>> the code path will not see exercise and split-meta will likely
>>>>>> fail and
>>>>>>>>>> go the way of
>>>>>>>>>> prefix tree and distributed log replay -- a feature that failed,
>>>>>>>>>> cluttered up the
>>>>>>>>>> code base, and only later was removed. Discussion. Was allowed this
>>>>>>>>>> could happen.
>>>>>>>>>> Counter examples: RegionServer Groups. A few of the attendees
>>>>>>>>>> volunteered they need
>>>>>>>>>> split meta for their production so would try to see it through.
>>>>>>>>>> 
>>>>>>>>>> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1
>>>>>> as
>>>>>>>>>> ROOT' if we don't
>>>>>>>>>> split; not much changes.
>>>>>>>>>> 
>>>>>>>>>> Comment on the PoC that its all
>>>>>>>>>> 
>>>>>>>>>> if 'first special meta region' do this
>>>>>>>>>> else do something else...
>>>>>>>>>> 
>>>>>>>>>> (all over the codebase) but it was suggested that this will be the
>>>>>> case
>>>>>>>>>> if ROOT table
>>>>>>>>>> added also and argued any implementation will have this issue (if
>>>>>> ROOT
>>>>>>>>>> then....) and
>>>>>>>>>> THAT ugly 'root' comparator too whether a ROOT table or the
>>>>>>>>>> 'hbase:meta,,1 as ROOT'
>>>>>>>>>> approach.
>>>>>>>>>> 
>>>>>>>>>> Clay asking if meta is a bottleneck (Clay played the 'operator' and
>>>>>>>>>> 'new-to-the-topic'
>>>>>>>>>> roles). Some discussion around how indeed it is and why we want to
>>>>>>> split
>>>>>>>>>> meta at all.
>>>>>>>>>> 
>>>>>>>>>> Clay mentioned how Master is inline for data now (Master-hosted
>>>>>>>>>> Registry)....
>>>>>>>>>> Discussion. Hopefully temporary state -- Registry doesn't need to
>>>>>> be
>>>>>>>>>> hosted on
>>>>>>>>>> Master -- and Master will return to its background role -- soon.
>>>>>>>>>> 
>>>>>>>>>> Clay brought up rollback after meta split. Discussion. Some
>>>>>> agreement
>>>>>>> it
>>>>>>>>>> could be done for
>>>>>>>>>> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root
>>>>>> table
>>>>>>> and
>>>>>>>>>> client had
>>>>>>>>>> started to use ROOT, it might be harder...
>>>>>>>>>> 
>>>>>>>>>> Duo suggested that we not change the client at all; that client
>>>>>> stays
>>>>>>>>>> same however split
>>>>>>>>>> meta is implemented (with addition of a root table or using
>>>>>>>>>> hbase:meta,,1 as 'root').
>>>>>>>>>> This sounded attractive. We discussed how this could be done in a
>>>>>>>>>> backward compatible way;
>>>>>>>>>> add simple location lookup API to Registry...A write-up on how this
>>>>>>>>>> might work will be posted
>>>>>>>>>> in next day or so for others to review (Need to figure getting
>>>>>> Registry
>>>>>>>>>> off Master).
>>>>>>>>>> 
>>>>>>>>>> Clay suggested, as an operator, how he thought the split meta
>>>>>> should
>>>>>>>>>> roll out. One of us
>>>>>>>>>> claimed this described the 'hbase:meta,,1 as ROOT' approach.
>>>>>>>>>> 
>>>>>>>>>> Duo had to go to work so we called it quits and said we'd meet
>>>>>> again
>>>>>>>>>> same time, next week.
>>>>>>>>>> 
>>>>>>>>>> 1.
>>>>>>>>>> 
>>>>>>> 
>>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>>>>>>>>>> 2. https://issues.apache.org/jira/browse/HBASE-25761
>>>>>>>>>> 
>>>>>>>>>> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <
>>>>>> palomino219@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Is it only me? I tried to join the meeting but it always tell me
>>>>>> that
>>>>>>> I
>>>>>>>>>>> need to wait for the presenter to invite me to join...
>>>>>>>>>>> 
>>>>>>>>>>> Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：
>>>>>>>>>>> 
>>>>>>>>>>>> Short notice but reminder that this meeting is today at 5pm PST
>>>>>> (We
>>>>>>>>>>> talked
>>>>>>>>>>>> of doing it earlier but in the end lets just keep the original
>>>>>> 5pm
>>>>>>>>>>> time).
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> S
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Lets do a quick chat on current state of hbase split-meta
>>>>>> project.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Francis has posted a suggested one-pager design in HBASE-25761
>>>>>>> which
>>>>>>>>>>>> would
>>>>>>>>>>>>> be good to review before the meeting if you can. Lets discuss
>>>>>> this
>>>>>>>>>>> and
>>>>>>>>>>>> any
>>>>>>>>>>>>> other suggestions for moving the project forward.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Meeting details below.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> S
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Topic Split Meta Design Reset Status
>>>>>>>>>>>>> Description Review one-pager design attached to
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-25761
>>>>>>>>>>>>> Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Meeting ID 736 3907 8852
>>>>>>>>>>>>> Security
>>>>>>>>>>>>> Passcode *hbase* Hide
>>>>>>>>>>>>> Waiting Room
>>>>>>>>>>>>> Invite Link
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>> 
>>>>>> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
>>>>>>>>>>>> Copy
>>>>>>>>>>>>> Invitation
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net>
>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Now the requirements are in [1], we're going to move to the
>>>>>> next
>>>>>>>>>>> stage
>>>>>>>>>>>> --
>>>>>>>>>>>>>> actual design for split-meta -- and have set up a chat for
>>>>>> this
>>>>>>>>>>> thursday
>>>>>>>>>>>>>> afternoon (4PM California time/8AM Beijing time) to get the
>>>>>> ball
>>>>>>>>>>>> rolling.
>>>>>>>>>>>>>> Please come if interested. Zoom details are below.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Yours,
>>>>>>>>>>>>>> S
>>>>>>>>>>>>>> 1.
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>> 
>>>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Topic: hbase split-meta design warmup chat
>>>>>>>>>>>>>> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Join Zoom Meeting
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>> 
>>>>>> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Meeting ID: 759 8800 3798
>>>>>>>>>>>>>> Passcode: hbase
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net>
>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> FYI, a few of us have been working on the redo/reset of the
>>>>>>> split
>>>>>>>>>>> meta
>>>>>>>>>>>>>>> design (HBASE-25382). We (think we've) finished the
>>>>>>> requirements.
>>>>>>>>>>> Are
>>>>>>>>>>>> there
>>>>>>>>>>>>>>> any others to consider?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Feedback and contribs welcome. Otherwise, on to the next
>>>>>> phase
>>>>>>> --
>>>>>>>>>>>> design.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> S
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 

Re: Split Meta Design Reset Status

["张铎 (Duo Zhang)" <pa...@gmail.com>]

This is not moving state to zookeeper, we already have this state in
zookeeper... The location of meta region is only stored in zookeeper,
and if it is cleaned, we will be in trouble...

And I filed HBASE-26193 to fix this problem. PTAL if you have interest.

Thanks.

Andrew Purtell <an...@gmail.com> 于2021年8月13日周五 下午11:30写道：
>
> Moving state into zookeeper is the wrong direction. Absolutely not. No, no, no. We can discuss it but it is very unlikely I would even not veto such a change. It doesn’t seem like a problem because you have other MUCH better and more viable options.
>
> > On Aug 13, 2021, at 7:50 AM, Stack <st...@duboce.net> wrote:
> >
> > Rough minutes from our meeting of  Wed Aug 11 17:04:44 PDT 2021
> >
> > Attendees: Francis, Duo, Francis.
> >
> > We moved to discuss the proposed agenda item, how to implement the
> > "ROOT" Region.
> >
> > Duo had a new proposal that we host meta Regions in zookeeper (either
> > all in a single znode or a znode per location).
> >
> > We talked about it. Mainly, it was noted that zookeeper doesn't scale,
> > that a metatable in zk would not be able to satisfy
> > "2.0.6 Support an hbase:meta Table of 10k Regions" [1], and that it also
> > in violation of the old invariant that we keep no permanent state in
> > zookeeper [2].
> >
> > Much back and forth on the fact that we violate the invariant currently; i.e.
> > there is info in zk now that can't be scrubbed; that we need to fix this before
> > we can go forward -- that the fact that meta location is currently persistent
> > means we could make split meta Region locations also zk persistent. Counter
> > arguments note that the original invariant script allowed that we had
> > violations,
> > that we have worked to undo, that addressing invariant violations was off-topic,
> > as well as variations on two wrongs do not make a right, etc.
> >
> > We tried to reset to talk of previously suggested options of ROOT as a
> > new table to host hbase:meta Regions or just use first-Region in
> > hbase:meta as special case Region. A new ROOT table was generally
> > dismissed as an impediment when small installs in no need of split meta.
> >
> > The second approach was called 'ugly'  -- I think 'ugly' was the term
> > used -- and too much if/else and possibly in the way of future evolutions.
> > Counter suggestions thought any uglyness could be hidden from client and
> > isolated and that this approach seemed to meet requirements.  Counter-arguments
> > to the counter-argument where this approach can not hide everything from client,
> > as opposed to separate ROOT table or master local region... see [1] for detail.
> >
> > We ran out of time. Will try again next week (meantime on-going
> > developments on design doc [1] and in master).
> >
> > (Would be good if we could get others on the call, other PMCers
> > with in an interest in split meta. Could be of help when we hit
> > blockage)
> > Thanks,
> > S
> >
> > 1. https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wv90yxilng75
> > 2. http://hbase.apache.org/book.html#design.invariants.zk.data
> >
> >
> >> On Tue, Aug 10, 2021 at 6:11 PM Stack <st...@duboce.net> wrote:
> >>
> >> Meeting tomorrow afternoon @ 5pm (to accommodate Beijing time). Lets
> >> continue our Split Meta Design Reset discussion.
> >>
> >> Time: Aug 11, 2021 05:00 PM Pacific Time (US and Canada)
> >>
> >> Join Zoom Meeting
> >> https://us04web.zoom.us/j/73900142989?pwd=b2s1UmJ2a2hiUFRsRERYQzZBV0NHdz09
> >>
> >> Meeting ID: 739 0014 2989
> >> Passcode: hbase
> >>
> >> Please review the design doc "Design/Brainstorming" Section 4.1 [1] before
> >> the meeting.
> >>
> >> Topics for discussion:
> >>
> >> * Reports on progress since last meeting.
> >> * How will we actually implement the ROOT table (as a distinct ROOT table,
> >> as the first region of hbase:meta, etc.)
> >>
> >> 1.
> >> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> >>
> >> Thanks,
> >> S
> >>
> >>
> >>> On Tue, Aug 3, 2021 at 4:13 PM Stack <st...@duboce.net> wrote:
> >>>
> >>> Any time Yu Li!
> >>>
> >>> No meeting tomorrow... Lets meet next week, the 10th.
> >>>
> >>> Thanks,
> >>> S
> >>>
> >>>> On Wed, Jul 28, 2021 at 11:25 PM Yu Li <ca...@gmail.com> wrote:
> >>>
> >>>> Thanks for the notes and efforts Stack, it's pretty helpful to know the
> >>>> progress and latest status of the work!
> >>>>
> >>>> Best Regards,
> >>>> Yu
> >>>>
> >>>>
> >>>> On Wed, 28 Jul 2021 at 12:13, Stack <st...@duboce.net> wrote:
> >>>>
> >>>>> Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of
> >>>>> August.
> >>>>> Thanks,
> >>>>> S
> >>>>>
> >>>>> On Thu, Jul 22, 2021 at 8:53 PM Stack <st...@duboce.net> wrote:
> >>>>>
> >>>>>> Notes from yesterday's meeting (attendees, please amend if I
> >>>> misrepresent
> >>>>>> or if you have anything extra to add!)
> >>>>>>
> >>>>>> Split Meta Design Reset Status
> >>>>>>
> >>>>>> Wed Jul 21 21:24:38 PDT 2021
> >>>>>>
> >>>>>> Attendees: Bharath, Stack, Duo, and Francis
> >>>>>>
> >>>>>> We went over the new updates to the Brainstorming [1] section under
> >>>>>>
> >>>>>> Design in the Super Split Meta Design doc [2].
> >>>>>>
> >>>>>> First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry;
> >>>>> hide
> >>>>>>
> >>>>>> ROOT from Client [3]. In particular, filling out how "ROOT" might be
> >>>>>>
> >>>>>> implemented behind the new API in ConnectionRegistry. On option 1.,
> >>>>>>
> >>>>>> replicating master-local Region to RegionServers, options considered
> >>>>>>
> >>>>>> included
> >>>>>>
> >>>>>> * Listener on master-local Region WAL to replicate.
> >>>>>>
> >>>>>> * Perhaps Read-Replica but master-local is not an actual Region
> >>>>>>
> >>>>>> * Needs to be incremental edits because ROOT could get too big to
> >>>> ship
> >>>>>>
> >>>>>>   in a lump; need to visit how...
> >>>>>>
> >>>>>> * Possibly in-memory-only Regions on RS replicated from master-local
> >>>>>>
> >>>>>>   Region via WAL tailing <= zhangduo@apache.org
> >>>>>>
> >>>>>> * Which RegionServers? Those hosting ROOT replicas?
> >>>>>>
> >>>>>> * How to bootstrap? Failure scenarios.
> >>>>>>
> >>>>>> * This would be a new replication system alongside current; could
> >>>>>>
> >>>>>>   evolve to replace/improve old?
> >>>>>>
> >>>>>> Duo offered to look into means of replicating the master-local Region
> >>>>>>
> >>>>>> out to RegionServers.
> >>>>>>
> >>>>>> Next up was discussion constrasting ROOT as a standalone table vs
> >>>>>>
> >>>>>> First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e.
> >>>>>>
> >>>>>> options 2 and 3 for how we'd implement a ROOT. One item that came up
> >>>>>>
> >>>>>> was whether a need to specify one replica count for a ROOT table vs
> >>>>>>
> >>>>>> another for hbase:meta. If so, then it would be argument for ROOT as
> >>>>>>
> >>>>>> standalone table (Others of us argued it not a concern of
> >>>> consequence).
> >>>>>>
> >>>>>> If ROOT access is behind a new simple API in ConnectionRegistry, how
> >>>>>>
> >>>>>> to stop clients reading hbase:meta table if not Master or fronted by
> >>>>>>
> >>>>>> a ConnectionRegistry request? (Should be able to switch on client
> >>>>>>
> >>>>>> identity/source). One suggestion for First-meta-Region-as-ROOT was
> >>>>>>
> >>>>>> NOT returning the first Region to the client post-meta split when
> >>>>>>
> >>>>>> accessing via the simple API. Some concern this would confuse old
> >>>>>>
> >>>>>> Clients (Francis was going to take a look).
> >>>>>>
> >>>>>> Moved to discussion how we'd move ConnectionRegistry from
> >>>>>>
> >>>>>> hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a
> >>>>>>
> >>>>>> system came up? Where do clients go? How do they know which
> >>>>>>
> >>>>>> RegionServers (special regionserver group?  Every RS fields
> >>>>>>
> >>>>>> ConnectionRegistry requests but only designated core serve the ROOT
> >>>>>>
> >>>>>> lookup APIs?). This was a TODO.
> >>>>>>
> >>>>>> This led naturally into 4.1.5 System RS group for client meta
> >>>> services
> >>>>>>
> >>>>>> [5], a new addition under Brainstorming. Discussion. Bharath to look
> >>>>>>
> >>>>>> into feedback.
> >>>>>>
> >>>>>> On the end of the discussion, group expressed support for adding
> >>>>>>
> >>>>>> simple API to the ConnectionRegistry to hide ROOT implementation
> >>>>>>
> >>>>>> detail from client. Support was expressed for moving
> >>>> ConnectionRegistry
> >>>>>>
> >>>>>> from Master to RegionServers. Intent is to move forward on design of
> >>>>>>
> >>>>>> these pieces: e.g. how client bootstraps.
> >>>>>>
> >>>>>> Support was expressed for getting at least the bones of a split meta
> >>>>>>
> >>>>>> into an hbase3 before the RCs.
> >>>>>>
> >>>>>> Where we'd actually store hbase:meta Region locations -- i.e. how a
> >>>>>>
> >>>>>> "ROOT' would be implemented -- was for our next meeting informed by
> >>>>>>
> >>>>>> research of the various approaches noted mostly above. It was
> >>>>>>
> >>>>>> also thought that the new ConnectionRegistry should not preclude
> >>>>>>
> >>>>>> making progress on the "ROOT" implementation.
> >>>>>>
> >>>>>> Will post notice of next meeting (next Weds or the one
> >>>>>>
> >>>>>> following).
> >>>>>>
> >>>>>> 1.
> >>>>>>
> >>>>>
> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
> >>>>>>
> >>>>>> 2.
> >>>>>>
> >>>>>
> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq
> >>>>>>
> >>>>>> 3.
> >>>>>>
> >>>>>
> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153
> >>>>>>
> >>>>>> 4.
> >>>>>>
> >>>>>
> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> >>>>>>
> >>>>>> 5.
> >>>>>>
> >>>>>
> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b
> >>>>>>
> >>>>>> On Tue, Jul 20, 2021 at 11:00 AM Stack <st...@duboce.net> wrote:
> >>>>>>
> >>>>>>> Lets meet tomorrow. Please review the design doc
> >>>> "Design/Brainstorming"
> >>>>>>> Section 4.1 [1] before the meeting if you can (No harm if a refresh
> >>>> of
> >>>>> the
> >>>>>>> requirements section while you are at it).
> >>>>>>>
> >>>>>>> Topic: Split Meta Design Reset Status
> >>>>>>> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada)
> >>>>>>>
> >>>>>>> Join Zoom Meeting
> >>>>>>>
> >>>>>
> >>>> https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09
> >>>>>>>
> >>>>>>> Meeting ID: 773 1892 0525
> >>>>>>> Passcode: hbase
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> S
> >>>>>>>
> >>>>>>> 1. 1.
> >>>>>>>
> >>>>>
> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
> >>>>>>>
> >>>>>>> On Thu, Jul 8, 2021 at 1:04 PM Stack <st...@duboce.net> wrote:
> >>>>>>>
> >>>>>>>> Meeting notes (Meeting happend after I figured the zoom had a
> >>>> 'waiting
> >>>>>>>> room' -- sorry Duo)
> >>>>>>>>
> >>>>>>>> Split Meta Status Zoom Meeting
> >>>>>>>> Wed Jul  7, 2021 @ 5pm for ~90minutes
> >>>>>>>> Attendees: Duo, Francis, Stack, and Clay
> >>>>>>>> Agenda: Mainly talk about the one-pager design and PoC proposed in
> >>>> [2]
> >>>>>>>> and added to the
> >>>>>>>> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as
> >>>> ROOT'
> >>>>>>>> approach)
> >>>>>>>>
> >>>>>>>> Duo suggested no advantage treating the first meta of hbase:meta
> >>>> table
> >>>>>>>> special; as a "root"
> >>>>>>>> and other comments (see remarks in [2]).
> >>>>>>>>
> >>>>>>>> Some pushback. When meta is NOT split, all works as it did before
> >>>> (big
> >>>>>>>> on backward-compatible).
> >>>>>>>>
> >>>>>>>> Duo suggested just intro a ROOT table altogether rather than treat
> >>>> the
> >>>>>>>> first Region
> >>>>>>>> in the hbase:meta table as a 'root' and then mirror to zk the first
> >>>>> meta
> >>>>>>>> region; if no
> >>>>>>>> split, then old clients should just work even though now hbase
> >>>> cluster
> >>>>>>>> has a ROOT table.
> >>>>>>>> Discussion. If no split, what's to do, etc.?
> >>>>>>>>
> >>>>>>>> Duo expressed concern that if split-meta is not on always --
> >>>> enabled
> >>>>>>>> optionally -- then
> >>>>>>>> the code path will not see exercise and split-meta will likely
> >>>> fail and
> >>>>>>>> go the way of
> >>>>>>>> prefix tree and distributed log replay -- a feature that failed,
> >>>>>>>> cluttered up the
> >>>>>>>> code base, and only later was removed. Discussion. Was allowed this
> >>>>>>>> could happen.
> >>>>>>>> Counter examples: RegionServer Groups. A few of the attendees
> >>>>>>>> volunteered they need
> >>>>>>>> split meta for their production so would try to see it through.
> >>>>>>>>
> >>>>>>>> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1
> >>>> as
> >>>>>>>> ROOT' if we don't
> >>>>>>>> split; not much changes.
> >>>>>>>>
> >>>>>>>> Comment on the PoC that its all
> >>>>>>>>
> >>>>>>>>  if 'first special meta region' do this
> >>>>>>>>  else do something else...
> >>>>>>>>
> >>>>>>>> (all over the codebase) but it was suggested that this will be the
> >>>> case
> >>>>>>>> if ROOT table
> >>>>>>>> added also and argued any implementation will have this issue (if
> >>>> ROOT
> >>>>>>>> then....) and
> >>>>>>>> THAT ugly 'root' comparator too whether a ROOT table or the
> >>>>>>>> 'hbase:meta,,1 as ROOT'
> >>>>>>>> approach.
> >>>>>>>>
> >>>>>>>> Clay asking if meta is a bottleneck (Clay played the 'operator' and
> >>>>>>>> 'new-to-the-topic'
> >>>>>>>> roles). Some discussion around how indeed it is and why we want to
> >>>>> split
> >>>>>>>> meta at all.
> >>>>>>>>
> >>>>>>>> Clay mentioned how Master is inline for data now (Master-hosted
> >>>>>>>> Registry)....
> >>>>>>>> Discussion. Hopefully temporary state -- Registry doesn't need to
> >>>> be
> >>>>>>>> hosted on
> >>>>>>>> Master -- and Master will return to its background role -- soon.
> >>>>>>>>
> >>>>>>>> Clay brought up rollback after meta split. Discussion. Some
> >>>> agreement
> >>>>> it
> >>>>>>>> could be done for
> >>>>>>>> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root
> >>>> table
> >>>>> and
> >>>>>>>> client had
> >>>>>>>> started to use ROOT, it might be harder...
> >>>>>>>>
> >>>>>>>> Duo suggested that we not change the client at all; that client
> >>>> stays
> >>>>>>>> same however split
> >>>>>>>> meta is implemented (with addition of a root table or using
> >>>>>>>> hbase:meta,,1 as 'root').
> >>>>>>>> This sounded attractive. We discussed how this could be done in a
> >>>>>>>> backward compatible way;
> >>>>>>>> add simple location lookup API to Registry...A write-up on how this
> >>>>>>>> might work will be posted
> >>>>>>>> in next day or so for others to review (Need to figure getting
> >>>> Registry
> >>>>>>>> off Master).
> >>>>>>>>
> >>>>>>>> Clay suggested, as an operator, how he thought the split meta
> >>>> should
> >>>>>>>> roll out. One of us
> >>>>>>>> claimed this described the 'hbase:meta,,1 as ROOT' approach.
> >>>>>>>>
> >>>>>>>> Duo had to go to work so we called it quits and said we'd meet
> >>>> again
> >>>>>>>> same time, next week.
> >>>>>>>>
> >>>>>>>> 1.
> >>>>>>>>
> >>>>>
> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> >>>>>>>> 2. https://issues.apache.org/jira/browse/HBASE-25761
> >>>>>>>>
> >>>>>>>> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <
> >>>> palomino219@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Is it only me? I tried to join the meeting but it always tell me
> >>>> that
> >>>>> I
> >>>>>>>>> need to wait for the presenter to invite me to join...
> >>>>>>>>>
> >>>>>>>>> Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：
> >>>>>>>>>
> >>>>>>>>>> Short notice but reminder that this meeting is today at 5pm PST
> >>>> (We
> >>>>>>>>> talked
> >>>>>>>>>> of doing it earlier but in the end lets just keep the original
> >>>> 5pm
> >>>>>>>>> time).
> >>>>>>>>>> Thanks,
> >>>>>>>>>> S
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Lets do a quick chat on current state of hbase split-meta
> >>>> project.
> >>>>>>>>>>>
> >>>>>>>>>>> Francis has posted a suggested one-pager design in HBASE-25761
> >>>>> which
> >>>>>>>>>> would
> >>>>>>>>>>> be good to review before the meeting if you can. Lets discuss
> >>>> this
> >>>>>>>>> and
> >>>>>>>>>> any
> >>>>>>>>>>> other suggestions for moving the project forward.
> >>>>>>>>>>>
> >>>>>>>>>>> Meeting details below.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> S
> >>>>>>>>>>>
> >>>>>>>>>>> Topic Split Meta Design Reset Status
> >>>>>>>>>>> Description Review one-pager design attached to
> >>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-25761
> >>>>>>>>>>> Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
> >>>>>>>>>>>
> >>>>>>>>>>> Meeting ID 736 3907 8852
> >>>>>>>>>>> Security
> >>>>>>>>>>> Passcode *hbase* Hide
> >>>>>>>>>>> Waiting Room
> >>>>>>>>>>> Invite Link
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>>> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
> >>>>>>>>>> Copy
> >>>>>>>>>>> Invitation
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net>
> >>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Now the requirements are in [1], we're going to move to the
> >>>> next
> >>>>>>>>> stage
> >>>>>>>>>> --
> >>>>>>>>>>>> actual design for split-meta -- and have set up a chat for
> >>>> this
> >>>>>>>>> thursday
> >>>>>>>>>>>> afternoon (4PM California time/8AM Beijing time) to get the
> >>>> ball
> >>>>>>>>>> rolling.
> >>>>>>>>>>>> Please come if interested. Zoom details are below.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Yours,
> >>>>>>>>>>>> S
> >>>>>>>>>>>> 1.
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Topic: hbase split-meta design warmup chat
> >>>>>>>>>>>> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Join Zoom Meeting
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>>> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
> >>>>>>>>>>>>
> >>>>>>>>>>>> Meeting ID: 759 8800 3798
> >>>>>>>>>>>> Passcode: hbase
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net>
> >>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> FYI, a few of us have been working on the redo/reset of the
> >>>>> split
> >>>>>>>>> meta
> >>>>>>>>>>>>> design (HBASE-25382). We (think we've) finished the
> >>>>> requirements.
> >>>>>>>>> Are
> >>>>>>>>>> there
> >>>>>>>>>>>>> any others to consider?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Feedback and contribs welcome. Otherwise, on to the next
> >>>> phase
> >>>>> --
> >>>>>>>>>> design.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> S
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>>
> >>>

Re: Split Meta Design Reset Status

[Andrew Purtell <an...@gmail.com>]

Moving state into zookeeper is the wrong direction. Absolutely not. No, no, no. We can discuss it but it is very unlikely I would even not veto such a change. It doesn’t seem like a problem because you have other MUCH better and more viable options. 

> On Aug 13, 2021, at 7:50 AM, Stack <st...@duboce.net> wrote:
> 
> Rough minutes from our meeting of  Wed Aug 11 17:04:44 PDT 2021
> 
> Attendees: Francis, Duo, Francis.
> 
> We moved to discuss the proposed agenda item, how to implement the
> "ROOT" Region.
> 
> Duo had a new proposal that we host meta Regions in zookeeper (either
> all in a single znode or a znode per location).
> 
> We talked about it. Mainly, it was noted that zookeeper doesn't scale,
> that a metatable in zk would not be able to satisfy
> "2.0.6 Support an hbase:meta Table of 10k Regions" [1], and that it also
> in violation of the old invariant that we keep no permanent state in
> zookeeper [2].
> 
> Much back and forth on the fact that we violate the invariant currently; i.e.
> there is info in zk now that can't be scrubbed; that we need to fix this before
> we can go forward -- that the fact that meta location is currently persistent
> means we could make split meta Region locations also zk persistent. Counter
> arguments note that the original invariant script allowed that we had
> violations,
> that we have worked to undo, that addressing invariant violations was off-topic,
> as well as variations on two wrongs do not make a right, etc.
> 
> We tried to reset to talk of previously suggested options of ROOT as a
> new table to host hbase:meta Regions or just use first-Region in
> hbase:meta as special case Region. A new ROOT table was generally
> dismissed as an impediment when small installs in no need of split meta.
> 
> The second approach was called 'ugly'  -- I think 'ugly' was the term
> used -- and too much if/else and possibly in the way of future evolutions.
> Counter suggestions thought any uglyness could be hidden from client and
> isolated and that this approach seemed to meet requirements.  Counter-arguments
> to the counter-argument where this approach can not hide everything from client,
> as opposed to separate ROOT table or master local region... see [1] for detail.
> 
> We ran out of time. Will try again next week (meantime on-going
> developments on design doc [1] and in master).
> 
> (Would be good if we could get others on the call, other PMCers
> with in an interest in split meta. Could be of help when we hit
> blockage)
> Thanks,
> S
> 
> 1. https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wv90yxilng75
> 2. http://hbase.apache.org/book.html#design.invariants.zk.data
> 
> 
>> On Tue, Aug 10, 2021 at 6:11 PM Stack <st...@duboce.net> wrote:
>> 
>> Meeting tomorrow afternoon @ 5pm (to accommodate Beijing time). Lets
>> continue our Split Meta Design Reset discussion.
>> 
>> Time: Aug 11, 2021 05:00 PM Pacific Time (US and Canada)
>> 
>> Join Zoom Meeting
>> https://us04web.zoom.us/j/73900142989?pwd=b2s1UmJ2a2hiUFRsRERYQzZBV0NHdz09
>> 
>> Meeting ID: 739 0014 2989
>> Passcode: hbase
>> 
>> Please review the design doc "Design/Brainstorming" Section 4.1 [1] before
>> the meeting.
>> 
>> Topics for discussion:
>> 
>> * Reports on progress since last meeting.
>> * How will we actually implement the ROOT table (as a distinct ROOT table,
>> as the first region of hbase:meta, etc.)
>> 
>> 1.
>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>> 
>> Thanks,
>> S
>> 
>> 
>>> On Tue, Aug 3, 2021 at 4:13 PM Stack <st...@duboce.net> wrote:
>>> 
>>> Any time Yu Li!
>>> 
>>> No meeting tomorrow... Lets meet next week, the 10th.
>>> 
>>> Thanks,
>>> S
>>> 
>>>> On Wed, Jul 28, 2021 at 11:25 PM Yu Li <ca...@gmail.com> wrote:
>>> 
>>>> Thanks for the notes and efforts Stack, it's pretty helpful to know the
>>>> progress and latest status of the work!
>>>> 
>>>> Best Regards,
>>>> Yu
>>>> 
>>>> 
>>>> On Wed, 28 Jul 2021 at 12:13, Stack <st...@duboce.net> wrote:
>>>> 
>>>>> Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of
>>>>> August.
>>>>> Thanks,
>>>>> S
>>>>> 
>>>>> On Thu, Jul 22, 2021 at 8:53 PM Stack <st...@duboce.net> wrote:
>>>>> 
>>>>>> Notes from yesterday's meeting (attendees, please amend if I
>>>> misrepresent
>>>>>> or if you have anything extra to add!)
>>>>>> 
>>>>>> Split Meta Design Reset Status
>>>>>> 
>>>>>> Wed Jul 21 21:24:38 PDT 2021
>>>>>> 
>>>>>> Attendees: Bharath, Stack, Duo, and Francis
>>>>>> 
>>>>>> We went over the new updates to the Brainstorming [1] section under
>>>>>> 
>>>>>> Design in the Super Split Meta Design doc [2].
>>>>>> 
>>>>>> First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry;
>>>>> hide
>>>>>> 
>>>>>> ROOT from Client [3]. In particular, filling out how "ROOT" might be
>>>>>> 
>>>>>> implemented behind the new API in ConnectionRegistry. On option 1.,
>>>>>> 
>>>>>> replicating master-local Region to RegionServers, options considered
>>>>>> 
>>>>>> included
>>>>>> 
>>>>>> * Listener on master-local Region WAL to replicate.
>>>>>> 
>>>>>> * Perhaps Read-Replica but master-local is not an actual Region
>>>>>> 
>>>>>> * Needs to be incremental edits because ROOT could get too big to
>>>> ship
>>>>>> 
>>>>>>   in a lump; need to visit how...
>>>>>> 
>>>>>> * Possibly in-memory-only Regions on RS replicated from master-local
>>>>>> 
>>>>>>   Region via WAL tailing <= zhangduo@apache.org
>>>>>> 
>>>>>> * Which RegionServers? Those hosting ROOT replicas?
>>>>>> 
>>>>>> * How to bootstrap? Failure scenarios.
>>>>>> 
>>>>>> * This would be a new replication system alongside current; could
>>>>>> 
>>>>>>   evolve to replace/improve old?
>>>>>> 
>>>>>> Duo offered to look into means of replicating the master-local Region
>>>>>> 
>>>>>> out to RegionServers.
>>>>>> 
>>>>>> Next up was discussion constrasting ROOT as a standalone table vs
>>>>>> 
>>>>>> First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e.
>>>>>> 
>>>>>> options 2 and 3 for how we'd implement a ROOT. One item that came up
>>>>>> 
>>>>>> was whether a need to specify one replica count for a ROOT table vs
>>>>>> 
>>>>>> another for hbase:meta. If so, then it would be argument for ROOT as
>>>>>> 
>>>>>> standalone table (Others of us argued it not a concern of
>>>> consequence).
>>>>>> 
>>>>>> If ROOT access is behind a new simple API in ConnectionRegistry, how
>>>>>> 
>>>>>> to stop clients reading hbase:meta table if not Master or fronted by
>>>>>> 
>>>>>> a ConnectionRegistry request? (Should be able to switch on client
>>>>>> 
>>>>>> identity/source). One suggestion for First-meta-Region-as-ROOT was
>>>>>> 
>>>>>> NOT returning the first Region to the client post-meta split when
>>>>>> 
>>>>>> accessing via the simple API. Some concern this would confuse old
>>>>>> 
>>>>>> Clients (Francis was going to take a look).
>>>>>> 
>>>>>> Moved to discussion how we'd move ConnectionRegistry from
>>>>>> 
>>>>>> hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a
>>>>>> 
>>>>>> system came up? Where do clients go? How do they know which
>>>>>> 
>>>>>> RegionServers (special regionserver group?  Every RS fields
>>>>>> 
>>>>>> ConnectionRegistry requests but only designated core serve the ROOT
>>>>>> 
>>>>>> lookup APIs?). This was a TODO.
>>>>>> 
>>>>>> This led naturally into 4.1.5 System RS group for client meta
>>>> services
>>>>>> 
>>>>>> [5], a new addition under Brainstorming. Discussion. Bharath to look
>>>>>> 
>>>>>> into feedback.
>>>>>> 
>>>>>> On the end of the discussion, group expressed support for adding
>>>>>> 
>>>>>> simple API to the ConnectionRegistry to hide ROOT implementation
>>>>>> 
>>>>>> detail from client. Support was expressed for moving
>>>> ConnectionRegistry
>>>>>> 
>>>>>> from Master to RegionServers. Intent is to move forward on design of
>>>>>> 
>>>>>> these pieces: e.g. how client bootstraps.
>>>>>> 
>>>>>> Support was expressed for getting at least the bones of a split meta
>>>>>> 
>>>>>> into an hbase3 before the RCs.
>>>>>> 
>>>>>> Where we'd actually store hbase:meta Region locations -- i.e. how a
>>>>>> 
>>>>>> "ROOT' would be implemented -- was for our next meeting informed by
>>>>>> 
>>>>>> research of the various approaches noted mostly above. It was
>>>>>> 
>>>>>> also thought that the new ConnectionRegistry should not preclude
>>>>>> 
>>>>>> making progress on the "ROOT" implementation.
>>>>>> 
>>>>>> Will post notice of next meeting (next Weds or the one
>>>>>> 
>>>>>> following).
>>>>>> 
>>>>>> 1.
>>>>>> 
>>>>> 
>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
>>>>>> 
>>>>>> 2.
>>>>>> 
>>>>> 
>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq
>>>>>> 
>>>>>> 3.
>>>>>> 
>>>>> 
>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153
>>>>>> 
>>>>>> 4.
>>>>>> 
>>>>> 
>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>>>>>> 
>>>>>> 5.
>>>>>> 
>>>>> 
>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b
>>>>>> 
>>>>>> On Tue, Jul 20, 2021 at 11:00 AM Stack <st...@duboce.net> wrote:
>>>>>> 
>>>>>>> Lets meet tomorrow. Please review the design doc
>>>> "Design/Brainstorming"
>>>>>>> Section 4.1 [1] before the meeting if you can (No harm if a refresh
>>>> of
>>>>> the
>>>>>>> requirements section while you are at it).
>>>>>>> 
>>>>>>> Topic: Split Meta Design Reset Status
>>>>>>> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada)
>>>>>>> 
>>>>>>> Join Zoom Meeting
>>>>>>> 
>>>>> 
>>>> https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09
>>>>>>> 
>>>>>>> Meeting ID: 773 1892 0525
>>>>>>> Passcode: hbase
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> S
>>>>>>> 
>>>>>>> 1. 1.
>>>>>>> 
>>>>> 
>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
>>>>>>> 
>>>>>>> On Thu, Jul 8, 2021 at 1:04 PM Stack <st...@duboce.net> wrote:
>>>>>>> 
>>>>>>>> Meeting notes (Meeting happend after I figured the zoom had a
>>>> 'waiting
>>>>>>>> room' -- sorry Duo)
>>>>>>>> 
>>>>>>>> Split Meta Status Zoom Meeting
>>>>>>>> Wed Jul  7, 2021 @ 5pm for ~90minutes
>>>>>>>> Attendees: Duo, Francis, Stack, and Clay
>>>>>>>> Agenda: Mainly talk about the one-pager design and PoC proposed in
>>>> [2]
>>>>>>>> and added to the
>>>>>>>> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as
>>>> ROOT'
>>>>>>>> approach)
>>>>>>>> 
>>>>>>>> Duo suggested no advantage treating the first meta of hbase:meta
>>>> table
>>>>>>>> special; as a "root"
>>>>>>>> and other comments (see remarks in [2]).
>>>>>>>> 
>>>>>>>> Some pushback. When meta is NOT split, all works as it did before
>>>> (big
>>>>>>>> on backward-compatible).
>>>>>>>> 
>>>>>>>> Duo suggested just intro a ROOT table altogether rather than treat
>>>> the
>>>>>>>> first Region
>>>>>>>> in the hbase:meta table as a 'root' and then mirror to zk the first
>>>>> meta
>>>>>>>> region; if no
>>>>>>>> split, then old clients should just work even though now hbase
>>>> cluster
>>>>>>>> has a ROOT table.
>>>>>>>> Discussion. If no split, what's to do, etc.?
>>>>>>>> 
>>>>>>>> Duo expressed concern that if split-meta is not on always --
>>>> enabled
>>>>>>>> optionally -- then
>>>>>>>> the code path will not see exercise and split-meta will likely
>>>> fail and
>>>>>>>> go the way of
>>>>>>>> prefix tree and distributed log replay -- a feature that failed,
>>>>>>>> cluttered up the
>>>>>>>> code base, and only later was removed. Discussion. Was allowed this
>>>>>>>> could happen.
>>>>>>>> Counter examples: RegionServer Groups. A few of the attendees
>>>>>>>> volunteered they need
>>>>>>>> split meta for their production so would try to see it through.
>>>>>>>> 
>>>>>>>> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1
>>>> as
>>>>>>>> ROOT' if we don't
>>>>>>>> split; not much changes.
>>>>>>>> 
>>>>>>>> Comment on the PoC that its all
>>>>>>>> 
>>>>>>>>  if 'first special meta region' do this
>>>>>>>>  else do something else...
>>>>>>>> 
>>>>>>>> (all over the codebase) but it was suggested that this will be the
>>>> case
>>>>>>>> if ROOT table
>>>>>>>> added also and argued any implementation will have this issue (if
>>>> ROOT
>>>>>>>> then....) and
>>>>>>>> THAT ugly 'root' comparator too whether a ROOT table or the
>>>>>>>> 'hbase:meta,,1 as ROOT'
>>>>>>>> approach.
>>>>>>>> 
>>>>>>>> Clay asking if meta is a bottleneck (Clay played the 'operator' and
>>>>>>>> 'new-to-the-topic'
>>>>>>>> roles). Some discussion around how indeed it is and why we want to
>>>>> split
>>>>>>>> meta at all.
>>>>>>>> 
>>>>>>>> Clay mentioned how Master is inline for data now (Master-hosted
>>>>>>>> Registry)....
>>>>>>>> Discussion. Hopefully temporary state -- Registry doesn't need to
>>>> be
>>>>>>>> hosted on
>>>>>>>> Master -- and Master will return to its background role -- soon.
>>>>>>>> 
>>>>>>>> Clay brought up rollback after meta split. Discussion. Some
>>>> agreement
>>>>> it
>>>>>>>> could be done for
>>>>>>>> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root
>>>> table
>>>>> and
>>>>>>>> client had
>>>>>>>> started to use ROOT, it might be harder...
>>>>>>>> 
>>>>>>>> Duo suggested that we not change the client at all; that client
>>>> stays
>>>>>>>> same however split
>>>>>>>> meta is implemented (with addition of a root table or using
>>>>>>>> hbase:meta,,1 as 'root').
>>>>>>>> This sounded attractive. We discussed how this could be done in a
>>>>>>>> backward compatible way;
>>>>>>>> add simple location lookup API to Registry...A write-up on how this
>>>>>>>> might work will be posted
>>>>>>>> in next day or so for others to review (Need to figure getting
>>>> Registry
>>>>>>>> off Master).
>>>>>>>> 
>>>>>>>> Clay suggested, as an operator, how he thought the split meta
>>>> should
>>>>>>>> roll out. One of us
>>>>>>>> claimed this described the 'hbase:meta,,1 as ROOT' approach.
>>>>>>>> 
>>>>>>>> Duo had to go to work so we called it quits and said we'd meet
>>>> again
>>>>>>>> same time, next week.
>>>>>>>> 
>>>>>>>> 1.
>>>>>>>> 
>>>>> 
>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>>>>>>>> 2. https://issues.apache.org/jira/browse/HBASE-25761
>>>>>>>> 
>>>>>>>> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <
>>>> palomino219@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Is it only me? I tried to join the meeting but it always tell me
>>>> that
>>>>> I
>>>>>>>>> need to wait for the presenter to invite me to join...
>>>>>>>>> 
>>>>>>>>> Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：
>>>>>>>>> 
>>>>>>>>>> Short notice but reminder that this meeting is today at 5pm PST
>>>> (We
>>>>>>>>> talked
>>>>>>>>>> of doing it earlier but in the end lets just keep the original
>>>> 5pm
>>>>>>>>> time).
>>>>>>>>>> Thanks,
>>>>>>>>>> S
>>>>>>>>>> 
>>>>>>>>>> On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Lets do a quick chat on current state of hbase split-meta
>>>> project.
>>>>>>>>>>> 
>>>>>>>>>>> Francis has posted a suggested one-pager design in HBASE-25761
>>>>> which
>>>>>>>>>> would
>>>>>>>>>>> be good to review before the meeting if you can. Lets discuss
>>>> this
>>>>>>>>> and
>>>>>>>>>> any
>>>>>>>>>>> other suggestions for moving the project forward.
>>>>>>>>>>> 
>>>>>>>>>>> Meeting details below.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> S
>>>>>>>>>>> 
>>>>>>>>>>> Topic Split Meta Design Reset Status
>>>>>>>>>>> Description Review one-pager design attached to
>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-25761
>>>>>>>>>>> Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
>>>>>>>>>>> 
>>>>>>>>>>> Meeting ID 736 3907 8852
>>>>>>>>>>> Security
>>>>>>>>>>> Passcode *hbase* Hide
>>>>>>>>>>> Waiting Room
>>>>>>>>>>> Invite Link
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>> 
>>>> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
>>>>>>>>>> Copy
>>>>>>>>>>> Invitation
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net>
>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Now the requirements are in [1], we're going to move to the
>>>> next
>>>>>>>>> stage
>>>>>>>>>> --
>>>>>>>>>>>> actual design for split-meta -- and have set up a chat for
>>>> this
>>>>>>>>> thursday
>>>>>>>>>>>> afternoon (4PM California time/8AM Beijing time) to get the
>>>> ball
>>>>>>>>>> rolling.
>>>>>>>>>>>> Please come if interested. Zoom details are below.
>>>>>>>>>>>> 
>>>>>>>>>>>> Yours,
>>>>>>>>>>>> S
>>>>>>>>>>>> 1.
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>> 
>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Topic: hbase split-meta design warmup chat
>>>>>>>>>>>> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
>>>>>>>>>>>> 
>>>>>>>>>>>> Join Zoom Meeting
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>> 
>>>> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
>>>>>>>>>>>> 
>>>>>>>>>>>> Meeting ID: 759 8800 3798
>>>>>>>>>>>> Passcode: hbase
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net>
>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> FYI, a few of us have been working on the redo/reset of the
>>>>> split
>>>>>>>>> meta
>>>>>>>>>>>>> design (HBASE-25382). We (think we've) finished the
>>>>> requirements.
>>>>>>>>> Are
>>>>>>>>>> there
>>>>>>>>>>>>> any others to consider?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Feedback and contribs welcome. Otherwise, on to the next
>>>> phase
>>>>> --
>>>>>>>>>> design.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> S
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> 
>>> 

Re: Split Meta Design Reset Status

[Stack <st...@duboce.net>]

Rough minutes from our meeting of  Wed Aug 11 17:04:44 PDT 2021

Attendees: Francis, Duo, Francis.

We moved to discuss the proposed agenda item, how to implement the
"ROOT" Region.

Duo had a new proposal that we host meta Regions in zookeeper (either
all in a single znode or a znode per location).

We talked about it. Mainly, it was noted that zookeeper doesn't scale,
that a metatable in zk would not be able to satisfy
"2.0.6 Support an hbase:meta Table of 10k Regions" [1], and that it also
in violation of the old invariant that we keep no permanent state in
zookeeper [2].

Much back and forth on the fact that we violate the invariant currently; i.e.
there is info in zk now that can't be scrubbed; that we need to fix this before
we can go forward -- that the fact that meta location is currently persistent
means we could make split meta Region locations also zk persistent. Counter
arguments note that the original invariant script allowed that we had
violations,
that we have worked to undo, that addressing invariant violations was off-topic,
as well as variations on two wrongs do not make a right, etc.

We tried to reset to talk of previously suggested options of ROOT as a
new table to host hbase:meta Regions or just use first-Region in
hbase:meta as special case Region. A new ROOT table was generally
dismissed as an impediment when small installs in no need of split meta.

The second approach was called 'ugly'  -- I think 'ugly' was the term
used -- and too much if/else and possibly in the way of future evolutions.
Counter suggestions thought any uglyness could be hidden from client and
isolated and that this approach seemed to meet requirements.  Counter-arguments
to the counter-argument where this approach can not hide everything from client,
as opposed to separate ROOT table or master local region... see [1] for detail.

We ran out of time. Will try again next week (meantime on-going
developments on design doc [1] and in master).

(Would be good if we could get others on the call, other PMCers
with in an interest in split meta. Could be of help when we hit
blockage)
Thanks,
S

1. https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wv90yxilng75
2. http://hbase.apache.org/book.html#design.invariants.zk.data


On Tue, Aug 10, 2021 at 6:11 PM Stack <st...@duboce.net> wrote:

> Meeting tomorrow afternoon @ 5pm (to accommodate Beijing time). Lets
> continue our Split Meta Design Reset discussion.
>
> Time: Aug 11, 2021 05:00 PM Pacific Time (US and Canada)
>
> Join Zoom Meeting
> https://us04web.zoom.us/j/73900142989?pwd=b2s1UmJ2a2hiUFRsRERYQzZBV0NHdz09
>
> Meeting ID: 739 0014 2989
> Passcode: hbase
>
> Please review the design doc "Design/Brainstorming" Section 4.1 [1] before
> the meeting.
>
> Topics for discussion:
>
> * Reports on progress since last meeting.
> * How will we actually implement the ROOT table (as a distinct ROOT table,
> as the first region of hbase:meta, etc.)
>
> 1.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>
> Thanks,
> S
>
>
> On Tue, Aug 3, 2021 at 4:13 PM Stack <st...@duboce.net> wrote:
>
>> Any time Yu Li!
>>
>> No meeting tomorrow... Lets meet next week, the 10th.
>>
>> Thanks,
>> S
>>
>> On Wed, Jul 28, 2021 at 11:25 PM Yu Li <ca...@gmail.com> wrote:
>>
>>> Thanks for the notes and efforts Stack, it's pretty helpful to know the
>>> progress and latest status of the work!
>>>
>>> Best Regards,
>>> Yu
>>>
>>>
>>> On Wed, 28 Jul 2021 at 12:13, Stack <st...@duboce.net> wrote:
>>>
>>> > Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of
>>> > August.
>>> > Thanks,
>>> > S
>>> >
>>> > On Thu, Jul 22, 2021 at 8:53 PM Stack <st...@duboce.net> wrote:
>>> >
>>> > > Notes from yesterday's meeting (attendees, please amend if I
>>> misrepresent
>>> > > or if you have anything extra to add!)
>>> > >
>>> > > Split Meta Design Reset Status
>>> > >
>>> > > Wed Jul 21 21:24:38 PDT 2021
>>> > >
>>> > > Attendees: Bharath, Stack, Duo, and Francis
>>> > >
>>> > > We went over the new updates to the Brainstorming [1] section under
>>> > >
>>> > > Design in the Super Split Meta Design doc [2].
>>> > >
>>> > > First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry;
>>> > hide
>>> > >
>>> > > ROOT from Client [3]. In particular, filling out how "ROOT" might be
>>> > >
>>> > > implemented behind the new API in ConnectionRegistry. On option 1.,
>>> > >
>>> > > replicating master-local Region to RegionServers, options considered
>>> > >
>>> > > included
>>> > >
>>> > >  * Listener on master-local Region WAL to replicate.
>>> > >
>>> > >  * Perhaps Read-Replica but master-local is not an actual Region
>>> > >
>>> > >  * Needs to be incremental edits because ROOT could get too big to
>>> ship
>>> > >
>>> > >    in a lump; need to visit how...
>>> > >
>>> > >  * Possibly in-memory-only Regions on RS replicated from master-local
>>> > >
>>> > >    Region via WAL tailing <= zhangduo@apache.org
>>> > >
>>> > >  * Which RegionServers? Those hosting ROOT replicas?
>>> > >
>>> > >  * How to bootstrap? Failure scenarios.
>>> > >
>>> > >  * This would be a new replication system alongside current; could
>>> > >
>>> > >    evolve to replace/improve old?
>>> > >
>>> > > Duo offered to look into means of replicating the master-local Region
>>> > >
>>> > > out to RegionServers.
>>> > >
>>> > > Next up was discussion constrasting ROOT as a standalone table vs
>>> > >
>>> > > First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e.
>>> > >
>>> > > options 2 and 3 for how we'd implement a ROOT. One item that came up
>>> > >
>>> > > was whether a need to specify one replica count for a ROOT table vs
>>> > >
>>> > > another for hbase:meta. If so, then it would be argument for ROOT as
>>> > >
>>> > > standalone table (Others of us argued it not a concern of
>>> consequence).
>>> > >
>>> > > If ROOT access is behind a new simple API in ConnectionRegistry, how
>>> > >
>>> > > to stop clients reading hbase:meta table if not Master or fronted by
>>> > >
>>> > > a ConnectionRegistry request? (Should be able to switch on client
>>> > >
>>> > > identity/source). One suggestion for First-meta-Region-as-ROOT was
>>> > >
>>> > > NOT returning the first Region to the client post-meta split when
>>> > >
>>> > > accessing via the simple API. Some concern this would confuse old
>>> > >
>>> > > Clients (Francis was going to take a look).
>>> > >
>>> > > Moved to discussion how we'd move ConnectionRegistry from
>>> > >
>>> > > hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a
>>> > >
>>> > > system came up? Where do clients go? How do they know which
>>> > >
>>> > > RegionServers (special regionserver group?  Every RS fields
>>> > >
>>> > > ConnectionRegistry requests but only designated core serve the ROOT
>>> > >
>>> > > lookup APIs?). This was a TODO.
>>> > >
>>> > > This led naturally into 4.1.5 System RS group for client meta
>>> services
>>> > >
>>> > > [5], a new addition under Brainstorming. Discussion. Bharath to look
>>> > >
>>> > > into feedback.
>>> > >
>>> > > On the end of the discussion, group expressed support for adding
>>> > >
>>> > > simple API to the ConnectionRegistry to hide ROOT implementation
>>> > >
>>> > > detail from client. Support was expressed for moving
>>> ConnectionRegistry
>>> > >
>>> > > from Master to RegionServers. Intent is to move forward on design of
>>> > >
>>> > > these pieces: e.g. how client bootstraps.
>>> > >
>>> > > Support was expressed for getting at least the bones of a split meta
>>> > >
>>> > > into an hbase3 before the RCs.
>>> > >
>>> > > Where we'd actually store hbase:meta Region locations -- i.e. how a
>>> > >
>>> > > "ROOT' would be implemented -- was for our next meeting informed by
>>> > >
>>> > > research of the various approaches noted mostly above. It was
>>> > >
>>> > > also thought that the new ConnectionRegistry should not preclude
>>> > >
>>> > > making progress on the "ROOT" implementation.
>>> > >
>>> > > Will post notice of next meeting (next Weds or the one
>>> > >
>>> > > following).
>>> > >
>>> > > 1.
>>> > >
>>> >
>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
>>> > >
>>> > > 2.
>>> > >
>>> >
>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq
>>> > >
>>> > > 3.
>>> > >
>>> >
>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153
>>> > >
>>> > > 4.
>>> > >
>>> >
>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>>> > >
>>> > > 5.
>>> > >
>>> >
>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b
>>> > >
>>> > > On Tue, Jul 20, 2021 at 11:00 AM Stack <st...@duboce.net> wrote:
>>> > >
>>> > >> Lets meet tomorrow. Please review the design doc
>>> "Design/Brainstorming"
>>> > >> Section 4.1 [1] before the meeting if you can (No harm if a refresh
>>> of
>>> > the
>>> > >> requirements section while you are at it).
>>> > >>
>>> > >> Topic: Split Meta Design Reset Status
>>> > >> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada)
>>> > >>
>>> > >> Join Zoom Meeting
>>> > >>
>>> >
>>> https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09
>>> > >>
>>> > >> Meeting ID: 773 1892 0525
>>> > >> Passcode: hbase
>>> > >>
>>> > >> Thanks,
>>> > >> S
>>> > >>
>>> > >> 1. 1.
>>> > >>
>>> >
>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
>>> > >>
>>> > >> On Thu, Jul 8, 2021 at 1:04 PM Stack <st...@duboce.net> wrote:
>>> > >>
>>> > >>> Meeting notes (Meeting happend after I figured the zoom had a
>>> 'waiting
>>> > >>> room' -- sorry Duo)
>>> > >>>
>>> > >>> Split Meta Status Zoom Meeting
>>> > >>> Wed Jul  7, 2021 @ 5pm for ~90minutes
>>> > >>> Attendees: Duo, Francis, Stack, and Clay
>>> > >>> Agenda: Mainly talk about the one-pager design and PoC proposed in
>>> [2]
>>> > >>> and added to the
>>> > >>> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as
>>> ROOT'
>>> > >>> approach)
>>> > >>>
>>> > >>> Duo suggested no advantage treating the first meta of hbase:meta
>>> table
>>> > >>> special; as a "root"
>>> > >>> and other comments (see remarks in [2]).
>>> > >>>
>>> > >>> Some pushback. When meta is NOT split, all works as it did before
>>> (big
>>> > >>> on backward-compatible).
>>> > >>>
>>> > >>> Duo suggested just intro a ROOT table altogether rather than treat
>>> the
>>> > >>> first Region
>>> > >>> in the hbase:meta table as a 'root' and then mirror to zk the first
>>> > meta
>>> > >>> region; if no
>>> > >>> split, then old clients should just work even though now hbase
>>> cluster
>>> > >>> has a ROOT table.
>>> > >>> Discussion. If no split, what's to do, etc.?
>>> > >>>
>>> > >>> Duo expressed concern that if split-meta is not on always --
>>> enabled
>>> > >>> optionally -- then
>>> > >>> the code path will not see exercise and split-meta will likely
>>> fail and
>>> > >>> go the way of
>>> > >>> prefix tree and distributed log replay -- a feature that failed,
>>> > >>> cluttered up the
>>> > >>> code base, and only later was removed. Discussion. Was allowed this
>>> > >>> could happen.
>>> > >>> Counter examples: RegionServer Groups. A few of the attendees
>>> > >>> volunteered they need
>>> > >>> split meta for their production so would try to see it through.
>>> > >>>
>>> > >>> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1
>>> as
>>> > >>> ROOT' if we don't
>>> > >>> split; not much changes.
>>> > >>>
>>> > >>> Comment on the PoC that its all
>>> > >>>
>>> > >>>   if 'first special meta region' do this
>>> > >>>   else do something else...
>>> > >>>
>>> > >>> (all over the codebase) but it was suggested that this will be the
>>> case
>>> > >>> if ROOT table
>>> > >>> added also and argued any implementation will have this issue (if
>>> ROOT
>>> > >>> then....) and
>>> > >>> THAT ugly 'root' comparator too whether a ROOT table or the
>>> > >>>  'hbase:meta,,1 as ROOT'
>>> > >>> approach.
>>> > >>>
>>> > >>> Clay asking if meta is a bottleneck (Clay played the 'operator' and
>>> > >>> 'new-to-the-topic'
>>> > >>> roles). Some discussion around how indeed it is and why we want to
>>> > split
>>> > >>> meta at all.
>>> > >>>
>>> > >>> Clay mentioned how Master is inline for data now (Master-hosted
>>> > >>> Registry)....
>>> > >>> Discussion. Hopefully temporary state -- Registry doesn't need to
>>> be
>>> > >>> hosted on
>>> > >>> Master -- and Master will return to its background role -- soon.
>>> > >>>
>>> > >>> Clay brought up rollback after meta split. Discussion. Some
>>> agreement
>>> > it
>>> > >>> could be done for
>>> > >>> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root
>>> table
>>> > and
>>> > >>> client had
>>> > >>> started to use ROOT, it might be harder...
>>> > >>>
>>> > >>> Duo suggested that we not change the client at all; that client
>>> stays
>>> > >>> same however split
>>> > >>> meta is implemented (with addition of a root table or using
>>> > >>> hbase:meta,,1 as 'root').
>>> > >>> This sounded attractive. We discussed how this could be done in a
>>> > >>> backward compatible way;
>>> > >>> add simple location lookup API to Registry...A write-up on how this
>>> > >>> might work will be posted
>>> > >>> in next day or so for others to review (Need to figure getting
>>> Registry
>>> > >>> off Master).
>>> > >>>
>>> > >>> Clay suggested, as an operator, how he thought the split meta
>>> should
>>> > >>> roll out. One of us
>>> > >>> claimed this described the 'hbase:meta,,1 as ROOT' approach.
>>> > >>>
>>> > >>> Duo had to go to work so we called it quits and said we'd meet
>>> again
>>> > >>> same time, next week.
>>> > >>>
>>> > >>> 1.
>>> > >>>
>>> >
>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>>> > >>> 2. https://issues.apache.org/jira/browse/HBASE-25761
>>> > >>>
>>> > >>> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <
>>> palomino219@gmail.com>
>>> > >>> wrote:
>>> > >>>
>>> > >>>> Is it only me? I tried to join the meeting but it always tell me
>>> that
>>> > I
>>> > >>>> need to wait for the presenter to invite me to join...
>>> > >>>>
>>> > >>>> Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：
>>> > >>>>
>>> > >>>> > Short notice but reminder that this meeting is today at 5pm PST
>>> (We
>>> > >>>> talked
>>> > >>>> > of doing it earlier but in the end lets just keep the original
>>> 5pm
>>> > >>>> time).
>>> > >>>> > Thanks,
>>> > >>>> > S
>>> > >>>> >
>>> > >>>> > On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net> wrote:
>>> > >>>> >
>>> > >>>> > > Lets do a quick chat on current state of hbase split-meta
>>> project.
>>> > >>>> > >
>>> > >>>> > > Francis has posted a suggested one-pager design in HBASE-25761
>>> > which
>>> > >>>> > would
>>> > >>>> > > be good to review before the meeting if you can. Lets discuss
>>> this
>>> > >>>> and
>>> > >>>> > any
>>> > >>>> > > other suggestions for moving the project forward.
>>> > >>>> > >
>>> > >>>> > > Meeting details below.
>>> > >>>> > >
>>> > >>>> > > Thanks,
>>> > >>>> > > S
>>> > >>>> > >
>>> > >>>> > > Topic Split Meta Design Reset Status
>>> > >>>> > > Description Review one-pager design attached to
>>> > >>>> > > https://issues.apache.org/jira/browse/HBASE-25761
>>> > >>>> > > Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
>>> > >>>> > >
>>> > >>>> > > Meeting ID 736 3907 8852
>>> > >>>> > > Security
>>> > >>>> > > Passcode *hbase* Hide
>>> > >>>> > > Waiting Room
>>> > >>>> > > Invite Link
>>> > >>>> > >
>>> > >>>> >
>>> > >>>>
>>> >
>>> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
>>> > >>>> > Copy
>>> > >>>> > > Invitation
>>> > >>>> > >
>>> > >>>> > > On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net>
>>> wrote:
>>> > >>>> > >
>>> > >>>> > >> Now the requirements are in [1], we're going to move to the
>>> next
>>> > >>>> stage
>>> > >>>> > --
>>> > >>>> > >> actual design for split-meta -- and have set up a chat for
>>> this
>>> > >>>> thursday
>>> > >>>> > >> afternoon (4PM California time/8AM Beijing time) to get the
>>> ball
>>> > >>>> > rolling.
>>> > >>>> > >> Please come if interested. Zoom details are below.
>>> > >>>> > >>
>>> > >>>> > >> Yours,
>>> > >>>> > >> S
>>> > >>>> > >> 1.
>>> > >>>> > >>
>>> > >>>> >
>>> > >>>>
>>> >
>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
>>> > >>>> > >>
>>> > >>>> > >>
>>> > >>>> > >> Topic: hbase split-meta design warmup chat
>>> > >>>> > >> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
>>> > >>>> > >>
>>> > >>>> > >> Join Zoom Meeting
>>> > >>>> > >>
>>> > >>>> >
>>> > >>>>
>>> >
>>> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
>>> > >>>> > >>
>>> > >>>> > >> Meeting ID: 759 8800 3798
>>> > >>>> > >> Passcode: hbase
>>> > >>>> > >>
>>> > >>>> > >>
>>> > >>>> > >> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net>
>>> wrote:
>>> > >>>> > >>
>>> > >>>> > >>> FYI, a few of us have been working on the redo/reset of the
>>> > split
>>> > >>>> meta
>>> > >>>> > >>> design (HBASE-25382). We (think we've) finished the
>>> > requirements.
>>> > >>>> Are
>>> > >>>> > there
>>> > >>>> > >>> any others to consider?
>>> > >>>> > >>>
>>> > >>>> > >>> Feedback and contribs welcome. Otherwise, on to the next
>>> phase
>>> > --
>>> > >>>> > design.
>>> > >>>> > >>>
>>> > >>>> > >>> Thanks,
>>> > >>>> > >>> S
>>> > >>>> > >>>
>>> > >>>> > >>
>>> > >>>> >
>>> > >>>>
>>> > >>>
>>> >
>>>
>>

Re: Split Meta Design Reset Status

[Stack <st...@duboce.net>]

Meeting tomorrow afternoon @ 5pm (to accommodate Beijing time). Lets
continue our Split Meta Design Reset discussion.

Time: Aug 11, 2021 05:00 PM Pacific Time (US and Canada)

Join Zoom Meeting
https://us04web.zoom.us/j/73900142989?pwd=b2s1UmJ2a2hiUFRsRERYQzZBV0NHdz09

Meeting ID: 739 0014 2989
Passcode: hbase

Please review the design doc "Design/Brainstorming" Section 4.1 [1] before
the meeting.

Topics for discussion:

* Reports on progress since last meeting.
* How will we actually implement the ROOT table (as a distinct ROOT table,
as the first region of hbase:meta, etc.)

1.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle

Thanks,
S


On Tue, Aug 3, 2021 at 4:13 PM Stack <st...@duboce.net> wrote:

> Any time Yu Li!
>
> No meeting tomorrow... Lets meet next week, the 10th.
>
> Thanks,
> S
>
> On Wed, Jul 28, 2021 at 11:25 PM Yu Li <ca...@gmail.com> wrote:
>
>> Thanks for the notes and efforts Stack, it's pretty helpful to know the
>> progress and latest status of the work!
>>
>> Best Regards,
>> Yu
>>
>>
>> On Wed, 28 Jul 2021 at 12:13, Stack <st...@duboce.net> wrote:
>>
>> > Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of
>> > August.
>> > Thanks,
>> > S
>> >
>> > On Thu, Jul 22, 2021 at 8:53 PM Stack <st...@duboce.net> wrote:
>> >
>> > > Notes from yesterday's meeting (attendees, please amend if I
>> misrepresent
>> > > or if you have anything extra to add!)
>> > >
>> > > Split Meta Design Reset Status
>> > >
>> > > Wed Jul 21 21:24:38 PDT 2021
>> > >
>> > > Attendees: Bharath, Stack, Duo, and Francis
>> > >
>> > > We went over the new updates to the Brainstorming [1] section under
>> > >
>> > > Design in the Super Split Meta Design doc [2].
>> > >
>> > > First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry;
>> > hide
>> > >
>> > > ROOT from Client [3]. In particular, filling out how "ROOT" might be
>> > >
>> > > implemented behind the new API in ConnectionRegistry. On option 1.,
>> > >
>> > > replicating master-local Region to RegionServers, options considered
>> > >
>> > > included
>> > >
>> > >  * Listener on master-local Region WAL to replicate.
>> > >
>> > >  * Perhaps Read-Replica but master-local is not an actual Region
>> > >
>> > >  * Needs to be incremental edits because ROOT could get too big to
>> ship
>> > >
>> > >    in a lump; need to visit how...
>> > >
>> > >  * Possibly in-memory-only Regions on RS replicated from master-local
>> > >
>> > >    Region via WAL tailing <= zhangduo@apache.org
>> > >
>> > >  * Which RegionServers? Those hosting ROOT replicas?
>> > >
>> > >  * How to bootstrap? Failure scenarios.
>> > >
>> > >  * This would be a new replication system alongside current; could
>> > >
>> > >    evolve to replace/improve old?
>> > >
>> > > Duo offered to look into means of replicating the master-local Region
>> > >
>> > > out to RegionServers.
>> > >
>> > > Next up was discussion constrasting ROOT as a standalone table vs
>> > >
>> > > First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e.
>> > >
>> > > options 2 and 3 for how we'd implement a ROOT. One item that came up
>> > >
>> > > was whether a need to specify one replica count for a ROOT table vs
>> > >
>> > > another for hbase:meta. If so, then it would be argument for ROOT as
>> > >
>> > > standalone table (Others of us argued it not a concern of
>> consequence).
>> > >
>> > > If ROOT access is behind a new simple API in ConnectionRegistry, how
>> > >
>> > > to stop clients reading hbase:meta table if not Master or fronted by
>> > >
>> > > a ConnectionRegistry request? (Should be able to switch on client
>> > >
>> > > identity/source). One suggestion for First-meta-Region-as-ROOT was
>> > >
>> > > NOT returning the first Region to the client post-meta split when
>> > >
>> > > accessing via the simple API. Some concern this would confuse old
>> > >
>> > > Clients (Francis was going to take a look).
>> > >
>> > > Moved to discussion how we'd move ConnectionRegistry from
>> > >
>> > > hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a
>> > >
>> > > system came up? Where do clients go? How do they know which
>> > >
>> > > RegionServers (special regionserver group?  Every RS fields
>> > >
>> > > ConnectionRegistry requests but only designated core serve the ROOT
>> > >
>> > > lookup APIs?). This was a TODO.
>> > >
>> > > This led naturally into 4.1.5 System RS group for client meta services
>> > >
>> > > [5], a new addition under Brainstorming. Discussion. Bharath to look
>> > >
>> > > into feedback.
>> > >
>> > > On the end of the discussion, group expressed support for adding
>> > >
>> > > simple API to the ConnectionRegistry to hide ROOT implementation
>> > >
>> > > detail from client. Support was expressed for moving
>> ConnectionRegistry
>> > >
>> > > from Master to RegionServers. Intent is to move forward on design of
>> > >
>> > > these pieces: e.g. how client bootstraps.
>> > >
>> > > Support was expressed for getting at least the bones of a split meta
>> > >
>> > > into an hbase3 before the RCs.
>> > >
>> > > Where we'd actually store hbase:meta Region locations -- i.e. how a
>> > >
>> > > "ROOT' would be implemented -- was for our next meeting informed by
>> > >
>> > > research of the various approaches noted mostly above. It was
>> > >
>> > > also thought that the new ConnectionRegistry should not preclude
>> > >
>> > > making progress on the "ROOT" implementation.
>> > >
>> > > Will post notice of next meeting (next Weds or the one
>> > >
>> > > following).
>> > >
>> > > 1.
>> > >
>> >
>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
>> > >
>> > > 2.
>> > >
>> >
>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq
>> > >
>> > > 3.
>> > >
>> >
>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153
>> > >
>> > > 4.
>> > >
>> >
>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>> > >
>> > > 5.
>> > >
>> >
>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b
>> > >
>> > > On Tue, Jul 20, 2021 at 11:00 AM Stack <st...@duboce.net> wrote:
>> > >
>> > >> Lets meet tomorrow. Please review the design doc
>> "Design/Brainstorming"
>> > >> Section 4.1 [1] before the meeting if you can (No harm if a refresh
>> of
>> > the
>> > >> requirements section while you are at it).
>> > >>
>> > >> Topic: Split Meta Design Reset Status
>> > >> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada)
>> > >>
>> > >> Join Zoom Meeting
>> > >>
>> >
>> https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09
>> > >>
>> > >> Meeting ID: 773 1892 0525
>> > >> Passcode: hbase
>> > >>
>> > >> Thanks,
>> > >> S
>> > >>
>> > >> 1. 1.
>> > >>
>> >
>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
>> > >>
>> > >> On Thu, Jul 8, 2021 at 1:04 PM Stack <st...@duboce.net> wrote:
>> > >>
>> > >>> Meeting notes (Meeting happend after I figured the zoom had a
>> 'waiting
>> > >>> room' -- sorry Duo)
>> > >>>
>> > >>> Split Meta Status Zoom Meeting
>> > >>> Wed Jul  7, 2021 @ 5pm for ~90minutes
>> > >>> Attendees: Duo, Francis, Stack, and Clay
>> > >>> Agenda: Mainly talk about the one-pager design and PoC proposed in
>> [2]
>> > >>> and added to the
>> > >>> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as
>> ROOT'
>> > >>> approach)
>> > >>>
>> > >>> Duo suggested no advantage treating the first meta of hbase:meta
>> table
>> > >>> special; as a "root"
>> > >>> and other comments (see remarks in [2]).
>> > >>>
>> > >>> Some pushback. When meta is NOT split, all works as it did before
>> (big
>> > >>> on backward-compatible).
>> > >>>
>> > >>> Duo suggested just intro a ROOT table altogether rather than treat
>> the
>> > >>> first Region
>> > >>> in the hbase:meta table as a 'root' and then mirror to zk the first
>> > meta
>> > >>> region; if no
>> > >>> split, then old clients should just work even though now hbase
>> cluster
>> > >>> has a ROOT table.
>> > >>> Discussion. If no split, what's to do, etc.?
>> > >>>
>> > >>> Duo expressed concern that if split-meta is not on always -- enabled
>> > >>> optionally -- then
>> > >>> the code path will not see exercise and split-meta will likely fail
>> and
>> > >>> go the way of
>> > >>> prefix tree and distributed log replay -- a feature that failed,
>> > >>> cluttered up the
>> > >>> code base, and only later was removed. Discussion. Was allowed this
>> > >>> could happen.
>> > >>> Counter examples: RegionServer Groups. A few of the attendees
>> > >>> volunteered they need
>> > >>> split meta for their production so would try to see it through.
>> > >>>
>> > >>> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1 as
>> > >>> ROOT' if we don't
>> > >>> split; not much changes.
>> > >>>
>> > >>> Comment on the PoC that its all
>> > >>>
>> > >>>   if 'first special meta region' do this
>> > >>>   else do something else...
>> > >>>
>> > >>> (all over the codebase) but it was suggested that this will be the
>> case
>> > >>> if ROOT table
>> > >>> added also and argued any implementation will have this issue (if
>> ROOT
>> > >>> then....) and
>> > >>> THAT ugly 'root' comparator too whether a ROOT table or the
>> > >>>  'hbase:meta,,1 as ROOT'
>> > >>> approach.
>> > >>>
>> > >>> Clay asking if meta is a bottleneck (Clay played the 'operator' and
>> > >>> 'new-to-the-topic'
>> > >>> roles). Some discussion around how indeed it is and why we want to
>> > split
>> > >>> meta at all.
>> > >>>
>> > >>> Clay mentioned how Master is inline for data now (Master-hosted
>> > >>> Registry)....
>> > >>> Discussion. Hopefully temporary state -- Registry doesn't need to be
>> > >>> hosted on
>> > >>> Master -- and Master will return to its background role -- soon.
>> > >>>
>> > >>> Clay brought up rollback after meta split. Discussion. Some
>> agreement
>> > it
>> > >>> could be done for
>> > >>> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root table
>> > and
>> > >>> client had
>> > >>> started to use ROOT, it might be harder...
>> > >>>
>> > >>> Duo suggested that we not change the client at all; that client
>> stays
>> > >>> same however split
>> > >>> meta is implemented (with addition of a root table or using
>> > >>> hbase:meta,,1 as 'root').
>> > >>> This sounded attractive. We discussed how this could be done in a
>> > >>> backward compatible way;
>> > >>> add simple location lookup API to Registry...A write-up on how this
>> > >>> might work will be posted
>> > >>> in next day or so for others to review (Need to figure getting
>> Registry
>> > >>> off Master).
>> > >>>
>> > >>> Clay suggested, as an operator, how he thought the split meta should
>> > >>> roll out. One of us
>> > >>> claimed this described the 'hbase:meta,,1 as ROOT' approach.
>> > >>>
>> > >>> Duo had to go to work so we called it quits and said we'd meet again
>> > >>> same time, next week.
>> > >>>
>> > >>> 1.
>> > >>>
>> >
>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>> > >>> 2. https://issues.apache.org/jira/browse/HBASE-25761
>> > >>>
>> > >>> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <palomino219@gmail.com
>> >
>> > >>> wrote:
>> > >>>
>> > >>>> Is it only me? I tried to join the meeting but it always tell me
>> that
>> > I
>> > >>>> need to wait for the presenter to invite me to join...
>> > >>>>
>> > >>>> Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：
>> > >>>>
>> > >>>> > Short notice but reminder that this meeting is today at 5pm PST
>> (We
>> > >>>> talked
>> > >>>> > of doing it earlier but in the end lets just keep the original
>> 5pm
>> > >>>> time).
>> > >>>> > Thanks,
>> > >>>> > S
>> > >>>> >
>> > >>>> > On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net> wrote:
>> > >>>> >
>> > >>>> > > Lets do a quick chat on current state of hbase split-meta
>> project.
>> > >>>> > >
>> > >>>> > > Francis has posted a suggested one-pager design in HBASE-25761
>> > which
>> > >>>> > would
>> > >>>> > > be good to review before the meeting if you can. Lets discuss
>> this
>> > >>>> and
>> > >>>> > any
>> > >>>> > > other suggestions for moving the project forward.
>> > >>>> > >
>> > >>>> > > Meeting details below.
>> > >>>> > >
>> > >>>> > > Thanks,
>> > >>>> > > S
>> > >>>> > >
>> > >>>> > > Topic Split Meta Design Reset Status
>> > >>>> > > Description Review one-pager design attached to
>> > >>>> > > https://issues.apache.org/jira/browse/HBASE-25761
>> > >>>> > > Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
>> > >>>> > >
>> > >>>> > > Meeting ID 736 3907 8852
>> > >>>> > > Security
>> > >>>> > > Passcode *hbase* Hide
>> > >>>> > > Waiting Room
>> > >>>> > > Invite Link
>> > >>>> > >
>> > >>>> >
>> > >>>>
>> >
>> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
>> > >>>> > Copy
>> > >>>> > > Invitation
>> > >>>> > >
>> > >>>> > > On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net>
>> wrote:
>> > >>>> > >
>> > >>>> > >> Now the requirements are in [1], we're going to move to the
>> next
>> > >>>> stage
>> > >>>> > --
>> > >>>> > >> actual design for split-meta -- and have set up a chat for
>> this
>> > >>>> thursday
>> > >>>> > >> afternoon (4PM California time/8AM Beijing time) to get the
>> ball
>> > >>>> > rolling.
>> > >>>> > >> Please come if interested. Zoom details are below.
>> > >>>> > >>
>> > >>>> > >> Yours,
>> > >>>> > >> S
>> > >>>> > >> 1.
>> > >>>> > >>
>> > >>>> >
>> > >>>>
>> >
>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
>> > >>>> > >>
>> > >>>> > >>
>> > >>>> > >> Topic: hbase split-meta design warmup chat
>> > >>>> > >> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
>> > >>>> > >>
>> > >>>> > >> Join Zoom Meeting
>> > >>>> > >>
>> > >>>> >
>> > >>>>
>> >
>> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
>> > >>>> > >>
>> > >>>> > >> Meeting ID: 759 8800 3798
>> > >>>> > >> Passcode: hbase
>> > >>>> > >>
>> > >>>> > >>
>> > >>>> > >> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net>
>> wrote:
>> > >>>> > >>
>> > >>>> > >>> FYI, a few of us have been working on the redo/reset of the
>> > split
>> > >>>> meta
>> > >>>> > >>> design (HBASE-25382). We (think we've) finished the
>> > requirements.
>> > >>>> Are
>> > >>>> > there
>> > >>>> > >>> any others to consider?
>> > >>>> > >>>
>> > >>>> > >>> Feedback and contribs welcome. Otherwise, on to the next
>> phase
>> > --
>> > >>>> > design.
>> > >>>> > >>>
>> > >>>> > >>> Thanks,
>> > >>>> > >>> S
>> > >>>> > >>>
>> > >>>> > >>
>> > >>>> >
>> > >>>>
>> > >>>
>> >
>>
>

Re: Split Meta Design Reset Status

[Stack <st...@duboce.net>]

Any time Yu Li!

No meeting tomorrow... Lets meet next week, the 10th.

Thanks,
S

On Wed, Jul 28, 2021 at 11:25 PM Yu Li <ca...@gmail.com> wrote:

> Thanks for the notes and efforts Stack, it's pretty helpful to know the
> progress and latest status of the work!
>
> Best Regards,
> Yu
>
>
> On Wed, 28 Jul 2021 at 12:13, Stack <st...@duboce.net> wrote:
>
> > Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of
> > August.
> > Thanks,
> > S
> >
> > On Thu, Jul 22, 2021 at 8:53 PM Stack <st...@duboce.net> wrote:
> >
> > > Notes from yesterday's meeting (attendees, please amend if I
> misrepresent
> > > or if you have anything extra to add!)
> > >
> > > Split Meta Design Reset Status
> > >
> > > Wed Jul 21 21:24:38 PDT 2021
> > >
> > > Attendees: Bharath, Stack, Duo, and Francis
> > >
> > > We went over the new updates to the Brainstorming [1] section under
> > >
> > > Design in the Super Split Meta Design doc [2].
> > >
> > > First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry;
> > hide
> > >
> > > ROOT from Client [3]. In particular, filling out how "ROOT" might be
> > >
> > > implemented behind the new API in ConnectionRegistry. On option 1.,
> > >
> > > replicating master-local Region to RegionServers, options considered
> > >
> > > included
> > >
> > >  * Listener on master-local Region WAL to replicate.
> > >
> > >  * Perhaps Read-Replica but master-local is not an actual Region
> > >
> > >  * Needs to be incremental edits because ROOT could get too big to ship
> > >
> > >    in a lump; need to visit how...
> > >
> > >  * Possibly in-memory-only Regions on RS replicated from master-local
> > >
> > >    Region via WAL tailing <= zhangduo@apache.org
> > >
> > >  * Which RegionServers? Those hosting ROOT replicas?
> > >
> > >  * How to bootstrap? Failure scenarios.
> > >
> > >  * This would be a new replication system alongside current; could
> > >
> > >    evolve to replace/improve old?
> > >
> > > Duo offered to look into means of replicating the master-local Region
> > >
> > > out to RegionServers.
> > >
> > > Next up was discussion constrasting ROOT as a standalone table vs
> > >
> > > First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e.
> > >
> > > options 2 and 3 for how we'd implement a ROOT. One item that came up
> > >
> > > was whether a need to specify one replica count for a ROOT table vs
> > >
> > > another for hbase:meta. If so, then it would be argument for ROOT as
> > >
> > > standalone table (Others of us argued it not a concern of consequence).
> > >
> > > If ROOT access is behind a new simple API in ConnectionRegistry, how
> > >
> > > to stop clients reading hbase:meta table if not Master or fronted by
> > >
> > > a ConnectionRegistry request? (Should be able to switch on client
> > >
> > > identity/source). One suggestion for First-meta-Region-as-ROOT was
> > >
> > > NOT returning the first Region to the client post-meta split when
> > >
> > > accessing via the simple API. Some concern this would confuse old
> > >
> > > Clients (Francis was going to take a look).
> > >
> > > Moved to discussion how we'd move ConnectionRegistry from
> > >
> > > hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a
> > >
> > > system came up? Where do clients go? How do they know which
> > >
> > > RegionServers (special regionserver group?  Every RS fields
> > >
> > > ConnectionRegistry requests but only designated core serve the ROOT
> > >
> > > lookup APIs?). This was a TODO.
> > >
> > > This led naturally into 4.1.5 System RS group for client meta services
> > >
> > > [5], a new addition under Brainstorming. Discussion. Bharath to look
> > >
> > > into feedback.
> > >
> > > On the end of the discussion, group expressed support for adding
> > >
> > > simple API to the ConnectionRegistry to hide ROOT implementation
> > >
> > > detail from client. Support was expressed for moving ConnectionRegistry
> > >
> > > from Master to RegionServers. Intent is to move forward on design of
> > >
> > > these pieces: e.g. how client bootstraps.
> > >
> > > Support was expressed for getting at least the bones of a split meta
> > >
> > > into an hbase3 before the RCs.
> > >
> > > Where we'd actually store hbase:meta Region locations -- i.e. how a
> > >
> > > "ROOT' would be implemented -- was for our next meeting informed by
> > >
> > > research of the various approaches noted mostly above. It was
> > >
> > > also thought that the new ConnectionRegistry should not preclude
> > >
> > > making progress on the "ROOT" implementation.
> > >
> > > Will post notice of next meeting (next Weds or the one
> > >
> > > following).
> > >
> > > 1.
> > >
> >
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
> > >
> > > 2.
> > >
> >
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq
> > >
> > > 3.
> > >
> >
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153
> > >
> > > 4.
> > >
> >
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> > >
> > > 5.
> > >
> >
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b
> > >
> > > On Tue, Jul 20, 2021 at 11:00 AM Stack <st...@duboce.net> wrote:
> > >
> > >> Lets meet tomorrow. Please review the design doc
> "Design/Brainstorming"
> > >> Section 4.1 [1] before the meeting if you can (No harm if a refresh of
> > the
> > >> requirements section while you are at it).
> > >>
> > >> Topic: Split Meta Design Reset Status
> > >> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada)
> > >>
> > >> Join Zoom Meeting
> > >>
> >
> https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09
> > >>
> > >> Meeting ID: 773 1892 0525
> > >> Passcode: hbase
> > >>
> > >> Thanks,
> > >> S
> > >>
> > >> 1. 1.
> > >>
> >
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
> > >>
> > >> On Thu, Jul 8, 2021 at 1:04 PM Stack <st...@duboce.net> wrote:
> > >>
> > >>> Meeting notes (Meeting happend after I figured the zoom had a
> 'waiting
> > >>> room' -- sorry Duo)
> > >>>
> > >>> Split Meta Status Zoom Meeting
> > >>> Wed Jul  7, 2021 @ 5pm for ~90minutes
> > >>> Attendees: Duo, Francis, Stack, and Clay
> > >>> Agenda: Mainly talk about the one-pager design and PoC proposed in
> [2]
> > >>> and added to the
> > >>> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as
> ROOT'
> > >>> approach)
> > >>>
> > >>> Duo suggested no advantage treating the first meta of hbase:meta
> table
> > >>> special; as a "root"
> > >>> and other comments (see remarks in [2]).
> > >>>
> > >>> Some pushback. When meta is NOT split, all works as it did before
> (big
> > >>> on backward-compatible).
> > >>>
> > >>> Duo suggested just intro a ROOT table altogether rather than treat
> the
> > >>> first Region
> > >>> in the hbase:meta table as a 'root' and then mirror to zk the first
> > meta
> > >>> region; if no
> > >>> split, then old clients should just work even though now hbase
> cluster
> > >>> has a ROOT table.
> > >>> Discussion. If no split, what's to do, etc.?
> > >>>
> > >>> Duo expressed concern that if split-meta is not on always -- enabled
> > >>> optionally -- then
> > >>> the code path will not see exercise and split-meta will likely fail
> and
> > >>> go the way of
> > >>> prefix tree and distributed log replay -- a feature that failed,
> > >>> cluttered up the
> > >>> code base, and only later was removed. Discussion. Was allowed this
> > >>> could happen.
> > >>> Counter examples: RegionServer Groups. A few of the attendees
> > >>> volunteered they need
> > >>> split meta for their production so would try to see it through.
> > >>>
> > >>> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1 as
> > >>> ROOT' if we don't
> > >>> split; not much changes.
> > >>>
> > >>> Comment on the PoC that its all
> > >>>
> > >>>   if 'first special meta region' do this
> > >>>   else do something else...
> > >>>
> > >>> (all over the codebase) but it was suggested that this will be the
> case
> > >>> if ROOT table
> > >>> added also and argued any implementation will have this issue (if
> ROOT
> > >>> then....) and
> > >>> THAT ugly 'root' comparator too whether a ROOT table or the
> > >>>  'hbase:meta,,1 as ROOT'
> > >>> approach.
> > >>>
> > >>> Clay asking if meta is a bottleneck (Clay played the 'operator' and
> > >>> 'new-to-the-topic'
> > >>> roles). Some discussion around how indeed it is and why we want to
> > split
> > >>> meta at all.
> > >>>
> > >>> Clay mentioned how Master is inline for data now (Master-hosted
> > >>> Registry)....
> > >>> Discussion. Hopefully temporary state -- Registry doesn't need to be
> > >>> hosted on
> > >>> Master -- and Master will return to its background role -- soon.
> > >>>
> > >>> Clay brought up rollback after meta split. Discussion. Some agreement
> > it
> > >>> could be done for
> > >>> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root table
> > and
> > >>> client had
> > >>> started to use ROOT, it might be harder...
> > >>>
> > >>> Duo suggested that we not change the client at all; that client stays
> > >>> same however split
> > >>> meta is implemented (with addition of a root table or using
> > >>> hbase:meta,,1 as 'root').
> > >>> This sounded attractive. We discussed how this could be done in a
> > >>> backward compatible way;
> > >>> add simple location lookup API to Registry...A write-up on how this
> > >>> might work will be posted
> > >>> in next day or so for others to review (Need to figure getting
> Registry
> > >>> off Master).
> > >>>
> > >>> Clay suggested, as an operator, how he thought the split meta should
> > >>> roll out. One of us
> > >>> claimed this described the 'hbase:meta,,1 as ROOT' approach.
> > >>>
> > >>> Duo had to go to work so we called it quits and said we'd meet again
> > >>> same time, next week.
> > >>>
> > >>> 1.
> > >>>
> >
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> > >>> 2. https://issues.apache.org/jira/browse/HBASE-25761
> > >>>
> > >>> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <pa...@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Is it only me? I tried to join the meeting but it always tell me
> that
> > I
> > >>>> need to wait for the presenter to invite me to join...
> > >>>>
> > >>>> Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：
> > >>>>
> > >>>> > Short notice but reminder that this meeting is today at 5pm PST
> (We
> > >>>> talked
> > >>>> > of doing it earlier but in the end lets just keep the original 5pm
> > >>>> time).
> > >>>> > Thanks,
> > >>>> > S
> > >>>> >
> > >>>> > On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net> wrote:
> > >>>> >
> > >>>> > > Lets do a quick chat on current state of hbase split-meta
> project.
> > >>>> > >
> > >>>> > > Francis has posted a suggested one-pager design in HBASE-25761
> > which
> > >>>> > would
> > >>>> > > be good to review before the meeting if you can. Lets discuss
> this
> > >>>> and
> > >>>> > any
> > >>>> > > other suggestions for moving the project forward.
> > >>>> > >
> > >>>> > > Meeting details below.
> > >>>> > >
> > >>>> > > Thanks,
> > >>>> > > S
> > >>>> > >
> > >>>> > > Topic Split Meta Design Reset Status
> > >>>> > > Description Review one-pager design attached to
> > >>>> > > https://issues.apache.org/jira/browse/HBASE-25761
> > >>>> > > Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
> > >>>> > >
> > >>>> > > Meeting ID 736 3907 8852
> > >>>> > > Security
> > >>>> > > Passcode *hbase* Hide
> > >>>> > > Waiting Room
> > >>>> > > Invite Link
> > >>>> > >
> > >>>> >
> > >>>>
> >
> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
> > >>>> > Copy
> > >>>> > > Invitation
> > >>>> > >
> > >>>> > > On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net> wrote:
> > >>>> > >
> > >>>> > >> Now the requirements are in [1], we're going to move to the
> next
> > >>>> stage
> > >>>> > --
> > >>>> > >> actual design for split-meta -- and have set up a chat for this
> > >>>> thursday
> > >>>> > >> afternoon (4PM California time/8AM Beijing time) to get the
> ball
> > >>>> > rolling.
> > >>>> > >> Please come if interested. Zoom details are below.
> > >>>> > >>
> > >>>> > >> Yours,
> > >>>> > >> S
> > >>>> > >> 1.
> > >>>> > >>
> > >>>> >
> > >>>>
> >
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
> > >>>> > >>
> > >>>> > >>
> > >>>> > >> Topic: hbase split-meta design warmup chat
> > >>>> > >> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
> > >>>> > >>
> > >>>> > >> Join Zoom Meeting
> > >>>> > >>
> > >>>> >
> > >>>>
> >
> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
> > >>>> > >>
> > >>>> > >> Meeting ID: 759 8800 3798
> > >>>> > >> Passcode: hbase
> > >>>> > >>
> > >>>> > >>
> > >>>> > >> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net> wrote:
> > >>>> > >>
> > >>>> > >>> FYI, a few of us have been working on the redo/reset of the
> > split
> > >>>> meta
> > >>>> > >>> design (HBASE-25382). We (think we've) finished the
> > requirements.
> > >>>> Are
> > >>>> > there
> > >>>> > >>> any others to consider?
> > >>>> > >>>
> > >>>> > >>> Feedback and contribs welcome. Otherwise, on to the next phase
> > --
> > >>>> > design.
> > >>>> > >>>
> > >>>> > >>> Thanks,
> > >>>> > >>> S
> > >>>> > >>>
> > >>>> > >>
> > >>>> >
> > >>>>
> > >>>
> >
>

Re: Split Meta Design Reset Status

[Yu Li <ca...@gmail.com>]

Thanks for the notes and efforts Stack, it's pretty helpful to know the
progress and latest status of the work!

Best Regards,
Yu


On Wed, 28 Jul 2021 at 12:13, Stack <st...@duboce.net> wrote:

> Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of
> August.
> Thanks,
> S
>
> On Thu, Jul 22, 2021 at 8:53 PM Stack <st...@duboce.net> wrote:
>
> > Notes from yesterday's meeting (attendees, please amend if I misrepresent
> > or if you have anything extra to add!)
> >
> > Split Meta Design Reset Status
> >
> > Wed Jul 21 21:24:38 PDT 2021
> >
> > Attendees: Bharath, Stack, Duo, and Francis
> >
> > We went over the new updates to the Brainstorming [1] section under
> >
> > Design in the Super Split Meta Design doc [2].
> >
> > First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry;
> hide
> >
> > ROOT from Client [3]. In particular, filling out how "ROOT" might be
> >
> > implemented behind the new API in ConnectionRegistry. On option 1.,
> >
> > replicating master-local Region to RegionServers, options considered
> >
> > included
> >
> >  * Listener on master-local Region WAL to replicate.
> >
> >  * Perhaps Read-Replica but master-local is not an actual Region
> >
> >  * Needs to be incremental edits because ROOT could get too big to ship
> >
> >    in a lump; need to visit how...
> >
> >  * Possibly in-memory-only Regions on RS replicated from master-local
> >
> >    Region via WAL tailing <= zhangduo@apache.org
> >
> >  * Which RegionServers? Those hosting ROOT replicas?
> >
> >  * How to bootstrap? Failure scenarios.
> >
> >  * This would be a new replication system alongside current; could
> >
> >    evolve to replace/improve old?
> >
> > Duo offered to look into means of replicating the master-local Region
> >
> > out to RegionServers.
> >
> > Next up was discussion constrasting ROOT as a standalone table vs
> >
> > First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e.
> >
> > options 2 and 3 for how we'd implement a ROOT. One item that came up
> >
> > was whether a need to specify one replica count for a ROOT table vs
> >
> > another for hbase:meta. If so, then it would be argument for ROOT as
> >
> > standalone table (Others of us argued it not a concern of consequence).
> >
> > If ROOT access is behind a new simple API in ConnectionRegistry, how
> >
> > to stop clients reading hbase:meta table if not Master or fronted by
> >
> > a ConnectionRegistry request? (Should be able to switch on client
> >
> > identity/source). One suggestion for First-meta-Region-as-ROOT was
> >
> > NOT returning the first Region to the client post-meta split when
> >
> > accessing via the simple API. Some concern this would confuse old
> >
> > Clients (Francis was going to take a look).
> >
> > Moved to discussion how we'd move ConnectionRegistry from
> >
> > hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a
> >
> > system came up? Where do clients go? How do they know which
> >
> > RegionServers (special regionserver group?  Every RS fields
> >
> > ConnectionRegistry requests but only designated core serve the ROOT
> >
> > lookup APIs?). This was a TODO.
> >
> > This led naturally into 4.1.5 System RS group for client meta services
> >
> > [5], a new addition under Brainstorming. Discussion. Bharath to look
> >
> > into feedback.
> >
> > On the end of the discussion, group expressed support for adding
> >
> > simple API to the ConnectionRegistry to hide ROOT implementation
> >
> > detail from client. Support was expressed for moving ConnectionRegistry
> >
> > from Master to RegionServers. Intent is to move forward on design of
> >
> > these pieces: e.g. how client bootstraps.
> >
> > Support was expressed for getting at least the bones of a split meta
> >
> > into an hbase3 before the RCs.
> >
> > Where we'd actually store hbase:meta Region locations -- i.e. how a
> >
> > "ROOT' would be implemented -- was for our next meeting informed by
> >
> > research of the various approaches noted mostly above. It was
> >
> > also thought that the new ConnectionRegistry should not preclude
> >
> > making progress on the "ROOT" implementation.
> >
> > Will post notice of next meeting (next Weds or the one
> >
> > following).
> >
> > 1.
> >
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
> >
> > 2.
> >
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq
> >
> > 3.
> >
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153
> >
> > 4.
> >
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> >
> > 5.
> >
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b
> >
> > On Tue, Jul 20, 2021 at 11:00 AM Stack <st...@duboce.net> wrote:
> >
> >> Lets meet tomorrow. Please review the design doc "Design/Brainstorming"
> >> Section 4.1 [1] before the meeting if you can (No harm if a refresh of
> the
> >> requirements section while you are at it).
> >>
> >> Topic: Split Meta Design Reset Status
> >> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada)
> >>
> >> Join Zoom Meeting
> >>
> https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09
> >>
> >> Meeting ID: 773 1892 0525
> >> Passcode: hbase
> >>
> >> Thanks,
> >> S
> >>
> >> 1. 1.
> >>
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
> >>
> >> On Thu, Jul 8, 2021 at 1:04 PM Stack <st...@duboce.net> wrote:
> >>
> >>> Meeting notes (Meeting happend after I figured the zoom had a 'waiting
> >>> room' -- sorry Duo)
> >>>
> >>> Split Meta Status Zoom Meeting
> >>> Wed Jul  7, 2021 @ 5pm for ~90minutes
> >>> Attendees: Duo, Francis, Stack, and Clay
> >>> Agenda: Mainly talk about the one-pager design and PoC proposed in [2]
> >>> and added to the
> >>> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as ROOT'
> >>> approach)
> >>>
> >>> Duo suggested no advantage treating the first meta of hbase:meta table
> >>> special; as a "root"
> >>> and other comments (see remarks in [2]).
> >>>
> >>> Some pushback. When meta is NOT split, all works as it did before (big
> >>> on backward-compatible).
> >>>
> >>> Duo suggested just intro a ROOT table altogether rather than treat the
> >>> first Region
> >>> in the hbase:meta table as a 'root' and then mirror to zk the first
> meta
> >>> region; if no
> >>> split, then old clients should just work even though now hbase cluster
> >>> has a ROOT table.
> >>> Discussion. If no split, what's to do, etc.?
> >>>
> >>> Duo expressed concern that if split-meta is not on always -- enabled
> >>> optionally -- then
> >>> the code path will not see exercise and split-meta will likely fail and
> >>> go the way of
> >>> prefix tree and distributed log replay -- a feature that failed,
> >>> cluttered up the
> >>> code base, and only later was removed. Discussion. Was allowed this
> >>> could happen.
> >>> Counter examples: RegionServer Groups. A few of the attendees
> >>> volunteered they need
> >>> split meta for their production so would try to see it through.
> >>>
> >>> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1 as
> >>> ROOT' if we don't
> >>> split; not much changes.
> >>>
> >>> Comment on the PoC that its all
> >>>
> >>>   if 'first special meta region' do this
> >>>   else do something else...
> >>>
> >>> (all over the codebase) but it was suggested that this will be the case
> >>> if ROOT table
> >>> added also and argued any implementation will have this issue (if ROOT
> >>> then....) and
> >>> THAT ugly 'root' comparator too whether a ROOT table or the
> >>>  'hbase:meta,,1 as ROOT'
> >>> approach.
> >>>
> >>> Clay asking if meta is a bottleneck (Clay played the 'operator' and
> >>> 'new-to-the-topic'
> >>> roles). Some discussion around how indeed it is and why we want to
> split
> >>> meta at all.
> >>>
> >>> Clay mentioned how Master is inline for data now (Master-hosted
> >>> Registry)....
> >>> Discussion. Hopefully temporary state -- Registry doesn't need to be
> >>> hosted on
> >>> Master -- and Master will return to its background role -- soon.
> >>>
> >>> Clay brought up rollback after meta split. Discussion. Some agreement
> it
> >>> could be done for
> >>> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root table
> and
> >>> client had
> >>> started to use ROOT, it might be harder...
> >>>
> >>> Duo suggested that we not change the client at all; that client stays
> >>> same however split
> >>> meta is implemented (with addition of a root table or using
> >>> hbase:meta,,1 as 'root').
> >>> This sounded attractive. We discussed how this could be done in a
> >>> backward compatible way;
> >>> add simple location lookup API to Registry...A write-up on how this
> >>> might work will be posted
> >>> in next day or so for others to review (Need to figure getting Registry
> >>> off Master).
> >>>
> >>> Clay suggested, as an operator, how he thought the split meta should
> >>> roll out. One of us
> >>> claimed this described the 'hbase:meta,,1 as ROOT' approach.
> >>>
> >>> Duo had to go to work so we called it quits and said we'd meet again
> >>> same time, next week.
> >>>
> >>> 1.
> >>>
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> >>> 2. https://issues.apache.org/jira/browse/HBASE-25761
> >>>
> >>> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <pa...@gmail.com>
> >>> wrote:
> >>>
> >>>> Is it only me? I tried to join the meeting but it always tell me that
> I
> >>>> need to wait for the presenter to invite me to join...
> >>>>
> >>>> Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：
> >>>>
> >>>> > Short notice but reminder that this meeting is today at 5pm PST (We
> >>>> talked
> >>>> > of doing it earlier but in the end lets just keep the original 5pm
> >>>> time).
> >>>> > Thanks,
> >>>> > S
> >>>> >
> >>>> > On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net> wrote:
> >>>> >
> >>>> > > Lets do a quick chat on current state of hbase split-meta project.
> >>>> > >
> >>>> > > Francis has posted a suggested one-pager design in HBASE-25761
> which
> >>>> > would
> >>>> > > be good to review before the meeting if you can. Lets discuss this
> >>>> and
> >>>> > any
> >>>> > > other suggestions for moving the project forward.
> >>>> > >
> >>>> > > Meeting details below.
> >>>> > >
> >>>> > > Thanks,
> >>>> > > S
> >>>> > >
> >>>> > > Topic Split Meta Design Reset Status
> >>>> > > Description Review one-pager design attached to
> >>>> > > https://issues.apache.org/jira/browse/HBASE-25761
> >>>> > > Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
> >>>> > >
> >>>> > > Meeting ID 736 3907 8852
> >>>> > > Security
> >>>> > > Passcode *hbase* Hide
> >>>> > > Waiting Room
> >>>> > > Invite Link
> >>>> > >
> >>>> >
> >>>>
> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
> >>>> > Copy
> >>>> > > Invitation
> >>>> > >
> >>>> > > On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net> wrote:
> >>>> > >
> >>>> > >> Now the requirements are in [1], we're going to move to the next
> >>>> stage
> >>>> > --
> >>>> > >> actual design for split-meta -- and have set up a chat for this
> >>>> thursday
> >>>> > >> afternoon (4PM California time/8AM Beijing time) to get the ball
> >>>> > rolling.
> >>>> > >> Please come if interested. Zoom details are below.
> >>>> > >>
> >>>> > >> Yours,
> >>>> > >> S
> >>>> > >> 1.
> >>>> > >>
> >>>> >
> >>>>
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
> >>>> > >>
> >>>> > >>
> >>>> > >> Topic: hbase split-meta design warmup chat
> >>>> > >> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
> >>>> > >>
> >>>> > >> Join Zoom Meeting
> >>>> > >>
> >>>> >
> >>>>
> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
> >>>> > >>
> >>>> > >> Meeting ID: 759 8800 3798
> >>>> > >> Passcode: hbase
> >>>> > >>
> >>>> > >>
> >>>> > >> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net> wrote:
> >>>> > >>
> >>>> > >>> FYI, a few of us have been working on the redo/reset of the
> split
> >>>> meta
> >>>> > >>> design (HBASE-25382). We (think we've) finished the
> requirements.
> >>>> Are
> >>>> > there
> >>>> > >>> any others to consider?
> >>>> > >>>
> >>>> > >>> Feedback and contribs welcome. Otherwise, on to the next phase
> --
> >>>> > design.
> >>>> > >>>
> >>>> > >>> Thanks,
> >>>> > >>> S
> >>>> > >>>
> >>>> > >>
> >>>> >
> >>>>
> >>>
>

Re: Split Meta Design Reset Status

[Stack <st...@duboce.net>]

Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of August.
Thanks,
S

On Thu, Jul 22, 2021 at 8:53 PM Stack <st...@duboce.net> wrote:

> Notes from yesterday's meeting (attendees, please amend if I misrepresent
> or if you have anything extra to add!)
>
> Split Meta Design Reset Status
>
> Wed Jul 21 21:24:38 PDT 2021
>
> Attendees: Bharath, Stack, Duo, and Francis
>
> We went over the new updates to the Brainstorming [1] section under
>
> Design in the Super Split Meta Design doc [2].
>
> First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry; hide
>
> ROOT from Client [3]. In particular, filling out how "ROOT" might be
>
> implemented behind the new API in ConnectionRegistry. On option 1.,
>
> replicating master-local Region to RegionServers, options considered
>
> included
>
>  * Listener on master-local Region WAL to replicate.
>
>  * Perhaps Read-Replica but master-local is not an actual Region
>
>  * Needs to be incremental edits because ROOT could get too big to ship
>
>    in a lump; need to visit how...
>
>  * Possibly in-memory-only Regions on RS replicated from master-local
>
>    Region via WAL tailing <= zhangduo@apache.org
>
>  * Which RegionServers? Those hosting ROOT replicas?
>
>  * How to bootstrap? Failure scenarios.
>
>  * This would be a new replication system alongside current; could
>
>    evolve to replace/improve old?
>
> Duo offered to look into means of replicating the master-local Region
>
> out to RegionServers.
>
> Next up was discussion constrasting ROOT as a standalone table vs
>
> First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e.
>
> options 2 and 3 for how we'd implement a ROOT. One item that came up
>
> was whether a need to specify one replica count for a ROOT table vs
>
> another for hbase:meta. If so, then it would be argument for ROOT as
>
> standalone table (Others of us argued it not a concern of consequence).
>
> If ROOT access is behind a new simple API in ConnectionRegistry, how
>
> to stop clients reading hbase:meta table if not Master or fronted by
>
> a ConnectionRegistry request? (Should be able to switch on client
>
> identity/source). One suggestion for First-meta-Region-as-ROOT was
>
> NOT returning the first Region to the client post-meta split when
>
> accessing via the simple API. Some concern this would confuse old
>
> Clients (Francis was going to take a look).
>
> Moved to discussion how we'd move ConnectionRegistry from
>
> hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a
>
> system came up? Where do clients go? How do they know which
>
> RegionServers (special regionserver group?  Every RS fields
>
> ConnectionRegistry requests but only designated core serve the ROOT
>
> lookup APIs?). This was a TODO.
>
> This led naturally into 4.1.5 System RS group for client meta services
>
> [5], a new addition under Brainstorming. Discussion. Bharath to look
>
> into feedback.
>
> On the end of the discussion, group expressed support for adding
>
> simple API to the ConnectionRegistry to hide ROOT implementation
>
> detail from client. Support was expressed for moving ConnectionRegistry
>
> from Master to RegionServers. Intent is to move forward on design of
>
> these pieces: e.g. how client bootstraps.
>
> Support was expressed for getting at least the bones of a split meta
>
> into an hbase3 before the RCs.
>
> Where we'd actually store hbase:meta Region locations -- i.e. how a
>
> "ROOT' would be implemented -- was for our next meeting informed by
>
> research of the various approaches noted mostly above. It was
>
> also thought that the new ConnectionRegistry should not preclude
>
> making progress on the "ROOT" implementation.
>
> Will post notice of next meeting (next Weds or the one
>
> following).
>
> 1.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
>
> 2.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq
>
> 3.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153
>
> 4.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>
> 5.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b
>
> On Tue, Jul 20, 2021 at 11:00 AM Stack <st...@duboce.net> wrote:
>
>> Lets meet tomorrow. Please review the design doc "Design/Brainstorming"
>> Section 4.1 [1] before the meeting if you can (No harm if a refresh of the
>> requirements section while you are at it).
>>
>> Topic: Split Meta Design Reset Status
>> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada)
>>
>> Join Zoom Meeting
>> https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09
>>
>> Meeting ID: 773 1892 0525
>> Passcode: hbase
>>
>> Thanks,
>> S
>>
>> 1. 1.
>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
>>
>> On Thu, Jul 8, 2021 at 1:04 PM Stack <st...@duboce.net> wrote:
>>
>>> Meeting notes (Meeting happend after I figured the zoom had a 'waiting
>>> room' -- sorry Duo)
>>>
>>> Split Meta Status Zoom Meeting
>>> Wed Jul  7, 2021 @ 5pm for ~90minutes
>>> Attendees: Duo, Francis, Stack, and Clay
>>> Agenda: Mainly talk about the one-pager design and PoC proposed in [2]
>>> and added to the
>>> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as ROOT'
>>> approach)
>>>
>>> Duo suggested no advantage treating the first meta of hbase:meta table
>>> special; as a "root"
>>> and other comments (see remarks in [2]).
>>>
>>> Some pushback. When meta is NOT split, all works as it did before (big
>>> on backward-compatible).
>>>
>>> Duo suggested just intro a ROOT table altogether rather than treat the
>>> first Region
>>> in the hbase:meta table as a 'root' and then mirror to zk the first meta
>>> region; if no
>>> split, then old clients should just work even though now hbase cluster
>>> has a ROOT table.
>>> Discussion. If no split, what's to do, etc.?
>>>
>>> Duo expressed concern that if split-meta is not on always -- enabled
>>> optionally -- then
>>> the code path will not see exercise and split-meta will likely fail and
>>> go the way of
>>> prefix tree and distributed log replay -- a feature that failed,
>>> cluttered up the
>>> code base, and only later was removed. Discussion. Was allowed this
>>> could happen.
>>> Counter examples: RegionServer Groups. A few of the attendees
>>> volunteered they need
>>> split meta for their production so would try to see it through.
>>>
>>> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1 as
>>> ROOT' if we don't
>>> split; not much changes.
>>>
>>> Comment on the PoC that its all
>>>
>>>   if 'first special meta region' do this
>>>   else do something else...
>>>
>>> (all over the codebase) but it was suggested that this will be the case
>>> if ROOT table
>>> added also and argued any implementation will have this issue (if ROOT
>>> then....) and
>>> THAT ugly 'root' comparator too whether a ROOT table or the
>>>  'hbase:meta,,1 as ROOT'
>>> approach.
>>>
>>> Clay asking if meta is a bottleneck (Clay played the 'operator' and
>>> 'new-to-the-topic'
>>> roles). Some discussion around how indeed it is and why we want to split
>>> meta at all.
>>>
>>> Clay mentioned how Master is inline for data now (Master-hosted
>>> Registry)....
>>> Discussion. Hopefully temporary state -- Registry doesn't need to be
>>> hosted on
>>> Master -- and Master will return to its background role -- soon.
>>>
>>> Clay brought up rollback after meta split. Discussion. Some agreement it
>>> could be done for
>>> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root table and
>>> client had
>>> started to use ROOT, it might be harder...
>>>
>>> Duo suggested that we not change the client at all; that client stays
>>> same however split
>>> meta is implemented (with addition of a root table or using
>>> hbase:meta,,1 as 'root').
>>> This sounded attractive. We discussed how this could be done in a
>>> backward compatible way;
>>> add simple location lookup API to Registry...A write-up on how this
>>> might work will be posted
>>> in next day or so for others to review (Need to figure getting Registry
>>> off Master).
>>>
>>> Clay suggested, as an operator, how he thought the split meta should
>>> roll out. One of us
>>> claimed this described the 'hbase:meta,,1 as ROOT' approach.
>>>
>>> Duo had to go to work so we called it quits and said we'd meet again
>>> same time, next week.
>>>
>>> 1.
>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>>> 2. https://issues.apache.org/jira/browse/HBASE-25761
>>>
>>> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <pa...@gmail.com>
>>> wrote:
>>>
>>>> Is it only me? I tried to join the meeting but it always tell me that I
>>>> need to wait for the presenter to invite me to join...
>>>>
>>>> Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：
>>>>
>>>> > Short notice but reminder that this meeting is today at 5pm PST (We
>>>> talked
>>>> > of doing it earlier but in the end lets just keep the original 5pm
>>>> time).
>>>> > Thanks,
>>>> > S
>>>> >
>>>> > On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net> wrote:
>>>> >
>>>> > > Lets do a quick chat on current state of hbase split-meta project.
>>>> > >
>>>> > > Francis has posted a suggested one-pager design in HBASE-25761 which
>>>> > would
>>>> > > be good to review before the meeting if you can. Lets discuss this
>>>> and
>>>> > any
>>>> > > other suggestions for moving the project forward.
>>>> > >
>>>> > > Meeting details below.
>>>> > >
>>>> > > Thanks,
>>>> > > S
>>>> > >
>>>> > > Topic Split Meta Design Reset Status
>>>> > > Description Review one-pager design attached to
>>>> > > https://issues.apache.org/jira/browse/HBASE-25761
>>>> > > Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
>>>> > >
>>>> > > Meeting ID 736 3907 8852
>>>> > > Security
>>>> > > Passcode *hbase* Hide
>>>> > > Waiting Room
>>>> > > Invite Link
>>>> > >
>>>> >
>>>> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
>>>> > Copy
>>>> > > Invitation
>>>> > >
>>>> > > On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net> wrote:
>>>> > >
>>>> > >> Now the requirements are in [1], we're going to move to the next
>>>> stage
>>>> > --
>>>> > >> actual design for split-meta -- and have set up a chat for this
>>>> thursday
>>>> > >> afternoon (4PM California time/8AM Beijing time) to get the ball
>>>> > rolling.
>>>> > >> Please come if interested. Zoom details are below.
>>>> > >>
>>>> > >> Yours,
>>>> > >> S
>>>> > >> 1.
>>>> > >>
>>>> >
>>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
>>>> > >>
>>>> > >>
>>>> > >> Topic: hbase split-meta design warmup chat
>>>> > >> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
>>>> > >>
>>>> > >> Join Zoom Meeting
>>>> > >>
>>>> >
>>>> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
>>>> > >>
>>>> > >> Meeting ID: 759 8800 3798
>>>> > >> Passcode: hbase
>>>> > >>
>>>> > >>
>>>> > >> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net> wrote:
>>>> > >>
>>>> > >>> FYI, a few of us have been working on the redo/reset of the split
>>>> meta
>>>> > >>> design (HBASE-25382). We (think we've) finished the requirements.
>>>> Are
>>>> > there
>>>> > >>> any others to consider?
>>>> > >>>
>>>> > >>> Feedback and contribs welcome. Otherwise, on to the next phase --
>>>> > design.
>>>> > >>>
>>>> > >>> Thanks,
>>>> > >>> S
>>>> > >>>
>>>> > >>
>>>> >
>>>>
>>>

Re: Split Meta Design Reset Status

[Stack <st...@duboce.net>]

Notes from yesterday's meeting (attendees, please amend if I misrepresent
or if you have anything extra to add!)

Split Meta Design Reset Status

Wed Jul 21 21:24:38 PDT 2021

Attendees: Bharath, Stack, Duo, and Francis

We went over the new updates to the Brainstorming [1] section under

Design in the Super Split Meta Design doc [2].

First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry; hide

ROOT from Client [3]. In particular, filling out how "ROOT" might be

implemented behind the new API in ConnectionRegistry. On option 1.,

replicating master-local Region to RegionServers, options considered

included

 * Listener on master-local Region WAL to replicate.

 * Perhaps Read-Replica but master-local is not an actual Region

 * Needs to be incremental edits because ROOT could get too big to ship

   in a lump; need to visit how...

 * Possibly in-memory-only Regions on RS replicated from master-local

   Region via WAL tailing <= zhangduo@apache.org

 * Which RegionServers? Those hosting ROOT replicas?

 * How to bootstrap? Failure scenarios.

 * This would be a new replication system alongside current; could

   evolve to replace/improve old?

Duo offered to look into means of replicating the master-local Region

out to RegionServers.

Next up was discussion constrasting ROOT as a standalone table vs

First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e.

options 2 and 3 for how we'd implement a ROOT. One item that came up

was whether a need to specify one replica count for a ROOT table vs

another for hbase:meta. If so, then it would be argument for ROOT as

standalone table (Others of us argued it not a concern of consequence).

If ROOT access is behind a new simple API in ConnectionRegistry, how

to stop clients reading hbase:meta table if not Master or fronted by

a ConnectionRegistry request? (Should be able to switch on client

identity/source). One suggestion for First-meta-Region-as-ROOT was

NOT returning the first Region to the client post-meta split when

accessing via the simple API. Some concern this would confuse old

Clients (Francis was going to take a look).

Moved to discussion how we'd move ConnectionRegistry from

hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a

system came up? Where do clients go? How do they know which

RegionServers (special regionserver group?  Every RS fields

ConnectionRegistry requests but only designated core serve the ROOT

lookup APIs?). This was a TODO.

This led naturally into 4.1.5 System RS group for client meta services

[5], a new addition under Brainstorming. Discussion. Bharath to look

into feedback.

On the end of the discussion, group expressed support for adding

simple API to the ConnectionRegistry to hide ROOT implementation

detail from client. Support was expressed for moving ConnectionRegistry

from Master to RegionServers. Intent is to move forward on design of

these pieces: e.g. how client bootstraps.

Support was expressed for getting at least the bones of a split meta

into an hbase3 before the RCs.

Where we'd actually store hbase:meta Region locations -- i.e. how a

"ROOT' would be implemented -- was for our next meeting informed by

research of the various approaches noted mostly above. It was

also thought that the new ConnectionRegistry should not preclude

making progress on the "ROOT" implementation.

Will post notice of next meeting (next Weds or the one

following).

1.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n

2.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq

3.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153

4.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle

5.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b

On Tue, Jul 20, 2021 at 11:00 AM Stack <st...@duboce.net> wrote:

> Lets meet tomorrow. Please review the design doc "Design/Brainstorming"
> Section 4.1 [1] before the meeting if you can (No harm if a refresh of the
> requirements section while you are at it).
>
> Topic: Split Meta Design Reset Status
> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada)
>
> Join Zoom Meeting
> https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09
>
> Meeting ID: 773 1892 0525
> Passcode: hbase
>
> Thanks,
> S
>
> 1. 1.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
>
> On Thu, Jul 8, 2021 at 1:04 PM Stack <st...@duboce.net> wrote:
>
>> Meeting notes (Meeting happend after I figured the zoom had a 'waiting
>> room' -- sorry Duo)
>>
>> Split Meta Status Zoom Meeting
>> Wed Jul  7, 2021 @ 5pm for ~90minutes
>> Attendees: Duo, Francis, Stack, and Clay
>> Agenda: Mainly talk about the one-pager design and PoC proposed in [2]
>> and added to the
>> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as ROOT'
>> approach)
>>
>> Duo suggested no advantage treating the first meta of hbase:meta table
>> special; as a "root"
>> and other comments (see remarks in [2]).
>>
>> Some pushback. When meta is NOT split, all works as it did before (big
>> on backward-compatible).
>>
>> Duo suggested just intro a ROOT table altogether rather than treat the
>> first Region
>> in the hbase:meta table as a 'root' and then mirror to zk the first meta
>> region; if no
>> split, then old clients should just work even though now hbase cluster
>> has a ROOT table.
>> Discussion. If no split, what's to do, etc.?
>>
>> Duo expressed concern that if split-meta is not on always -- enabled
>> optionally -- then
>> the code path will not see exercise and split-meta will likely fail and
>> go the way of
>> prefix tree and distributed log replay -- a feature that failed,
>> cluttered up the
>> code base, and only later was removed. Discussion. Was allowed this could
>> happen.
>> Counter examples: RegionServer Groups. A few of the attendees volunteered
>> they need
>> split meta for their production so would try to see it through.
>>
>> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1 as
>> ROOT' if we don't
>> split; not much changes.
>>
>> Comment on the PoC that its all
>>
>>   if 'first special meta region' do this
>>   else do something else...
>>
>> (all over the codebase) but it was suggested that this will be the case
>> if ROOT table
>> added also and argued any implementation will have this issue (if ROOT
>> then....) and
>> THAT ugly 'root' comparator too whether a ROOT table or the
>>  'hbase:meta,,1 as ROOT'
>> approach.
>>
>> Clay asking if meta is a bottleneck (Clay played the 'operator' and
>> 'new-to-the-topic'
>> roles). Some discussion around how indeed it is and why we want to split
>> meta at all.
>>
>> Clay mentioned how Master is inline for data now (Master-hosted
>> Registry)....
>> Discussion. Hopefully temporary state -- Registry doesn't need to be
>> hosted on
>> Master -- and Master will return to its background role -- soon.
>>
>> Clay brought up rollback after meta split. Discussion. Some agreement it
>> could be done for
>> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root table and
>> client had
>> started to use ROOT, it might be harder...
>>
>> Duo suggested that we not change the client at all; that client stays
>> same however split
>> meta is implemented (with addition of a root table or using hbase:meta,,1
>> as 'root').
>> This sounded attractive. We discussed how this could be done in a
>> backward compatible way;
>> add simple location lookup API to Registry...A write-up on how this might
>> work will be posted
>> in next day or so for others to review (Need to figure getting Registry
>> off Master).
>>
>> Clay suggested, as an operator, how he thought the split meta should roll
>> out. One of us
>> claimed this described the 'hbase:meta,,1 as ROOT' approach.
>>
>> Duo had to go to work so we called it quits and said we'd meet again same
>> time, next week.
>>
>> 1.
>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>> 2. https://issues.apache.org/jira/browse/HBASE-25761
>>
>> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <pa...@gmail.com>
>> wrote:
>>
>>> Is it only me? I tried to join the meeting but it always tell me that I
>>> need to wait for the presenter to invite me to join...
>>>
>>> Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：
>>>
>>> > Short notice but reminder that this meeting is today at 5pm PST (We
>>> talked
>>> > of doing it earlier but in the end lets just keep the original 5pm
>>> time).
>>> > Thanks,
>>> > S
>>> >
>>> > On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net> wrote:
>>> >
>>> > > Lets do a quick chat on current state of hbase split-meta project.
>>> > >
>>> > > Francis has posted a suggested one-pager design in HBASE-25761 which
>>> > would
>>> > > be good to review before the meeting if you can. Lets discuss this
>>> and
>>> > any
>>> > > other suggestions for moving the project forward.
>>> > >
>>> > > Meeting details below.
>>> > >
>>> > > Thanks,
>>> > > S
>>> > >
>>> > > Topic Split Meta Design Reset Status
>>> > > Description Review one-pager design attached to
>>> > > https://issues.apache.org/jira/browse/HBASE-25761
>>> > > Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
>>> > >
>>> > > Meeting ID 736 3907 8852
>>> > > Security
>>> > > Passcode *hbase* Hide
>>> > > Waiting Room
>>> > > Invite Link
>>> > >
>>> >
>>> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
>>> > Copy
>>> > > Invitation
>>> > >
>>> > > On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net> wrote:
>>> > >
>>> > >> Now the requirements are in [1], we're going to move to the next
>>> stage
>>> > --
>>> > >> actual design for split-meta -- and have set up a chat for this
>>> thursday
>>> > >> afternoon (4PM California time/8AM Beijing time) to get the ball
>>> > rolling.
>>> > >> Please come if interested. Zoom details are below.
>>> > >>
>>> > >> Yours,
>>> > >> S
>>> > >> 1.
>>> > >>
>>> >
>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
>>> > >>
>>> > >>
>>> > >> Topic: hbase split-meta design warmup chat
>>> > >> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
>>> > >>
>>> > >> Join Zoom Meeting
>>> > >>
>>> >
>>> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
>>> > >>
>>> > >> Meeting ID: 759 8800 3798
>>> > >> Passcode: hbase
>>> > >>
>>> > >>
>>> > >> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net> wrote:
>>> > >>
>>> > >>> FYI, a few of us have been working on the redo/reset of the split
>>> meta
>>> > >>> design (HBASE-25382). We (think we've) finished the requirements.
>>> Are
>>> > there
>>> > >>> any others to consider?
>>> > >>>
>>> > >>> Feedback and contribs welcome. Otherwise, on to the next phase --
>>> > design.
>>> > >>>
>>> > >>> Thanks,
>>> > >>> S
>>> > >>>
>>> > >>
>>> >
>>>
>>

Re: Split Meta Design Reset Status

[Stack <st...@duboce.net>]

Lets meet tomorrow. Please review the design doc "Design/Brainstorming"
Section 4.1 [1] before the meeting if you can (No harm if a refresh of the
requirements section while you are at it).

Topic: Split Meta Design Reset Status
Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada)

Join Zoom Meeting
https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09

Meeting ID: 773 1892 0525
Passcode: hbase

Thanks,
S

1. 1.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n

On Thu, Jul 8, 2021 at 1:04 PM Stack <st...@duboce.net> wrote:

> Meeting notes (Meeting happend after I figured the zoom had a 'waiting
> room' -- sorry Duo)
>
> Split Meta Status Zoom Meeting
> Wed Jul  7, 2021 @ 5pm for ~90minutes
> Attendees: Duo, Francis, Stack, and Clay
> Agenda: Mainly talk about the one-pager design and PoC proposed in [2] and
> added to the
> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as ROOT'
> approach)
>
> Duo suggested no advantage treating the first meta of hbase:meta table
> special; as a "root"
> and other comments (see remarks in [2]).
>
> Some pushback. When meta is NOT split, all works as it did before (big
> on backward-compatible).
>
> Duo suggested just intro a ROOT table altogether rather than treat the
> first Region
> in the hbase:meta table as a 'root' and then mirror to zk the first meta
> region; if no
> split, then old clients should just work even though now hbase cluster has
> a ROOT table.
> Discussion. If no split, what's to do, etc.?
>
> Duo expressed concern that if split-meta is not on always -- enabled
> optionally -- then
> the code path will not see exercise and split-meta will likely fail and go
> the way of
> prefix tree and distributed log replay -- a feature that failed, cluttered
> up the
> code base, and only later was removed. Discussion. Was allowed this could
> happen.
> Counter examples: RegionServer Groups. A few of the attendees volunteered
> they need
> split meta for their production so would try to see it through.
>
> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1 as ROOT'
> if we don't
> split; not much changes.
>
> Comment on the PoC that its all
>
>   if 'first special meta region' do this
>   else do something else...
>
> (all over the codebase) but it was suggested that this will be the case if
> ROOT table
> added also and argued any implementation will have this issue (if ROOT
> then....) and
> THAT ugly 'root' comparator too whether a ROOT table or the
>  'hbase:meta,,1 as ROOT'
> approach.
>
> Clay asking if meta is a bottleneck (Clay played the 'operator' and
> 'new-to-the-topic'
> roles). Some discussion around how indeed it is and why we want to split
> meta at all.
>
> Clay mentioned how Master is inline for data now (Master-hosted
> Registry)....
> Discussion. Hopefully temporary state -- Registry doesn't need to be
> hosted on
> Master -- and Master will return to its background role -- soon.
>
> Clay brought up rollback after meta split. Discussion. Some agreement it
> could be done for
> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root table and
> client had
> started to use ROOT, it might be harder...
>
> Duo suggested that we not change the client at all; that client stays same
> however split
> meta is implemented (with addition of a root table or using hbase:meta,,1
> as 'root').
> This sounded attractive. We discussed how this could be done in a backward
> compatible way;
> add simple location lookup API to Registry...A write-up on how this might
> work will be posted
> in next day or so for others to review (Need to figure getting Registry
> off Master).
>
> Clay suggested, as an operator, how he thought the split meta should roll
> out. One of us
> claimed this described the 'hbase:meta,,1 as ROOT' approach.
>
> Duo had to go to work so we called it quits and said we'd meet again same
> time, next week.
>
> 1.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> 2. https://issues.apache.org/jira/browse/HBASE-25761
>
> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
>> Is it only me? I tried to join the meeting but it always tell me that I
>> need to wait for the presenter to invite me to join...
>>
>> Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：
>>
>> > Short notice but reminder that this meeting is today at 5pm PST (We
>> talked
>> > of doing it earlier but in the end lets just keep the original 5pm
>> time).
>> > Thanks,
>> > S
>> >
>> > On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net> wrote:
>> >
>> > > Lets do a quick chat on current state of hbase split-meta project.
>> > >
>> > > Francis has posted a suggested one-pager design in HBASE-25761 which
>> > would
>> > > be good to review before the meeting if you can. Lets discuss this and
>> > any
>> > > other suggestions for moving the project forward.
>> > >
>> > > Meeting details below.
>> > >
>> > > Thanks,
>> > > S
>> > >
>> > > Topic Split Meta Design Reset Status
>> > > Description Review one-pager design attached to
>> > > https://issues.apache.org/jira/browse/HBASE-25761
>> > > Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
>> > >
>> > > Meeting ID 736 3907 8852
>> > > Security
>> > > Passcode *hbase* Hide
>> > > Waiting Room
>> > > Invite Link
>> > >
>> >
>> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
>> > Copy
>> > > Invitation
>> > >
>> > > On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net> wrote:
>> > >
>> > >> Now the requirements are in [1], we're going to move to the next
>> stage
>> > --
>> > >> actual design for split-meta -- and have set up a chat for this
>> thursday
>> > >> afternoon (4PM California time/8AM Beijing time) to get the ball
>> > rolling.
>> > >> Please come if interested. Zoom details are below.
>> > >>
>> > >> Yours,
>> > >> S
>> > >> 1.
>> > >>
>> >
>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
>> > >>
>> > >>
>> > >> Topic: hbase split-meta design warmup chat
>> > >> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
>> > >>
>> > >> Join Zoom Meeting
>> > >>
>> >
>> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
>> > >>
>> > >> Meeting ID: 759 8800 3798
>> > >> Passcode: hbase
>> > >>
>> > >>
>> > >> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net> wrote:
>> > >>
>> > >>> FYI, a few of us have been working on the redo/reset of the split
>> meta
>> > >>> design (HBASE-25382). We (think we've) finished the requirements.
>> Are
>> > there
>> > >>> any others to consider?
>> > >>>
>> > >>> Feedback and contribs welcome. Otherwise, on to the next phase --
>> > design.
>> > >>>
>> > >>> Thanks,
>> > >>> S
>> > >>>
>> > >>
>> >
>>
>

Re: Split Meta Design Reset Status

[Stack <st...@duboce.net>]

No meeting tomorrow (A few of us can't make it). Lets do it next week.
Meantime, some updates on the design doc under brainstorming. Check it out.
Feedback/opinions/input appreciated.
S

On Thu, Jul 8, 2021 at 1:04 PM Stack <st...@duboce.net> wrote:

> Meeting notes (Meeting happend after I figured the zoom had a 'waiting
> room' -- sorry Duo)
>
> Split Meta Status Zoom Meeting
> Wed Jul  7, 2021 @ 5pm for ~90minutes
> Attendees: Duo, Francis, Stack, and Clay
> Agenda: Mainly talk about the one-pager design and PoC proposed in [2] and
> added to the
> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as ROOT'
> approach)
>
> Duo suggested no advantage treating the first meta of hbase:meta table
> special; as a "root"
> and other comments (see remarks in [2]).
>
> Some pushback. When meta is NOT split, all works as it did before (big
> on backward-compatible).
>
> Duo suggested just intro a ROOT table altogether rather than treat the
> first Region
> in the hbase:meta table as a 'root' and then mirror to zk the first meta
> region; if no
> split, then old clients should just work even though now hbase cluster has
> a ROOT table.
> Discussion. If no split, what's to do, etc.?
>
> Duo expressed concern that if split-meta is not on always -- enabled
> optionally -- then
> the code path will not see exercise and split-meta will likely fail and go
> the way of
> prefix tree and distributed log replay -- a feature that failed, cluttered
> up the
> code base, and only later was removed. Discussion. Was allowed this could
> happen.
> Counter examples: RegionServer Groups. A few of the attendees volunteered
> they need
> split meta for their production so would try to see it through.
>
> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1 as ROOT'
> if we don't
> split; not much changes.
>
> Comment on the PoC that its all
>
>   if 'first special meta region' do this
>   else do something else...
>
> (all over the codebase) but it was suggested that this will be the case if
> ROOT table
> added also and argued any implementation will have this issue (if ROOT
> then....) and
> THAT ugly 'root' comparator too whether a ROOT table or the
>  'hbase:meta,,1 as ROOT'
> approach.
>
> Clay asking if meta is a bottleneck (Clay played the 'operator' and
> 'new-to-the-topic'
> roles). Some discussion around how indeed it is and why we want to split
> meta at all.
>
> Clay mentioned how Master is inline for data now (Master-hosted
> Registry)....
> Discussion. Hopefully temporary state -- Registry doesn't need to be
> hosted on
> Master -- and Master will return to its background role -- soon.
>
> Clay brought up rollback after meta split. Discussion. Some agreement it
> could be done for
> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root table and
> client had
> started to use ROOT, it might be harder...
>
> Duo suggested that we not change the client at all; that client stays same
> however split
> meta is implemented (with addition of a root table or using hbase:meta,,1
> as 'root').
> This sounded attractive. We discussed how this could be done in a backward
> compatible way;
> add simple location lookup API to Registry...A write-up on how this might
> work will be posted
> in next day or so for others to review (Need to figure getting Registry
> off Master).
>
> Clay suggested, as an operator, how he thought the split meta should roll
> out. One of us
> claimed this described the 'hbase:meta,,1 as ROOT' approach.
>
> Duo had to go to work so we called it quits and said we'd meet again same
> time, next week.
>
> 1.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> 2. https://issues.apache.org/jira/browse/HBASE-25761
>
> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
>
>> Is it only me? I tried to join the meeting but it always tell me that I
>> need to wait for the presenter to invite me to join...
>>
>> Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：
>>
>> > Short notice but reminder that this meeting is today at 5pm PST (We
>> talked
>> > of doing it earlier but in the end lets just keep the original 5pm
>> time).
>> > Thanks,
>> > S
>> >
>> > On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net> wrote:
>> >
>> > > Lets do a quick chat on current state of hbase split-meta project.
>> > >
>> > > Francis has posted a suggested one-pager design in HBASE-25761 which
>> > would
>> > > be good to review before the meeting if you can. Lets discuss this and
>> > any
>> > > other suggestions for moving the project forward.
>> > >
>> > > Meeting details below.
>> > >
>> > > Thanks,
>> > > S
>> > >
>> > > Topic Split Meta Design Reset Status
>> > > Description Review one-pager design attached to
>> > > https://issues.apache.org/jira/browse/HBASE-25761
>> > > Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
>> > >
>> > > Meeting ID 736 3907 8852
>> > > Security
>> > > Passcode *hbase* Hide
>> > > Waiting Room
>> > > Invite Link
>> > >
>> >
>> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
>> > Copy
>> > > Invitation
>> > >
>> > > On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net> wrote:
>> > >
>> > >> Now the requirements are in [1], we're going to move to the next
>> stage
>> > --
>> > >> actual design for split-meta -- and have set up a chat for this
>> thursday
>> > >> afternoon (4PM California time/8AM Beijing time) to get the ball
>> > rolling.
>> > >> Please come if interested. Zoom details are below.
>> > >>
>> > >> Yours,
>> > >> S
>> > >> 1.
>> > >>
>> >
>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
>> > >>
>> > >>
>> > >> Topic: hbase split-meta design warmup chat
>> > >> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
>> > >>
>> > >> Join Zoom Meeting
>> > >>
>> >
>> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
>> > >>
>> > >> Meeting ID: 759 8800 3798
>> > >> Passcode: hbase
>> > >>
>> > >>
>> > >> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net> wrote:
>> > >>
>> > >>> FYI, a few of us have been working on the redo/reset of the split
>> meta
>> > >>> design (HBASE-25382). We (think we've) finished the requirements.
>> Are
>> > there
>> > >>> any others to consider?
>> > >>>
>> > >>> Feedback and contribs welcome. Otherwise, on to the next phase --
>> > design.
>> > >>>
>> > >>> Thanks,
>> > >>> S
>> > >>>
>> > >>
>> >
>>
>

Re: Split Meta Design Reset Status

[Stack <st...@duboce.net>]

Meeting notes (Meeting happend after I figured the zoom had a 'waiting
room' -- sorry Duo)

Split Meta Status Zoom Meeting
Wed Jul  7, 2021 @ 5pm for ~90minutes
Attendees: Duo, Francis, Stack, and Clay
Agenda: Mainly talk about the one-pager design and PoC proposed in [2] and
added to the
split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as ROOT'
approach)

Duo suggested no advantage treating the first meta of hbase:meta table
special; as a "root"
and other comments (see remarks in [2]).

Some pushback. When meta is NOT split, all works as it did before (big
on backward-compatible).

Duo suggested just intro a ROOT table altogether rather than treat the
first Region
in the hbase:meta table as a 'root' and then mirror to zk the first meta
region; if no
split, then old clients should just work even though now hbase cluster has
a ROOT table.
Discussion. If no split, what's to do, etc.?

Duo expressed concern that if split-meta is not on always -- enabled
optionally -- then
the code path will not see exercise and split-meta will likely fail and go
the way of
prefix tree and distributed log replay -- a feature that failed, cluttered
up the
code base, and only later was removed. Discussion. Was allowed this could
happen.
Counter examples: RegionServer Groups. A few of the attendees volunteered
they need
split meta for their production so would try to see it through.

Some back and forth. Duo allowed that merit to the 'hbase:meta,,1 as ROOT'
if we don't
split; not much changes.

Comment on the PoC that its all

  if 'first special meta region' do this
  else do something else...

(all over the codebase) but it was suggested that this will be the case if
ROOT table
added also and argued any implementation will have this issue (if ROOT
then....) and
THAT ugly 'root' comparator too whether a ROOT table or the  'hbase:meta,,1
as ROOT'
approach.

Clay asking if meta is a bottleneck (Clay played the 'operator' and
'new-to-the-topic'
roles). Some discussion around how indeed it is and why we want to split
meta at all.

Clay mentioned how Master is inline for data now (Master-hosted
Registry)....
Discussion. Hopefully temporary state -- Registry doesn't need to be hosted
on
Master -- and Master will return to its background role -- soon.

Clay brought up rollback after meta split. Discussion. Some agreement it
could be done for
'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root table and
client had
started to use ROOT, it might be harder...

Duo suggested that we not change the client at all; that client stays same
however split
meta is implemented (with addition of a root table or using hbase:meta,,1
as 'root').
This sounded attractive. We discussed how this could be done in a backward
compatible way;
add simple location lookup API to Registry...A write-up on how this might
work will be posted
in next day or so for others to review (Need to figure getting Registry off
Master).

Clay suggested, as an operator, how he thought the split meta should roll
out. One of us
claimed this described the 'hbase:meta,,1 as ROOT' approach.

Duo had to go to work so we called it quits and said we'd meet again same
time, next week.

1.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
2. https://issues.apache.org/jira/browse/HBASE-25761

On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <pa...@gmail.com> wrote:

> Is it only me? I tried to join the meeting but it always tell me that I
> need to wait for the presenter to invite me to join...
>
> Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：
>
> > Short notice but reminder that this meeting is today at 5pm PST (We
> talked
> > of doing it earlier but in the end lets just keep the original 5pm time).
> > Thanks,
> > S
> >
> > On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net> wrote:
> >
> > > Lets do a quick chat on current state of hbase split-meta project.
> > >
> > > Francis has posted a suggested one-pager design in HBASE-25761 which
> > would
> > > be good to review before the meeting if you can. Lets discuss this and
> > any
> > > other suggestions for moving the project forward.
> > >
> > > Meeting details below.
> > >
> > > Thanks,
> > > S
> > >
> > > Topic Split Meta Design Reset Status
> > > Description Review one-pager design attached to
> > > https://issues.apache.org/jira/browse/HBASE-25761
> > > Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
> > >
> > > Meeting ID 736 3907 8852
> > > Security
> > > Passcode *hbase* Hide
> > > Waiting Room
> > > Invite Link
> > >
> >
> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
> > Copy
> > > Invitation
> > >
> > > On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net> wrote:
> > >
> > >> Now the requirements are in [1], we're going to move to the next stage
> > --
> > >> actual design for split-meta -- and have set up a chat for this
> thursday
> > >> afternoon (4PM California time/8AM Beijing time) to get the ball
> > rolling.
> > >> Please come if interested. Zoom details are below.
> > >>
> > >> Yours,
> > >> S
> > >> 1.
> > >>
> >
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
> > >>
> > >>
> > >> Topic: hbase split-meta design warmup chat
> > >> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
> > >>
> > >> Join Zoom Meeting
> > >>
> >
> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
> > >>
> > >> Meeting ID: 759 8800 3798
> > >> Passcode: hbase
> > >>
> > >>
> > >> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net> wrote:
> > >>
> > >>> FYI, a few of us have been working on the redo/reset of the split
> meta
> > >>> design (HBASE-25382). We (think we've) finished the requirements. Are
> > there
> > >>> any others to consider?
> > >>>
> > >>> Feedback and contribs welcome. Otherwise, on to the next phase --
> > design.
> > >>>
> > >>> Thanks,
> > >>> S
> > >>>
> > >>
> >
>

Re: Split Meta Design Reset Status

["张铎 (Duo Zhang)" <pa...@gmail.com>]

Is it only me? I tried to join the meeting but it always tell me that I
need to wait for the presenter to invite me to join...

Stack <st...@duboce.net> 于2021年7月8日周四 上午1:04写道：

> Short notice but reminder that this meeting is today at 5pm PST (We talked
> of doing it earlier but in the end lets just keep the original 5pm time).
> Thanks,
> S
>
> On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net> wrote:
>
> > Lets do a quick chat on current state of hbase split-meta project.
> >
> > Francis has posted a suggested one-pager design in HBASE-25761 which
> would
> > be good to review before the meeting if you can. Lets discuss this and
> any
> > other suggestions for moving the project forward.
> >
> > Meeting details below.
> >
> > Thanks,
> > S
> >
> > Topic Split Meta Design Reset Status
> > Description Review one-pager design attached to
> > https://issues.apache.org/jira/browse/HBASE-25761
> > Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
> >
> > Meeting ID 736 3907 8852
> > Security
> > Passcode *hbase* Hide
> > Waiting Room
> > Invite Link
> >
> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
> Copy
> > Invitation
> >
> > On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net> wrote:
> >
> >> Now the requirements are in [1], we're going to move to the next stage
> --
> >> actual design for split-meta -- and have set up a chat for this thursday
> >> afternoon (4PM California time/8AM Beijing time) to get the ball
> rolling.
> >> Please come if interested. Zoom details are below.
> >>
> >> Yours,
> >> S
> >> 1.
> >>
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
> >>
> >>
> >> Topic: hbase split-meta design warmup chat
> >> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
> >>
> >> Join Zoom Meeting
> >>
> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
> >>
> >> Meeting ID: 759 8800 3798
> >> Passcode: hbase
> >>
> >>
> >> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net> wrote:
> >>
> >>> FYI, a few of us have been working on the redo/reset of the split meta
> >>> design (HBASE-25382). We (think we've) finished the requirements. Are
> there
> >>> any others to consider?
> >>>
> >>> Feedback and contribs welcome. Otherwise, on to the next phase --
> design.
> >>>
> >>> Thanks,
> >>> S
> >>>
> >>
>

Re: Split Meta Design Reset Status

[Stack <st...@duboce.net>]

Short notice but reminder that this meeting is today at 5pm PST (We talked
of doing it earlier but in the end lets just keep the original 5pm time).
Thanks,
S

On Tue, Jul 6, 2021 at 11:36 AM Stack <st...@duboce.net> wrote:

> Lets do a quick chat on current state of hbase split-meta project.
>
> Francis has posted a suggested one-pager design in HBASE-25761 which would
> be good to review before the meeting if you can. Lets discuss this and any
> other suggestions for moving the project forward.
>
> Meeting details below.
>
> Thanks,
> S
>
> Topic Split Meta Design Reset Status
> Description Review one-pager design attached to
> https://issues.apache.org/jira/browse/HBASE-25761
> Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
>
> Meeting ID 736 3907 8852
> Security
> Passcode *hbase* Hide
> Waiting Room
> Invite Link
> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09 Copy
> Invitation
>
> On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net> wrote:
>
>> Now the requirements are in [1], we're going to move to the next stage --
>> actual design for split-meta -- and have set up a chat for this thursday
>> afternoon (4PM California time/8AM Beijing time) to get the ball rolling.
>> Please come if interested. Zoom details are below.
>>
>> Yours,
>> S
>> 1.
>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
>>
>>
>> Topic: hbase split-meta design warmup chat
>> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
>>
>> Join Zoom Meeting
>> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
>>
>> Meeting ID: 759 8800 3798
>> Passcode: hbase
>>
>>
>> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net> wrote:
>>
>>> FYI, a few of us have been working on the redo/reset of the split meta
>>> design (HBASE-25382). We (think we've) finished the requirements. Are there
>>> any others to consider?
>>>
>>> Feedback and contribs welcome. Otherwise, on to the next phase -- design.
>>>
>>> Thanks,
>>> S
>>>
>>

Re: Split Meta Design Reset Status

[Stack <st...@duboce.net>]

 Lets do a quick chat on current state of hbase split-meta project.

Francis has posted a suggested one-pager design in HBASE-25761 which would
be good to review before the meeting if you can. Lets discuss this and any
other suggestions for moving the project forward.

Meeting details below.

Thanks,
S

Topic Split Meta Design Reset Status
Description Review one-pager design attached to
https://issues.apache.org/jira/browse/HBASE-25761
Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)

Meeting ID 736 3907 8852
Security
Passcode *hbase* Hide
Waiting Room
Invite Link
https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09 Copy
Invitation

On Mon, Mar 22, 2021 at 4:28 PM Stack <st...@duboce.net> wrote:

> Now the requirements are in [1], we're going to move to the next stage --
> actual design for split-meta -- and have set up a chat for this thursday
> afternoon (4PM California time/8AM Beijing time) to get the ball rolling.
> Please come if interested. Zoom details are below.
>
> Yours,
> S
> 1.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
>
>
> Topic: hbase split-meta design warmup chat
> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
>
> Join Zoom Meeting
> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
>
> Meeting ID: 759 8800 3798
> Passcode: hbase
>
>
> On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net> wrote:
>
>> FYI, a few of us have been working on the redo/reset of the split meta
>> design (HBASE-25382). We (think we've) finished the requirements. Are there
>> any others to consider?
>>
>> Feedback and contribs welcome. Otherwise, on to the next phase -- design.
>>
>> Thanks,
>> S
>>
>

Re: Split Meta Design Reset Status

[Stack <st...@duboce.net>]

Now the requirements are in [1], we're going to move to the next stage --
actual design for split-meta -- and have set up a chat for this thursday
afternoon (4PM California time/8AM Beijing time) to get the ball rolling.
Please come if interested. Zoom details are below.

Yours,
S
1.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2


Topic: hbase split-meta design warmup chat
Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)

Join Zoom Meeting
https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09

Meeting ID: 759 8800 3798
Passcode: hbase


On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net> wrote:

> FYI, a few of us have been working on the redo/reset of the split meta
> design (HBASE-25382). We (think we've) finished the requirements. Are there
> any others to consider?
>
> Feedback and contribs welcome. Otherwise, on to the next phase -- design.
>
> Thanks,
> S
>

Re: Split Meta Design Reset Status

[Stack <st...@duboce.net>]

No meeting tomorrow, August 17th. Lets try for next week, the 24th.
S

On Tue, Jan 5, 2021 at 9:13 AM Stack <st...@duboce.net> wrote:

> FYI, a few of us have been working on the redo/reset of the split meta
> design (HBASE-25382). We (think we've) finished the requirements. Are there
> any others to consider?
>
> Feedback and contribs welcome. Otherwise, on to the next phase -- design.
>
> Thanks,
> S
>