You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Stack <st...@duboce.net> on 2019/09/13 23:44:14 UTC
Re: [DISCUSS] Moving towards a branch-2 line that can get the
'stable' pointer.
HBASE-21745 <https://issues.apache.org/jira/browse/HBASE-21745>, the issue
addressing gaps between hbck1 and hbck2 was closed a few days back after a
bunch of work by a kaleidoscope of folks. The release notes section tries
to describe what was added by HBASE-21745. Shout if you think the claim at
the end of the release notes section that hbck2 now is on par or beyond
what hbck1 offered is problematic. Otherwise, will proceed as though it is
the case.
Suggestion: Given that hbase 2.2.1 will ship soon and hbase-operator-tools
1.0.0 with latest hbase-hbck2 should get an RC inside the next week or so,
if feedback that 2.2.1 looks good, give 2.2.2 (with bug fixes only) the
stable pointer?
Thanks,
S
On Sat, Jan 26, 2019 at 11:31 AM Stack <st...@duboce.net> wrote:
> As per Sean, bypass with optional 'force' (override) and recurse for case
> where a procedure had spawned children was the mechanism Allan implemented
> after a chat about merits of procedure delete. I found it of use doing
> fixup to clusters I'd intentionally damaged testing candidates. Procedures
> are usually part of a fabric with relations that an operator might have
> trouble unraveling. It was thought that the bypass would be safer than a
> delete, likely to cause more damage than solution.
>
> Interested in the issues you are seeing on Master branch Sergey.
>
> Thanks,
> S
>
> On Fri, Jan 25, 2019 at 1:54 PM Sean Busbey <bu...@apache.org> wrote:
>
>> that's already present, see the README for the "bypass" command:
>>
>> https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2
>>
>> On Fri, Jan 25, 2019 at 3:40 PM Sergey Shelukhin
>> <Se...@microsoft.com.invalid> wrote:
>> >
>> > I think one thing that is needed for HBCK2 for AMv2 is to be able to
>> delete single procedures from store.
>> > We are evaluating master (whose assignment is very similar to branch-2)
>> right now and I have to delete proc WAL pretty much every day because some
>> procedure(s) are in bad state, but deleting the entire WAL also causes
>> other issues.
>> > It should be possible to remove some offending procedure while master
>> is offline and/or online.
>> >
>> > -----Original Message-----
>> > From: 张铎(Duo Zhang) <pa...@gmail.com>
>> > Sent: Friday, January 18, 2019 5:52 PM
>> > To: HBase Dev List <de...@hbase.apache.org>
>> > Subject: Re: [DISCUSS] Moving towards a branch-2 line that can get the
>> 'stable' pointer.
>> >
>> > OK, the original issue is HBCK2 for AMv2, but here we need to do more,
>> not only for AMv2.
>> >
>> > Let me open a new issue and post what Andrew said above there.
>> >
>>
>
Re: [DISCUSS] Moving towards a branch-2 line that can get the
'stable' pointer.
Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.
We have made some progress, especially on HBCK2.
And we plan to cut 2.2.2 after resolving HBSE-23079, maybe this could be
the stable pointer candidate.
But anyway, I think we should have a good documentation on how to make use
HBCK2.
Not sure if this one is up to date...
https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2
OpenInx <op...@gmail.com> 于2019年9月29日周日 下午7:36写道：
> > Let's gather a few stories on it working for folks in production and move
> the pointer then.
> Agree, XiaoMi is making parts of the online clusters upgrade to
> HBase2.2.x, I think
> Guanghao will share the practices some time later.
> Thanks.
>
> On Sun, Sep 15, 2019 at 12:09 AM Stack <st...@duboce.net> wrote:
>
> > On Fri, Sep 13, 2019 at 4:58 PM Andrew Purtell <andrew.purtell@gmail.com
> >
> > wrote:
> >
> > > For what it’s worth I had previously been concerned about the disparity
> > > between hbck capability in 1.x and 2.x but after review of the recent
> > work
> > > I believe that is no longer true. Put another way, it is reasonable to
> > > claim it on par.
> > >
> > >
> > Thanks Andrew for chiming in.
> >
> >
> >
> > > As for moving the stable pointer I don’t personally have enough
> > experience
> > > with HBase 2 to weigh in but will trust the opinions of those that do.
> > >
> > >
> > >
> > Let's gather a few stories on it working for folks in production and move
> > the pointer then.
> >
> > Thanks,
> > S
> >
> >
> >
> > > > On Sep 14, 2019, at 8:44 AM, Stack <st...@duboce.net> wrote:
> > > >
> > > > HBASE-21745 <https://issues.apache.org/jira/browse/HBASE-21745>, the
> > > issue
> > > > addressing gaps between hbck1 and hbck2 was closed a few days back
> > after
> > > a
> > > > bunch of work by a kaleidoscope of folks. The release notes section
> > tries
> > > > to describe what was added by HBASE-21745. Shout if you think the
> claim
> > > at
> > > > the end of the release notes section that hbck2 now is on par or
> beyond
> > > > what hbck1 offered is problematic. Otherwise, will proceed as though
> it
> > > is
> > > > the case.
> > > >
> > > > Suggestion: Given that hbase 2.2.1 will ship soon and
> > > hbase-operator-tools
> > > > 1.0.0 with latest hbase-hbck2 should get an RC inside the next week
> or
> > > so,
> > > > if feedback that 2.2.1 looks good, give 2.2.2 (with bug fixes only)
> the
> > > > stable pointer?
> > > >
> > > > Thanks,
> > > > S
> > > >
> > > >> On Sat, Jan 26, 2019 at 11:31 AM Stack <st...@duboce.net> wrote:
> > > >>
> > > >> As per Sean, bypass with optional 'force' (override) and recurse for
> > > case
> > > >> where a procedure had spawned children was the mechanism Allan
> > > implemented
> > > >> after a chat about merits of procedure delete. I found it of use
> doing
> > > >> fixup to clusters I'd intentionally damaged testing candidates.
> > > Procedures
> > > >> are usually part of a fabric with relations that an operator might
> > have
> > > >> trouble unraveling. It was thought that the bypass would be safer
> > than a
> > > >> delete, likely to cause more damage than solution.
> > > >>
> > > >> Interested in the issues you are seeing on Master branch Sergey.
> > > >>
> > > >> Thanks,
> > > >> S
> > > >>
> > > >>> On Fri, Jan 25, 2019 at 1:54 PM Sean Busbey <bu...@apache.org>
> > wrote:
> > > >>>
> > > >>> that's already present, see the README for the "bypass" command:
> > > >>>
> > > >>>
> > https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2
> > > >>>
> > > >>> On Fri, Jan 25, 2019 at 3:40 PM Sergey Shelukhin
> > > >>> <Se...@microsoft.com.invalid> wrote:
> > > >>>>
> > > >>>> I think one thing that is needed for HBCK2 for AMv2 is to be able
> to
> > > >>> delete single procedures from store.
> > > >>>> We are evaluating master (whose assignment is very similar to
> > > branch-2)
> > > >>> right now and I have to delete proc WAL pretty much every day
> because
> > > some
> > > >>> procedure(s) are in bad state, but deleting the entire WAL also
> > causes
> > > >>> other issues.
> > > >>>> It should be possible to remove some offending procedure while
> > master
> > > >>> is offline and/or online.
> > > >>>>
> > > >>>> -----Original Message-----
> > > >>>> From: 张铎(Duo Zhang) <pa...@gmail.com>
> > > >>>> Sent: Friday, January 18, 2019 5:52 PM
> > > >>>> To: HBase Dev List <de...@hbase.apache.org>
> > > >>>> Subject: Re: [DISCUSS] Moving towards a branch-2 line that can get
> > the
> > > >>> 'stable' pointer.
> > > >>>>
> > > >>>> OK, the original issue is HBCK2 for AMv2, but here we need to do
> > more,
> > > >>> not only for AMv2.
> > > >>>>
> > > >>>> Let me open a new issue and post what Andrew said above there.
> > > >>>>
> > > >>>
> > > >>
> > >
> >
>
Re: [DISCUSS] Moving towards a branch-2 line that can get the
'stable' pointer.
Posted by OpenInx <op...@gmail.com>.
> Let's gather a few stories on it working for folks in production and move
the pointer then.
Agree, XiaoMi is making parts of the online clusters upgrade to
HBase2.2.x, I think
Guanghao will share the practices some time later.
Thanks.
On Sun, Sep 15, 2019 at 12:09 AM Stack <st...@duboce.net> wrote:
> On Fri, Sep 13, 2019 at 4:58 PM Andrew Purtell <an...@gmail.com>
> wrote:
>
> > For what it’s worth I had previously been concerned about the disparity
> > between hbck capability in 1.x and 2.x but after review of the recent
> work
> > I believe that is no longer true. Put another way, it is reasonable to
> > claim it on par.
> >
> >
> Thanks Andrew for chiming in.
>
>
>
> > As for moving the stable pointer I don’t personally have enough
> experience
> > with HBase 2 to weigh in but will trust the opinions of those that do.
> >
> >
> >
> Let's gather a few stories on it working for folks in production and move
> the pointer then.
>
> Thanks,
> S
>
>
>
> > > On Sep 14, 2019, at 8:44 AM, Stack <st...@duboce.net> wrote:
> > >
> > > HBASE-21745 <https://issues.apache.org/jira/browse/HBASE-21745>, the
> > issue
> > > addressing gaps between hbck1 and hbck2 was closed a few days back
> after
> > a
> > > bunch of work by a kaleidoscope of folks. The release notes section
> tries
> > > to describe what was added by HBASE-21745. Shout if you think the claim
> > at
> > > the end of the release notes section that hbck2 now is on par or beyond
> > > what hbck1 offered is problematic. Otherwise, will proceed as though it
> > is
> > > the case.
> > >
> > > Suggestion: Given that hbase 2.2.1 will ship soon and
> > hbase-operator-tools
> > > 1.0.0 with latest hbase-hbck2 should get an RC inside the next week or
> > so,
> > > if feedback that 2.2.1 looks good, give 2.2.2 (with bug fixes only) the
> > > stable pointer?
> > >
> > > Thanks,
> > > S
> > >
> > >> On Sat, Jan 26, 2019 at 11:31 AM Stack <st...@duboce.net> wrote:
> > >>
> > >> As per Sean, bypass with optional 'force' (override) and recurse for
> > case
> > >> where a procedure had spawned children was the mechanism Allan
> > implemented
> > >> after a chat about merits of procedure delete. I found it of use doing
> > >> fixup to clusters I'd intentionally damaged testing candidates.
> > Procedures
> > >> are usually part of a fabric with relations that an operator might
> have
> > >> trouble unraveling. It was thought that the bypass would be safer
> than a
> > >> delete, likely to cause more damage than solution.
> > >>
> > >> Interested in the issues you are seeing on Master branch Sergey.
> > >>
> > >> Thanks,
> > >> S
> > >>
> > >>> On Fri, Jan 25, 2019 at 1:54 PM Sean Busbey <bu...@apache.org>
> wrote:
> > >>>
> > >>> that's already present, see the README for the "bypass" command:
> > >>>
> > >>>
> https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2
> > >>>
> > >>> On Fri, Jan 25, 2019 at 3:40 PM Sergey Shelukhin
> > >>> <Se...@microsoft.com.invalid> wrote:
> > >>>>
> > >>>> I think one thing that is needed for HBCK2 for AMv2 is to be able to
> > >>> delete single procedures from store.
> > >>>> We are evaluating master (whose assignment is very similar to
> > branch-2)
> > >>> right now and I have to delete proc WAL pretty much every day because
> > some
> > >>> procedure(s) are in bad state, but deleting the entire WAL also
> causes
> > >>> other issues.
> > >>>> It should be possible to remove some offending procedure while
> master
> > >>> is offline and/or online.
> > >>>>
> > >>>> -----Original Message-----
> > >>>> From: 张铎(Duo Zhang) <pa...@gmail.com>
> > >>>> Sent: Friday, January 18, 2019 5:52 PM
> > >>>> To: HBase Dev List <de...@hbase.apache.org>
> > >>>> Subject: Re: [DISCUSS] Moving towards a branch-2 line that can get
> the
> > >>> 'stable' pointer.
> > >>>>
> > >>>> OK, the original issue is HBCK2 for AMv2, but here we need to do
> more,
> > >>> not only for AMv2.
> > >>>>
> > >>>> Let me open a new issue and post what Andrew said above there.
> > >>>>
> > >>>
> > >>
> >
>
Re: [DISCUSS] Moving towards a branch-2 line that can get the
'stable' pointer.
Posted by Stack <st...@duboce.net>.
On Fri, Sep 13, 2019 at 4:58 PM Andrew Purtell <an...@gmail.com>
wrote:
> For what it’s worth I had previously been concerned about the disparity
> between hbck capability in 1.x and 2.x but after review of the recent work
> I believe that is no longer true. Put another way, it is reasonable to
> claim it on par.
>
>
Thanks Andrew for chiming in.
> As for moving the stable pointer I don’t personally have enough experience
> with HBase 2 to weigh in but will trust the opinions of those that do.
>
>
>
Let's gather a few stories on it working for folks in production and move
the pointer then.
Thanks,
S
> > On Sep 14, 2019, at 8:44 AM, Stack <st...@duboce.net> wrote:
> >
> > HBASE-21745 <https://issues.apache.org/jira/browse/HBASE-21745>, the
> issue
> > addressing gaps between hbck1 and hbck2 was closed a few days back after
> a
> > bunch of work by a kaleidoscope of folks. The release notes section tries
> > to describe what was added by HBASE-21745. Shout if you think the claim
> at
> > the end of the release notes section that hbck2 now is on par or beyond
> > what hbck1 offered is problematic. Otherwise, will proceed as though it
> is
> > the case.
> >
> > Suggestion: Given that hbase 2.2.1 will ship soon and
> hbase-operator-tools
> > 1.0.0 with latest hbase-hbck2 should get an RC inside the next week or
> so,
> > if feedback that 2.2.1 looks good, give 2.2.2 (with bug fixes only) the
> > stable pointer?
> >
> > Thanks,
> > S
> >
> >> On Sat, Jan 26, 2019 at 11:31 AM Stack <st...@duboce.net> wrote:
> >>
> >> As per Sean, bypass with optional 'force' (override) and recurse for
> case
> >> where a procedure had spawned children was the mechanism Allan
> implemented
> >> after a chat about merits of procedure delete. I found it of use doing
> >> fixup to clusters I'd intentionally damaged testing candidates.
> Procedures
> >> are usually part of a fabric with relations that an operator might have
> >> trouble unraveling. It was thought that the bypass would be safer than a
> >> delete, likely to cause more damage than solution.
> >>
> >> Interested in the issues you are seeing on Master branch Sergey.
> >>
> >> Thanks,
> >> S
> >>
> >>> On Fri, Jan 25, 2019 at 1:54 PM Sean Busbey <bu...@apache.org> wrote:
> >>>
> >>> that's already present, see the README for the "bypass" command:
> >>>
> >>> https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2
> >>>
> >>> On Fri, Jan 25, 2019 at 3:40 PM Sergey Shelukhin
> >>> <Se...@microsoft.com.invalid> wrote:
> >>>>
> >>>> I think one thing that is needed for HBCK2 for AMv2 is to be able to
> >>> delete single procedures from store.
> >>>> We are evaluating master (whose assignment is very similar to
> branch-2)
> >>> right now and I have to delete proc WAL pretty much every day because
> some
> >>> procedure(s) are in bad state, but deleting the entire WAL also causes
> >>> other issues.
> >>>> It should be possible to remove some offending procedure while master
> >>> is offline and/or online.
> >>>>
> >>>> -----Original Message-----
> >>>> From: 张铎(Duo Zhang) <pa...@gmail.com>
> >>>> Sent: Friday, January 18, 2019 5:52 PM
> >>>> To: HBase Dev List <de...@hbase.apache.org>
> >>>> Subject: Re: [DISCUSS] Moving towards a branch-2 line that can get the
> >>> 'stable' pointer.
> >>>>
> >>>> OK, the original issue is HBCK2 for AMv2, but here we need to do more,
> >>> not only for AMv2.
> >>>>
> >>>> Let me open a new issue and post what Andrew said above there.
> >>>>
> >>>
> >>
>
Re: [DISCUSS] Moving towards a branch-2 line that can get the 'stable' pointer.
Posted by Andrew Purtell <an...@gmail.com>.
For what it’s worth I had previously been concerned about the disparity between hbck capability in 1.x and 2.x but after review of the recent work I believe that is no longer true. Put another way, it is reasonable to claim it on par.
As for moving the stable pointer I don’t personally have enough experience with HBase 2 to weigh in but will trust the opinions of those that do.
> On Sep 14, 2019, at 8:44 AM, Stack <st...@duboce.net> wrote:
>
> HBASE-21745 <https://issues.apache.org/jira/browse/HBASE-21745>, the issue
> addressing gaps between hbck1 and hbck2 was closed a few days back after a
> bunch of work by a kaleidoscope of folks. The release notes section tries
> to describe what was added by HBASE-21745. Shout if you think the claim at
> the end of the release notes section that hbck2 now is on par or beyond
> what hbck1 offered is problematic. Otherwise, will proceed as though it is
> the case.
>
> Suggestion: Given that hbase 2.2.1 will ship soon and hbase-operator-tools
> 1.0.0 with latest hbase-hbck2 should get an RC inside the next week or so,
> if feedback that 2.2.1 looks good, give 2.2.2 (with bug fixes only) the
> stable pointer?
>
> Thanks,
> S
>
>> On Sat, Jan 26, 2019 at 11:31 AM Stack <st...@duboce.net> wrote:
>>
>> As per Sean, bypass with optional 'force' (override) and recurse for case
>> where a procedure had spawned children was the mechanism Allan implemented
>> after a chat about merits of procedure delete. I found it of use doing
>> fixup to clusters I'd intentionally damaged testing candidates. Procedures
>> are usually part of a fabric with relations that an operator might have
>> trouble unraveling. It was thought that the bypass would be safer than a
>> delete, likely to cause more damage than solution.
>>
>> Interested in the issues you are seeing on Master branch Sergey.
>>
>> Thanks,
>> S
>>
>>> On Fri, Jan 25, 2019 at 1:54 PM Sean Busbey <bu...@apache.org> wrote:
>>>
>>> that's already present, see the README for the "bypass" command:
>>>
>>> https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2
>>>
>>> On Fri, Jan 25, 2019 at 3:40 PM Sergey Shelukhin
>>> <Se...@microsoft.com.invalid> wrote:
>>>>
>>>> I think one thing that is needed for HBCK2 for AMv2 is to be able to
>>> delete single procedures from store.
>>>> We are evaluating master (whose assignment is very similar to branch-2)
>>> right now and I have to delete proc WAL pretty much every day because some
>>> procedure(s) are in bad state, but deleting the entire WAL also causes
>>> other issues.
>>>> It should be possible to remove some offending procedure while master
>>> is offline and/or online.
>>>>
>>>> -----Original Message-----
>>>> From: 张铎(Duo Zhang) <pa...@gmail.com>
>>>> Sent: Friday, January 18, 2019 5:52 PM
>>>> To: HBase Dev List <de...@hbase.apache.org>
>>>> Subject: Re: [DISCUSS] Moving towards a branch-2 line that can get the
>>> 'stable' pointer.
>>>>
>>>> OK, the original issue is HBCK2 for AMv2, but here we need to do more,
>>> not only for AMv2.
>>>>
>>>> Let me open a new issue and post what Andrew said above there.
>>>>
>>>
>>