You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Mohammad arshad <mo...@huawei.com> on 2020/08/07 05:10:00 UTC
Fixing Catalog Janitor and HBCK chore reported inconsistencies
automatically
Hello HBase Folks
Currently Catalog Janitor (CJ) and HBCK chore reported inconsistencies to be fixed by manually by executing HBCK2 commands.
HBCK2 requires high HBase skills. It is bit difficult for maintenance personals to figure out which command, when and in which order to be executed.
Is there any effort going on in community to automate fixing these inconsistencies? I also would like to contribute there.
I was thinking, maybe we can expose CJ and HBCK chore reported inconsistencies through a new master API and then provide option to fix these inconsistencies. Basically adding two new commands in HBCK2
-listInconsistencies list CJ and HBCK chore reported inconsistencies
-fixInconsistencies fix CJ and HBCK chore reported inconsistencies (Not sure if possible to fix all inconsistencies, need to analyze all inconsistencies case by case, but some are very straight forward for example holes and overlap)
Any thoughts/inputs highly appreciated.
Regards
-Arshad
RE: Fixing Catalog Janitor and HBCK chore reported inconsistencies
automatically
Posted by Mohammad arshad <mo...@huawei.com>.
> In hbase1?
No not in hbase1, Inconsistency occurred in hbase 2.2.3.
> One suggestion is that you move to 2.3.1 rather than 2.2.3. It has the benefit of experience running the 2.2 line and has even more fixes for anomalies applied....
Good point. Thanks for suggesting, will surely consider upgrade to 2.3.x
Thanks
-Arshad
-----Original Message-----
From: Stack [mailto:stack@duboce.net]
Sent: Monday, August 24, 2020 5:01 AM
To: HBase Dev List <de...@hbase.apache.org>
Cc: Hbase-User <us...@hbase.apache.org>
Subject: Re: Fixing Catalog Janitor and HBCK chore reported inconsistencies automatically
On Thu, Aug 20, 2020 at 11:48 PM Mohammad arshad <mo...@huawei.com>
wrote:
> ....
> > Are you regularly experiencing consistency issues? If so, what
> > version
> of hbase are you running?
> We have experienced some inconsistencies like unknown servers, double
> assignment or RIT (HBASE-24885).
In hbase1?
> In future we may witness some other inconsistencies, so we should make
> HBCK2 mature enough to handle them. Off course we will learn from the
> new bug and fix them in code rather than depending on HBCK2.
>
> Agree and if you run into issues, talk out loud here because we want
> to
hear about them and fix them as the come up.
> Currently we are upgrading from 1.x to 2.2.3 version. Just a thought
> process to ease HBase operator job as per some operator feedback, we
> can discuss and come up with some ideas to handle this.
>
>
Appreciated.
One suggestion is that you move to 2.3.1 rather than 2.2.3. It has the benefit of experience running the 2.2 line and has even more fixes for anomalies applied. Also, the master-side of the hbck2 invocations has had a bunch of improvement made so when they run, they are more thorough.
Yours,
S
> Thanks
> -Arshad
>
> -----Original Message-----
> From: Stack [mailto:stack@duboce.net]
> Sent: Tuesday, August 18, 2020 9:20 AM
> To: Hbase-User <us...@hbase.apache.org>
> Cc: dev@hbase.apache.org
> Subject: Re: Fixing Catalog Janitor and HBCK chore reported
> inconsistencies automatically
>
> On Thu, Aug 6, 2020 at 10:10 PM Mohammad arshad <
> mohammad.arshad@huawei.com>
> wrote:
>
> > Hello HBase Folks
> >
> > Currently Catalog Janitor (CJ) and HBCK chore reported
> > inconsistencies to be fixed by manually by executing HBCK2 commands.
> > HBCK2 requires high HBase skills. It is bit difficult for
> > maintenance personals to figure out which command, when and in which
> > order to be executed.
> >
> >
> True.
>
> Is there any effort going on in community to automate fixing these
> > inconsistencies? I also would like to contribute there.
> >
> > I was thinking, maybe we can expose CJ and HBCK chore reported
> > inconsistencies through a new master API and then provide option to
> > fix these inconsistencies. Basically adding two new commands in
> > HBCK2 -listInconsistencies list CJ and HBCK chore reported
> > inconsistencies -fixInconsistencies fix CJ and HBCK chore reported
> > inconsistencies (Not sure if possible to fix all inconsistencies,
> > need to analyze all inconsistencies case by case, but some are very
> > straight forward for example holes and overlap)
> >
> >
> So, a 'god' command that will fix any issue found?
>
> That is tough. You've seen the philosophy section on hbck2, of how it
> makes no claims to being so capable [1]?
>
> We are trying to get to a place where hbck2 is increasingly less necessary.
> The general idea is that inconsistencies are caused by bugs or oversight.
> As time goes by, we've been plugging the holes. Upgrading hbase gains
> you the fixes making the need for hbck2 less.
>
> But as you state above, when there is an issue, it can be hard for the
> operator to figure how to make fixes. We've been trying to improve
> this state with documentation in the UI up on the 'HBCK Report' page
> and elsewhere but there is room for improvement.
>
> We've also been trying to aggregate on the hbck2 side so that commands
> become increasingly 'macro', fixing a whole category of problem types
> rather than an affliction at a time. This should make the tool easier
> to use. The 'fixMeta' command is a good example here as it fixes any
> holes or overlaps found in hbase:meta (This is probably ripe for
> conversion into an auto-repair run on occasion by the Master). Another
> way in which we've been trying to make improvement is by obsoleting
> commands in hbck2 as we fix the root cause that required the hbck2
> command option to be needed in the first place.
>
> CJ and the HBCK Chore can report on inconsistencies found. It is
> another thing altogether having them go ahead and repair any issues
> found mostly because we are not yet confident the repair won't cause
> more damage than it fixes.
>
>
> > Any thoughts/inputs highly appreciated.
> >
> >
> Are you regularly experiencing consistency issues? If so, what version
> of hbase are you running?
>
> Thanks,
> S
>
>
> 1.
>
> https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2
> #philosophy
>
>
> > Regards
> > -Arshad
> >
> >
> >
>
RE: Fixing Catalog Janitor and HBCK chore reported inconsistencies
automatically
Posted by Mohammad arshad <mo...@huawei.com>.
> In hbase1?
No not in hbase1, Inconsistency occurred in hbase 2.2.3.
> One suggestion is that you move to 2.3.1 rather than 2.2.3. It has the benefit of experience running the 2.2 line and has even more fixes for anomalies applied....
Good point. Thanks for suggesting, will surely consider upgrade to 2.3.x
Thanks
-Arshad
-----Original Message-----
From: Stack [mailto:stack@duboce.net]
Sent: Monday, August 24, 2020 5:01 AM
To: HBase Dev List <de...@hbase.apache.org>
Cc: Hbase-User <us...@hbase.apache.org>
Subject: Re: Fixing Catalog Janitor and HBCK chore reported inconsistencies automatically
On Thu, Aug 20, 2020 at 11:48 PM Mohammad arshad <mo...@huawei.com>
wrote:
> ....
> > Are you regularly experiencing consistency issues? If so, what
> > version
> of hbase are you running?
> We have experienced some inconsistencies like unknown servers, double
> assignment or RIT (HBASE-24885).
In hbase1?
> In future we may witness some other inconsistencies, so we should make
> HBCK2 mature enough to handle them. Off course we will learn from the
> new bug and fix them in code rather than depending on HBCK2.
>
> Agree and if you run into issues, talk out loud here because we want
> to
hear about them and fix them as the come up.
> Currently we are upgrading from 1.x to 2.2.3 version. Just a thought
> process to ease HBase operator job as per some operator feedback, we
> can discuss and come up with some ideas to handle this.
>
>
Appreciated.
One suggestion is that you move to 2.3.1 rather than 2.2.3. It has the benefit of experience running the 2.2 line and has even more fixes for anomalies applied. Also, the master-side of the hbck2 invocations has had a bunch of improvement made so when they run, they are more thorough.
Yours,
S
> Thanks
> -Arshad
>
> -----Original Message-----
> From: Stack [mailto:stack@duboce.net]
> Sent: Tuesday, August 18, 2020 9:20 AM
> To: Hbase-User <us...@hbase.apache.org>
> Cc: dev@hbase.apache.org
> Subject: Re: Fixing Catalog Janitor and HBCK chore reported
> inconsistencies automatically
>
> On Thu, Aug 6, 2020 at 10:10 PM Mohammad arshad <
> mohammad.arshad@huawei.com>
> wrote:
>
> > Hello HBase Folks
> >
> > Currently Catalog Janitor (CJ) and HBCK chore reported
> > inconsistencies to be fixed by manually by executing HBCK2 commands.
> > HBCK2 requires high HBase skills. It is bit difficult for
> > maintenance personals to figure out which command, when and in which
> > order to be executed.
> >
> >
> True.
>
> Is there any effort going on in community to automate fixing these
> > inconsistencies? I also would like to contribute there.
> >
> > I was thinking, maybe we can expose CJ and HBCK chore reported
> > inconsistencies through a new master API and then provide option to
> > fix these inconsistencies. Basically adding two new commands in
> > HBCK2 -listInconsistencies list CJ and HBCK chore reported
> > inconsistencies -fixInconsistencies fix CJ and HBCK chore reported
> > inconsistencies (Not sure if possible to fix all inconsistencies,
> > need to analyze all inconsistencies case by case, but some are very
> > straight forward for example holes and overlap)
> >
> >
> So, a 'god' command that will fix any issue found?
>
> That is tough. You've seen the philosophy section on hbck2, of how it
> makes no claims to being so capable [1]?
>
> We are trying to get to a place where hbck2 is increasingly less necessary.
> The general idea is that inconsistencies are caused by bugs or oversight.
> As time goes by, we've been plugging the holes. Upgrading hbase gains
> you the fixes making the need for hbck2 less.
>
> But as you state above, when there is an issue, it can be hard for the
> operator to figure how to make fixes. We've been trying to improve
> this state with documentation in the UI up on the 'HBCK Report' page
> and elsewhere but there is room for improvement.
>
> We've also been trying to aggregate on the hbck2 side so that commands
> become increasingly 'macro', fixing a whole category of problem types
> rather than an affliction at a time. This should make the tool easier
> to use. The 'fixMeta' command is a good example here as it fixes any
> holes or overlaps found in hbase:meta (This is probably ripe for
> conversion into an auto-repair run on occasion by the Master). Another
> way in which we've been trying to make improvement is by obsoleting
> commands in hbck2 as we fix the root cause that required the hbck2
> command option to be needed in the first place.
>
> CJ and the HBCK Chore can report on inconsistencies found. It is
> another thing altogether having them go ahead and repair any issues
> found mostly because we are not yet confident the repair won't cause
> more damage than it fixes.
>
>
> > Any thoughts/inputs highly appreciated.
> >
> >
> Are you regularly experiencing consistency issues? If so, what version
> of hbase are you running?
>
> Thanks,
> S
>
>
> 1.
>
> https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2
> #philosophy
>
>
> > Regards
> > -Arshad
> >
> >
> >
>
Re: Fixing Catalog Janitor and HBCK chore reported inconsistencies automatically
Posted by Stack <st...@duboce.net>.
On Thu, Aug 20, 2020 at 11:48 PM Mohammad arshad <mo...@huawei.com>
wrote:
> ....
> > Are you regularly experiencing consistency issues? If so, what version
> of hbase are you running?
> We have experienced some inconsistencies like unknown servers, double
> assignment or RIT (HBASE-24885).
In hbase1?
> In future we may witness some other inconsistencies, so we should make
> HBCK2 mature enough to handle them. Off course we will learn from the new
> bug and fix them in code rather than depending on HBCK2.
>
> Agree and if you run into issues, talk out loud here because we want to
hear about them and fix them as the come up.
> Currently we are upgrading from 1.x to 2.2.3 version. Just a thought
> process to ease HBase operator job as per some operator feedback, we can
> discuss and come up with some ideas to handle this.
>
>
Appreciated.
One suggestion is that you move to 2.3.1 rather than 2.2.3. It has the
benefit of experience running the 2.2 line and has even more fixes for
anomalies applied. Also, the master-side of the hbck2 invocations has had a
bunch of improvement made so when they run, they are more thorough.
Yours,
S
> Thanks
> -Arshad
>
> -----Original Message-----
> From: Stack [mailto:stack@duboce.net]
> Sent: Tuesday, August 18, 2020 9:20 AM
> To: Hbase-User <us...@hbase.apache.org>
> Cc: dev@hbase.apache.org
> Subject: Re: Fixing Catalog Janitor and HBCK chore reported
> inconsistencies automatically
>
> On Thu, Aug 6, 2020 at 10:10 PM Mohammad arshad <
> mohammad.arshad@huawei.com>
> wrote:
>
> > Hello HBase Folks
> >
> > Currently Catalog Janitor (CJ) and HBCK chore reported inconsistencies
> > to be fixed by manually by executing HBCK2 commands.
> > HBCK2 requires high HBase skills. It is bit difficult for maintenance
> > personals to figure out which command, when and in which order to be
> > executed.
> >
> >
> True.
>
> Is there any effort going on in community to automate fixing these
> > inconsistencies? I also would like to contribute there.
> >
> > I was thinking, maybe we can expose CJ and HBCK chore reported
> > inconsistencies through a new master API and then provide option to
> > fix these inconsistencies. Basically adding two new commands in HBCK2
> > -listInconsistencies list CJ and HBCK chore reported inconsistencies
> > -fixInconsistencies fix CJ and HBCK chore reported inconsistencies
> > (Not sure if possible to fix all inconsistencies, need to analyze all
> > inconsistencies case by case, but some are very straight forward for
> > example holes and overlap)
> >
> >
> So, a 'god' command that will fix any issue found?
>
> That is tough. You've seen the philosophy section on hbck2, of how it
> makes no claims to being so capable [1]?
>
> We are trying to get to a place where hbck2 is increasingly less necessary.
> The general idea is that inconsistencies are caused by bugs or oversight.
> As time goes by, we've been plugging the holes. Upgrading hbase gains you
> the fixes making the need for hbck2 less.
>
> But as you state above, when there is an issue, it can be hard for the
> operator to figure how to make fixes. We've been trying to improve this
> state with documentation in the UI up on the 'HBCK Report' page and
> elsewhere but there is room for improvement.
>
> We've also been trying to aggregate on the hbck2 side so that commands
> become increasingly 'macro', fixing a whole category of problem types
> rather than an affliction at a time. This should make the tool easier to
> use. The 'fixMeta' command is a good example here as it fixes any holes or
> overlaps found in hbase:meta (This is probably ripe for conversion into an
> auto-repair run on occasion by the Master). Another way in which we've been
> trying to make improvement is by obsoleting commands in hbck2 as we fix the
> root cause that required the hbck2 command option to be needed in the first
> place.
>
> CJ and the HBCK Chore can report on inconsistencies found. It is another
> thing altogether having them go ahead and repair any issues found mostly
> because we are not yet confident the repair won't cause more damage than it
> fixes.
>
>
> > Any thoughts/inputs highly appreciated.
> >
> >
> Are you regularly experiencing consistency issues? If so, what version of
> hbase are you running?
>
> Thanks,
> S
>
>
> 1.
>
> https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2#philosophy
>
>
> > Regards
> > -Arshad
> >
> >
> >
>
Re: Fixing Catalog Janitor and HBCK chore reported inconsistencies automatically
Posted by Stack <st...@duboce.net>.
On Thu, Aug 20, 2020 at 11:48 PM Mohammad arshad <mo...@huawei.com>
wrote:
> ....
> > Are you regularly experiencing consistency issues? If so, what version
> of hbase are you running?
> We have experienced some inconsistencies like unknown servers, double
> assignment or RIT (HBASE-24885).
In hbase1?
> In future we may witness some other inconsistencies, so we should make
> HBCK2 mature enough to handle them. Off course we will learn from the new
> bug and fix them in code rather than depending on HBCK2.
>
> Agree and if you run into issues, talk out loud here because we want to
hear about them and fix them as the come up.
> Currently we are upgrading from 1.x to 2.2.3 version. Just a thought
> process to ease HBase operator job as per some operator feedback, we can
> discuss and come up with some ideas to handle this.
>
>
Appreciated.
One suggestion is that you move to 2.3.1 rather than 2.2.3. It has the
benefit of experience running the 2.2 line and has even more fixes for
anomalies applied. Also, the master-side of the hbck2 invocations has had a
bunch of improvement made so when they run, they are more thorough.
Yours,
S
> Thanks
> -Arshad
>
> -----Original Message-----
> From: Stack [mailto:stack@duboce.net]
> Sent: Tuesday, August 18, 2020 9:20 AM
> To: Hbase-User <us...@hbase.apache.org>
> Cc: dev@hbase.apache.org
> Subject: Re: Fixing Catalog Janitor and HBCK chore reported
> inconsistencies automatically
>
> On Thu, Aug 6, 2020 at 10:10 PM Mohammad arshad <
> mohammad.arshad@huawei.com>
> wrote:
>
> > Hello HBase Folks
> >
> > Currently Catalog Janitor (CJ) and HBCK chore reported inconsistencies
> > to be fixed by manually by executing HBCK2 commands.
> > HBCK2 requires high HBase skills. It is bit difficult for maintenance
> > personals to figure out which command, when and in which order to be
> > executed.
> >
> >
> True.
>
> Is there any effort going on in community to automate fixing these
> > inconsistencies? I also would like to contribute there.
> >
> > I was thinking, maybe we can expose CJ and HBCK chore reported
> > inconsistencies through a new master API and then provide option to
> > fix these inconsistencies. Basically adding two new commands in HBCK2
> > -listInconsistencies list CJ and HBCK chore reported inconsistencies
> > -fixInconsistencies fix CJ and HBCK chore reported inconsistencies
> > (Not sure if possible to fix all inconsistencies, need to analyze all
> > inconsistencies case by case, but some are very straight forward for
> > example holes and overlap)
> >
> >
> So, a 'god' command that will fix any issue found?
>
> That is tough. You've seen the philosophy section on hbck2, of how it
> makes no claims to being so capable [1]?
>
> We are trying to get to a place where hbck2 is increasingly less necessary.
> The general idea is that inconsistencies are caused by bugs or oversight.
> As time goes by, we've been plugging the holes. Upgrading hbase gains you
> the fixes making the need for hbck2 less.
>
> But as you state above, when there is an issue, it can be hard for the
> operator to figure how to make fixes. We've been trying to improve this
> state with documentation in the UI up on the 'HBCK Report' page and
> elsewhere but there is room for improvement.
>
> We've also been trying to aggregate on the hbck2 side so that commands
> become increasingly 'macro', fixing a whole category of problem types
> rather than an affliction at a time. This should make the tool easier to
> use. The 'fixMeta' command is a good example here as it fixes any holes or
> overlaps found in hbase:meta (This is probably ripe for conversion into an
> auto-repair run on occasion by the Master). Another way in which we've been
> trying to make improvement is by obsoleting commands in hbck2 as we fix the
> root cause that required the hbck2 command option to be needed in the first
> place.
>
> CJ and the HBCK Chore can report on inconsistencies found. It is another
> thing altogether having them go ahead and repair any issues found mostly
> because we are not yet confident the repair won't cause more damage than it
> fixes.
>
>
> > Any thoughts/inputs highly appreciated.
> >
> >
> Are you regularly experiencing consistency issues? If so, what version of
> hbase are you running?
>
> Thanks,
> S
>
>
> 1.
>
> https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2#philosophy
>
>
> > Regards
> > -Arshad
> >
> >
> >
>
RE: Fixing Catalog Janitor and HBCK chore reported inconsistencies
automatically
Posted by Mohammad arshad <mo...@huawei.com>.
Thank you sir for your very informative response.
> So, a 'god' command that will fix any issue found?
Yes, the idea was to make -fixInconsistencies god command, fixing all inconstancies reported in both CJ and hbck chore. But as you rightly pointed out that we are not yet confident the repair won't cause more damage than it fixes, on second thought automating fixes at this point may not be appropriate as efficacy of all the fixes is not yet proved.
> Are you regularly experiencing consistency issues? If so, what version of hbase are you running?
We have experienced some inconsistencies like unknown servers, double assignment or RIT (HBASE-24885). In future we may witness some other inconsistencies, so we should make HBCK2 mature enough to handle them. Off course we will learn from the new bug and fix them in code rather than depending on HBCK2.
Currently we are upgrading from 1.x to 2.2.3 version. Just a thought process to ease HBase operator job as per some operator feedback, we can discuss and come up with some ideas to handle this.
Thanks
-Arshad
-----Original Message-----
From: Stack [mailto:stack@duboce.net]
Sent: Tuesday, August 18, 2020 9:20 AM
To: Hbase-User <us...@hbase.apache.org>
Cc: dev@hbase.apache.org
Subject: Re: Fixing Catalog Janitor and HBCK chore reported inconsistencies automatically
On Thu, Aug 6, 2020 at 10:10 PM Mohammad arshad <mo...@huawei.com>
wrote:
> Hello HBase Folks
>
> Currently Catalog Janitor (CJ) and HBCK chore reported inconsistencies
> to be fixed by manually by executing HBCK2 commands.
> HBCK2 requires high HBase skills. It is bit difficult for maintenance
> personals to figure out which command, when and in which order to be
> executed.
>
>
True.
Is there any effort going on in community to automate fixing these
> inconsistencies? I also would like to contribute there.
>
> I was thinking, maybe we can expose CJ and HBCK chore reported
> inconsistencies through a new master API and then provide option to
> fix these inconsistencies. Basically adding two new commands in HBCK2
> -listInconsistencies list CJ and HBCK chore reported inconsistencies
> -fixInconsistencies fix CJ and HBCK chore reported inconsistencies
> (Not sure if possible to fix all inconsistencies, need to analyze all
> inconsistencies case by case, but some are very straight forward for
> example holes and overlap)
>
>
So, a 'god' command that will fix any issue found?
That is tough. You've seen the philosophy section on hbck2, of how it makes no claims to being so capable [1]?
We are trying to get to a place where hbck2 is increasingly less necessary.
The general idea is that inconsistencies are caused by bugs or oversight.
As time goes by, we've been plugging the holes. Upgrading hbase gains you the fixes making the need for hbck2 less.
But as you state above, when there is an issue, it can be hard for the operator to figure how to make fixes. We've been trying to improve this state with documentation in the UI up on the 'HBCK Report' page and elsewhere but there is room for improvement.
We've also been trying to aggregate on the hbck2 side so that commands become increasingly 'macro', fixing a whole category of problem types rather than an affliction at a time. This should make the tool easier to use. The 'fixMeta' command is a good example here as it fixes any holes or overlaps found in hbase:meta (This is probably ripe for conversion into an auto-repair run on occasion by the Master). Another way in which we've been trying to make improvement is by obsoleting commands in hbck2 as we fix the root cause that required the hbck2 command option to be needed in the first place.
CJ and the HBCK Chore can report on inconsistencies found. It is another thing altogether having them go ahead and repair any issues found mostly because we are not yet confident the repair won't cause more damage than it fixes.
> Any thoughts/inputs highly appreciated.
>
>
Are you regularly experiencing consistency issues? If so, what version of hbase are you running?
Thanks,
S
1.
https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2#philosophy
> Regards
> -Arshad
>
>
>
RE: Fixing Catalog Janitor and HBCK chore reported inconsistencies
automatically
Posted by Mohammad arshad <mo...@huawei.com>.
Thank you sir for your very informative response.
> So, a 'god' command that will fix any issue found?
Yes, the idea was to make -fixInconsistencies god command, fixing all inconstancies reported in both CJ and hbck chore. But as you rightly pointed out that we are not yet confident the repair won't cause more damage than it fixes, on second thought automating fixes at this point may not be appropriate as efficacy of all the fixes is not yet proved.
> Are you regularly experiencing consistency issues? If so, what version of hbase are you running?
We have experienced some inconsistencies like unknown servers, double assignment or RIT (HBASE-24885). In future we may witness some other inconsistencies, so we should make HBCK2 mature enough to handle them. Off course we will learn from the new bug and fix them in code rather than depending on HBCK2.
Currently we are upgrading from 1.x to 2.2.3 version. Just a thought process to ease HBase operator job as per some operator feedback, we can discuss and come up with some ideas to handle this.
Thanks
-Arshad
-----Original Message-----
From: Stack [mailto:stack@duboce.net]
Sent: Tuesday, August 18, 2020 9:20 AM
To: Hbase-User <us...@hbase.apache.org>
Cc: dev@hbase.apache.org
Subject: Re: Fixing Catalog Janitor and HBCK chore reported inconsistencies automatically
On Thu, Aug 6, 2020 at 10:10 PM Mohammad arshad <mo...@huawei.com>
wrote:
> Hello HBase Folks
>
> Currently Catalog Janitor (CJ) and HBCK chore reported inconsistencies
> to be fixed by manually by executing HBCK2 commands.
> HBCK2 requires high HBase skills. It is bit difficult for maintenance
> personals to figure out which command, when and in which order to be
> executed.
>
>
True.
Is there any effort going on in community to automate fixing these
> inconsistencies? I also would like to contribute there.
>
> I was thinking, maybe we can expose CJ and HBCK chore reported
> inconsistencies through a new master API and then provide option to
> fix these inconsistencies. Basically adding two new commands in HBCK2
> -listInconsistencies list CJ and HBCK chore reported inconsistencies
> -fixInconsistencies fix CJ and HBCK chore reported inconsistencies
> (Not sure if possible to fix all inconsistencies, need to analyze all
> inconsistencies case by case, but some are very straight forward for
> example holes and overlap)
>
>
So, a 'god' command that will fix any issue found?
That is tough. You've seen the philosophy section on hbck2, of how it makes no claims to being so capable [1]?
We are trying to get to a place where hbck2 is increasingly less necessary.
The general idea is that inconsistencies are caused by bugs or oversight.
As time goes by, we've been plugging the holes. Upgrading hbase gains you the fixes making the need for hbck2 less.
But as you state above, when there is an issue, it can be hard for the operator to figure how to make fixes. We've been trying to improve this state with documentation in the UI up on the 'HBCK Report' page and elsewhere but there is room for improvement.
We've also been trying to aggregate on the hbck2 side so that commands become increasingly 'macro', fixing a whole category of problem types rather than an affliction at a time. This should make the tool easier to use. The 'fixMeta' command is a good example here as it fixes any holes or overlaps found in hbase:meta (This is probably ripe for conversion into an auto-repair run on occasion by the Master). Another way in which we've been trying to make improvement is by obsoleting commands in hbck2 as we fix the root cause that required the hbck2 command option to be needed in the first place.
CJ and the HBCK Chore can report on inconsistencies found. It is another thing altogether having them go ahead and repair any issues found mostly because we are not yet confident the repair won't cause more damage than it fixes.
> Any thoughts/inputs highly appreciated.
>
>
Are you regularly experiencing consistency issues? If so, what version of hbase are you running?
Thanks,
S
1.
https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2#philosophy
> Regards
> -Arshad
>
>
>
Re: Fixing Catalog Janitor and HBCK chore reported inconsistencies automatically
Posted by Stack <st...@duboce.net>.
On Thu, Aug 6, 2020 at 10:10 PM Mohammad arshad <mo...@huawei.com>
wrote:
> Hello HBase Folks
>
> Currently Catalog Janitor (CJ) and HBCK chore reported inconsistencies to
> be fixed by manually by executing HBCK2 commands.
> HBCK2 requires high HBase skills. It is bit difficult for maintenance
> personals to figure out which command, when and in which order to be
> executed.
>
>
True.
Is there any effort going on in community to automate fixing these
> inconsistencies? I also would like to contribute there.
>
> I was thinking, maybe we can expose CJ and HBCK chore reported
> inconsistencies through a new master API and then provide option to fix
> these inconsistencies. Basically adding two new commands in HBCK2
> -listInconsistencies list CJ and HBCK chore reported inconsistencies
> -fixInconsistencies fix CJ and HBCK chore reported inconsistencies (Not
> sure if possible to fix all inconsistencies, need to analyze all
> inconsistencies case by case, but some are very straight forward for
> example holes and overlap)
>
>
So, a 'god' command that will fix any issue found?
That is tough. You've seen the philosophy section on hbck2, of how it makes
no claims to being so capable [1]?
We are trying to get to a place where hbck2 is increasingly less necessary.
The general idea is that inconsistencies are caused by bugs or oversight.
As time goes by, we've been plugging the holes. Upgrading hbase gains you
the fixes making the need for hbck2 less.
But as you state above, when there is an issue, it can be hard for the
operator to figure how to make fixes. We've been trying to improve this
state with documentation in the UI up on the 'HBCK Report' page and
elsewhere but there is room for improvement.
We've also been trying to aggregate on the hbck2 side so that commands
become increasingly 'macro', fixing a whole category of problem types
rather than an affliction at a time. This should make the tool easier to
use. The 'fixMeta' command is a good example here as it fixes any holes or
overlaps found in hbase:meta (This is probably ripe for conversion into an
auto-repair run on occasion by the Master). Another way in which we've been
trying to make improvement is by obsoleting commands in hbck2 as we fix the
root cause that required the hbck2 command option to be needed in the first
place.
CJ and the HBCK Chore can report on inconsistencies found. It is another
thing altogether having them go ahead and repair any issues found mostly
because we are not yet confident the repair won't cause more damage than it
fixes.
> Any thoughts/inputs highly appreciated.
>
>
Are you regularly experiencing consistency issues? If so, what version of
hbase are you running?
Thanks,
S
1.
https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2#philosophy
> Regards
> -Arshad
>
>
>
Re: Fixing Catalog Janitor and HBCK chore reported inconsistencies automatically
Posted by Sukumar Maddineni <sm...@salesforce.com.INVALID>.
Huge +1 and love to have that option in 1.6+ version as well, based on the
experience we always recommend to be extra careful running HBCK fix
inconsistencies directly as we had incidents where running hbck fix without
complete knowledge ended up causing more damage to already broken clusters.
--
Sukumar
On Thu, Aug 6, 2020 at 10:10 PM Mohammad arshad <mo...@huawei.com>
wrote:
> Hello HBase Folks
>
> Currently Catalog Janitor (CJ) and HBCK chore reported inconsistencies to
> be fixed by manually by executing HBCK2 commands.
> HBCK2 requires high HBase skills. It is bit difficult for maintenance
> personals to figure out which command, when and in which order to be
> executed.
>
> Is there any effort going on in community to automate fixing these
> inconsistencies? I also would like to contribute there.
>
> I was thinking, maybe we can expose CJ and HBCK chore reported
> inconsistencies through a new master API and then provide option to fix
> these inconsistencies. Basically adding two new commands in HBCK2
> -listInconsistencies list CJ and HBCK chore reported inconsistencies
> -fixInconsistencies fix CJ and HBCK chore reported inconsistencies (Not
> sure if possible to fix all inconsistencies, need to analyze all
> inconsistencies case by case, but some are very straight forward for
> example holes and overlap)
>
> Any thoughts/inputs highly appreciated.
>
> Regards
> -Arshad
>
>
>
--
<https://smart.salesforce.com/sig/smaddineni//us_mb/default/link.html>
Re: Fixing Catalog Janitor and HBCK chore reported inconsistencies automatically
Posted by Stack <st...@duboce.net>.
On Thu, Aug 6, 2020 at 10:10 PM Mohammad arshad <mo...@huawei.com>
wrote:
> Hello HBase Folks
>
> Currently Catalog Janitor (CJ) and HBCK chore reported inconsistencies to
> be fixed by manually by executing HBCK2 commands.
> HBCK2 requires high HBase skills. It is bit difficult for maintenance
> personals to figure out which command, when and in which order to be
> executed.
>
>
True.
Is there any effort going on in community to automate fixing these
> inconsistencies? I also would like to contribute there.
>
> I was thinking, maybe we can expose CJ and HBCK chore reported
> inconsistencies through a new master API and then provide option to fix
> these inconsistencies. Basically adding two new commands in HBCK2
> -listInconsistencies list CJ and HBCK chore reported inconsistencies
> -fixInconsistencies fix CJ and HBCK chore reported inconsistencies (Not
> sure if possible to fix all inconsistencies, need to analyze all
> inconsistencies case by case, but some are very straight forward for
> example holes and overlap)
>
>
So, a 'god' command that will fix any issue found?
That is tough. You've seen the philosophy section on hbck2, of how it makes
no claims to being so capable [1]?
We are trying to get to a place where hbck2 is increasingly less necessary.
The general idea is that inconsistencies are caused by bugs or oversight.
As time goes by, we've been plugging the holes. Upgrading hbase gains you
the fixes making the need for hbck2 less.
But as you state above, when there is an issue, it can be hard for the
operator to figure how to make fixes. We've been trying to improve this
state with documentation in the UI up on the 'HBCK Report' page and
elsewhere but there is room for improvement.
We've also been trying to aggregate on the hbck2 side so that commands
become increasingly 'macro', fixing a whole category of problem types
rather than an affliction at a time. This should make the tool easier to
use. The 'fixMeta' command is a good example here as it fixes any holes or
overlaps found in hbase:meta (This is probably ripe for conversion into an
auto-repair run on occasion by the Master). Another way in which we've been
trying to make improvement is by obsoleting commands in hbck2 as we fix the
root cause that required the hbck2 command option to be needed in the first
place.
CJ and the HBCK Chore can report on inconsistencies found. It is another
thing altogether having them go ahead and repair any issues found mostly
because we are not yet confident the repair won't cause more damage than it
fixes.
> Any thoughts/inputs highly appreciated.
>
>
Are you regularly experiencing consistency issues? If so, what version of
hbase are you running?
Thanks,
S
1.
https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2#philosophy
> Regards
> -Arshad
>
>
>