You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Peter Somogyi <ps...@apache.org> on 2019/08/29 12:12:17 UTC
[DISCUSS] HBCK2 compatibility
Hi everyone,
This topic came up a couple of times already but now we reached a point
when the recent HBCK2 is incompatible with older HBase releases,
specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2 against one of
the previously mentioned releases you will get compilation errors. (mvn
clean install -DskipTests -Dhbase.version=2.0.6)
Our previous goal was to maintain compatibility with HBCK2 and all HBase 2
releases. Now we missed this target.
One option that we could do is to have different branches/releases of HBCK2
targeted for specific HBase releases (e.g. branch-2.0 version of HBCK2).
This probably makes the development on HBCK2 a bit harder since we'll have
to take care of multiple branches.
Another option I could think of is to always build HBCK2 with the latest
HBase release but have version checks on individual commands where we could
decide if it is supported on that release line.
What are your opinions on this?
Thanks,
Peter
Re: [DISCUSS] HBCK2 compatibility
Posted by Andrew Purtell <an...@gmail.com>.
It does seem better to put it back in tree with core because core will be branched, and would allow for the usual way we vary features and compatibility by code line.
> On Aug 29, 2019, at 11:14 AM, Stack <st...@duboce.net> wrote:
>
>> On Thu, Aug 29, 2019 at 5:12 AM Peter Somogyi <ps...@apache.org> wrote:
>>
>> Hi everyone,
>>
>> This topic came up a couple of times already but now we reached a point
>> when the recent HBCK2 is incompatible with older HBase releases,
>> specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2 against one of
>> the previously mentioned releases you will get compilation errors. (mvn
>> clean install -DskipTests -Dhbase.version=2.0.6)
>>
>> Our previous goal was to maintain compatibility with HBCK2 and all HBase 2
>> releases. Now we missed this target.
>>
>>
> Was being able to build against all hbase2s a target? Didn't know.
>
>
>
>> One option that we could do is to have different branches/releases of HBCK2
>> targeted for specific HBase releases (e.g. branch-2.0 version of HBCK2).
>> This probably makes the development on HBCK2 a bit harder since we'll have
>> to take care of multiple branches.
>>
>>
> This would defeat the whole reason for moving hbck2 out of core. Better to
> just have hbck2 in hbase rather than do this.
>
>
>> Another option I could think of is to always build HBCK2 with the latest
>> HBase release but have version checks on individual commands where we could
>> decide if it is supported on that release line.
>>
>>
> This is what we have now [1].
>
> S
>
> 1.
> https://github.com/apache/hbase-operator-tools/blob/master/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java#L105
>
>
>> What are your opinions on this?
>>
>> Thanks,
>> Peter
>>
Re: [DISCUSS] HBCK2 compatibility
Posted by Stack <st...@duboce.net>.
On Thu, Aug 29, 2019 at 5:12 AM Peter Somogyi <ps...@apache.org> wrote:
> Hi everyone,
>
> This topic came up a couple of times already but now we reached a point
> when the recent HBCK2 is incompatible with older HBase releases,
> specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2 against one of
> the previously mentioned releases you will get compilation errors. (mvn
> clean install -DskipTests -Dhbase.version=2.0.6)
>
> Our previous goal was to maintain compatibility with HBCK2 and all HBase 2
> releases. Now we missed this target.
>
>
Was being able to build against all hbase2s a target? Didn't know.
> One option that we could do is to have different branches/releases of HBCK2
> targeted for specific HBase releases (e.g. branch-2.0 version of HBCK2).
> This probably makes the development on HBCK2 a bit harder since we'll have
> to take care of multiple branches.
>
>
This would defeat the whole reason for moving hbck2 out of core. Better to
just have hbck2 in hbase rather than do this.
> Another option I could think of is to always build HBCK2 with the latest
> HBase release but have version checks on individual commands where we could
> decide if it is supported on that release line.
>
>
This is what we have now [1].
S
1.
https://github.com/apache/hbase-operator-tools/blob/master/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java#L105
> What are your opinions on this?
>
> Thanks,
> Peter
>
Re: [DISCUSS] HBCK2 compatibility
Posted by Sean Busbey <bu...@apache.org>.
> > what would folks think about going with an hbck2 alpha release?
>
> I'm fine with an alpha release but since "HBCK2 should continuously evolve"
> it might be better to always use the latest codebase whenever you need to
> use the tool.
We can't do this, as a matter of ASF policy. Anything that's supposed
to be used by folks outside of the dev@hbase mailing list needs to
have artifacts that the PMC has signed off on and placed into
dist.a.o. It's fine if they're just source releases and we expect
folks to always build it. But it can't be pointing them at the git
repo.
On Thu, Aug 29, 2019 at 8:09 AM Peter Somogyi <ps...@apache.org> wrote:
>
> bq. Is it possible to put some hacks into HBCK2 to work around
> the compatibility to fix the current state
>
> There are some classes around Replication which were introduced in 2.1.0+
> so I don't think we could easily solve it for 2.0.
> For 2.1.1 the missing method is Hbck#scheduleServerCrashProcedure, probably
> that could be solved with some workarounds or using reflection.
>
> bq. focus more on automation to let us know the next time we inevitably
> break it again? ;)
>
> Sure! Based on this I think it should be a strong goal. We can set up
> nightly builds for hbase-operator-tools repo that builds against the latest
> development branches as well as checking compatibility with released
> versions.
>
> bq. what would folks think about going with an hbck2 alpha release?
> I'm fine with an alpha release but since "HBCK2 should continuously evolve"
> it might be better to always use the latest codebase whenever you need to
> use the tool.
>
> On Thu, Aug 29, 2019 at 2:30 PM Wellington Chevreuil <
> wellington.chevreuil@gmail.com> wrote:
>
> > I would favour having hbck2 release branches. Temporary compatibility
> > breaks at compile time may be inevitable if we always point to the latest
> > release. That could cause problems for operators trying to build hbck2 (we
> > are already seeing this happening with our support team). Another argument
> > for starting having hbck2 releases is that we already have quite a few
> > hbase 2 releases, yet, the main tool to fix inconsistencies is not easily
> > available. And there's been considerable efforts lately to bring many of
> > the fix options from hbck1 into hbck2, so what would folks think about
> > going with an hbck2 alpha release?
> >
> > Em qui, 29 de ago de 2019 às 13:20, Josh Elser <el...@apache.org>
> > escreveu:
> >
> > > I still like one HBCK2 release as the goal.
> > >
> > > Is it possible to put some hacks into HBCK2 to work around the
> > > compatibility to fix the current state and focus more on automation to
> > > let us know the next time we inevitably break it again? ;)
> > >
> > > On 8/29/19 8:12 AM, Peter Somogyi wrote:
> > > > Hi everyone,
> > > >
> > > > This topic came up a couple of times already but now we reached a point
> > > > when the recent HBCK2 is incompatible with older HBase releases,
> > > > specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2 against one
> > of
> > > > the previously mentioned releases you will get compilation errors. (mvn
> > > > clean install -DskipTests -Dhbase.version=2.0.6)
> > > >
> > > > Our previous goal was to maintain compatibility with HBCK2 and all
> > HBase
> > > 2
> > > > releases. Now we missed this target.
> > > >
> > > > One option that we could do is to have different branches/releases of
> > > HBCK2
> > > > targeted for specific HBase releases (e.g. branch-2.0 version of
> > HBCK2).
> > > > This probably makes the development on HBCK2 a bit harder since we'll
> > > have
> > > > to take care of multiple branches.
> > > >
> > > > Another option I could think of is to always build HBCK2 with the
> > latest
> > > > HBase release but have version checks on individual commands where we
> > > could
> > > > decide if it is supported on that release line.
> > > >
> > > > What are your opinions on this?
> > > >
> > > > Thanks,
> > > > Peter
> > > >
> > >
> >
Re: [DISCUSS] HBCK2 compatibility
Posted by Peter Somogyi <ps...@apache.org>.
bq. I do not think we need to compile HBCK2 with every releases? We just
need
make sure that it can work with all the releases. If there are missing
methods, we just tell users you can not use several features.
We could do this but in this case we need to put version-specific checks
for each HBCK2 commands.
On Thu, Aug 29, 2019 at 4:34 PM Wellington Chevreuil <
wellington.chevreuil@gmail.com> wrote:
> >
> > I do not think we need to compile HBCK2 with every releases?
> >
> Well, not with every release, was thinking in doing it whenever an hbase
> release breaks compatibility.
>
> We just need
> > make sure that it can work with all the releases.
>
> This could be a solution as well, but I believe it would be harder to
> guarantee. Here the problem is:
> 1) A new hbase release changes one or more interfaces currently used by
> hbck2;
> 2) We update hbck2 to depend on this new hbase release, and change hbck2
> accordingly;
> 3) Operators need to run hbck2 to a previous hbase release. If they try to
> build hbck2 against that version, it won't compile. If they build it with
> latest hbase version, it may give a runtime error, and now they have no
> working tool to fix the problem.
> To avoid #3, we would need to add extra checks on the changes applied on
> #2. Might become too complex.
>
> If there are missing
> > methods, we just tell users you can not use several features.
>
> Fine for new fix options added, but what if some changes break basic,
> already working hbck2 methods.
>
>
> Em qui, 29 de ago de 2019 às 14:51, 张铎(Duo Zhang) <pa...@gmail.com>
> escreveu:
>
> > I do not think we need to compile HBCK2 with every releases? We just need
> > make sure that it can work with all the releases. If there are missing
> > methods, we just tell users you can not use several features.
> >
> > Wellington Chevreuil <we...@gmail.com> 于2019年8月29日周四
> > 下午9:39写道：
> >
> > > >
> > > > bq. what would folks think about going with an hbck2 alpha release?
> > > > I'm fine with an alpha release but since "HBCK2 should continuously
> > > evolve"
> > > > it might be better to always use the latest codebase whenever you
> need
> > to
> > > > use the tool.
> > > >
> > > Ideally yes, but that might not always be possible, as hbase API might
> > > change ahead of hbck2. Operators could then have problems to get a
> > working
> > > version of hbck2. Since hbck2 already has now many equivalent options
> for
> > > the ones from hbck1, I guess a first release would provide a working
> > hbck2
> > > that already brings a considerable number of fix methods to help with
> > most
> > > common inconsistencies issues seen in hbase 2 so far. And we could
> still
> > > fulfil the "HBCK2 should continuously evolve" principle by building
> from
> > > its master branch, in scenarios where a new fix was needed and
> > implemented
> > > back into hbck2.
> > >
> > > Em qui, 29 de ago de 2019 às 14:09, Peter Somogyi <psomogyi@apache.org
> >
> > > escreveu:
> > >
> > > > bq. Is it possible to put some hacks into HBCK2 to work around
> > > > the compatibility to fix the current state
> > > >
> > > > There are some classes around Replication which were introduced in
> > 2.1.0+
> > > > so I don't think we could easily solve it for 2.0.
> > > > For 2.1.1 the missing method is Hbck#scheduleServerCrashProcedure,
> > > probably
> > > > that could be solved with some workarounds or using reflection.
> > > >
> > > > bq. focus more on automation to let us know the next time we
> inevitably
> > > > break it again? ;)
> > > >
> > > > Sure! Based on this I think it should be a strong goal. We can set up
> > > > nightly builds for hbase-operator-tools repo that builds against the
> > > latest
> > > > development branches as well as checking compatibility with released
> > > > versions.
> > > >
> > > > bq. what would folks think about going with an hbck2 alpha release?
> > > > I'm fine with an alpha release but since "HBCK2 should continuously
> > > evolve"
> > > > it might be better to always use the latest codebase whenever you
> need
> > to
> > > > use the tool.
> > > >
> > > > On Thu, Aug 29, 2019 at 2:30 PM Wellington Chevreuil <
> > > > wellington.chevreuil@gmail.com> wrote:
> > > >
> > > > > I would favour having hbck2 release branches. Temporary
> compatibility
> > > > > breaks at compile time may be inevitable if we always point to the
> > > latest
> > > > > release. That could cause problems for operators trying to build
> > hbck2
> > > > (we
> > > > > are already seeing this happening with our support team). Another
> > > > argument
> > > > > for starting having hbck2 releases is that we already have quite a
> > few
> > > > > hbase 2 releases, yet, the main tool to fix inconsistencies is not
> > > easily
> > > > > available. And there's been considerable efforts lately to bring
> many
> > > of
> > > > > the fix options from hbck1 into hbck2, so what would folks think
> > about
> > > > > going with an hbck2 alpha release?
> > > > >
> > > > > Em qui, 29 de ago de 2019 às 13:20, Josh Elser <el...@apache.org>
> > > > > escreveu:
> > > > >
> > > > > > I still like one HBCK2 release as the goal.
> > > > > >
> > > > > > Is it possible to put some hacks into HBCK2 to work around the
> > > > > > compatibility to fix the current state and focus more on
> automation
> > > to
> > > > > > let us know the next time we inevitably break it again? ;)
> > > > > >
> > > > > > On 8/29/19 8:12 AM, Peter Somogyi wrote:
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > This topic came up a couple of times already but now we
> reached a
> > > > point
> > > > > > > when the recent HBCK2 is incompatible with older HBase
> releases,
> > > > > > > specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2
> against
> > > one
> > > > > of
> > > > > > > the previously mentioned releases you will get compilation
> > errors.
> > > > (mvn
> > > > > > > clean install -DskipTests -Dhbase.version=2.0.6)
> > > > > > >
> > > > > > > Our previous goal was to maintain compatibility with HBCK2 and
> > all
> > > > > HBase
> > > > > > 2
> > > > > > > releases. Now we missed this target.
> > > > > > >
> > > > > > > One option that we could do is to have different
> > branches/releases
> > > of
> > > > > > HBCK2
> > > > > > > targeted for specific HBase releases (e.g. branch-2.0 version
> of
> > > > > HBCK2).
> > > > > > > This probably makes the development on HBCK2 a bit harder since
> > > we'll
> > > > > > have
> > > > > > > to take care of multiple branches.
> > > > > > >
> > > > > > > Another option I could think of is to always build HBCK2 with
> the
> > > > > latest
> > > > > > > HBase release but have version checks on individual commands
> > where
> > > we
> > > > > > could
> > > > > > > decide if it is supported on that release line.
> > > > > > >
> > > > > > > What are your opinions on this?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Peter
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] HBCK2 compatibility
Posted by Stack <st...@duboce.net>.
I filed, https://issues.apache.org/jira/browse/HBASE-22956, for a first
release of hbck2.
S
On Fri, Aug 30, 2019 at 8:31 AM Peter Somogyi <ps...@apache.org> wrote:
> Thanks for all the feedback!
>
> Looks like my information about compatibility was incorrect and we do not
> need to build HBCK2 targeted for the actual HBase version. Having
> command-specific version checks that Stack mentioned looks good enough to
> deal with compatibility.
> Currently there are some issues with Replication on 2.0 releases. I'll work
> on some corrections and once it's done with hbck1+ coverage we can start to
> work on the first release!
>
> Peter
>
> On Thu, Aug 29, 2019 at 11:07 PM Sakthi <sa...@apache.org> wrote:
>
> > I like the last idea suggested by Stack. This way the core idea of
> keeping
> > both the dev process separate stays intact and also the operators have a
> > version of hbck to fix their clusters without worrying about
> compatibility
> > issues.
> >
> > On Thu, Aug 29, 2019 at 2:03 PM Stack <st...@duboce.net> wrote:
> >
> > > Could also just make a release now of hbase-operator-tools (or in a
> week
> > or
> > > so when we should have hbck1+ coverage in place) built against an
> > > up-to-date hbase release. It has the check version before running a
> > feature
> > > in place where it matters. Operators could use this on all currently
> > > shipping hbase2s?
> > >
> > > S
> > >
> > > On Thu, Aug 29, 2019 at 11:42 AM Stack <st...@duboce.net> wrote:
> > >
> > > > On Thu, Aug 29, 2019 at 7:34 AM Wellington Chevreuil <
> > > > wellington.chevreuil@gmail.com> wrote:
> > > >
> > > >> >
> > > >> > I do not think we need to compile HBCK2 with every releases?
> > > >> >
> > > >> Well, not with every release, was thinking in doing it whenever an
> > hbase
> > > >> release breaks compatibility.
> > > >>
> > > >> We just need
> > > >> > make sure that it can work with all the releases.
> > > >>
> > > >> This could be a solution as well, but I believe it would be harder
> to
> > > >> guarantee. Here the problem is:
> > > >> 1) A new hbase release changes one or more interfaces currently used
> > by
> > > >> hbck2;
> > > >> 2) We update hbck2 to depend on this new hbase release, and change
> > hbck2
> > > >> accordingly;
> > > >> 3) Operators need to run hbck2 to a previous hbase release. If they
> > try
> > > to
> > > >> build hbck2 against that version, it won't compile. If they build it
> > > with
> > > >> latest hbase version, it may give a runtime error, and now they have
> > no
> > > >> working tool to fix the problem.
> > > >> To avoid #3, we would need to add extra checks on the changes
> applied
> > on
> > > >> #2. Might become too complex.
> > > >>
> > > >> Thanks Wellington for above. I see issue now.
> > > >
> > > > So, we should make retroactive releases of hbase-operator-tools at
> > points
> > > > just before compat broke? Release could be named for the hbase2
> > versions
> > > > supported. Releases would make it easier on operators making it so
> they
> > > > don't have to build themselves? There'd be one only? Two maybe?
> > > >
> > > > Looking at changes to the Hbck Interface -- using this as gauge for
> > > > possible breakage points -- there aren't many. One release? Maybe
> two?
> > > >
> > > > S
> > > >
> > > >
> > > >
> > > >> If there are missing
> > > >> > methods, we just tell users you can not use several features.
> > > >>
> > > >> Fine for new fix options added, but what if some changes break
> basic,
> > > >> already working hbck2 methods.
> > > >>
> > > >>
> > > >> Em qui, 29 de ago de 2019 às 14:51, 张铎(Duo Zhang) <
> > > palomino219@gmail.com>
> > > >> escreveu:
> > > >>
> > > >> > I do not think we need to compile HBCK2 with every releases? We
> just
> > > >> need
> > > >> > make sure that it can work with all the releases. If there are
> > missing
> > > >> > methods, we just tell users you can not use several features.
> > > >> >
> > > >> > Wellington Chevreuil <we...@gmail.com>
> 于2019年8月29日周四
> > > >> > 下午9:39写道：
> > > >> >
> > > >> > > >
> > > >> > > > bq. what would folks think about going with an hbck2 alpha
> > > release?
> > > >> > > > I'm fine with an alpha release but since "HBCK2 should
> > > continuously
> > > >> > > evolve"
> > > >> > > > it might be better to always use the latest codebase whenever
> > you
> > > >> need
> > > >> > to
> > > >> > > > use the tool.
> > > >> > > >
> > > >> > > Ideally yes, but that might not always be possible, as hbase API
> > > might
> > > >> > > change ahead of hbck2. Operators could then have problems to
> get a
> > > >> > working
> > > >> > > version of hbck2. Since hbck2 already has now many equivalent
> > > options
> > > >> for
> > > >> > > the ones from hbck1, I guess a first release would provide a
> > working
> > > >> > hbck2
> > > >> > > that already brings a considerable number of fix methods to help
> > > with
> > > >> > most
> > > >> > > common inconsistencies issues seen in hbase 2 so far. And we
> could
> > > >> still
> > > >> > > fulfil the "HBCK2 should continuously evolve" principle by
> > building
> > > >> from
> > > >> > > its master branch, in scenarios where a new fix was needed and
> > > >> > implemented
> > > >> > > back into hbck2.
> > > >> > >
> > > >> > > Em qui, 29 de ago de 2019 às 14:09, Peter Somogyi <
> > > >> psomogyi@apache.org>
> > > >> > > escreveu:
> > > >> > >
> > > >> > > > bq. Is it possible to put some hacks into HBCK2 to work around
> > > >> > > > the compatibility to fix the current state
> > > >> > > >
> > > >> > > > There are some classes around Replication which were
> introduced
> > in
> > > >> > 2.1.0+
> > > >> > > > so I don't think we could easily solve it for 2.0.
> > > >> > > > For 2.1.1 the missing method is
> > Hbck#scheduleServerCrashProcedure,
> > > >> > > probably
> > > >> > > > that could be solved with some workarounds or using
> reflection.
> > > >> > > >
> > > >> > > > bq. focus more on automation to let us know the next time we
> > > >> inevitably
> > > >> > > > break it again? ;)
> > > >> > > >
> > > >> > > > Sure! Based on this I think it should be a strong goal. We can
> > set
> > > >> up
> > > >> > > > nightly builds for hbase-operator-tools repo that builds
> against
> > > the
> > > >> > > latest
> > > >> > > > development branches as well as checking compatibility with
> > > released
> > > >> > > > versions.
> > > >> > > >
> > > >> > > > bq. what would folks think about going with an hbck2 alpha
> > > release?
> > > >> > > > I'm fine with an alpha release but since "HBCK2 should
> > > continuously
> > > >> > > evolve"
> > > >> > > > it might be better to always use the latest codebase whenever
> > you
> > > >> need
> > > >> > to
> > > >> > > > use the tool.
> > > >> > > >
> > > >> > > > On Thu, Aug 29, 2019 at 2:30 PM Wellington Chevreuil <
> > > >> > > > wellington.chevreuil@gmail.com> wrote:
> > > >> > > >
> > > >> > > > > I would favour having hbck2 release branches. Temporary
> > > >> compatibility
> > > >> > > > > breaks at compile time may be inevitable if we always point
> to
> > > the
> > > >> > > latest
> > > >> > > > > release. That could cause problems for operators trying to
> > build
> > > >> > hbck2
> > > >> > > > (we
> > > >> > > > > are already seeing this happening with our support team).
> > > Another
> > > >> > > > argument
> > > >> > > > > for starting having hbck2 releases is that we already have
> > > quite a
> > > >> > few
> > > >> > > > > hbase 2 releases, yet, the main tool to fix inconsistencies
> is
> > > not
> > > >> > > easily
> > > >> > > > > available. And there's been considerable efforts lately to
> > bring
> > > >> many
> > > >> > > of
> > > >> > > > > the fix options from hbck1 into hbck2, so what would folks
> > think
> > > >> > about
> > > >> > > > > going with an hbck2 alpha release?
> > > >> > > > >
> > > >> > > > > Em qui, 29 de ago de 2019 às 13:20, Josh Elser <
> > > elserj@apache.org
> > > >> >
> > > >> > > > > escreveu:
> > > >> > > > >
> > > >> > > > > > I still like one HBCK2 release as the goal.
> > > >> > > > > >
> > > >> > > > > > Is it possible to put some hacks into HBCK2 to work around
> > the
> > > >> > > > > > compatibility to fix the current state and focus more on
> > > >> automation
> > > >> > > to
> > > >> > > > > > let us know the next time we inevitably break it again? ;)
> > > >> > > > > >
> > > >> > > > > > On 8/29/19 8:12 AM, Peter Somogyi wrote:
> > > >> > > > > > > Hi everyone,
> > > >> > > > > > >
> > > >> > > > > > > This topic came up a couple of times already but now we
> > > >> reached a
> > > >> > > > point
> > > >> > > > > > > when the recent HBCK2 is incompatible with older HBase
> > > >> releases,
> > > >> > > > > > > specifically 2.0.x, 2.1.0 and 2.1.1. When you build
> HBCK2
> > > >> against
> > > >> > > one
> > > >> > > > > of
> > > >> > > > > > > the previously mentioned releases you will get
> compilation
> > > >> > errors.
> > > >> > > > (mvn
> > > >> > > > > > > clean install -DskipTests -Dhbase.version=2.0.6)
> > > >> > > > > > >
> > > >> > > > > > > Our previous goal was to maintain compatibility with
> HBCK2
> > > and
> > > >> > all
> > > >> > > > > HBase
> > > >> > > > > > 2
> > > >> > > > > > > releases. Now we missed this target.
> > > >> > > > > > >
> > > >> > > > > > > One option that we could do is to have different
> > > >> > branches/releases
> > > >> > > of
> > > >> > > > > > HBCK2
> > > >> > > > > > > targeted for specific HBase releases (e.g. branch-2.0
> > > version
> > > >> of
> > > >> > > > > HBCK2).
> > > >> > > > > > > This probably makes the development on HBCK2 a bit
> harder
> > > >> since
> > > >> > > we'll
> > > >> > > > > > have
> > > >> > > > > > > to take care of multiple branches.
> > > >> > > > > > >
> > > >> > > > > > > Another option I could think of is to always build HBCK2
> > > with
> > > >> the
> > > >> > > > > latest
> > > >> > > > > > > HBase release but have version checks on individual
> > commands
> > > >> > where
> > > >> > > we
> > > >> > > > > > could
> > > >> > > > > > > decide if it is supported on that release line.
> > > >> > > > > > >
> > > >> > > > > > > What are your opinions on this?
> > > >> > > > > > >
> > > >> > > > > > > Thanks,
> > > >> > > > > > > Peter
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] HBCK2 compatibility
Posted by Peter Somogyi <ps...@apache.org>.
Thanks for all the feedback!
Looks like my information about compatibility was incorrect and we do not
need to build HBCK2 targeted for the actual HBase version. Having
command-specific version checks that Stack mentioned looks good enough to
deal with compatibility.
Currently there are some issues with Replication on 2.0 releases. I'll work
on some corrections and once it's done with hbck1+ coverage we can start to
work on the first release!
Peter
On Thu, Aug 29, 2019 at 11:07 PM Sakthi <sa...@apache.org> wrote:
> I like the last idea suggested by Stack. This way the core idea of keeping
> both the dev process separate stays intact and also the operators have a
> version of hbck to fix their clusters without worrying about compatibility
> issues.
>
> On Thu, Aug 29, 2019 at 2:03 PM Stack <st...@duboce.net> wrote:
>
> > Could also just make a release now of hbase-operator-tools (or in a week
> or
> > so when we should have hbck1+ coverage in place) built against an
> > up-to-date hbase release. It has the check version before running a
> feature
> > in place where it matters. Operators could use this on all currently
> > shipping hbase2s?
> >
> > S
> >
> > On Thu, Aug 29, 2019 at 11:42 AM Stack <st...@duboce.net> wrote:
> >
> > > On Thu, Aug 29, 2019 at 7:34 AM Wellington Chevreuil <
> > > wellington.chevreuil@gmail.com> wrote:
> > >
> > >> >
> > >> > I do not think we need to compile HBCK2 with every releases?
> > >> >
> > >> Well, not with every release, was thinking in doing it whenever an
> hbase
> > >> release breaks compatibility.
> > >>
> > >> We just need
> > >> > make sure that it can work with all the releases.
> > >>
> > >> This could be a solution as well, but I believe it would be harder to
> > >> guarantee. Here the problem is:
> > >> 1) A new hbase release changes one or more interfaces currently used
> by
> > >> hbck2;
> > >> 2) We update hbck2 to depend on this new hbase release, and change
> hbck2
> > >> accordingly;
> > >> 3) Operators need to run hbck2 to a previous hbase release. If they
> try
> > to
> > >> build hbck2 against that version, it won't compile. If they build it
> > with
> > >> latest hbase version, it may give a runtime error, and now they have
> no
> > >> working tool to fix the problem.
> > >> To avoid #3, we would need to add extra checks on the changes applied
> on
> > >> #2. Might become too complex.
> > >>
> > >> Thanks Wellington for above. I see issue now.
> > >
> > > So, we should make retroactive releases of hbase-operator-tools at
> points
> > > just before compat broke? Release could be named for the hbase2
> versions
> > > supported. Releases would make it easier on operators making it so they
> > > don't have to build themselves? There'd be one only? Two maybe?
> > >
> > > Looking at changes to the Hbck Interface -- using this as gauge for
> > > possible breakage points -- there aren't many. One release? Maybe two?
> > >
> > > S
> > >
> > >
> > >
> > >> If there are missing
> > >> > methods, we just tell users you can not use several features.
> > >>
> > >> Fine for new fix options added, but what if some changes break basic,
> > >> already working hbck2 methods.
> > >>
> > >>
> > >> Em qui, 29 de ago de 2019 às 14:51, 张铎(Duo Zhang) <
> > palomino219@gmail.com>
> > >> escreveu:
> > >>
> > >> > I do not think we need to compile HBCK2 with every releases? We just
> > >> need
> > >> > make sure that it can work with all the releases. If there are
> missing
> > >> > methods, we just tell users you can not use several features.
> > >> >
> > >> > Wellington Chevreuil <we...@gmail.com> 于2019年8月29日周四
> > >> > 下午9:39写道：
> > >> >
> > >> > > >
> > >> > > > bq. what would folks think about going with an hbck2 alpha
> > release?
> > >> > > > I'm fine with an alpha release but since "HBCK2 should
> > continuously
> > >> > > evolve"
> > >> > > > it might be better to always use the latest codebase whenever
> you
> > >> need
> > >> > to
> > >> > > > use the tool.
> > >> > > >
> > >> > > Ideally yes, but that might not always be possible, as hbase API
> > might
> > >> > > change ahead of hbck2. Operators could then have problems to get a
> > >> > working
> > >> > > version of hbck2. Since hbck2 already has now many equivalent
> > options
> > >> for
> > >> > > the ones from hbck1, I guess a first release would provide a
> working
> > >> > hbck2
> > >> > > that already brings a considerable number of fix methods to help
> > with
> > >> > most
> > >> > > common inconsistencies issues seen in hbase 2 so far. And we could
> > >> still
> > >> > > fulfil the "HBCK2 should continuously evolve" principle by
> building
> > >> from
> > >> > > its master branch, in scenarios where a new fix was needed and
> > >> > implemented
> > >> > > back into hbck2.
> > >> > >
> > >> > > Em qui, 29 de ago de 2019 às 14:09, Peter Somogyi <
> > >> psomogyi@apache.org>
> > >> > > escreveu:
> > >> > >
> > >> > > > bq. Is it possible to put some hacks into HBCK2 to work around
> > >> > > > the compatibility to fix the current state
> > >> > > >
> > >> > > > There are some classes around Replication which were introduced
> in
> > >> > 2.1.0+
> > >> > > > so I don't think we could easily solve it for 2.0.
> > >> > > > For 2.1.1 the missing method is
> Hbck#scheduleServerCrashProcedure,
> > >> > > probably
> > >> > > > that could be solved with some workarounds or using reflection.
> > >> > > >
> > >> > > > bq. focus more on automation to let us know the next time we
> > >> inevitably
> > >> > > > break it again? ;)
> > >> > > >
> > >> > > > Sure! Based on this I think it should be a strong goal. We can
> set
> > >> up
> > >> > > > nightly builds for hbase-operator-tools repo that builds against
> > the
> > >> > > latest
> > >> > > > development branches as well as checking compatibility with
> > released
> > >> > > > versions.
> > >> > > >
> > >> > > > bq. what would folks think about going with an hbck2 alpha
> > release?
> > >> > > > I'm fine with an alpha release but since "HBCK2 should
> > continuously
> > >> > > evolve"
> > >> > > > it might be better to always use the latest codebase whenever
> you
> > >> need
> > >> > to
> > >> > > > use the tool.
> > >> > > >
> > >> > > > On Thu, Aug 29, 2019 at 2:30 PM Wellington Chevreuil <
> > >> > > > wellington.chevreuil@gmail.com> wrote:
> > >> > > >
> > >> > > > > I would favour having hbck2 release branches. Temporary
> > >> compatibility
> > >> > > > > breaks at compile time may be inevitable if we always point to
> > the
> > >> > > latest
> > >> > > > > release. That could cause problems for operators trying to
> build
> > >> > hbck2
> > >> > > > (we
> > >> > > > > are already seeing this happening with our support team).
> > Another
> > >> > > > argument
> > >> > > > > for starting having hbck2 releases is that we already have
> > quite a
> > >> > few
> > >> > > > > hbase 2 releases, yet, the main tool to fix inconsistencies is
> > not
> > >> > > easily
> > >> > > > > available. And there's been considerable efforts lately to
> bring
> > >> many
> > >> > > of
> > >> > > > > the fix options from hbck1 into hbck2, so what would folks
> think
> > >> > about
> > >> > > > > going with an hbck2 alpha release?
> > >> > > > >
> > >> > > > > Em qui, 29 de ago de 2019 às 13:20, Josh Elser <
> > elserj@apache.org
> > >> >
> > >> > > > > escreveu:
> > >> > > > >
> > >> > > > > > I still like one HBCK2 release as the goal.
> > >> > > > > >
> > >> > > > > > Is it possible to put some hacks into HBCK2 to work around
> the
> > >> > > > > > compatibility to fix the current state and focus more on
> > >> automation
> > >> > > to
> > >> > > > > > let us know the next time we inevitably break it again? ;)
> > >> > > > > >
> > >> > > > > > On 8/29/19 8:12 AM, Peter Somogyi wrote:
> > >> > > > > > > Hi everyone,
> > >> > > > > > >
> > >> > > > > > > This topic came up a couple of times already but now we
> > >> reached a
> > >> > > > point
> > >> > > > > > > when the recent HBCK2 is incompatible with older HBase
> > >> releases,
> > >> > > > > > > specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2
> > >> against
> > >> > > one
> > >> > > > > of
> > >> > > > > > > the previously mentioned releases you will get compilation
> > >> > errors.
> > >> > > > (mvn
> > >> > > > > > > clean install -DskipTests -Dhbase.version=2.0.6)
> > >> > > > > > >
> > >> > > > > > > Our previous goal was to maintain compatibility with HBCK2
> > and
> > >> > all
> > >> > > > > HBase
> > >> > > > > > 2
> > >> > > > > > > releases. Now we missed this target.
> > >> > > > > > >
> > >> > > > > > > One option that we could do is to have different
> > >> > branches/releases
> > >> > > of
> > >> > > > > > HBCK2
> > >> > > > > > > targeted for specific HBase releases (e.g. branch-2.0
> > version
> > >> of
> > >> > > > > HBCK2).
> > >> > > > > > > This probably makes the development on HBCK2 a bit harder
> > >> since
> > >> > > we'll
> > >> > > > > > have
> > >> > > > > > > to take care of multiple branches.
> > >> > > > > > >
> > >> > > > > > > Another option I could think of is to always build HBCK2
> > with
> > >> the
> > >> > > > > latest
> > >> > > > > > > HBase release but have version checks on individual
> commands
> > >> > where
> > >> > > we
> > >> > > > > > could
> > >> > > > > > > decide if it is supported on that release line.
> > >> > > > > > >
> > >> > > > > > > What are your opinions on this?
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > Peter
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] HBCK2 compatibility
Posted by Wellington Chevreuil <we...@gmail.com>.
>
> Could also just make a release now of hbase-operator-tools (or in a week or
> so when we should have hbck1+ coverage in place) built against an
> up-to-date hbase release.
+1, I think current version has a reasonable number of fix options, and if
we can ensure it runs for valid 2.x releases, would be of great help for
users/operators.
Em qui, 29 de ago de 2019 às 22:07, Sakthi <sa...@apache.org> escreveu:
> I like the last idea suggested by Stack. This way the core idea of keeping
> both the dev process separate stays intact and also the operators have a
> version of hbck to fix their clusters without worrying about compatibility
> issues.
>
> On Thu, Aug 29, 2019 at 2:03 PM Stack <st...@duboce.net> wrote:
>
> > Could also just make a release now of hbase-operator-tools (or in a week
> or
> > so when we should have hbck1+ coverage in place) built against an
> > up-to-date hbase release. It has the check version before running a
> feature
> > in place where it matters. Operators could use this on all currently
> > shipping hbase2s?
> >
> > S
> >
> > On Thu, Aug 29, 2019 at 11:42 AM Stack <st...@duboce.net> wrote:
> >
> > > On Thu, Aug 29, 2019 at 7:34 AM Wellington Chevreuil <
> > > wellington.chevreuil@gmail.com> wrote:
> > >
> > >> >
> > >> > I do not think we need to compile HBCK2 with every releases?
> > >> >
> > >> Well, not with every release, was thinking in doing it whenever an
> hbase
> > >> release breaks compatibility.
> > >>
> > >> We just need
> > >> > make sure that it can work with all the releases.
> > >>
> > >> This could be a solution as well, but I believe it would be harder to
> > >> guarantee. Here the problem is:
> > >> 1) A new hbase release changes one or more interfaces currently used
> by
> > >> hbck2;
> > >> 2) We update hbck2 to depend on this new hbase release, and change
> hbck2
> > >> accordingly;
> > >> 3) Operators need to run hbck2 to a previous hbase release. If they
> try
> > to
> > >> build hbck2 against that version, it won't compile. If they build it
> > with
> > >> latest hbase version, it may give a runtime error, and now they have
> no
> > >> working tool to fix the problem.
> > >> To avoid #3, we would need to add extra checks on the changes applied
> on
> > >> #2. Might become too complex.
> > >>
> > >> Thanks Wellington for above. I see issue now.
> > >
> > > So, we should make retroactive releases of hbase-operator-tools at
> points
> > > just before compat broke? Release could be named for the hbase2
> versions
> > > supported. Releases would make it easier on operators making it so they
> > > don't have to build themselves? There'd be one only? Two maybe?
> > >
> > > Looking at changes to the Hbck Interface -- using this as gauge for
> > > possible breakage points -- there aren't many. One release? Maybe two?
> > >
> > > S
> > >
> > >
> > >
> > >> If there are missing
> > >> > methods, we just tell users you can not use several features.
> > >>
> > >> Fine for new fix options added, but what if some changes break basic,
> > >> already working hbck2 methods.
> > >>
> > >>
> > >> Em qui, 29 de ago de 2019 às 14:51, 张铎(Duo Zhang) <
> > palomino219@gmail.com>
> > >> escreveu:
> > >>
> > >> > I do not think we need to compile HBCK2 with every releases? We just
> > >> need
> > >> > make sure that it can work with all the releases. If there are
> missing
> > >> > methods, we just tell users you can not use several features.
> > >> >
> > >> > Wellington Chevreuil <we...@gmail.com> 于2019年8月29日周四
> > >> > 下午9:39写道：
> > >> >
> > >> > > >
> > >> > > > bq. what would folks think about going with an hbck2 alpha
> > release?
> > >> > > > I'm fine with an alpha release but since "HBCK2 should
> > continuously
> > >> > > evolve"
> > >> > > > it might be better to always use the latest codebase whenever
> you
> > >> need
> > >> > to
> > >> > > > use the tool.
> > >> > > >
> > >> > > Ideally yes, but that might not always be possible, as hbase API
> > might
> > >> > > change ahead of hbck2. Operators could then have problems to get a
> > >> > working
> > >> > > version of hbck2. Since hbck2 already has now many equivalent
> > options
> > >> for
> > >> > > the ones from hbck1, I guess a first release would provide a
> working
> > >> > hbck2
> > >> > > that already brings a considerable number of fix methods to help
> > with
> > >> > most
> > >> > > common inconsistencies issues seen in hbase 2 so far. And we could
> > >> still
> > >> > > fulfil the "HBCK2 should continuously evolve" principle by
> building
> > >> from
> > >> > > its master branch, in scenarios where a new fix was needed and
> > >> > implemented
> > >> > > back into hbck2.
> > >> > >
> > >> > > Em qui, 29 de ago de 2019 às 14:09, Peter Somogyi <
> > >> psomogyi@apache.org>
> > >> > > escreveu:
> > >> > >
> > >> > > > bq. Is it possible to put some hacks into HBCK2 to work around
> > >> > > > the compatibility to fix the current state
> > >> > > >
> > >> > > > There are some classes around Replication which were introduced
> in
> > >> > 2.1.0+
> > >> > > > so I don't think we could easily solve it for 2.0.
> > >> > > > For 2.1.1 the missing method is
> Hbck#scheduleServerCrashProcedure,
> > >> > > probably
> > >> > > > that could be solved with some workarounds or using reflection.
> > >> > > >
> > >> > > > bq. focus more on automation to let us know the next time we
> > >> inevitably
> > >> > > > break it again? ;)
> > >> > > >
> > >> > > > Sure! Based on this I think it should be a strong goal. We can
> set
> > >> up
> > >> > > > nightly builds for hbase-operator-tools repo that builds against
> > the
> > >> > > latest
> > >> > > > development branches as well as checking compatibility with
> > released
> > >> > > > versions.
> > >> > > >
> > >> > > > bq. what would folks think about going with an hbck2 alpha
> > release?
> > >> > > > I'm fine with an alpha release but since "HBCK2 should
> > continuously
> > >> > > evolve"
> > >> > > > it might be better to always use the latest codebase whenever
> you
> > >> need
> > >> > to
> > >> > > > use the tool.
> > >> > > >
> > >> > > > On Thu, Aug 29, 2019 at 2:30 PM Wellington Chevreuil <
> > >> > > > wellington.chevreuil@gmail.com> wrote:
> > >> > > >
> > >> > > > > I would favour having hbck2 release branches. Temporary
> > >> compatibility
> > >> > > > > breaks at compile time may be inevitable if we always point to
> > the
> > >> > > latest
> > >> > > > > release. That could cause problems for operators trying to
> build
> > >> > hbck2
> > >> > > > (we
> > >> > > > > are already seeing this happening with our support team).
> > Another
> > >> > > > argument
> > >> > > > > for starting having hbck2 releases is that we already have
> > quite a
> > >> > few
> > >> > > > > hbase 2 releases, yet, the main tool to fix inconsistencies is
> > not
> > >> > > easily
> > >> > > > > available. And there's been considerable efforts lately to
> bring
> > >> many
> > >> > > of
> > >> > > > > the fix options from hbck1 into hbck2, so what would folks
> think
> > >> > about
> > >> > > > > going with an hbck2 alpha release?
> > >> > > > >
> > >> > > > > Em qui, 29 de ago de 2019 às 13:20, Josh Elser <
> > elserj@apache.org
> > >> >
> > >> > > > > escreveu:
> > >> > > > >
> > >> > > > > > I still like one HBCK2 release as the goal.
> > >> > > > > >
> > >> > > > > > Is it possible to put some hacks into HBCK2 to work around
> the
> > >> > > > > > compatibility to fix the current state and focus more on
> > >> automation
> > >> > > to
> > >> > > > > > let us know the next time we inevitably break it again? ;)
> > >> > > > > >
> > >> > > > > > On 8/29/19 8:12 AM, Peter Somogyi wrote:
> > >> > > > > > > Hi everyone,
> > >> > > > > > >
> > >> > > > > > > This topic came up a couple of times already but now we
> > >> reached a
> > >> > > > point
> > >> > > > > > > when the recent HBCK2 is incompatible with older HBase
> > >> releases,
> > >> > > > > > > specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2
> > >> against
> > >> > > one
> > >> > > > > of
> > >> > > > > > > the previously mentioned releases you will get compilation
> > >> > errors.
> > >> > > > (mvn
> > >> > > > > > > clean install -DskipTests -Dhbase.version=2.0.6)
> > >> > > > > > >
> > >> > > > > > > Our previous goal was to maintain compatibility with HBCK2
> > and
> > >> > all
> > >> > > > > HBase
> > >> > > > > > 2
> > >> > > > > > > releases. Now we missed this target.
> > >> > > > > > >
> > >> > > > > > > One option that we could do is to have different
> > >> > branches/releases
> > >> > > of
> > >> > > > > > HBCK2
> > >> > > > > > > targeted for specific HBase releases (e.g. branch-2.0
> > version
> > >> of
> > >> > > > > HBCK2).
> > >> > > > > > > This probably makes the development on HBCK2 a bit harder
> > >> since
> > >> > > we'll
> > >> > > > > > have
> > >> > > > > > > to take care of multiple branches.
> > >> > > > > > >
> > >> > > > > > > Another option I could think of is to always build HBCK2
> > with
> > >> the
> > >> > > > > latest
> > >> > > > > > > HBase release but have version checks on individual
> commands
> > >> > where
> > >> > > we
> > >> > > > > > could
> > >> > > > > > > decide if it is supported on that release line.
> > >> > > > > > >
> > >> > > > > > > What are your opinions on this?
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > Peter
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] HBCK2 compatibility
Posted by Sakthi <sa...@apache.org>.
I like the last idea suggested by Stack. This way the core idea of keeping
both the dev process separate stays intact and also the operators have a
version of hbck to fix their clusters without worrying about compatibility
issues.
On Thu, Aug 29, 2019 at 2:03 PM Stack <st...@duboce.net> wrote:
> Could also just make a release now of hbase-operator-tools (or in a week or
> so when we should have hbck1+ coverage in place) built against an
> up-to-date hbase release. It has the check version before running a feature
> in place where it matters. Operators could use this on all currently
> shipping hbase2s?
>
> S
>
> On Thu, Aug 29, 2019 at 11:42 AM Stack <st...@duboce.net> wrote:
>
> > On Thu, Aug 29, 2019 at 7:34 AM Wellington Chevreuil <
> > wellington.chevreuil@gmail.com> wrote:
> >
> >> >
> >> > I do not think we need to compile HBCK2 with every releases?
> >> >
> >> Well, not with every release, was thinking in doing it whenever an hbase
> >> release breaks compatibility.
> >>
> >> We just need
> >> > make sure that it can work with all the releases.
> >>
> >> This could be a solution as well, but I believe it would be harder to
> >> guarantee. Here the problem is:
> >> 1) A new hbase release changes one or more interfaces currently used by
> >> hbck2;
> >> 2) We update hbck2 to depend on this new hbase release, and change hbck2
> >> accordingly;
> >> 3) Operators need to run hbck2 to a previous hbase release. If they try
> to
> >> build hbck2 against that version, it won't compile. If they build it
> with
> >> latest hbase version, it may give a runtime error, and now they have no
> >> working tool to fix the problem.
> >> To avoid #3, we would need to add extra checks on the changes applied on
> >> #2. Might become too complex.
> >>
> >> Thanks Wellington for above. I see issue now.
> >
> > So, we should make retroactive releases of hbase-operator-tools at points
> > just before compat broke? Release could be named for the hbase2 versions
> > supported. Releases would make it easier on operators making it so they
> > don't have to build themselves? There'd be one only? Two maybe?
> >
> > Looking at changes to the Hbck Interface -- using this as gauge for
> > possible breakage points -- there aren't many. One release? Maybe two?
> >
> > S
> >
> >
> >
> >> If there are missing
> >> > methods, we just tell users you can not use several features.
> >>
> >> Fine for new fix options added, but what if some changes break basic,
> >> already working hbck2 methods.
> >>
> >>
> >> Em qui, 29 de ago de 2019 às 14:51, 张铎(Duo Zhang) <
> palomino219@gmail.com>
> >> escreveu:
> >>
> >> > I do not think we need to compile HBCK2 with every releases? We just
> >> need
> >> > make sure that it can work with all the releases. If there are missing
> >> > methods, we just tell users you can not use several features.
> >> >
> >> > Wellington Chevreuil <we...@gmail.com> 于2019年8月29日周四
> >> > 下午9:39写道：
> >> >
> >> > > >
> >> > > > bq. what would folks think about going with an hbck2 alpha
> release?
> >> > > > I'm fine with an alpha release but since "HBCK2 should
> continuously
> >> > > evolve"
> >> > > > it might be better to always use the latest codebase whenever you
> >> need
> >> > to
> >> > > > use the tool.
> >> > > >
> >> > > Ideally yes, but that might not always be possible, as hbase API
> might
> >> > > change ahead of hbck2. Operators could then have problems to get a
> >> > working
> >> > > version of hbck2. Since hbck2 already has now many equivalent
> options
> >> for
> >> > > the ones from hbck1, I guess a first release would provide a working
> >> > hbck2
> >> > > that already brings a considerable number of fix methods to help
> with
> >> > most
> >> > > common inconsistencies issues seen in hbase 2 so far. And we could
> >> still
> >> > > fulfil the "HBCK2 should continuously evolve" principle by building
> >> from
> >> > > its master branch, in scenarios where a new fix was needed and
> >> > implemented
> >> > > back into hbck2.
> >> > >
> >> > > Em qui, 29 de ago de 2019 às 14:09, Peter Somogyi <
> >> psomogyi@apache.org>
> >> > > escreveu:
> >> > >
> >> > > > bq. Is it possible to put some hacks into HBCK2 to work around
> >> > > > the compatibility to fix the current state
> >> > > >
> >> > > > There are some classes around Replication which were introduced in
> >> > 2.1.0+
> >> > > > so I don't think we could easily solve it for 2.0.
> >> > > > For 2.1.1 the missing method is Hbck#scheduleServerCrashProcedure,
> >> > > probably
> >> > > > that could be solved with some workarounds or using reflection.
> >> > > >
> >> > > > bq. focus more on automation to let us know the next time we
> >> inevitably
> >> > > > break it again? ;)
> >> > > >
> >> > > > Sure! Based on this I think it should be a strong goal. We can set
> >> up
> >> > > > nightly builds for hbase-operator-tools repo that builds against
> the
> >> > > latest
> >> > > > development branches as well as checking compatibility with
> released
> >> > > > versions.
> >> > > >
> >> > > > bq. what would folks think about going with an hbck2 alpha
> release?
> >> > > > I'm fine with an alpha release but since "HBCK2 should
> continuously
> >> > > evolve"
> >> > > > it might be better to always use the latest codebase whenever you
> >> need
> >> > to
> >> > > > use the tool.
> >> > > >
> >> > > > On Thu, Aug 29, 2019 at 2:30 PM Wellington Chevreuil <
> >> > > > wellington.chevreuil@gmail.com> wrote:
> >> > > >
> >> > > > > I would favour having hbck2 release branches. Temporary
> >> compatibility
> >> > > > > breaks at compile time may be inevitable if we always point to
> the
> >> > > latest
> >> > > > > release. That could cause problems for operators trying to build
> >> > hbck2
> >> > > > (we
> >> > > > > are already seeing this happening with our support team).
> Another
> >> > > > argument
> >> > > > > for starting having hbck2 releases is that we already have
> quite a
> >> > few
> >> > > > > hbase 2 releases, yet, the main tool to fix inconsistencies is
> not
> >> > > easily
> >> > > > > available. And there's been considerable efforts lately to bring
> >> many
> >> > > of
> >> > > > > the fix options from hbck1 into hbck2, so what would folks think
> >> > about
> >> > > > > going with an hbck2 alpha release?
> >> > > > >
> >> > > > > Em qui, 29 de ago de 2019 às 13:20, Josh Elser <
> elserj@apache.org
> >> >
> >> > > > > escreveu:
> >> > > > >
> >> > > > > > I still like one HBCK2 release as the goal.
> >> > > > > >
> >> > > > > > Is it possible to put some hacks into HBCK2 to work around the
> >> > > > > > compatibility to fix the current state and focus more on
> >> automation
> >> > > to
> >> > > > > > let us know the next time we inevitably break it again? ;)
> >> > > > > >
> >> > > > > > On 8/29/19 8:12 AM, Peter Somogyi wrote:
> >> > > > > > > Hi everyone,
> >> > > > > > >
> >> > > > > > > This topic came up a couple of times already but now we
> >> reached a
> >> > > > point
> >> > > > > > > when the recent HBCK2 is incompatible with older HBase
> >> releases,
> >> > > > > > > specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2
> >> against
> >> > > one
> >> > > > > of
> >> > > > > > > the previously mentioned releases you will get compilation
> >> > errors.
> >> > > > (mvn
> >> > > > > > > clean install -DskipTests -Dhbase.version=2.0.6)
> >> > > > > > >
> >> > > > > > > Our previous goal was to maintain compatibility with HBCK2
> and
> >> > all
> >> > > > > HBase
> >> > > > > > 2
> >> > > > > > > releases. Now we missed this target.
> >> > > > > > >
> >> > > > > > > One option that we could do is to have different
> >> > branches/releases
> >> > > of
> >> > > > > > HBCK2
> >> > > > > > > targeted for specific HBase releases (e.g. branch-2.0
> version
> >> of
> >> > > > > HBCK2).
> >> > > > > > > This probably makes the development on HBCK2 a bit harder
> >> since
> >> > > we'll
> >> > > > > > have
> >> > > > > > > to take care of multiple branches.
> >> > > > > > >
> >> > > > > > > Another option I could think of is to always build HBCK2
> with
> >> the
> >> > > > > latest
> >> > > > > > > HBase release but have version checks on individual commands
> >> > where
> >> > > we
> >> > > > > > could
> >> > > > > > > decide if it is supported on that release line.
> >> > > > > > >
> >> > > > > > > What are your opinions on this?
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > Peter
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: [DISCUSS] HBCK2 compatibility
Posted by Stack <st...@duboce.net>.
Could also just make a release now of hbase-operator-tools (or in a week or
so when we should have hbck1+ coverage in place) built against an
up-to-date hbase release. It has the check version before running a feature
in place where it matters. Operators could use this on all currently
shipping hbase2s?
S
On Thu, Aug 29, 2019 at 11:42 AM Stack <st...@duboce.net> wrote:
> On Thu, Aug 29, 2019 at 7:34 AM Wellington Chevreuil <
> wellington.chevreuil@gmail.com> wrote:
>
>> >
>> > I do not think we need to compile HBCK2 with every releases?
>> >
>> Well, not with every release, was thinking in doing it whenever an hbase
>> release breaks compatibility.
>>
>> We just need
>> > make sure that it can work with all the releases.
>>
>> This could be a solution as well, but I believe it would be harder to
>> guarantee. Here the problem is:
>> 1) A new hbase release changes one or more interfaces currently used by
>> hbck2;
>> 2) We update hbck2 to depend on this new hbase release, and change hbck2
>> accordingly;
>> 3) Operators need to run hbck2 to a previous hbase release. If they try to
>> build hbck2 against that version, it won't compile. If they build it with
>> latest hbase version, it may give a runtime error, and now they have no
>> working tool to fix the problem.
>> To avoid #3, we would need to add extra checks on the changes applied on
>> #2. Might become too complex.
>>
>> Thanks Wellington for above. I see issue now.
>
> So, we should make retroactive releases of hbase-operator-tools at points
> just before compat broke? Release could be named for the hbase2 versions
> supported. Releases would make it easier on operators making it so they
> don't have to build themselves? There'd be one only? Two maybe?
>
> Looking at changes to the Hbck Interface -- using this as gauge for
> possible breakage points -- there aren't many. One release? Maybe two?
>
> S
>
>
>
>> If there are missing
>> > methods, we just tell users you can not use several features.
>>
>> Fine for new fix options added, but what if some changes break basic,
>> already working hbck2 methods.
>>
>>
>> Em qui, 29 de ago de 2019 às 14:51, 张铎(Duo Zhang) <pa...@gmail.com>
>> escreveu:
>>
>> > I do not think we need to compile HBCK2 with every releases? We just
>> need
>> > make sure that it can work with all the releases. If there are missing
>> > methods, we just tell users you can not use several features.
>> >
>> > Wellington Chevreuil <we...@gmail.com> 于2019年8月29日周四
>> > 下午9:39写道：
>> >
>> > > >
>> > > > bq. what would folks think about going with an hbck2 alpha release?
>> > > > I'm fine with an alpha release but since "HBCK2 should continuously
>> > > evolve"
>> > > > it might be better to always use the latest codebase whenever you
>> need
>> > to
>> > > > use the tool.
>> > > >
>> > > Ideally yes, but that might not always be possible, as hbase API might
>> > > change ahead of hbck2. Operators could then have problems to get a
>> > working
>> > > version of hbck2. Since hbck2 already has now many equivalent options
>> for
>> > > the ones from hbck1, I guess a first release would provide a working
>> > hbck2
>> > > that already brings a considerable number of fix methods to help with
>> > most
>> > > common inconsistencies issues seen in hbase 2 so far. And we could
>> still
>> > > fulfil the "HBCK2 should continuously evolve" principle by building
>> from
>> > > its master branch, in scenarios where a new fix was needed and
>> > implemented
>> > > back into hbck2.
>> > >
>> > > Em qui, 29 de ago de 2019 às 14:09, Peter Somogyi <
>> psomogyi@apache.org>
>> > > escreveu:
>> > >
>> > > > bq. Is it possible to put some hacks into HBCK2 to work around
>> > > > the compatibility to fix the current state
>> > > >
>> > > > There are some classes around Replication which were introduced in
>> > 2.1.0+
>> > > > so I don't think we could easily solve it for 2.0.
>> > > > For 2.1.1 the missing method is Hbck#scheduleServerCrashProcedure,
>> > > probably
>> > > > that could be solved with some workarounds or using reflection.
>> > > >
>> > > > bq. focus more on automation to let us know the next time we
>> inevitably
>> > > > break it again? ;)
>> > > >
>> > > > Sure! Based on this I think it should be a strong goal. We can set
>> up
>> > > > nightly builds for hbase-operator-tools repo that builds against the
>> > > latest
>> > > > development branches as well as checking compatibility with released
>> > > > versions.
>> > > >
>> > > > bq. what would folks think about going with an hbck2 alpha release?
>> > > > I'm fine with an alpha release but since "HBCK2 should continuously
>> > > evolve"
>> > > > it might be better to always use the latest codebase whenever you
>> need
>> > to
>> > > > use the tool.
>> > > >
>> > > > On Thu, Aug 29, 2019 at 2:30 PM Wellington Chevreuil <
>> > > > wellington.chevreuil@gmail.com> wrote:
>> > > >
>> > > > > I would favour having hbck2 release branches. Temporary
>> compatibility
>> > > > > breaks at compile time may be inevitable if we always point to the
>> > > latest
>> > > > > release. That could cause problems for operators trying to build
>> > hbck2
>> > > > (we
>> > > > > are already seeing this happening with our support team). Another
>> > > > argument
>> > > > > for starting having hbck2 releases is that we already have quite a
>> > few
>> > > > > hbase 2 releases, yet, the main tool to fix inconsistencies is not
>> > > easily
>> > > > > available. And there's been considerable efforts lately to bring
>> many
>> > > of
>> > > > > the fix options from hbck1 into hbck2, so what would folks think
>> > about
>> > > > > going with an hbck2 alpha release?
>> > > > >
>> > > > > Em qui, 29 de ago de 2019 às 13:20, Josh Elser <elserj@apache.org
>> >
>> > > > > escreveu:
>> > > > >
>> > > > > > I still like one HBCK2 release as the goal.
>> > > > > >
>> > > > > > Is it possible to put some hacks into HBCK2 to work around the
>> > > > > > compatibility to fix the current state and focus more on
>> automation
>> > > to
>> > > > > > let us know the next time we inevitably break it again? ;)
>> > > > > >
>> > > > > > On 8/29/19 8:12 AM, Peter Somogyi wrote:
>> > > > > > > Hi everyone,
>> > > > > > >
>> > > > > > > This topic came up a couple of times already but now we
>> reached a
>> > > > point
>> > > > > > > when the recent HBCK2 is incompatible with older HBase
>> releases,
>> > > > > > > specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2
>> against
>> > > one
>> > > > > of
>> > > > > > > the previously mentioned releases you will get compilation
>> > errors.
>> > > > (mvn
>> > > > > > > clean install -DskipTests -Dhbase.version=2.0.6)
>> > > > > > >
>> > > > > > > Our previous goal was to maintain compatibility with HBCK2 and
>> > all
>> > > > > HBase
>> > > > > > 2
>> > > > > > > releases. Now we missed this target.
>> > > > > > >
>> > > > > > > One option that we could do is to have different
>> > branches/releases
>> > > of
>> > > > > > HBCK2
>> > > > > > > targeted for specific HBase releases (e.g. branch-2.0 version
>> of
>> > > > > HBCK2).
>> > > > > > > This probably makes the development on HBCK2 a bit harder
>> since
>> > > we'll
>> > > > > > have
>> > > > > > > to take care of multiple branches.
>> > > > > > >
>> > > > > > > Another option I could think of is to always build HBCK2 with
>> the
>> > > > > latest
>> > > > > > > HBase release but have version checks on individual commands
>> > where
>> > > we
>> > > > > > could
>> > > > > > > decide if it is supported on that release line.
>> > > > > > >
>> > > > > > > What are your opinions on this?
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Peter
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] HBCK2 compatibility
Posted by Stack <st...@duboce.net>.
On Thu, Aug 29, 2019 at 7:34 AM Wellington Chevreuil <
wellington.chevreuil@gmail.com> wrote:
> >
> > I do not think we need to compile HBCK2 with every releases?
> >
> Well, not with every release, was thinking in doing it whenever an hbase
> release breaks compatibility.
>
> We just need
> > make sure that it can work with all the releases.
>
> This could be a solution as well, but I believe it would be harder to
> guarantee. Here the problem is:
> 1) A new hbase release changes one or more interfaces currently used by
> hbck2;
> 2) We update hbck2 to depend on this new hbase release, and change hbck2
> accordingly;
> 3) Operators need to run hbck2 to a previous hbase release. If they try to
> build hbck2 against that version, it won't compile. If they build it with
> latest hbase version, it may give a runtime error, and now they have no
> working tool to fix the problem.
> To avoid #3, we would need to add extra checks on the changes applied on
> #2. Might become too complex.
>
> Thanks Wellington for above. I see issue now.
So, we should make retroactive releases of hbase-operator-tools at points
just before compat broke? Release could be named for the hbase2 versions
supported. Releases would make it easier on operators making it so they
don't have to build themselves? There'd be one only? Two maybe?
Looking at changes to the Hbck Interface -- using this as gauge for
possible breakage points -- there aren't many. One release? Maybe two?
S
> If there are missing
> > methods, we just tell users you can not use several features.
>
> Fine for new fix options added, but what if some changes break basic,
> already working hbck2 methods.
>
>
> Em qui, 29 de ago de 2019 às 14:51, 张铎(Duo Zhang) <pa...@gmail.com>
> escreveu:
>
> > I do not think we need to compile HBCK2 with every releases? We just need
> > make sure that it can work with all the releases. If there are missing
> > methods, we just tell users you can not use several features.
> >
> > Wellington Chevreuil <we...@gmail.com> 于2019年8月29日周四
> > 下午9:39写道：
> >
> > > >
> > > > bq. what would folks think about going with an hbck2 alpha release?
> > > > I'm fine with an alpha release but since "HBCK2 should continuously
> > > evolve"
> > > > it might be better to always use the latest codebase whenever you
> need
> > to
> > > > use the tool.
> > > >
> > > Ideally yes, but that might not always be possible, as hbase API might
> > > change ahead of hbck2. Operators could then have problems to get a
> > working
> > > version of hbck2. Since hbck2 already has now many equivalent options
> for
> > > the ones from hbck1, I guess a first release would provide a working
> > hbck2
> > > that already brings a considerable number of fix methods to help with
> > most
> > > common inconsistencies issues seen in hbase 2 so far. And we could
> still
> > > fulfil the "HBCK2 should continuously evolve" principle by building
> from
> > > its master branch, in scenarios where a new fix was needed and
> > implemented
> > > back into hbck2.
> > >
> > > Em qui, 29 de ago de 2019 às 14:09, Peter Somogyi <psomogyi@apache.org
> >
> > > escreveu:
> > >
> > > > bq. Is it possible to put some hacks into HBCK2 to work around
> > > > the compatibility to fix the current state
> > > >
> > > > There are some classes around Replication which were introduced in
> > 2.1.0+
> > > > so I don't think we could easily solve it for 2.0.
> > > > For 2.1.1 the missing method is Hbck#scheduleServerCrashProcedure,
> > > probably
> > > > that could be solved with some workarounds or using reflection.
> > > >
> > > > bq. focus more on automation to let us know the next time we
> inevitably
> > > > break it again? ;)
> > > >
> > > > Sure! Based on this I think it should be a strong goal. We can set up
> > > > nightly builds for hbase-operator-tools repo that builds against the
> > > latest
> > > > development branches as well as checking compatibility with released
> > > > versions.
> > > >
> > > > bq. what would folks think about going with an hbck2 alpha release?
> > > > I'm fine with an alpha release but since "HBCK2 should continuously
> > > evolve"
> > > > it might be better to always use the latest codebase whenever you
> need
> > to
> > > > use the tool.
> > > >
> > > > On Thu, Aug 29, 2019 at 2:30 PM Wellington Chevreuil <
> > > > wellington.chevreuil@gmail.com> wrote:
> > > >
> > > > > I would favour having hbck2 release branches. Temporary
> compatibility
> > > > > breaks at compile time may be inevitable if we always point to the
> > > latest
> > > > > release. That could cause problems for operators trying to build
> > hbck2
> > > > (we
> > > > > are already seeing this happening with our support team). Another
> > > > argument
> > > > > for starting having hbck2 releases is that we already have quite a
> > few
> > > > > hbase 2 releases, yet, the main tool to fix inconsistencies is not
> > > easily
> > > > > available. And there's been considerable efforts lately to bring
> many
> > > of
> > > > > the fix options from hbck1 into hbck2, so what would folks think
> > about
> > > > > going with an hbck2 alpha release?
> > > > >
> > > > > Em qui, 29 de ago de 2019 às 13:20, Josh Elser <el...@apache.org>
> > > > > escreveu:
> > > > >
> > > > > > I still like one HBCK2 release as the goal.
> > > > > >
> > > > > > Is it possible to put some hacks into HBCK2 to work around the
> > > > > > compatibility to fix the current state and focus more on
> automation
> > > to
> > > > > > let us know the next time we inevitably break it again? ;)
> > > > > >
> > > > > > On 8/29/19 8:12 AM, Peter Somogyi wrote:
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > This topic came up a couple of times already but now we
> reached a
> > > > point
> > > > > > > when the recent HBCK2 is incompatible with older HBase
> releases,
> > > > > > > specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2
> against
> > > one
> > > > > of
> > > > > > > the previously mentioned releases you will get compilation
> > errors.
> > > > (mvn
> > > > > > > clean install -DskipTests -Dhbase.version=2.0.6)
> > > > > > >
> > > > > > > Our previous goal was to maintain compatibility with HBCK2 and
> > all
> > > > > HBase
> > > > > > 2
> > > > > > > releases. Now we missed this target.
> > > > > > >
> > > > > > > One option that we could do is to have different
> > branches/releases
> > > of
> > > > > > HBCK2
> > > > > > > targeted for specific HBase releases (e.g. branch-2.0 version
> of
> > > > > HBCK2).
> > > > > > > This probably makes the development on HBCK2 a bit harder since
> > > we'll
> > > > > > have
> > > > > > > to take care of multiple branches.
> > > > > > >
> > > > > > > Another option I could think of is to always build HBCK2 with
> the
> > > > > latest
> > > > > > > HBase release but have version checks on individual commands
> > where
> > > we
> > > > > > could
> > > > > > > decide if it is supported on that release line.
> > > > > > >
> > > > > > > What are your opinions on this?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Peter
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] HBCK2 compatibility
Posted by Wellington Chevreuil <we...@gmail.com>.
>
> I do not think we need to compile HBCK2 with every releases?
>
Well, not with every release, was thinking in doing it whenever an hbase
release breaks compatibility.
We just need
> make sure that it can work with all the releases.
This could be a solution as well, but I believe it would be harder to
guarantee. Here the problem is:
1) A new hbase release changes one or more interfaces currently used by
hbck2;
2) We update hbck2 to depend on this new hbase release, and change hbck2
accordingly;
3) Operators need to run hbck2 to a previous hbase release. If they try to
build hbck2 against that version, it won't compile. If they build it with
latest hbase version, it may give a runtime error, and now they have no
working tool to fix the problem.
To avoid #3, we would need to add extra checks on the changes applied on
#2. Might become too complex.
If there are missing
> methods, we just tell users you can not use several features.
Fine for new fix options added, but what if some changes break basic,
already working hbck2 methods.
Em qui, 29 de ago de 2019 às 14:51, 张铎(Duo Zhang) <pa...@gmail.com>
escreveu:
> I do not think we need to compile HBCK2 with every releases? We just need
> make sure that it can work with all the releases. If there are missing
> methods, we just tell users you can not use several features.
>
> Wellington Chevreuil <we...@gmail.com> 于2019年8月29日周四
> 下午9:39写道：
>
> > >
> > > bq. what would folks think about going with an hbck2 alpha release?
> > > I'm fine with an alpha release but since "HBCK2 should continuously
> > evolve"
> > > it might be better to always use the latest codebase whenever you need
> to
> > > use the tool.
> > >
> > Ideally yes, but that might not always be possible, as hbase API might
> > change ahead of hbck2. Operators could then have problems to get a
> working
> > version of hbck2. Since hbck2 already has now many equivalent options for
> > the ones from hbck1, I guess a first release would provide a working
> hbck2
> > that already brings a considerable number of fix methods to help with
> most
> > common inconsistencies issues seen in hbase 2 so far. And we could still
> > fulfil the "HBCK2 should continuously evolve" principle by building from
> > its master branch, in scenarios where a new fix was needed and
> implemented
> > back into hbck2.
> >
> > Em qui, 29 de ago de 2019 às 14:09, Peter Somogyi <ps...@apache.org>
> > escreveu:
> >
> > > bq. Is it possible to put some hacks into HBCK2 to work around
> > > the compatibility to fix the current state
> > >
> > > There are some classes around Replication which were introduced in
> 2.1.0+
> > > so I don't think we could easily solve it for 2.0.
> > > For 2.1.1 the missing method is Hbck#scheduleServerCrashProcedure,
> > probably
> > > that could be solved with some workarounds or using reflection.
> > >
> > > bq. focus more on automation to let us know the next time we inevitably
> > > break it again? ;)
> > >
> > > Sure! Based on this I think it should be a strong goal. We can set up
> > > nightly builds for hbase-operator-tools repo that builds against the
> > latest
> > > development branches as well as checking compatibility with released
> > > versions.
> > >
> > > bq. what would folks think about going with an hbck2 alpha release?
> > > I'm fine with an alpha release but since "HBCK2 should continuously
> > evolve"
> > > it might be better to always use the latest codebase whenever you need
> to
> > > use the tool.
> > >
> > > On Thu, Aug 29, 2019 at 2:30 PM Wellington Chevreuil <
> > > wellington.chevreuil@gmail.com> wrote:
> > >
> > > > I would favour having hbck2 release branches. Temporary compatibility
> > > > breaks at compile time may be inevitable if we always point to the
> > latest
> > > > release. That could cause problems for operators trying to build
> hbck2
> > > (we
> > > > are already seeing this happening with our support team). Another
> > > argument
> > > > for starting having hbck2 releases is that we already have quite a
> few
> > > > hbase 2 releases, yet, the main tool to fix inconsistencies is not
> > easily
> > > > available. And there's been considerable efforts lately to bring many
> > of
> > > > the fix options from hbck1 into hbck2, so what would folks think
> about
> > > > going with an hbck2 alpha release?
> > > >
> > > > Em qui, 29 de ago de 2019 às 13:20, Josh Elser <el...@apache.org>
> > > > escreveu:
> > > >
> > > > > I still like one HBCK2 release as the goal.
> > > > >
> > > > > Is it possible to put some hacks into HBCK2 to work around the
> > > > > compatibility to fix the current state and focus more on automation
> > to
> > > > > let us know the next time we inevitably break it again? ;)
> > > > >
> > > > > On 8/29/19 8:12 AM, Peter Somogyi wrote:
> > > > > > Hi everyone,
> > > > > >
> > > > > > This topic came up a couple of times already but now we reached a
> > > point
> > > > > > when the recent HBCK2 is incompatible with older HBase releases,
> > > > > > specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2 against
> > one
> > > > of
> > > > > > the previously mentioned releases you will get compilation
> errors.
> > > (mvn
> > > > > > clean install -DskipTests -Dhbase.version=2.0.6)
> > > > > >
> > > > > > Our previous goal was to maintain compatibility with HBCK2 and
> all
> > > > HBase
> > > > > 2
> > > > > > releases. Now we missed this target.
> > > > > >
> > > > > > One option that we could do is to have different
> branches/releases
> > of
> > > > > HBCK2
> > > > > > targeted for specific HBase releases (e.g. branch-2.0 version of
> > > > HBCK2).
> > > > > > This probably makes the development on HBCK2 a bit harder since
> > we'll
> > > > > have
> > > > > > to take care of multiple branches.
> > > > > >
> > > > > > Another option I could think of is to always build HBCK2 with the
> > > > latest
> > > > > > HBase release but have version checks on individual commands
> where
> > we
> > > > > could
> > > > > > decide if it is supported on that release line.
> > > > > >
> > > > > > What are your opinions on this?
> > > > > >
> > > > > > Thanks,
> > > > > > Peter
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] HBCK2 compatibility
Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.
I do not think we need to compile HBCK2 with every releases? We just need
make sure that it can work with all the releases. If there are missing
methods, we just tell users you can not use several features.
Wellington Chevreuil <we...@gmail.com> 于2019年8月29日周四
下午9:39写道：
> >
> > bq. what would folks think about going with an hbck2 alpha release?
> > I'm fine with an alpha release but since "HBCK2 should continuously
> evolve"
> > it might be better to always use the latest codebase whenever you need to
> > use the tool.
> >
> Ideally yes, but that might not always be possible, as hbase API might
> change ahead of hbck2. Operators could then have problems to get a working
> version of hbck2. Since hbck2 already has now many equivalent options for
> the ones from hbck1, I guess a first release would provide a working hbck2
> that already brings a considerable number of fix methods to help with most
> common inconsistencies issues seen in hbase 2 so far. And we could still
> fulfil the "HBCK2 should continuously evolve" principle by building from
> its master branch, in scenarios where a new fix was needed and implemented
> back into hbck2.
>
> Em qui, 29 de ago de 2019 às 14:09, Peter Somogyi <ps...@apache.org>
> escreveu:
>
> > bq. Is it possible to put some hacks into HBCK2 to work around
> > the compatibility to fix the current state
> >
> > There are some classes around Replication which were introduced in 2.1.0+
> > so I don't think we could easily solve it for 2.0.
> > For 2.1.1 the missing method is Hbck#scheduleServerCrashProcedure,
> probably
> > that could be solved with some workarounds or using reflection.
> >
> > bq. focus more on automation to let us know the next time we inevitably
> > break it again? ;)
> >
> > Sure! Based on this I think it should be a strong goal. We can set up
> > nightly builds for hbase-operator-tools repo that builds against the
> latest
> > development branches as well as checking compatibility with released
> > versions.
> >
> > bq. what would folks think about going with an hbck2 alpha release?
> > I'm fine with an alpha release but since "HBCK2 should continuously
> evolve"
> > it might be better to always use the latest codebase whenever you need to
> > use the tool.
> >
> > On Thu, Aug 29, 2019 at 2:30 PM Wellington Chevreuil <
> > wellington.chevreuil@gmail.com> wrote:
> >
> > > I would favour having hbck2 release branches. Temporary compatibility
> > > breaks at compile time may be inevitable if we always point to the
> latest
> > > release. That could cause problems for operators trying to build hbck2
> > (we
> > > are already seeing this happening with our support team). Another
> > argument
> > > for starting having hbck2 releases is that we already have quite a few
> > > hbase 2 releases, yet, the main tool to fix inconsistencies is not
> easily
> > > available. And there's been considerable efforts lately to bring many
> of
> > > the fix options from hbck1 into hbck2, so what would folks think about
> > > going with an hbck2 alpha release?
> > >
> > > Em qui, 29 de ago de 2019 às 13:20, Josh Elser <el...@apache.org>
> > > escreveu:
> > >
> > > > I still like one HBCK2 release as the goal.
> > > >
> > > > Is it possible to put some hacks into HBCK2 to work around the
> > > > compatibility to fix the current state and focus more on automation
> to
> > > > let us know the next time we inevitably break it again? ;)
> > > >
> > > > On 8/29/19 8:12 AM, Peter Somogyi wrote:
> > > > > Hi everyone,
> > > > >
> > > > > This topic came up a couple of times already but now we reached a
> > point
> > > > > when the recent HBCK2 is incompatible with older HBase releases,
> > > > > specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2 against
> one
> > > of
> > > > > the previously mentioned releases you will get compilation errors.
> > (mvn
> > > > > clean install -DskipTests -Dhbase.version=2.0.6)
> > > > >
> > > > > Our previous goal was to maintain compatibility with HBCK2 and all
> > > HBase
> > > > 2
> > > > > releases. Now we missed this target.
> > > > >
> > > > > One option that we could do is to have different branches/releases
> of
> > > > HBCK2
> > > > > targeted for specific HBase releases (e.g. branch-2.0 version of
> > > HBCK2).
> > > > > This probably makes the development on HBCK2 a bit harder since
> we'll
> > > > have
> > > > > to take care of multiple branches.
> > > > >
> > > > > Another option I could think of is to always build HBCK2 with the
> > > latest
> > > > > HBase release but have version checks on individual commands where
> we
> > > > could
> > > > > decide if it is supported on that release line.
> > > > >
> > > > > What are your opinions on this?
> > > > >
> > > > > Thanks,
> > > > > Peter
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] HBCK2 compatibility
Posted by Wellington Chevreuil <we...@gmail.com>.
>
> bq. what would folks think about going with an hbck2 alpha release?
> I'm fine with an alpha release but since "HBCK2 should continuously evolve"
> it might be better to always use the latest codebase whenever you need to
> use the tool.
>
Ideally yes, but that might not always be possible, as hbase API might
change ahead of hbck2. Operators could then have problems to get a working
version of hbck2. Since hbck2 already has now many equivalent options for
the ones from hbck1, I guess a first release would provide a working hbck2
that already brings a considerable number of fix methods to help with most
common inconsistencies issues seen in hbase 2 so far. And we could still
fulfil the "HBCK2 should continuously evolve" principle by building from
its master branch, in scenarios where a new fix was needed and implemented
back into hbck2.
Em qui, 29 de ago de 2019 às 14:09, Peter Somogyi <ps...@apache.org>
escreveu:
> bq. Is it possible to put some hacks into HBCK2 to work around
> the compatibility to fix the current state
>
> There are some classes around Replication which were introduced in 2.1.0+
> so I don't think we could easily solve it for 2.0.
> For 2.1.1 the missing method is Hbck#scheduleServerCrashProcedure, probably
> that could be solved with some workarounds or using reflection.
>
> bq. focus more on automation to let us know the next time we inevitably
> break it again? ;)
>
> Sure! Based on this I think it should be a strong goal. We can set up
> nightly builds for hbase-operator-tools repo that builds against the latest
> development branches as well as checking compatibility with released
> versions.
>
> bq. what would folks think about going with an hbck2 alpha release?
> I'm fine with an alpha release but since "HBCK2 should continuously evolve"
> it might be better to always use the latest codebase whenever you need to
> use the tool.
>
> On Thu, Aug 29, 2019 at 2:30 PM Wellington Chevreuil <
> wellington.chevreuil@gmail.com> wrote:
>
> > I would favour having hbck2 release branches. Temporary compatibility
> > breaks at compile time may be inevitable if we always point to the latest
> > release. That could cause problems for operators trying to build hbck2
> (we
> > are already seeing this happening with our support team). Another
> argument
> > for starting having hbck2 releases is that we already have quite a few
> > hbase 2 releases, yet, the main tool to fix inconsistencies is not easily
> > available. And there's been considerable efforts lately to bring many of
> > the fix options from hbck1 into hbck2, so what would folks think about
> > going with an hbck2 alpha release?
> >
> > Em qui, 29 de ago de 2019 às 13:20, Josh Elser <el...@apache.org>
> > escreveu:
> >
> > > I still like one HBCK2 release as the goal.
> > >
> > > Is it possible to put some hacks into HBCK2 to work around the
> > > compatibility to fix the current state and focus more on automation to
> > > let us know the next time we inevitably break it again? ;)
> > >
> > > On 8/29/19 8:12 AM, Peter Somogyi wrote:
> > > > Hi everyone,
> > > >
> > > > This topic came up a couple of times already but now we reached a
> point
> > > > when the recent HBCK2 is incompatible with older HBase releases,
> > > > specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2 against one
> > of
> > > > the previously mentioned releases you will get compilation errors.
> (mvn
> > > > clean install -DskipTests -Dhbase.version=2.0.6)
> > > >
> > > > Our previous goal was to maintain compatibility with HBCK2 and all
> > HBase
> > > 2
> > > > releases. Now we missed this target.
> > > >
> > > > One option that we could do is to have different branches/releases of
> > > HBCK2
> > > > targeted for specific HBase releases (e.g. branch-2.0 version of
> > HBCK2).
> > > > This probably makes the development on HBCK2 a bit harder since we'll
> > > have
> > > > to take care of multiple branches.
> > > >
> > > > Another option I could think of is to always build HBCK2 with the
> > latest
> > > > HBase release but have version checks on individual commands where we
> > > could
> > > > decide if it is supported on that release line.
> > > >
> > > > What are your opinions on this?
> > > >
> > > > Thanks,
> > > > Peter
> > > >
> > >
> >
>
Re: [DISCUSS] HBCK2 compatibility
Posted by Peter Somogyi <ps...@apache.org>.
bq. Is it possible to put some hacks into HBCK2 to work around
the compatibility to fix the current state
There are some classes around Replication which were introduced in 2.1.0+
so I don't think we could easily solve it for 2.0.
For 2.1.1 the missing method is Hbck#scheduleServerCrashProcedure, probably
that could be solved with some workarounds or using reflection.
bq. focus more on automation to let us know the next time we inevitably
break it again? ;)
Sure! Based on this I think it should be a strong goal. We can set up
nightly builds for hbase-operator-tools repo that builds against the latest
development branches as well as checking compatibility with released
versions.
bq. what would folks think about going with an hbck2 alpha release?
I'm fine with an alpha release but since "HBCK2 should continuously evolve"
it might be better to always use the latest codebase whenever you need to
use the tool.
On Thu, Aug 29, 2019 at 2:30 PM Wellington Chevreuil <
wellington.chevreuil@gmail.com> wrote:
> I would favour having hbck2 release branches. Temporary compatibility
> breaks at compile time may be inevitable if we always point to the latest
> release. That could cause problems for operators trying to build hbck2 (we
> are already seeing this happening with our support team). Another argument
> for starting having hbck2 releases is that we already have quite a few
> hbase 2 releases, yet, the main tool to fix inconsistencies is not easily
> available. And there's been considerable efforts lately to bring many of
> the fix options from hbck1 into hbck2, so what would folks think about
> going with an hbck2 alpha release?
>
> Em qui, 29 de ago de 2019 às 13:20, Josh Elser <el...@apache.org>
> escreveu:
>
> > I still like one HBCK2 release as the goal.
> >
> > Is it possible to put some hacks into HBCK2 to work around the
> > compatibility to fix the current state and focus more on automation to
> > let us know the next time we inevitably break it again? ;)
> >
> > On 8/29/19 8:12 AM, Peter Somogyi wrote:
> > > Hi everyone,
> > >
> > > This topic came up a couple of times already but now we reached a point
> > > when the recent HBCK2 is incompatible with older HBase releases,
> > > specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2 against one
> of
> > > the previously mentioned releases you will get compilation errors. (mvn
> > > clean install -DskipTests -Dhbase.version=2.0.6)
> > >
> > > Our previous goal was to maintain compatibility with HBCK2 and all
> HBase
> > 2
> > > releases. Now we missed this target.
> > >
> > > One option that we could do is to have different branches/releases of
> > HBCK2
> > > targeted for specific HBase releases (e.g. branch-2.0 version of
> HBCK2).
> > > This probably makes the development on HBCK2 a bit harder since we'll
> > have
> > > to take care of multiple branches.
> > >
> > > Another option I could think of is to always build HBCK2 with the
> latest
> > > HBase release but have version checks on individual commands where we
> > could
> > > decide if it is supported on that release line.
> > >
> > > What are your opinions on this?
> > >
> > > Thanks,
> > > Peter
> > >
> >
>
Re: [DISCUSS] HBCK2 compatibility
Posted by Wellington Chevreuil <we...@gmail.com>.
I would favour having hbck2 release branches. Temporary compatibility
breaks at compile time may be inevitable if we always point to the latest
release. That could cause problems for operators trying to build hbck2 (we
are already seeing this happening with our support team). Another argument
for starting having hbck2 releases is that we already have quite a few
hbase 2 releases, yet, the main tool to fix inconsistencies is not easily
available. And there's been considerable efforts lately to bring many of
the fix options from hbck1 into hbck2, so what would folks think about
going with an hbck2 alpha release?
Em qui, 29 de ago de 2019 às 13:20, Josh Elser <el...@apache.org> escreveu:
> I still like one HBCK2 release as the goal.
>
> Is it possible to put some hacks into HBCK2 to work around the
> compatibility to fix the current state and focus more on automation to
> let us know the next time we inevitably break it again? ;)
>
> On 8/29/19 8:12 AM, Peter Somogyi wrote:
> > Hi everyone,
> >
> > This topic came up a couple of times already but now we reached a point
> > when the recent HBCK2 is incompatible with older HBase releases,
> > specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2 against one of
> > the previously mentioned releases you will get compilation errors. (mvn
> > clean install -DskipTests -Dhbase.version=2.0.6)
> >
> > Our previous goal was to maintain compatibility with HBCK2 and all HBase
> 2
> > releases. Now we missed this target.
> >
> > One option that we could do is to have different branches/releases of
> HBCK2
> > targeted for specific HBase releases (e.g. branch-2.0 version of HBCK2).
> > This probably makes the development on HBCK2 a bit harder since we'll
> have
> > to take care of multiple branches.
> >
> > Another option I could think of is to always build HBCK2 with the latest
> > HBase release but have version checks on individual commands where we
> could
> > decide if it is supported on that release line.
> >
> > What are your opinions on this?
> >
> > Thanks,
> > Peter
> >
>
Re: [DISCUSS] HBCK2 compatibility
Posted by Josh Elser <el...@apache.org>.
I still like one HBCK2 release as the goal.
Is it possible to put some hacks into HBCK2 to work around the
compatibility to fix the current state and focus more on automation to
let us know the next time we inevitably break it again? ;)
On 8/29/19 8:12 AM, Peter Somogyi wrote:
> Hi everyone,
>
> This topic came up a couple of times already but now we reached a point
> when the recent HBCK2 is incompatible with older HBase releases,
> specifically 2.0.x, 2.1.0 and 2.1.1. When you build HBCK2 against one of
> the previously mentioned releases you will get compilation errors. (mvn
> clean install -DskipTests -Dhbase.version=2.0.6)
>
> Our previous goal was to maintain compatibility with HBCK2 and all HBase 2
> releases. Now we missed this target.
>
> One option that we could do is to have different branches/releases of HBCK2
> targeted for specific HBase releases (e.g. branch-2.0 version of HBCK2).
> This probably makes the development on HBCK2 a bit harder since we'll have
> to take care of multiple branches.
>
> Another option I could think of is to always build HBCK2 with the latest
> HBase release but have version checks on individual commands where we could
> decide if it is supported on that release line.
>
> What are your opinions on this?
>
> Thanks,
> Peter
>