You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Stack <st...@duboce.net> on 2017/08/02 23:38:02 UTC
[DISCUSS] hbase-2.0.0 compatibility expectations
What are our expectations regards compatibility between hbase1 and hbase2?
Lets have a chat about it. Here are some goal posts.
+ You have to upgrade to hbase-1.x before you can migrate to hbase-2. No
migration from < hbase-1 (Is this too onerous? Should we support 0.98 =>
2.0?).
+ You do NOT have to upgrade to the latest release of hbase1 to migrate to
hbase2; being up on hbase-1.0.0+ will be sufficient.
+ You'll have to update your hbase1 coprocessors to deploy them on hbase2.
A bunch of CP API has/will change by the time hbase2 comes out; e.g.
watching for region split on RegionServer no longer makes sense given
Master runs all splits now.
+ An hbase1 client can run against an hbase2 cluster but it will only be
able to do DML (Get/Put/Scan, etc.). We do not allow being able to do admin
ops using an hbase1 Admin client against an hbase2 cluster. We have some
egregious API violations in branch-1; e.g. we have protobuf in our API (See
HBASE-15607). The notion is that we can't afford a deprecation cycle
purging this stuff from our Admin API.
What you all think?
St.Ack
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Jerry He <je...@gmail.com>.
Thanks.
I filed a JIRA to track it.
https://issues.apache.org/jira/browse/HBASE-19145
On Mon, Oct 30, 2017 at 9:36 PM, Stack <st...@duboce.net> wrote:
> On Mon, Oct 30, 2017 at 11:46 AM, Jerry He <je...@gmail.com> wrote:
>
>> Hi, Stack
>>
>> Coming back to this thread, i have meant to ask this question long ago.
>> Do we support hbase-2 client going against hbase-1 server?
>>
>
> It has not been an objective. It might work doing basic Client API on a
> later branch-1 but will fail doing Admin functions (and figuring if a Table
> is online). I've not tried it Jerry. If it was a small thing to make it
> work, lets get it in.
>
> St.Ack
>
>
>
>> We seem to be fine mix-and-match the clients and servers within the
>> hbase-1 releases. And IIRC hbase-1 client is ok against 0.98 server.
>> Suppose I have a product that depends and bundles HBase client. I
>> want to upgrade the dependency to hbase-2 so that it can take
>> advantages of and claim support of hbase-2.
>> But does it mean that I will need drop the claims that the new version
>> of the product support any hbase-1 backend?
>>
>> Thanks.
>>
>>
>>
>> On Tue, Sep 12, 2017 at 10:21 AM, Stack <st...@duboce.net> wrote:
>> > Let me put this one on this thread, G1GC on by default in hbase2?
>> > St.Ack
>> >
>> > On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
>> >
>> >> What are our expectations regards compatibility between hbase1 and
>> hbase2?
>> >>
>> >> Lets have a chat about it. Here are some goal posts.
>> >>
>> >> + You have to upgrade to hbase-1.x before you can migrate to hbase-2. No
>> >> migration from < hbase-1 (Is this too onerous? Should we support 0.98 =>
>> >> 2.0?).
>> >> + You do NOT have to upgrade to the latest release of hbase1 to migrate
>> to
>> >> hbase2; being up on hbase-1.0.0+ will be sufficient.
>> >> + You'll have to update your hbase1 coprocessors to deploy them on
>> hbase2.
>> >> A bunch of CP API has/will change by the time hbase2 comes out; e.g.
>> >> watching for region split on RegionServer no longer makes sense given
>> >> Master runs all splits now.
>> >> + An hbase1 client can run against an hbase2 cluster but it will only be
>> >> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
>> admin
>> >> ops using an hbase1 Admin client against an hbase2 cluster. We have some
>> >> egregious API violations in branch-1; e.g. we have protobuf in our API
>> (See
>> >> HBASE-15607). The notion is that we can't afford a deprecation cycle
>> >> purging this stuff from our Admin API.
>> >>
>> >> What you all think?
>> >>
>> >> St.Ack
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Stack <st...@duboce.net>.
On Mon, Oct 30, 2017 at 11:46 AM, Jerry He <je...@gmail.com> wrote:
> Hi, Stack
>
> Coming back to this thread, i have meant to ask this question long ago.
> Do we support hbase-2 client going against hbase-1 server?
>
It has not been an objective. It might work doing basic Client API on a
later branch-1 but will fail doing Admin functions (and figuring if a Table
is online). I've not tried it Jerry. If it was a small thing to make it
work, lets get it in.
St.Ack
> We seem to be fine mix-and-match the clients and servers within the
> hbase-1 releases. And IIRC hbase-1 client is ok against 0.98 server.
> Suppose I have a product that depends and bundles HBase client. I
> want to upgrade the dependency to hbase-2 so that it can take
> advantages of and claim support of hbase-2.
> But does it mean that I will need drop the claims that the new version
> of the product support any hbase-1 backend?
>
> Thanks.
>
>
>
> On Tue, Sep 12, 2017 at 10:21 AM, Stack <st...@duboce.net> wrote:
> > Let me put this one on this thread, G1GC on by default in hbase2?
> > St.Ack
> >
> > On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> >
> >> What are our expectations regards compatibility between hbase1 and
> hbase2?
> >>
> >> Lets have a chat about it. Here are some goal posts.
> >>
> >> + You have to upgrade to hbase-1.x before you can migrate to hbase-2. No
> >> migration from < hbase-1 (Is this too onerous? Should we support 0.98 =>
> >> 2.0?).
> >> + You do NOT have to upgrade to the latest release of hbase1 to migrate
> to
> >> hbase2; being up on hbase-1.0.0+ will be sufficient.
> >> + You'll have to update your hbase1 coprocessors to deploy them on
> hbase2.
> >> A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> >> watching for region split on RegionServer no longer makes sense given
> >> Master runs all splits now.
> >> + An hbase1 client can run against an hbase2 cluster but it will only be
> >> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> admin
> >> ops using an hbase1 Admin client against an hbase2 cluster. We have some
> >> egregious API violations in branch-1; e.g. we have protobuf in our API
> (See
> >> HBASE-15607). The notion is that we can't afford a deprecation cycle
> >> purging this stuff from our Admin API.
> >>
> >> What you all think?
> >>
> >> St.Ack
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Jerry He <je...@gmail.com>.
Hi, Stack
Coming back to this thread, i have meant to ask this question long ago.
Do we support hbase-2 client going against hbase-1 server?
We seem to be fine mix-and-match the clients and servers within the
hbase-1 releases. And IIRC hbase-1 client is ok against 0.98 server.
Suppose I have a product that depends and bundles HBase client. I
want to upgrade the dependency to hbase-2 so that it can take
advantages of and claim support of hbase-2.
But does it mean that I will need drop the claims that the new version
of the product support any hbase-1 backend?
Thanks.
On Tue, Sep 12, 2017 at 10:21 AM, Stack <st...@duboce.net> wrote:
> Let me put this one on this thread, G1GC on by default in hbase2?
> St.Ack
>
> On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
>
>> What are our expectations regards compatibility between hbase1 and hbase2?
>>
>> Lets have a chat about it. Here are some goal posts.
>>
>> + You have to upgrade to hbase-1.x before you can migrate to hbase-2. No
>> migration from < hbase-1 (Is this too onerous? Should we support 0.98 =>
>> 2.0?).
>> + You do NOT have to upgrade to the latest release of hbase1 to migrate to
>> hbase2; being up on hbase-1.0.0+ will be sufficient.
>> + You'll have to update your hbase1 coprocessors to deploy them on hbase2.
>> A bunch of CP API has/will change by the time hbase2 comes out; e.g.
>> watching for region split on RegionServer no longer makes sense given
>> Master runs all splits now.
>> + An hbase1 client can run against an hbase2 cluster but it will only be
>> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do admin
>> ops using an hbase1 Admin client against an hbase2 cluster. We have some
>> egregious API violations in branch-1; e.g. we have protobuf in our API (See
>> HBASE-15607). The notion is that we can't afford a deprecation cycle
>> purging this stuff from our Admin API.
>>
>> What you all think?
>>
>> St.Ack
>>
>>
>>
>>
>>
>>
>>
>>
>>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Stack <st...@duboce.net>.
Let me put this one on this thread, G1GC on by default in hbase2?
St.Ack
On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> What are our expectations regards compatibility between hbase1 and hbase2?
>
> Lets have a chat about it. Here are some goal posts.
>
> + You have to upgrade to hbase-1.x before you can migrate to hbase-2. No
> migration from < hbase-1 (Is this too onerous? Should we support 0.98 =>
> 2.0?).
> + You do NOT have to upgrade to the latest release of hbase1 to migrate to
> hbase2; being up on hbase-1.0.0+ will be sufficient.
> + You'll have to update your hbase1 coprocessors to deploy them on hbase2.
> A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> watching for region split on RegionServer no longer makes sense given
> Master runs all splits now.
> + An hbase1 client can run against an hbase2 cluster but it will only be
> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do admin
> ops using an hbase1 Admin client against an hbase2 cluster. We have some
> egregious API violations in branch-1; e.g. we have protobuf in our API (See
> HBASE-15607). The notion is that we can't afford a deprecation cycle
> purging this stuff from our Admin API.
>
> What you all think?
>
> St.Ack
>
>
>
>
>
>
>
>
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Stack <st...@duboce.net>.
On Fri, Sep 1, 2017 at 3:12 PM, Sean Busbey <bu...@apache.org> wrote:
> Giving rolling-upgradable ability within branch-1, I'm inclined to
> push for our current stable release line,1.2.
>
>
I like this suggestion. Any other opinions on minimum version from which
we'd support rolling upgrade?
S
> On Fri, Sep 1, 2017 at 2:24 PM, Stack <st...@duboce.net> wrote:
> > On Fri, Sep 1, 2017 at 2:23 PM, Stack <st...@duboce.net> wrote:
> >
> >> What versions should you be able to upgrade from? 1.0.0? 0.98.x? 1.2?
> I'm
> >> thinking 0.98 unless it means more work.
> >>
> >
> > Scratch that. Minimum is 1.0.0 I think given that is what I'm doing all
> > compat compares against and given we did API cleanup before we shipped
> > 1.0.0.
> > St.Ack
> >
> >
> >
> >> Thanks,
> >> St.Ack
> >>
> >> On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> >>
> >>> What are our expectations regards compatibility between hbase1 and
> hbase2?
> >>>
> >>> Lets have a chat about it. Here are some goal posts.
> >>>
> >>> + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
> No
> >>> migration from < hbase-1 (Is this too onerous? Should we support 0.98
> =>
> >>> 2.0?).
> >>> + You do NOT have to upgrade to the latest release of hbase1 to migrate
> >>> to hbase2; being up on hbase-1.0.0+ will be sufficient.
> >>> + You'll have to update your hbase1 coprocessors to deploy them on
> >>> hbase2. A bunch of CP API has/will change by the time hbase2 comes out;
> >>> e.g. watching for region split on RegionServer no longer makes sense
> given
> >>> Master runs all splits now.
> >>> + An hbase1 client can run against an hbase2 cluster but it will only
> be
> >>> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> admin
> >>> ops using an hbase1 Admin client against an hbase2 cluster. We have
> some
> >>> egregious API violations in branch-1; e.g. we have protobuf in our API
> (See
> >>> HBASE-15607). The notion is that we can't afford a deprecation cycle
> >>> purging this stuff from our Admin API.
> >>>
> >>> What you all think?
> >>>
> >>> St.Ack
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Sean Busbey <bu...@apache.org>.
Giving rolling-upgradable ability within branch-1, I'm inclined to
push for our current stable release line,1.2.
On Fri, Sep 1, 2017 at 2:24 PM, Stack <st...@duboce.net> wrote:
> On Fri, Sep 1, 2017 at 2:23 PM, Stack <st...@duboce.net> wrote:
>
>> What versions should you be able to upgrade from? 1.0.0? 0.98.x? 1.2? I'm
>> thinking 0.98 unless it means more work.
>>
>
> Scratch that. Minimum is 1.0.0 I think given that is what I'm doing all
> compat compares against and given we did API cleanup before we shipped
> 1.0.0.
> St.Ack
>
>
>
>> Thanks,
>> St.Ack
>>
>> On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
>>
>>> What are our expectations regards compatibility between hbase1 and hbase2?
>>>
>>> Lets have a chat about it. Here are some goal posts.
>>>
>>> + You have to upgrade to hbase-1.x before you can migrate to hbase-2. No
>>> migration from < hbase-1 (Is this too onerous? Should we support 0.98 =>
>>> 2.0?).
>>> + You do NOT have to upgrade to the latest release of hbase1 to migrate
>>> to hbase2; being up on hbase-1.0.0+ will be sufficient.
>>> + You'll have to update your hbase1 coprocessors to deploy them on
>>> hbase2. A bunch of CP API has/will change by the time hbase2 comes out;
>>> e.g. watching for region split on RegionServer no longer makes sense given
>>> Master runs all splits now.
>>> + An hbase1 client can run against an hbase2 cluster but it will only be
>>> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do admin
>>> ops using an hbase1 Admin client against an hbase2 cluster. We have some
>>> egregious API violations in branch-1; e.g. we have protobuf in our API (See
>>> HBASE-15607). The notion is that we can't afford a deprecation cycle
>>> purging this stuff from our Admin API.
>>>
>>> What you all think?
>>>
>>> St.Ack
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Stack <st...@duboce.net>.
On Fri, Sep 1, 2017 at 2:23 PM, Stack <st...@duboce.net> wrote:
> What versions should you be able to upgrade from? 1.0.0? 0.98.x? 1.2? I'm
> thinking 0.98 unless it means more work.
>
Scratch that. Minimum is 1.0.0 I think given that is what I'm doing all
compat compares against and given we did API cleanup before we shipped
1.0.0.
St.Ack
> Thanks,
> St.Ack
>
> On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
>
>> What are our expectations regards compatibility between hbase1 and hbase2?
>>
>> Lets have a chat about it. Here are some goal posts.
>>
>> + You have to upgrade to hbase-1.x before you can migrate to hbase-2. No
>> migration from < hbase-1 (Is this too onerous? Should we support 0.98 =>
>> 2.0?).
>> + You do NOT have to upgrade to the latest release of hbase1 to migrate
>> to hbase2; being up on hbase-1.0.0+ will be sufficient.
>> + You'll have to update your hbase1 coprocessors to deploy them on
>> hbase2. A bunch of CP API has/will change by the time hbase2 comes out;
>> e.g. watching for region split on RegionServer no longer makes sense given
>> Master runs all splits now.
>> + An hbase1 client can run against an hbase2 cluster but it will only be
>> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do admin
>> ops using an hbase1 Admin client against an hbase2 cluster. We have some
>> egregious API violations in branch-1; e.g. we have protobuf in our API (See
>> HBASE-15607). The notion is that we can't afford a deprecation cycle
>> purging this stuff from our Admin API.
>>
>> What you all think?
>>
>> St.Ack
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Stack <st...@duboce.net>.
What versions should you be able to upgrade from? 1.0.0? 0.98.x? 1.2? I'm
thinking 0.98 unless it means more work.
Thanks,
St.Ack
On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> What are our expectations regards compatibility between hbase1 and hbase2?
>
> Lets have a chat about it. Here are some goal posts.
>
> + You have to upgrade to hbase-1.x before you can migrate to hbase-2. No
> migration from < hbase-1 (Is this too onerous? Should we support 0.98 =>
> 2.0?).
> + You do NOT have to upgrade to the latest release of hbase1 to migrate to
> hbase2; being up on hbase-1.0.0+ will be sufficient.
> + You'll have to update your hbase1 coprocessors to deploy them on hbase2.
> A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> watching for region split on RegionServer no longer makes sense given
> Master runs all splits now.
> + An hbase1 client can run against an hbase2 cluster but it will only be
> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do admin
> ops using an hbase1 Admin client against an hbase2 cluster. We have some
> egregious API violations in branch-1; e.g. we have protobuf in our API (See
> HBASE-15607). The notion is that we can't afford a deprecation cycle
> purging this stuff from our Admin API.
>
> What you all think?
>
> St.Ack
>
>
>
>
>
>
>
>
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Stack <st...@duboce.net>.
On Mon, Sep 4, 2017 at 6:55 PM, Francis Christopher Liu <
toffer.liu@gmail.com> wrote:
> Just some high-level thoughts on rolling upgradeability (I'm repeating a
> few things already said).
>
> 1. No service interruption for 1.x clients while a cluster is being
> ugpraded to 2.x.
- This includes support for apis commonly used in a running system:
> delegation tokens, getting region locations, etc.
>
Agreed. Having hbase-1 clients read the basic info they need to operate
seems doable w/ a little bit of work (having meta table edits mirrored in
zk). Let me add note to test delegation tokens continue to work.
> 2. The somewhat reverse of #1 is also necessary for Master DMLs to meta.
>
Come again. hbase1 won't be able to admin a hbase2 (basic read-only admin
function will work like checking if table is online or not, etc.)
> 3. No interruption for replication between 1.x and 2.x cluster
> 4. Restart master first before any other service sounds reasonable to me.
> Will we support RU for zkless mode?
>
I'll not be testing RU for ZKLESS mode. Any one up for this?
> 5. 2.x should be either able to understand formats written by 1.x or online
> migration is done as a preparation step.
>
Yep. Auto-migration preferred.
> 6. Data generated by 2.x daemons should be readable in 1.x (ie HFile,
> zookeeper, etc). Some deploys may have large amounts of data could be
> cumbersome/impractical (ie cost, time, etc) to copy the data.
>
>
Thanks Francis,
S
> Thanks,
> Francis
>
>
>
> On Fri, Sep 1, 2017 at 10:01 PM Chia-Ping Tsai <ch...@apache.org>
> wrote:
>
> > The most of issues use hadoop flag, so the filter looks like this.
> > project = HBASE AND fixVersion in (2.0.0, 2.0.0-alpha-1, 2.0.0-alpha-2,
> > 2.0.0-alpha-3,
> > 3.0) AND (labels in (incompatibleChange, incompatible, incompatibility)
> OR
> > "Hadoop Flags" in ("Incompatible change"))
> >
> > On 2017-08-04 05:46, Ted Yu <yu...@gmail.com> wrote:
> > > I expanded the condition in the filter like this:
> > >
> > > project = HBASE AND fixVersion in (2.0.0, 2.0.0-alpha-1,
> 2.0.0-alpha-2,
> > > 3.0) AND labels in (incompatibleChange, incompatible, incompatibility)
> > >
> > > Still there're only two showing up.
> > >
> > > On Thu, Aug 3, 2017 at 9:07 AM, Zach York <
> zyork.contribution@gmail.com>
> > > wrote:
> > >
> > > > I tried to filter based on imcompatible labels and there were only
> two
> > > > JIRAs returned [1]. I have a hard time believing that there were only
> > two
> > > > breaking changes from 1.x to 2.x.
> > > >
> > > > [1]
> > > > https://issues.apache.org/jira/browse/HBASE-17957?jql=
> > > > project%20%3D%20HBASE%20AND%20fixVersion%20%3D%202.0.0%
> > > > 20AND%20labels%20in%20(incompatibleChange%2C%20incompatible%2C%
> > > > 20incompatibility)
> > > >
> > > > On Thu, Aug 3, 2017 at 8:54 AM, Zach York <
> > zyork.contribution@gmail.com>
> > > > wrote:
> > > >
> > > > > This kinda helps, but these seem more like expectations. I was
> going
> > more
> > > > > for things like HFile format changed, meta table structure changed,
> > > > > coprocessor implementations changed (these are just examples, I
> don't
> > > > know
> > > > > if any of these actually changed).
> > > > >
> > > > > More technical differences between branch-1 and branch-2 which then
> > can
> > > > > help us get the right expectations for compatibility.
> > > > >
> > > > > On Wed, Aug 2, 2017 at 6:34 PM, Stack <st...@duboce.net> wrote:
> > > > >
> > > > >> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <
> > zyork.contribution@gmail.com
> > > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Do we know what the major pain points for migration are? Can we
> > > > discuss
> > > > >> > that/get a list going?
> > > > >> >
> > > > >> >
> > > > >> Here's a few in outline:
> > > > >>
> > > > >> + There is issue of formats, of hbase-2.x being able to read
> > hbase-1.x
> > > > >> data
> > > > >> whether from HDFS or ZooKeeper or off the wire.
> > > > >> + An hbase-1.x client should be able to Get/Put and Scan an
> > hbase-2.x
> > > > >> cluster; no holes in the API or unintelligible serializations.
> > > > >> + There is then the little dance that has us rolling restart from
> an
> > > > >> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then
> > it
> > > > will
> > > > >> assign regions to the new hbase-2.x regionservers as they come on
> > line.
> > > > >> TBD.
> > > > >>
> > > > >> Is this what you mean sir?
> > > > >>
> > > > >> S
> > > > >>
> > > > >>
> > > > >> > I think without that knowledge it is hard (for me at least :) )
> to
> > > > >> > determine where we should set our sights in terms of migration.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Zach
> > > > >> >
> > > > >> > On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> > > > >> >
> > > > >> > > What are our expectations regards compatibility between hbase1
> > and
> > > > >> > hbase2?
> > > > >> > >
> > > > >> > > Lets have a chat about it. Here are some goal posts.
> > > > >> > >
> > > > >> > > + You have to upgrade to hbase-1.x before you can migrate to
> > > > hbase-2.
> > > > >> No
> > > > >> > > migration from < hbase-1 (Is this too onerous? Should we
> support
> > > > 0.98
> > > > >> =>
> > > > >> > > 2.0?).
> > > > >> > > + You do NOT have to upgrade to the latest release of hbase1
> to
> > > > >> migrate
> > > > >> > to
> > > > >> > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > > >> > > + You'll have to update your hbase1 coprocessors to deploy
> them
> > on
> > > > >> > hbase2.
> > > > >> > > A bunch of CP API has/will change by the time hbase2 comes
> out;
> > e.g.
> > > > >> > > watching for region split on RegionServer no longer makes
> sense
> > > > given
> > > > >> > > Master runs all splits now.
> > > > >> > > + An hbase1 client can run against an hbase2 cluster but it
> will
> > > > only
> > > > >> be
> > > > >> > > able to do DML (Get/Put/Scan, etc.). We do not allow being
> able
> > to
> > > > do
> > > > >> > admin
> > > > >> > > ops using an hbase1 Admin client against an hbase2 cluster. We
> > have
> > > > >> some
> > > > >> > > egregious API violations in branch-1; e.g. we have protobuf in
> > our
> > > > API
> > > > >> > (See
> > > > >> > > HBASE-15607). The notion is that we can't afford a deprecation
> > cycle
> > > > >> > > purging this stuff from our Admin API.
> > > > >> > >
> > > > >> > > What you all think?
> > > > >> > >
> > > > >> > > St.Ack
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Francis Christopher Liu <to...@gmail.com>.
Just some high-level thoughts on rolling upgradeability (I'm repeating a
few things already said).
1. No service interruption for 1.x clients while a cluster is being
ugpraded to 2.x.
- This includes support for apis commonly used in a running system:
delegation tokens, getting region locations, etc.
2. The somewhat reverse of #1 is also necessary for Master DMLs to meta.
3. No interruption for replication between 1.x and 2.x cluster
4. Restart master first before any other service sounds reasonable to me.
Will we support RU for zkless mode?
5. 2.x should be either able to understand formats written by 1.x or online
migration is done as a preparation step.
6. Data generated by 2.x daemons should be readable in 1.x (ie HFile,
zookeeper, etc). Some deploys may have large amounts of data could be
cumbersome/impractical (ie cost, time, etc) to copy the data.
Thanks,
Francis
On Fri, Sep 1, 2017 at 10:01 PM Chia-Ping Tsai <ch...@apache.org> wrote:
> The most of issues use hadoop flag, so the filter looks like this.
> project = HBASE AND fixVersion in (2.0.0, 2.0.0-alpha-1, 2.0.0-alpha-2,
> 2.0.0-alpha-3,
> 3.0) AND (labels in (incompatibleChange, incompatible, incompatibility) OR
> "Hadoop Flags" in ("Incompatible change"))
>
> On 2017-08-04 05:46, Ted Yu <yu...@gmail.com> wrote:
> > I expanded the condition in the filter like this:
> >
> > project = HBASE AND fixVersion in (2.0.0, 2.0.0-alpha-1, 2.0.0-alpha-2,
> > 3.0) AND labels in (incompatibleChange, incompatible, incompatibility)
> >
> > Still there're only two showing up.
> >
> > On Thu, Aug 3, 2017 at 9:07 AM, Zach York <zy...@gmail.com>
> > wrote:
> >
> > > I tried to filter based on imcompatible labels and there were only two
> > > JIRAs returned [1]. I have a hard time believing that there were only
> two
> > > breaking changes from 1.x to 2.x.
> > >
> > > [1]
> > > https://issues.apache.org/jira/browse/HBASE-17957?jql=
> > > project%20%3D%20HBASE%20AND%20fixVersion%20%3D%202.0.0%
> > > 20AND%20labels%20in%20(incompatibleChange%2C%20incompatible%2C%
> > > 20incompatibility)
> > >
> > > On Thu, Aug 3, 2017 at 8:54 AM, Zach York <
> zyork.contribution@gmail.com>
> > > wrote:
> > >
> > > > This kinda helps, but these seem more like expectations. I was going
> more
> > > > for things like HFile format changed, meta table structure changed,
> > > > coprocessor implementations changed (these are just examples, I don't
> > > know
> > > > if any of these actually changed).
> > > >
> > > > More technical differences between branch-1 and branch-2 which then
> can
> > > > help us get the right expectations for compatibility.
> > > >
> > > > On Wed, Aug 2, 2017 at 6:34 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > >> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <
> zyork.contribution@gmail.com
> > > >
> > > >> wrote:
> > > >>
> > > >> > Do we know what the major pain points for migration are? Can we
> > > discuss
> > > >> > that/get a list going?
> > > >> >
> > > >> >
> > > >> Here's a few in outline:
> > > >>
> > > >> + There is issue of formats, of hbase-2.x being able to read
> hbase-1.x
> > > >> data
> > > >> whether from HDFS or ZooKeeper or off the wire.
> > > >> + An hbase-1.x client should be able to Get/Put and Scan an
> hbase-2.x
> > > >> cluster; no holes in the API or unintelligible serializations.
> > > >> + There is then the little dance that has us rolling restart from an
> > > >> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then
> it
> > > will
> > > >> assign regions to the new hbase-2.x regionservers as they come on
> line.
> > > >> TBD.
> > > >>
> > > >> Is this what you mean sir?
> > > >>
> > > >> S
> > > >>
> > > >>
> > > >> > I think without that knowledge it is hard (for me at least :) ) to
> > > >> > determine where we should set our sights in terms of migration.
> > > >> >
> > > >> > Thanks,
> > > >> > Zach
> > > >> >
> > > >> > On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> > > >> >
> > > >> > > What are our expectations regards compatibility between hbase1
> and
> > > >> > hbase2?
> > > >> > >
> > > >> > > Lets have a chat about it. Here are some goal posts.
> > > >> > >
> > > >> > > + You have to upgrade to hbase-1.x before you can migrate to
> > > hbase-2.
> > > >> No
> > > >> > > migration from < hbase-1 (Is this too onerous? Should we support
> > > 0.98
> > > >> =>
> > > >> > > 2.0?).
> > > >> > > + You do NOT have to upgrade to the latest release of hbase1 to
> > > >> migrate
> > > >> > to
> > > >> > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > >> > > + You'll have to update your hbase1 coprocessors to deploy them
> on
> > > >> > hbase2.
> > > >> > > A bunch of CP API has/will change by the time hbase2 comes out;
> e.g.
> > > >> > > watching for region split on RegionServer no longer makes sense
> > > given
> > > >> > > Master runs all splits now.
> > > >> > > + An hbase1 client can run against an hbase2 cluster but it will
> > > only
> > > >> be
> > > >> > > able to do DML (Get/Put/Scan, etc.). We do not allow being able
> to
> > > do
> > > >> > admin
> > > >> > > ops using an hbase1 Admin client against an hbase2 cluster. We
> have
> > > >> some
> > > >> > > egregious API violations in branch-1; e.g. we have protobuf in
> our
> > > API
> > > >> > (See
> > > >> > > HBASE-15607). The notion is that we can't afford a deprecation
> cycle
> > > >> > > purging this stuff from our Admin API.
> > > >> > >
> > > >> > > What you all think?
> > > >> > >
> > > >> > > St.Ack
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Chia-Ping Tsai <ch...@apache.org>.
The most of issues use hadoop flag, so the filter looks like this.
project = HBASE AND fixVersion in (2.0.0, 2.0.0-alpha-1, 2.0.0-alpha-2, 2.0.0-alpha-3,
3.0) AND (labels in (incompatibleChange, incompatible, incompatibility) OR "Hadoop Flags" in ("Incompatible change"))
On 2017-08-04 05:46, Ted Yu <yu...@gmail.com> wrote:
> I expanded the condition in the filter like this:
>
> project = HBASE AND fixVersion in (2.0.0, 2.0.0-alpha-1, 2.0.0-alpha-2,
> 3.0) AND labels in (incompatibleChange, incompatible, incompatibility)
>
> Still there're only two showing up.
>
> On Thu, Aug 3, 2017 at 9:07 AM, Zach York <zy...@gmail.com>
> wrote:
>
> > I tried to filter based on imcompatible labels and there were only two
> > JIRAs returned [1]. I have a hard time believing that there were only two
> > breaking changes from 1.x to 2.x.
> >
> > [1]
> > https://issues.apache.org/jira/browse/HBASE-17957?jql=
> > project%20%3D%20HBASE%20AND%20fixVersion%20%3D%202.0.0%
> > 20AND%20labels%20in%20(incompatibleChange%2C%20incompatible%2C%
> > 20incompatibility)
> >
> > On Thu, Aug 3, 2017 at 8:54 AM, Zach York <zy...@gmail.com>
> > wrote:
> >
> > > This kinda helps, but these seem more like expectations. I was going more
> > > for things like HFile format changed, meta table structure changed,
> > > coprocessor implementations changed (these are just examples, I don't
> > know
> > > if any of these actually changed).
> > >
> > > More technical differences between branch-1 and branch-2 which then can
> > > help us get the right expectations for compatibility.
> > >
> > > On Wed, Aug 2, 2017 at 6:34 PM, Stack <st...@duboce.net> wrote:
> > >
> > >> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <zyork.contribution@gmail.com
> > >
> > >> wrote:
> > >>
> > >> > Do we know what the major pain points for migration are? Can we
> > discuss
> > >> > that/get a list going?
> > >> >
> > >> >
> > >> Here's a few in outline:
> > >>
> > >> + There is issue of formats, of hbase-2.x being able to read hbase-1.x
> > >> data
> > >> whether from HDFS or ZooKeeper or off the wire.
> > >> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> > >> cluster; no holes in the API or unintelligible serializations.
> > >> + There is then the little dance that has us rolling restart from an
> > >> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> > will
> > >> assign regions to the new hbase-2.x regionservers as they come on line.
> > >> TBD.
> > >>
> > >> Is this what you mean sir?
> > >>
> > >> S
> > >>
> > >>
> > >> > I think without that knowledge it is hard (for me at least :) ) to
> > >> > determine where we should set our sights in terms of migration.
> > >> >
> > >> > Thanks,
> > >> > Zach
> > >> >
> > >> > On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> > >> >
> > >> > > What are our expectations regards compatibility between hbase1 and
> > >> > hbase2?
> > >> > >
> > >> > > Lets have a chat about it. Here are some goal posts.
> > >> > >
> > >> > > + You have to upgrade to hbase-1.x before you can migrate to
> > hbase-2.
> > >> No
> > >> > > migration from < hbase-1 (Is this too onerous? Should we support
> > 0.98
> > >> =>
> > >> > > 2.0?).
> > >> > > + You do NOT have to upgrade to the latest release of hbase1 to
> > >> migrate
> > >> > to
> > >> > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > >> > > + You'll have to update your hbase1 coprocessors to deploy them on
> > >> > hbase2.
> > >> > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> > >> > > watching for region split on RegionServer no longer makes sense
> > given
> > >> > > Master runs all splits now.
> > >> > > + An hbase1 client can run against an hbase2 cluster but it will
> > only
> > >> be
> > >> > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to
> > do
> > >> > admin
> > >> > > ops using an hbase1 Admin client against an hbase2 cluster. We have
> > >> some
> > >> > > egregious API violations in branch-1; e.g. we have protobuf in our
> > API
> > >> > (See
> > >> > > HBASE-15607). The notion is that we can't afford a deprecation cycle
> > >> > > purging this stuff from our Admin API.
> > >> > >
> > >> > > What you all think?
> > >> > >
> > >> > > St.Ack
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Ted Yu <yu...@gmail.com>.
I expanded the condition in the filter like this:
project = HBASE AND fixVersion in (2.0.0, 2.0.0-alpha-1, 2.0.0-alpha-2,
3.0) AND labels in (incompatibleChange, incompatible, incompatibility)
Still there're only two showing up.
On Thu, Aug 3, 2017 at 9:07 AM, Zach York <zy...@gmail.com>
wrote:
> I tried to filter based on imcompatible labels and there were only two
> JIRAs returned [1]. I have a hard time believing that there were only two
> breaking changes from 1.x to 2.x.
>
> [1]
> https://issues.apache.org/jira/browse/HBASE-17957?jql=
> project%20%3D%20HBASE%20AND%20fixVersion%20%3D%202.0.0%
> 20AND%20labels%20in%20(incompatibleChange%2C%20incompatible%2C%
> 20incompatibility)
>
> On Thu, Aug 3, 2017 at 8:54 AM, Zach York <zy...@gmail.com>
> wrote:
>
> > This kinda helps, but these seem more like expectations. I was going more
> > for things like HFile format changed, meta table structure changed,
> > coprocessor implementations changed (these are just examples, I don't
> know
> > if any of these actually changed).
> >
> > More technical differences between branch-1 and branch-2 which then can
> > help us get the right expectations for compatibility.
> >
> > On Wed, Aug 2, 2017 at 6:34 PM, Stack <st...@duboce.net> wrote:
> >
> >> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <zyork.contribution@gmail.com
> >
> >> wrote:
> >>
> >> > Do we know what the major pain points for migration are? Can we
> discuss
> >> > that/get a list going?
> >> >
> >> >
> >> Here's a few in outline:
> >>
> >> + There is issue of formats, of hbase-2.x being able to read hbase-1.x
> >> data
> >> whether from HDFS or ZooKeeper or off the wire.
> >> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> >> cluster; no holes in the API or unintelligible serializations.
> >> + There is then the little dance that has us rolling restart from an
> >> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> will
> >> assign regions to the new hbase-2.x regionservers as they come on line.
> >> TBD.
> >>
> >> Is this what you mean sir?
> >>
> >> S
> >>
> >>
> >> > I think without that knowledge it is hard (for me at least :) ) to
> >> > determine where we should set our sights in terms of migration.
> >> >
> >> > Thanks,
> >> > Zach
> >> >
> >> > On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> >> >
> >> > > What are our expectations regards compatibility between hbase1 and
> >> > hbase2?
> >> > >
> >> > > Lets have a chat about it. Here are some goal posts.
> >> > >
> >> > > + You have to upgrade to hbase-1.x before you can migrate to
> hbase-2.
> >> No
> >> > > migration from < hbase-1 (Is this too onerous? Should we support
> 0.98
> >> =>
> >> > > 2.0?).
> >> > > + You do NOT have to upgrade to the latest release of hbase1 to
> >> migrate
> >> > to
> >> > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> >> > > + You'll have to update your hbase1 coprocessors to deploy them on
> >> > hbase2.
> >> > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> >> > > watching for region split on RegionServer no longer makes sense
> given
> >> > > Master runs all splits now.
> >> > > + An hbase1 client can run against an hbase2 cluster but it will
> only
> >> be
> >> > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to
> do
> >> > admin
> >> > > ops using an hbase1 Admin client against an hbase2 cluster. We have
> >> some
> >> > > egregious API violations in branch-1; e.g. we have protobuf in our
> API
> >> > (See
> >> > > HBASE-15607). The notion is that we can't afford a deprecation cycle
> >> > > purging this stuff from our Admin API.
> >> > >
> >> > > What you all think?
> >> > >
> >> > > St.Ack
> >> > >
> >> >
> >>
> >
> >
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Zach York <zy...@gmail.com>.
I tried to filter based on imcompatible labels and there were only two
JIRAs returned [1]. I have a hard time believing that there were only two
breaking changes from 1.x to 2.x.
[1]
https://issues.apache.org/jira/browse/HBASE-17957?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%202.0.0%20AND%20labels%20in%20(incompatibleChange%2C%20incompatible%2C%20incompatibility)
On Thu, Aug 3, 2017 at 8:54 AM, Zach York <zy...@gmail.com>
wrote:
> This kinda helps, but these seem more like expectations. I was going more
> for things like HFile format changed, meta table structure changed,
> coprocessor implementations changed (these are just examples, I don't know
> if any of these actually changed).
>
> More technical differences between branch-1 and branch-2 which then can
> help us get the right expectations for compatibility.
>
> On Wed, Aug 2, 2017 at 6:34 PM, Stack <st...@duboce.net> wrote:
>
>> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <zy...@gmail.com>
>> wrote:
>>
>> > Do we know what the major pain points for migration are? Can we discuss
>> > that/get a list going?
>> >
>> >
>> Here's a few in outline:
>>
>> + There is issue of formats, of hbase-2.x being able to read hbase-1.x
>> data
>> whether from HDFS or ZooKeeper or off the wire.
>> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
>> cluster; no holes in the API or unintelligible serializations.
>> + There is then the little dance that has us rolling restart from an
>> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it will
>> assign regions to the new hbase-2.x regionservers as they come on line.
>> TBD.
>>
>> Is this what you mean sir?
>>
>> S
>>
>>
>> > I think without that knowledge it is hard (for me at least :) ) to
>> > determine where we should set our sights in terms of migration.
>> >
>> > Thanks,
>> > Zach
>> >
>> > On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
>> >
>> > > What are our expectations regards compatibility between hbase1 and
>> > hbase2?
>> > >
>> > > Lets have a chat about it. Here are some goal posts.
>> > >
>> > > + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
>> No
>> > > migration from < hbase-1 (Is this too onerous? Should we support 0.98
>> =>
>> > > 2.0?).
>> > > + You do NOT have to upgrade to the latest release of hbase1 to
>> migrate
>> > to
>> > > hbase2; being up on hbase-1.0.0+ will be sufficient.
>> > > + You'll have to update your hbase1 coprocessors to deploy them on
>> > hbase2.
>> > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
>> > > watching for region split on RegionServer no longer makes sense given
>> > > Master runs all splits now.
>> > > + An hbase1 client can run against an hbase2 cluster but it will only
>> be
>> > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
>> > admin
>> > > ops using an hbase1 Admin client against an hbase2 cluster. We have
>> some
>> > > egregious API violations in branch-1; e.g. we have protobuf in our API
>> > (See
>> > > HBASE-15607). The notion is that we can't afford a deprecation cycle
>> > > purging this stuff from our Admin API.
>> > >
>> > > What you all think?
>> > >
>> > > St.Ack
>> > >
>> >
>>
>
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by stack <sa...@gmail.com>.
On Thu, Aug 24, 2017 at 9:13 AM, stack <sa...@gmail.com> wrote:
> On Thu, Aug 3, 2017 at 11:05 PM, stack <sa...@gmail.com> wrote:
>
>> Thanks Zach for clarification. Let me work up a list and then come back
>> to this thread. Jira needs an edit pass to.
>>
>>
> JIRA editing reflecting incompatibles has been coming along. It is still a
> work-in-progress though. Meantime, let me air out here the incompatibles as
> I encounter them. The complete list may never be known (smile) and the best
> knowledge will only be had on the day before release which will be giving
> notice too late for folks to do much about it.
>
> Here is a known breaking incompatible that just went in: HBASE-15982
> "Interface ReplicationEndpoint extends Guava's Service". See the release
> note for why the breakage unavoidable. Anyone who has implemented our
> Replication Endpoint will have a refactor on their hands deploying their
> replication machine atop hbase2 (Side note: Estebans' initial work has it
> that hbase native replication seems to work going from hbase1 to hbase2 and
> even the other way around; more to do....). Upside, if there is the will,
> now is the time for HBASE-10504 "Define Replication Interface". It is late
> in the game but am up for helping out if interest.
>
> Here is a breaking compatible we *could* (maybe) avoid but the amount of
> work involved in the workaround and how the patch-up would muddy our
> messaging around protobufs in the HBase public API is not worth the price
> IMO. What do you all think? The issue is HBASE-15607 "Remove PB references
> from Admin for 2.0". The plan is to (re-)commit Ram's change. We'd then
> tell users that you cannot administer an hbase2 cluster using an hbase1
> client.
>
>
Ok. The "could" above has changed to a "can't" on review of changes in
Admin Interface since hbase1.0.0. The amount of work involved would be too
much and the methods we'd be restoring compatibility on are wonky in the
first place; i.e. async calls that gave you no means of querying state of
the call and protobuf Messages that would require 'translation' from our
internal protobuf to protobuf 2.5. Admin Interface in 2.0 will be
incompatible with hbase 1.x Admin.
I've a WIP incompatibles list as a messy subsection of the hbase2 doc [1].
St.Ack
1.
https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#heading=h.723jjn18p2jr
> Coprocessors will require a refactor to deploy on hbase2. More detail to
> follow.
>
> St.Ack
>
>
>
>
>
>
>> S
>>
>> On Aug 3, 2017 23:54, "Zach York" <zy...@gmail.com> wrote:
>>
>> This kinda helps, but these seem more like expectations. I was going more
>> for things like HFile format changed, meta table structure changed,
>> coprocessor implementations changed (these are just examples, I don't know
>> if any of these actually changed).
>>
>> More technical differences between branch-1 and branch-2 which then can
>> help us get the right expectations for compatibility.
>>
>> On Wed, Aug 2, 2017 at 6:34 PM, Stack <st...@duboce.net> wrote:
>>
>> > On Wed, Aug 2, 2017 at 5:25 PM, Zach York <zyork.contribution@gmail.com
>> >
>> > wrote:
>> >
>> > > Do we know what the major pain points for migration are? Can we
>> discuss
>> > > that/get a list going?
>> > >
>> > >
>> > Here's a few in outline:
>> >
>> > + There is issue of formats, of hbase-2.x being able to read hbase-1.x
>> data
>> > whether from HDFS or ZooKeeper or off the wire.
>> > + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
>> > cluster; no holes in the API or unintelligible serializations.
>> > + There is then the little dance that has us rolling restart from an
>> > hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
>> will
>> > assign regions to the new hbase-2.x regionservers as they come on line.
>> > TBD.
>> >
>> > Is this what you mean sir?
>> >
>> > S
>> >
>> >
>> > > I think without that knowledge it is hard (for me at least :) ) to
>> > > determine where we should set our sights in terms of migration.
>> > >
>> > > Thanks,
>> > > Zach
>> > >
>> > > On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
>> > >
>> > > > What are our expectations regards compatibility between hbase1 and
>> > > hbase2?
>> > > >
>> > > > Lets have a chat about it. Here are some goal posts.
>> > > >
>> > > > + You have to upgrade to hbase-1.x before you can migrate to
>> hbase-2.
>> > No
>> > > > migration from < hbase-1 (Is this too onerous? Should we support
>> 0.98
>> > =>
>> > > > 2.0?).
>> > > > + You do NOT have to upgrade to the latest release of hbase1 to
>> migrate
>> > > to
>> > > > hbase2; being up on hbase-1.0.0+ will be sufficient.
>> > > > + You'll have to update your hbase1 coprocessors to deploy them on
>> > > hbase2.
>> > > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
>> > > > watching for region split on RegionServer no longer makes sense
>> given
>> > > > Master runs all splits now.
>> > > > + An hbase1 client can run against an hbase2 cluster but it will
>> only
>> > be
>> > > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to
>> do
>> > > admin
>> > > > ops using an hbase1 Admin client against an hbase2 cluster. We have
>> > some
>> > > > egregious API violations in branch-1; e.g. we have protobuf in our
>> API
>> > > (See
>> > > > HBASE-15607). The notion is that we can't afford a deprecation cycle
>> > > > purging this stuff from our Admin API.
>> > > >
>> > > > What you all think?
>> > > >
>> > > > St.Ack
>> > > >
>> > >
>> >
>>
>>
>>
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by stack <sa...@gmail.com>.
On Thu, Aug 3, 2017 at 11:05 PM, stack <sa...@gmail.com> wrote:
> Thanks Zach for clarification. Let me work up a list and then come back to
> this thread. Jira needs an edit pass to.
>
>
JIRA editing reflecting incompatibles has been coming along. It is still a
work-in-progress though. Meantime, let me air out here the incompatibles as
I encounter them. The complete list may never be known (smile) and the best
knowledge will only be had on the day before release which will be giving
notice too late for folks to do much about it.
Here is a known breaking incompatible that just went in: HBASE-15982
"Interface ReplicationEndpoint extends Guava's Service". See the release
note for why the breakage unavoidable. Anyone who has implemented our
Replication Endpoint will have a refactor on their hands deploying their
replication machine atop hbase2 (Side note: Estebans' initial work has it
that hbase native replication seems to work going from hbase1 to hbase2 and
even the other way around; more to do....). Upside, if there is the will,
now is the time for HBASE-10504 "Define Replication Interface". It is late
in the game but am up for helping out if interest.
Here is a breaking compatible we *could* (maybe) avoid but the amount of
work involved in the workaround and how the patch-up would muddy our
messaging around protobufs in the HBase public API is not worth the price
IMO. What do you all think? The issue is HBASE-15607 "Remove PB references
from Admin for 2.0". The plan is to (re-)commit Ram's change. We'd then
tell users that you cannot administer an hbase2 cluster using an hbase1
client.
Coprocessors will require a refactor to deploy on hbase2. More detail to
follow.
St.Ack
> S
>
> On Aug 3, 2017 23:54, "Zach York" <zy...@gmail.com> wrote:
>
> This kinda helps, but these seem more like expectations. I was going more
> for things like HFile format changed, meta table structure changed,
> coprocessor implementations changed (these are just examples, I don't know
> if any of these actually changed).
>
> More technical differences between branch-1 and branch-2 which then can
> help us get the right expectations for compatibility.
>
> On Wed, Aug 2, 2017 at 6:34 PM, Stack <st...@duboce.net> wrote:
>
> > On Wed, Aug 2, 2017 at 5:25 PM, Zach York <zy...@gmail.com>
> > wrote:
> >
> > > Do we know what the major pain points for migration are? Can we discuss
> > > that/get a list going?
> > >
> > >
> > Here's a few in outline:
> >
> > + There is issue of formats, of hbase-2.x being able to read hbase-1.x
> data
> > whether from HDFS or ZooKeeper or off the wire.
> > + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> > cluster; no holes in the API or unintelligible serializations.
> > + There is then the little dance that has us rolling restart from an
> > hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> will
> > assign regions to the new hbase-2.x regionservers as they come on line.
> > TBD.
> >
> > Is this what you mean sir?
> >
> > S
> >
> >
> > > I think without that knowledge it is hard (for me at least :) ) to
> > > determine where we should set our sights in terms of migration.
> > >
> > > Thanks,
> > > Zach
> > >
> > > On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > What are our expectations regards compatibility between hbase1 and
> > > hbase2?
> > > >
> > > > Lets have a chat about it. Here are some goal posts.
> > > >
> > > > + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
> > No
> > > > migration from < hbase-1 (Is this too onerous? Should we support 0.98
> > =>
> > > > 2.0?).
> > > > + You do NOT have to upgrade to the latest release of hbase1 to
> migrate
> > > to
> > > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > > + You'll have to update your hbase1 coprocessors to deploy them on
> > > hbase2.
> > > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> > > > watching for region split on RegionServer no longer makes sense given
> > > > Master runs all splits now.
> > > > + An hbase1 client can run against an hbase2 cluster but it will only
> > be
> > > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> > > admin
> > > > ops using an hbase1 Admin client against an hbase2 cluster. We have
> > some
> > > > egregious API violations in branch-1; e.g. we have protobuf in our
> API
> > > (See
> > > > HBASE-15607). The notion is that we can't afford a deprecation cycle
> > > > purging this stuff from our Admin API.
> > > >
> > > > What you all think?
> > > >
> > > > St.Ack
> > > >
> > >
> >
>
>
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Stack <st...@duboce.net>.
On Fri, Aug 11, 2017 at 5:07 PM, Apekshit Sharma <ap...@cloudera.com> wrote:
> Ref: deleting/deprecating some methods in HBaseAdmin
> https://github.com/apache/hbase/commit/de696cf6b653749c6bf105ef3d62d7
> a6c6923c57
> From first message:
> *" An hbase1 client can run against an hbase2 cluster but it will only be
> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do admin
> ops using an hbase1 Admin client against an hbase2 cluster."*
>
> Question 1:
> These guarantees are with what version of jars?
> Does the client has to keep using corresponding 1.x jars with which it was
> compiled, OR can the 2.x jars just replace earlier ones and client is still
> expected to work correctly? (i would say no!)
>
>
Dropping hbase2 jars into an hbase1 deploy? No. No guarantee of binary
compat.
I was thinking of wire compatibility here Appy.
> Question second:
> How do we handle clients doing deprecated admin ops?
> a) make admin functions no-op by using empty method.
> b) throw exception. May/maynot crash the client - ideally shouldn't since
> methods are define with 'throws IOException'.
>
I think throwing exception the way to go. Its clean signal of change.
> c) delete the method - will crash the client
> But the question only makes sense if expect clients to work with 2.x jars.
> If not, we can just delete them, simple!
>
>
I'm not sure I follow. If a hbase1 client calls a deprecated/unimplemented
method, I was thinking it'd get an exception.
> Question 3: AMv2 related
> There are ops in old HBaseAdmin that'll be destructive for 2.0
> (closeRegion() fn).
> If clients continue to use 1.x jars and make these calls, that's not
> acceptable. We need to handle such deprecation on server sider itself.
> If only there was way of versioning the rpc service!
> Another way, but bad one is, deprecate existing ones and move logic to a
> new one. I think there are just 1-2 that need this treatment.
> Either way, need to come up with a solution!
>
>
Throw an exception pointing at the alternative API?
Thanks Appy,
S
>
> On Fri, Aug 4, 2017 at 10:08 AM, Andrew Purtell <an...@gmail.com>
> wrote:
>
> > This is a really good question. I think many operators will give a lot of
> > leeway to data format changes as long as data can be copied from A to B
> > (perhaps with batch rewrite to upgrade (ideally, not required)) and
> > replication can be enabled to sync up to the current moment for cut over.
> >
> >
> > > On Aug 4, 2017, at 10:00 AM, Esteban Gutierrez <es...@cloudera.com>
> > wrote:
> > >
> > > Should we add additional details around replication as well? for
> > instance,
> > > shall we consider a hbase-1.x cluster as a client for a hbase-2.x
> > cluster?
> > >
> > > Thanks for starting this discussion Stack,
> > >
> > > esteban.
> > >
> > > --
> > > Cloudera, Inc.
> > >
> > >
> > >> On Fri, Aug 4, 2017 at 1:05 AM, stack <sa...@gmail.com> wrote:
> > >>
> > >> Thanks Zach for clarification. Let me work up a list and then come
> back
> > to
> > >> this thread. Jira needs an edit pass to.
> > >>
> > >> S
> > >>
> > >> On Aug 3, 2017 23:54, "Zach York" <zy...@gmail.com>
> wrote:
> > >>
> > >> This kinda helps, but these seem more like expectations. I was going
> > more
> > >> for things like HFile format changed, meta table structure changed,
> > >> coprocessor implementations changed (these are just examples, I don't
> > know
> > >> if any of these actually changed).
> > >>
> > >> More technical differences between branch-1 and branch-2 which then
> can
> > >> help us get the right expectations for compatibility.
> > >>
> > >>> On Wed, Aug 2, 2017 at 6:34 PM, Stack <st...@duboce.net> wrote:
> > >>>
> > >>> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <
> > zyork.contribution@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Do we know what the major pain points for migration are? Can we
> > discuss
> > >>>> that/get a list going?
> > >>>>
> > >>>>
> > >>> Here's a few in outline:
> > >>>
> > >>> + There is issue of formats, of hbase-2.x being able to read
> hbase-1.x
> > >> data
> > >>> whether from HDFS or ZooKeeper or off the wire.
> > >>> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> > >>> cluster; no holes in the API or unintelligible serializations.
> > >>> + There is then the little dance that has us rolling restart from an
> > >>> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> > >> will
> > >>> assign regions to the new hbase-2.x regionservers as they come on
> line.
> > >>> TBD.
> > >>>
> > >>> Is this what you mean sir?
> > >>>
> > >>> S
> > >>>
> > >>>
> > >>>> I think without that knowledge it is hard (for me at least :) ) to
> > >>>> determine where we should set our sights in terms of migration.
> > >>>>
> > >>>> Thanks,
> > >>>> Zach
> > >>>>
> > >>>>> On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> > >>>>>
> > >>>>> What are our expectations regards compatibility between hbase1 and
> > >>>> hbase2?
> > >>>>>
> > >>>>> Lets have a chat about it. Here are some goal posts.
> > >>>>>
> > >>>>> + You have to upgrade to hbase-1.x before you can migrate to
> hbase-2.
> > >>> No
> > >>>>> migration from < hbase-1 (Is this too onerous? Should we support
> 0.98
> > >>> =>
> > >>>>> 2.0?).
> > >>>>> + You do NOT have to upgrade to the latest release of hbase1 to
> > >> migrate
> > >>>> to
> > >>>>> hbase2; being up on hbase-1.0.0+ will be sufficient.
> > >>>>> + You'll have to update your hbase1 coprocessors to deploy them on
> > >>>> hbase2.
> > >>>>> A bunch of CP API has/will change by the time hbase2 comes out;
> e.g.
> > >>>>> watching for region split on RegionServer no longer makes sense
> given
> > >>>>> Master runs all splits now.
> > >>>>> + An hbase1 client can run against an hbase2 cluster but it will
> only
> > >>> be
> > >>>>> able to do DML (Get/Put/Scan, etc.). We do not allow being able to
> do
> > >>>> admin
> > >>>>> ops using an hbase1 Admin client against an hbase2 cluster. We have
> > >>> some
> > >>>>> egregious API violations in branch-1; e.g. we have protobuf in our
> > >> API
> > >>>> (See
> > >>>>> HBASE-15607). The notion is that we can't afford a deprecation
> cycle
> > >>>>> purging this stuff from our Admin API.
> > >>>>>
> > >>>>> What you all think?
> > >>>>>
> > >>>>> St.Ack
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
>
>
>
> --
>
> -- Appy
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Apekshit Sharma <ap...@cloudera.com>.
Ref: deleting/deprecating some methods in HBaseAdmin
https://github.com/apache/hbase/commit/de696cf6b653749c6bf105ef3d62d7a6c6923c57
From first message:
*" An hbase1 client can run against an hbase2 cluster but it will only be
able to do DML (Get/Put/Scan, etc.). We do not allow being able to do admin
ops using an hbase1 Admin client against an hbase2 cluster."*
Question 1:
These guarantees are with what version of jars?
Does the client has to keep using corresponding 1.x jars with which it was
compiled, OR can the 2.x jars just replace earlier ones and client is still
expected to work correctly? (i would say no!)
Question second:
How do we handle clients doing deprecated admin ops?
a) make admin functions no-op by using empty method.
b) throw exception. May/maynot crash the client - ideally shouldn't since
methods are define with 'throws IOException'.
c) delete the method - will crash the client
But the question only makes sense if expect clients to work with 2.x jars.
If not, we can just delete them, simple!
Question 3: AMv2 related
There are ops in old HBaseAdmin that'll be destructive for 2.0
(closeRegion() fn).
If clients continue to use 1.x jars and make these calls, that's not
acceptable. We need to handle such deprecation on server sider itself.
If only there was way of versioning the rpc service!
Another way, but bad one is, deprecate existing ones and move logic to a
new one. I think there are just 1-2 that need this treatment.
Either way, need to come up with a solution!
On Fri, Aug 4, 2017 at 10:08 AM, Andrew Purtell <an...@gmail.com>
wrote:
> This is a really good question. I think many operators will give a lot of
> leeway to data format changes as long as data can be copied from A to B
> (perhaps with batch rewrite to upgrade (ideally, not required)) and
> replication can be enabled to sync up to the current moment for cut over.
>
>
> > On Aug 4, 2017, at 10:00 AM, Esteban Gutierrez <es...@cloudera.com>
> wrote:
> >
> > Should we add additional details around replication as well? for
> instance,
> > shall we consider a hbase-1.x cluster as a client for a hbase-2.x
> cluster?
> >
> > Thanks for starting this discussion Stack,
> >
> > esteban.
> >
> > --
> > Cloudera, Inc.
> >
> >
> >> On Fri, Aug 4, 2017 at 1:05 AM, stack <sa...@gmail.com> wrote:
> >>
> >> Thanks Zach for clarification. Let me work up a list and then come back
> to
> >> this thread. Jira needs an edit pass to.
> >>
> >> S
> >>
> >> On Aug 3, 2017 23:54, "Zach York" <zy...@gmail.com> wrote:
> >>
> >> This kinda helps, but these seem more like expectations. I was going
> more
> >> for things like HFile format changed, meta table structure changed,
> >> coprocessor implementations changed (these are just examples, I don't
> know
> >> if any of these actually changed).
> >>
> >> More technical differences between branch-1 and branch-2 which then can
> >> help us get the right expectations for compatibility.
> >>
> >>> On Wed, Aug 2, 2017 at 6:34 PM, Stack <st...@duboce.net> wrote:
> >>>
> >>> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <
> zyork.contribution@gmail.com>
> >>> wrote:
> >>>
> >>>> Do we know what the major pain points for migration are? Can we
> discuss
> >>>> that/get a list going?
> >>>>
> >>>>
> >>> Here's a few in outline:
> >>>
> >>> + There is issue of formats, of hbase-2.x being able to read hbase-1.x
> >> data
> >>> whether from HDFS or ZooKeeper or off the wire.
> >>> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> >>> cluster; no holes in the API or unintelligible serializations.
> >>> + There is then the little dance that has us rolling restart from an
> >>> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> >> will
> >>> assign regions to the new hbase-2.x regionservers as they come on line.
> >>> TBD.
> >>>
> >>> Is this what you mean sir?
> >>>
> >>> S
> >>>
> >>>
> >>>> I think without that knowledge it is hard (for me at least :) ) to
> >>>> determine where we should set our sights in terms of migration.
> >>>>
> >>>> Thanks,
> >>>> Zach
> >>>>
> >>>>> On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> >>>>>
> >>>>> What are our expectations regards compatibility between hbase1 and
> >>>> hbase2?
> >>>>>
> >>>>> Lets have a chat about it. Here are some goal posts.
> >>>>>
> >>>>> + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
> >>> No
> >>>>> migration from < hbase-1 (Is this too onerous? Should we support 0.98
> >>> =>
> >>>>> 2.0?).
> >>>>> + You do NOT have to upgrade to the latest release of hbase1 to
> >> migrate
> >>>> to
> >>>>> hbase2; being up on hbase-1.0.0+ will be sufficient.
> >>>>> + You'll have to update your hbase1 coprocessors to deploy them on
> >>>> hbase2.
> >>>>> A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> >>>>> watching for region split on RegionServer no longer makes sense given
> >>>>> Master runs all splits now.
> >>>>> + An hbase1 client can run against an hbase2 cluster but it will only
> >>> be
> >>>>> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> >>>> admin
> >>>>> ops using an hbase1 Admin client against an hbase2 cluster. We have
> >>> some
> >>>>> egregious API violations in branch-1; e.g. we have protobuf in our
> >> API
> >>>> (See
> >>>>> HBASE-15607). The notion is that we can't afford a deprecation cycle
> >>>>> purging this stuff from our Admin API.
> >>>>>
> >>>>> What you all think?
> >>>>>
> >>>>> St.Ack
> >>>>>
> >>>>
> >>>
> >>
>
--
-- Appy
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Andrew Purtell <an...@gmail.com>.
This is a really good question. I think many operators will give a lot of leeway to data format changes as long as data can be copied from A to B (perhaps with batch rewrite to upgrade (ideally, not required)) and replication can be enabled to sync up to the current moment for cut over.
> On Aug 4, 2017, at 10:00 AM, Esteban Gutierrez <es...@cloudera.com> wrote:
>
> Should we add additional details around replication as well? for instance,
> shall we consider a hbase-1.x cluster as a client for a hbase-2.x cluster?
>
> Thanks for starting this discussion Stack,
>
> esteban.
>
> --
> Cloudera, Inc.
>
>
>> On Fri, Aug 4, 2017 at 1:05 AM, stack <sa...@gmail.com> wrote:
>>
>> Thanks Zach for clarification. Let me work up a list and then come back to
>> this thread. Jira needs an edit pass to.
>>
>> S
>>
>> On Aug 3, 2017 23:54, "Zach York" <zy...@gmail.com> wrote:
>>
>> This kinda helps, but these seem more like expectations. I was going more
>> for things like HFile format changed, meta table structure changed,
>> coprocessor implementations changed (these are just examples, I don't know
>> if any of these actually changed).
>>
>> More technical differences between branch-1 and branch-2 which then can
>> help us get the right expectations for compatibility.
>>
>>> On Wed, Aug 2, 2017 at 6:34 PM, Stack <st...@duboce.net> wrote:
>>>
>>> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <zy...@gmail.com>
>>> wrote:
>>>
>>>> Do we know what the major pain points for migration are? Can we discuss
>>>> that/get a list going?
>>>>
>>>>
>>> Here's a few in outline:
>>>
>>> + There is issue of formats, of hbase-2.x being able to read hbase-1.x
>> data
>>> whether from HDFS or ZooKeeper or off the wire.
>>> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
>>> cluster; no holes in the API or unintelligible serializations.
>>> + There is then the little dance that has us rolling restart from an
>>> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
>> will
>>> assign regions to the new hbase-2.x regionservers as they come on line.
>>> TBD.
>>>
>>> Is this what you mean sir?
>>>
>>> S
>>>
>>>
>>>> I think without that knowledge it is hard (for me at least :) ) to
>>>> determine where we should set our sights in terms of migration.
>>>>
>>>> Thanks,
>>>> Zach
>>>>
>>>>> On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
>>>>>
>>>>> What are our expectations regards compatibility between hbase1 and
>>>> hbase2?
>>>>>
>>>>> Lets have a chat about it. Here are some goal posts.
>>>>>
>>>>> + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
>>> No
>>>>> migration from < hbase-1 (Is this too onerous? Should we support 0.98
>>> =>
>>>>> 2.0?).
>>>>> + You do NOT have to upgrade to the latest release of hbase1 to
>> migrate
>>>> to
>>>>> hbase2; being up on hbase-1.0.0+ will be sufficient.
>>>>> + You'll have to update your hbase1 coprocessors to deploy them on
>>>> hbase2.
>>>>> A bunch of CP API has/will change by the time hbase2 comes out; e.g.
>>>>> watching for region split on RegionServer no longer makes sense given
>>>>> Master runs all splits now.
>>>>> + An hbase1 client can run against an hbase2 cluster but it will only
>>> be
>>>>> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
>>>> admin
>>>>> ops using an hbase1 Admin client against an hbase2 cluster. We have
>>> some
>>>>> egregious API violations in branch-1; e.g. we have protobuf in our
>> API
>>>> (See
>>>>> HBASE-15607). The notion is that we can't afford a deprecation cycle
>>>>> purging this stuff from our Admin API.
>>>>>
>>>>> What you all think?
>>>>>
>>>>> St.Ack
>>>>>
>>>>
>>>
>>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Esteban Gutierrez <es...@cloudera.com>.
Thanks Stack!
--
Cloudera, Inc.
On Mon, Aug 14, 2017 at 1:05 PM, Stack <st...@duboce.net> wrote:
> On Fri, Aug 4, 2017 at 10:00 AM, Esteban Gutierrez <es...@cloudera.com>
> wrote:
>
> > Should we add additional details around replication as well? for
> instance,
> > shall we consider a hbase-1.x cluster as a client for a hbase-2.x
> cluster?
> >
> >
> Yes. I'd say this should be a blocker Esteban. Filed HBASE-18596.
> Thanks,
> S
>
>
>
>
>
> > Thanks for starting this discussion Stack,
> >
> > esteban.
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Fri, Aug 4, 2017 at 1:05 AM, stack <sa...@gmail.com> wrote:
> >
> > > Thanks Zach for clarification. Let me work up a list and then come back
> > to
> > > this thread. Jira needs an edit pass to.
> > >
> > > S
> > >
> > > On Aug 3, 2017 23:54, "Zach York" <zy...@gmail.com>
> wrote:
> > >
> > > This kinda helps, but these seem more like expectations. I was going
> more
> > > for things like HFile format changed, meta table structure changed,
> > > coprocessor implementations changed (these are just examples, I don't
> > know
> > > if any of these actually changed).
> > >
> > > More technical differences between branch-1 and branch-2 which then can
> > > help us get the right expectations for compatibility.
> > >
> > > On Wed, Aug 2, 2017 at 6:34 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > On Wed, Aug 2, 2017 at 5:25 PM, Zach York <
> > zyork.contribution@gmail.com>
> > > > wrote:
> > > >
> > > > > Do we know what the major pain points for migration are? Can we
> > discuss
> > > > > that/get a list going?
> > > > >
> > > > >
> > > > Here's a few in outline:
> > > >
> > > > + There is issue of formats, of hbase-2.x being able to read
> hbase-1.x
> > > data
> > > > whether from HDFS or ZooKeeper or off the wire.
> > > > + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> > > > cluster; no holes in the API or unintelligible serializations.
> > > > + There is then the little dance that has us rolling restart from an
> > > > hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> > > will
> > > > assign regions to the new hbase-2.x regionservers as they come on
> line.
> > > > TBD.
> > > >
> > > > Is this what you mean sir?
> > > >
> > > > S
> > > >
> > > >
> > > > > I think without that knowledge it is hard (for me at least :) ) to
> > > > > determine where we should set our sights in terms of migration.
> > > > >
> > > > > Thanks,
> > > > > Zach
> > > > >
> > > > > On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> > > > >
> > > > > > What are our expectations regards compatibility between hbase1
> and
> > > > > hbase2?
> > > > > >
> > > > > > Lets have a chat about it. Here are some goal posts.
> > > > > >
> > > > > > + You have to upgrade to hbase-1.x before you can migrate to
> > hbase-2.
> > > > No
> > > > > > migration from < hbase-1 (Is this too onerous? Should we support
> > 0.98
> > > > =>
> > > > > > 2.0?).
> > > > > > + You do NOT have to upgrade to the latest release of hbase1 to
> > > migrate
> > > > > to
> > > > > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > > > > + You'll have to update your hbase1 coprocessors to deploy them
> on
> > > > > hbase2.
> > > > > > A bunch of CP API has/will change by the time hbase2 comes out;
> > e.g.
> > > > > > watching for region split on RegionServer no longer makes sense
> > given
> > > > > > Master runs all splits now.
> > > > > > + An hbase1 client can run against an hbase2 cluster but it will
> > only
> > > > be
> > > > > > able to do DML (Get/Put/Scan, etc.). We do not allow being able
> to
> > do
> > > > > admin
> > > > > > ops using an hbase1 Admin client against an hbase2 cluster. We
> have
> > > > some
> > > > > > egregious API violations in branch-1; e.g. we have protobuf in
> our
> > > API
> > > > > (See
> > > > > > HBASE-15607). The notion is that we can't afford a deprecation
> > cycle
> > > > > > purging this stuff from our Admin API.
> > > > > >
> > > > > > What you all think?
> > > > > >
> > > > > > St.Ack
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Stack <st...@duboce.net>.
On Fri, Aug 4, 2017 at 10:00 AM, Esteban Gutierrez <es...@cloudera.com>
wrote:
> Should we add additional details around replication as well? for instance,
> shall we consider a hbase-1.x cluster as a client for a hbase-2.x cluster?
>
>
Yes. I'd say this should be a blocker Esteban. Filed HBASE-18596.
Thanks,
S
> Thanks for starting this discussion Stack,
>
> esteban.
>
> --
> Cloudera, Inc.
>
>
> On Fri, Aug 4, 2017 at 1:05 AM, stack <sa...@gmail.com> wrote:
>
> > Thanks Zach for clarification. Let me work up a list and then come back
> to
> > this thread. Jira needs an edit pass to.
> >
> > S
> >
> > On Aug 3, 2017 23:54, "Zach York" <zy...@gmail.com> wrote:
> >
> > This kinda helps, but these seem more like expectations. I was going more
> > for things like HFile format changed, meta table structure changed,
> > coprocessor implementations changed (these are just examples, I don't
> know
> > if any of these actually changed).
> >
> > More technical differences between branch-1 and branch-2 which then can
> > help us get the right expectations for compatibility.
> >
> > On Wed, Aug 2, 2017 at 6:34 PM, Stack <st...@duboce.net> wrote:
> >
> > > On Wed, Aug 2, 2017 at 5:25 PM, Zach York <
> zyork.contribution@gmail.com>
> > > wrote:
> > >
> > > > Do we know what the major pain points for migration are? Can we
> discuss
> > > > that/get a list going?
> > > >
> > > >
> > > Here's a few in outline:
> > >
> > > + There is issue of formats, of hbase-2.x being able to read hbase-1.x
> > data
> > > whether from HDFS or ZooKeeper or off the wire.
> > > + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> > > cluster; no holes in the API or unintelligible serializations.
> > > + There is then the little dance that has us rolling restart from an
> > > hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> > will
> > > assign regions to the new hbase-2.x regionservers as they come on line.
> > > TBD.
> > >
> > > Is this what you mean sir?
> > >
> > > S
> > >
> > >
> > > > I think without that knowledge it is hard (for me at least :) ) to
> > > > determine where we should set our sights in terms of migration.
> > > >
> > > > Thanks,
> > > > Zach
> > > >
> > > > On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > What are our expectations regards compatibility between hbase1 and
> > > > hbase2?
> > > > >
> > > > > Lets have a chat about it. Here are some goal posts.
> > > > >
> > > > > + You have to upgrade to hbase-1.x before you can migrate to
> hbase-2.
> > > No
> > > > > migration from < hbase-1 (Is this too onerous? Should we support
> 0.98
> > > =>
> > > > > 2.0?).
> > > > > + You do NOT have to upgrade to the latest release of hbase1 to
> > migrate
> > > > to
> > > > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > > > + You'll have to update your hbase1 coprocessors to deploy them on
> > > > hbase2.
> > > > > A bunch of CP API has/will change by the time hbase2 comes out;
> e.g.
> > > > > watching for region split on RegionServer no longer makes sense
> given
> > > > > Master runs all splits now.
> > > > > + An hbase1 client can run against an hbase2 cluster but it will
> only
> > > be
> > > > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to
> do
> > > > admin
> > > > > ops using an hbase1 Admin client against an hbase2 cluster. We have
> > > some
> > > > > egregious API violations in branch-1; e.g. we have protobuf in our
> > API
> > > > (See
> > > > > HBASE-15607). The notion is that we can't afford a deprecation
> cycle
> > > > > purging this stuff from our Admin API.
> > > > >
> > > > > What you all think?
> > > > >
> > > > > St.Ack
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Esteban Gutierrez <es...@cloudera.com>.
Should we add additional details around replication as well? for instance,
shall we consider a hbase-1.x cluster as a client for a hbase-2.x cluster?
Thanks for starting this discussion Stack,
esteban.
--
Cloudera, Inc.
On Fri, Aug 4, 2017 at 1:05 AM, stack <sa...@gmail.com> wrote:
> Thanks Zach for clarification. Let me work up a list and then come back to
> this thread. Jira needs an edit pass to.
>
> S
>
> On Aug 3, 2017 23:54, "Zach York" <zy...@gmail.com> wrote:
>
> This kinda helps, but these seem more like expectations. I was going more
> for things like HFile format changed, meta table structure changed,
> coprocessor implementations changed (these are just examples, I don't know
> if any of these actually changed).
>
> More technical differences between branch-1 and branch-2 which then can
> help us get the right expectations for compatibility.
>
> On Wed, Aug 2, 2017 at 6:34 PM, Stack <st...@duboce.net> wrote:
>
> > On Wed, Aug 2, 2017 at 5:25 PM, Zach York <zy...@gmail.com>
> > wrote:
> >
> > > Do we know what the major pain points for migration are? Can we discuss
> > > that/get a list going?
> > >
> > >
> > Here's a few in outline:
> >
> > + There is issue of formats, of hbase-2.x being able to read hbase-1.x
> data
> > whether from HDFS or ZooKeeper or off the wire.
> > + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> > cluster; no holes in the API or unintelligible serializations.
> > + There is then the little dance that has us rolling restart from an
> > hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it
> will
> > assign regions to the new hbase-2.x regionservers as they come on line.
> > TBD.
> >
> > Is this what you mean sir?
> >
> > S
> >
> >
> > > I think without that knowledge it is hard (for me at least :) ) to
> > > determine where we should set our sights in terms of migration.
> > >
> > > Thanks,
> > > Zach
> > >
> > > On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > What are our expectations regards compatibility between hbase1 and
> > > hbase2?
> > > >
> > > > Lets have a chat about it. Here are some goal posts.
> > > >
> > > > + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
> > No
> > > > migration from < hbase-1 (Is this too onerous? Should we support 0.98
> > =>
> > > > 2.0?).
> > > > + You do NOT have to upgrade to the latest release of hbase1 to
> migrate
> > > to
> > > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > > + You'll have to update your hbase1 coprocessors to deploy them on
> > > hbase2.
> > > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> > > > watching for region split on RegionServer no longer makes sense given
> > > > Master runs all splits now.
> > > > + An hbase1 client can run against an hbase2 cluster but it will only
> > be
> > > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> > > admin
> > > > ops using an hbase1 Admin client against an hbase2 cluster. We have
> > some
> > > > egregious API violations in branch-1; e.g. we have protobuf in our
> API
> > > (See
> > > > HBASE-15607). The notion is that we can't afford a deprecation cycle
> > > > purging this stuff from our Admin API.
> > > >
> > > > What you all think?
> > > >
> > > > St.Ack
> > > >
> > >
> >
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by stack <sa...@gmail.com>.
Thanks Zach for clarification. Let me work up a list and then come back to
this thread. Jira needs an edit pass to.
S
On Aug 3, 2017 23:54, "Zach York" <zy...@gmail.com> wrote:
This kinda helps, but these seem more like expectations. I was going more
for things like HFile format changed, meta table structure changed,
coprocessor implementations changed (these are just examples, I don't know
if any of these actually changed).
More technical differences between branch-1 and branch-2 which then can
help us get the right expectations for compatibility.
On Wed, Aug 2, 2017 at 6:34 PM, Stack <st...@duboce.net> wrote:
> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <zy...@gmail.com>
> wrote:
>
> > Do we know what the major pain points for migration are? Can we discuss
> > that/get a list going?
> >
> >
> Here's a few in outline:
>
> + There is issue of formats, of hbase-2.x being able to read hbase-1.x
data
> whether from HDFS or ZooKeeper or off the wire.
> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> cluster; no holes in the API or unintelligible serializations.
> + There is then the little dance that has us rolling restart from an
> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it will
> assign regions to the new hbase-2.x regionservers as they come on line.
> TBD.
>
> Is this what you mean sir?
>
> S
>
>
> > I think without that knowledge it is hard (for me at least :) ) to
> > determine where we should set our sights in terms of migration.
> >
> > Thanks,
> > Zach
> >
> > On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> >
> > > What are our expectations regards compatibility between hbase1 and
> > hbase2?
> > >
> > > Lets have a chat about it. Here are some goal posts.
> > >
> > > + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
> No
> > > migration from < hbase-1 (Is this too onerous? Should we support 0.98
> =>
> > > 2.0?).
> > > + You do NOT have to upgrade to the latest release of hbase1 to
migrate
> > to
> > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > + You'll have to update your hbase1 coprocessors to deploy them on
> > hbase2.
> > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> > > watching for region split on RegionServer no longer makes sense given
> > > Master runs all splits now.
> > > + An hbase1 client can run against an hbase2 cluster but it will only
> be
> > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> > admin
> > > ops using an hbase1 Admin client against an hbase2 cluster. We have
> some
> > > egregious API violations in branch-1; e.g. we have protobuf in our API
> > (See
> > > HBASE-15607). The notion is that we can't afford a deprecation cycle
> > > purging this stuff from our Admin API.
> > >
> > > What you all think?
> > >
> > > St.Ack
> > >
> >
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Zach York <zy...@gmail.com>.
This kinda helps, but these seem more like expectations. I was going more
for things like HFile format changed, meta table structure changed,
coprocessor implementations changed (these are just examples, I don't know
if any of these actually changed).
More technical differences between branch-1 and branch-2 which then can
help us get the right expectations for compatibility.
On Wed, Aug 2, 2017 at 6:34 PM, Stack <st...@duboce.net> wrote:
> On Wed, Aug 2, 2017 at 5:25 PM, Zach York <zy...@gmail.com>
> wrote:
>
> > Do we know what the major pain points for migration are? Can we discuss
> > that/get a list going?
> >
> >
> Here's a few in outline:
>
> + There is issue of formats, of hbase-2.x being able to read hbase-1.x data
> whether from HDFS or ZooKeeper or off the wire.
> + An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
> cluster; no holes in the API or unintelligible serializations.
> + There is then the little dance that has us rolling restart from an
> hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it will
> assign regions to the new hbase-2.x regionservers as they come on line.
> TBD.
>
> Is this what you mean sir?
>
> S
>
>
> > I think without that knowledge it is hard (for me at least :) ) to
> > determine where we should set our sights in terms of migration.
> >
> > Thanks,
> > Zach
> >
> > On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> >
> > > What are our expectations regards compatibility between hbase1 and
> > hbase2?
> > >
> > > Lets have a chat about it. Here are some goal posts.
> > >
> > > + You have to upgrade to hbase-1.x before you can migrate to hbase-2.
> No
> > > migration from < hbase-1 (Is this too onerous? Should we support 0.98
> =>
> > > 2.0?).
> > > + You do NOT have to upgrade to the latest release of hbase1 to migrate
> > to
> > > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > > + You'll have to update your hbase1 coprocessors to deploy them on
> > hbase2.
> > > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> > > watching for region split on RegionServer no longer makes sense given
> > > Master runs all splits now.
> > > + An hbase1 client can run against an hbase2 cluster but it will only
> be
> > > able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> > admin
> > > ops using an hbase1 Admin client against an hbase2 cluster. We have
> some
> > > egregious API violations in branch-1; e.g. we have protobuf in our API
> > (See
> > > HBASE-15607). The notion is that we can't afford a deprecation cycle
> > > purging this stuff from our Admin API.
> > >
> > > What you all think?
> > >
> > > St.Ack
> > >
> >
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Stack <st...@duboce.net>.
On Wed, Aug 2, 2017 at 5:25 PM, Zach York <zy...@gmail.com>
wrote:
> Do we know what the major pain points for migration are? Can we discuss
> that/get a list going?
>
>
Here's a few in outline:
+ There is issue of formats, of hbase-2.x being able to read hbase-1.x data
whether from HDFS or ZooKeeper or off the wire.
+ An hbase-1.x client should be able to Get/Put and Scan an hbase-2.x
cluster; no holes in the API or unintelligible serializations.
+ There is then the little dance that has us rolling restart from an
hbase-1.x cluster to hbase-2.x; i.e. upgrade master first and then it will
assign regions to the new hbase-2.x regionservers as they come on line. TBD.
Is this what you mean sir?
S
> I think without that knowledge it is hard (for me at least :) ) to
> determine where we should set our sights in terms of migration.
>
> Thanks,
> Zach
>
> On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
>
> > What are our expectations regards compatibility between hbase1 and
> hbase2?
> >
> > Lets have a chat about it. Here are some goal posts.
> >
> > + You have to upgrade to hbase-1.x before you can migrate to hbase-2. No
> > migration from < hbase-1 (Is this too onerous? Should we support 0.98 =>
> > 2.0?).
> > + You do NOT have to upgrade to the latest release of hbase1 to migrate
> to
> > hbase2; being up on hbase-1.0.0+ will be sufficient.
> > + You'll have to update your hbase1 coprocessors to deploy them on
> hbase2.
> > A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> > watching for region split on RegionServer no longer makes sense given
> > Master runs all splits now.
> > + An hbase1 client can run against an hbase2 cluster but it will only be
> > able to do DML (Get/Put/Scan, etc.). We do not allow being able to do
> admin
> > ops using an hbase1 Admin client against an hbase2 cluster. We have some
> > egregious API violations in branch-1; e.g. we have protobuf in our API
> (See
> > HBASE-15607). The notion is that we can't afford a deprecation cycle
> > purging this stuff from our Admin API.
> >
> > What you all think?
> >
> > St.Ack
> >
>
Re: [DISCUSS] hbase-2.0.0 compatibility expectations
Posted by Zach York <zy...@gmail.com>.
Do we know what the major pain points for migration are? Can we discuss
that/get a list going?
I think without that knowledge it is hard (for me at least :) ) to
determine where we should set our sights in terms of migration.
Thanks,
Zach
On Wed, Aug 2, 2017 at 4:38 PM, Stack <st...@duboce.net> wrote:
> What are our expectations regards compatibility between hbase1 and hbase2?
>
> Lets have a chat about it. Here are some goal posts.
>
> + You have to upgrade to hbase-1.x before you can migrate to hbase-2. No
> migration from < hbase-1 (Is this too onerous? Should we support 0.98 =>
> 2.0?).
> + You do NOT have to upgrade to the latest release of hbase1 to migrate to
> hbase2; being up on hbase-1.0.0+ will be sufficient.
> + You'll have to update your hbase1 coprocessors to deploy them on hbase2.
> A bunch of CP API has/will change by the time hbase2 comes out; e.g.
> watching for region split on RegionServer no longer makes sense given
> Master runs all splits now.
> + An hbase1 client can run against an hbase2 cluster but it will only be
> able to do DML (Get/Put/Scan, etc.). We do not allow being able to do admin
> ops using an hbase1 Admin client against an hbase2 cluster. We have some
> egregious API violations in branch-1; e.g. we have protobuf in our API (See
> HBASE-15607). The notion is that we can't afford a deprecation cycle
> purging this stuff from our Admin API.
>
> What you all think?
>
> St.Ack
>