You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Jonathan Hsieh <jo...@cloudera.com> on 2014/09/10 14:10:12 UTC
Re: thinking about supporting upgrades to HBase 1.x and 2.x
We can go down the 1.x path to 1.1, 1.9, all the way to 1.42.x and I'll
agree with you.
But some of the things being proposed could change not just the rpcs (which
the protobufs would help with) but the communications patterns as well
(which protobuf cannot help with).
Jon.
On Fri, Aug 29, 2014 at 10:58 PM, Suraj Varma <sv...@gmail.com> wrote:
> Wasn't all the effort to go to end to end "protobuf messaging" meant to
> support rolling upgrades across major versions?
>
> Perhaps I may be missing the point, but for us post-Singularity release,
> the assumption was that all upgrades, major & minor, could be done
> "rolling" as proto bufs would ensure backward compatibility.
>
> This was a pretty important feature to allow us to upgrade live clusters
> without down times.
> --Suraj
>
>
>
> On Fri, Aug 29, 2014 at 3:16 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> > I don't think we can or should guarantee a clean *rolling* upgrade from
> > hbase 1.0 to 2.0. However, we absolutely should have a shutdown restart
> > 1.0 -> 2.0 upgrade.
> >
> > The whole point of a major version is to allow for api and compat
> breaking.
> >
> > There are a lot of things in flight that will likely make rolling upgrade
> > hard to do -- for example removing zk and some of the proposals for
> > consensus protocols that are trying to get in to 2.0 won't be compatible
> > with older clients. Also, the deployment will likely be different due to
> > the combined master/meta options and some of the proposals for having
> meta
> > splitting/sharded again will break a 1.0 client.
> >
> > Jon.
> >
> >
> > On Fri, Aug 29, 2014 at 2:35 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > bq. 1.0 to 2.0 we need to sure
> > >
> > > +1 on supporting rolling upgrade from 1.0 to 2.0
> > >
> > > Cheers
> > >
> > >
> > > On Fri, Aug 29, 2014 at 11:24 AM, Jean-Marc Spaggiari <
> > > jean-marc@spaggiari.org> wrote:
> > >
> > > > My opinion:
> > > >
> > > > If we have support for 0.98 to 1.00, support from 0.94 to 1.00 might
> be
> > > > pretty the same thing.
> > > >
> > > > After that, 2,0 might be far in the future. So 0.94 to 2.0 direct I'm
> > not
> > > > sure it's required. 1.0 to 2.0 we need to sure. People looking to
> > upgrade
> > > > from 0.94 to 2.0 might have to go to 1.0 first? I don't think it will
> > be
> > > a
> > > > big constraint. But still, it all depends on the effort of work
> > required
> > > to
> > > > implement upgrade from 0.94 to 2.0. If it's simple, let's to it.
> Else,
> > > > let's ask people to migrate to 1.0 first.
> > > >
> > > > Just my 2 ¢ ;)
> > > >
> > > >
> > > > 2014-08-29 14:19 GMT-04:00 Esteban Gutierrez <es...@cloudera.com>:
> > > >
> > > > > Hello all,
> > > > >
> > > > > Per suggestion of Sean in HBASE-11860 I'm sending this to the list
> to
> > > > > discuss this idea:
> > > > >
> > > > > I've been thinking that we should support upgrades from HBase
> > clusters
> > > > > running 0.94 to HBase 1.x initially. Do you guys concur that we
> > should
> > > > > support that upgrade path to HBase 1.x and depending the adoption
> of
> > > 1.x
> > > > > consider to extend or deprecate the same functionality in HBase
> 2.x?
> > > > >
> > > > > regards,
> > > > > esteban.
> > > > > --
> > > > > Cloudera, Inc.
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // jon@cloudera.com // @jmhsieh
> >
>
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: thinking about supporting upgrades to HBase 1.x and 2.x
Posted by Stack <st...@duboce.net>.
On Tue, Dec 2, 2014 at 2:52 PM, Sean Busbey <bu...@cloudera.com> wrote:
> Resurrecting this thread to confirm our plans around upgrades to 2.0.
>
> Early on there was mention of requiring an upgrade to 1.x (from prior to
> 1.0 release) before users can upgrade to 2.0.
>
> At the moment I'm looking at fixing a problem in the NamespaceUpgrade code
> that is only in 2.0. Since the code isn't actually needed for 1.0+
> upgrades, I'd prefer to just rip it out.
>
> I didn't see any opposition to this idea, but I just want to confirm. To be
> clear, the actual tooling to use this upgrade code has already been
> removed, so it would take work to get the upgrade path going again.
>
Just to say that the first sentence in our upgrading chapter is: "You
cannot skip major versions upgrading." [1]
St.Ack
1. http://hbase.apache.org/book.html#upgrading
Re: thinking about supporting upgrades to HBase 1.x and 2.x
Posted by Sean Busbey <bu...@cloudera.com>.
Resurrecting this thread to confirm our plans around upgrades to 2.0.
Early on there was mention of requiring an upgrade to 1.x (from prior to
1.0 release) before users can upgrade to 2.0.
At the moment I'm looking at fixing a problem in the NamespaceUpgrade code
that is only in 2.0. Since the code isn't actually needed for 1.0+
upgrades, I'd prefer to just rip it out.
I didn't see any opposition to this idea, but I just want to confirm. To be
clear, the actual tooling to use this upgrade code has already been
removed, so it would take work to get the upgrade path going again.
On Wed, Sep 10, 2014 at 4:17 PM, Andrew Purtell <ap...@apache.org> wrote:
> 0.98 was wire compatible with 0.96.
>
> I think we will can manage 1.0 as wire compatible with 0.98 as long as
> some new features remain off until the fleet is uniformly at 1.0.
>
> For 2.0 there are architectural changes being considered which make
> wire compatibility moot, I think this is what people are mostly
> talking about.
>
> On Wed, Sep 10, 2014 at 5:25 AM, Kevin O'dell <ke...@cloudera.com>
> wrote:
> > I believe that Jon has it right. Previously we had been treating 9x as
> > major upgrades and breaking compatibility between 90 -> 92 for example(I
> > know it is old). Going forward from a supportability standpoint we
> should
> > guarantee rolling upgrades to 1.0, 1.1, 1.2...1.9. Major version
> upgrades
> > should always reserve the right to break compatibility, this will give us
> > the ability to make more drastic HFile version changes, RPC changes, etc.
> > That is not to say we should make all major version upgrades painful,
> but
> > the guarantee should not be mandatory.
> >
> >
> > On Wed, Sep 10, 2014 at 8:10 AM, Jonathan Hsieh <jo...@cloudera.com>
> wrote:
> >
> >> We can go down the 1.x path to 1.1, 1.9, all the way to 1.42.x and I'll
> >> agree with you.
> >>
> >> But some of the things being proposed could change not just the rpcs
> (which
> >> the protobufs would help with) but the communications patterns as well
> >> (which protobuf cannot help with).
> >>
> >> Jon.
> >>
> >> On Fri, Aug 29, 2014 at 10:58 PM, Suraj Varma <sv...@gmail.com>
> wrote:
> >>
> >> > Wasn't all the effort to go to end to end "protobuf messaging" meant
> to
> >> > support rolling upgrades across major versions?
> >> >
> >> > Perhaps I may be missing the point, but for us post-Singularity
> release,
> >> > the assumption was that all upgrades, major & minor, could be done
> >> > "rolling" as proto bufs would ensure backward compatibility.
> >> >
> >> > This was a pretty important feature to allow us to upgrade live
> clusters
> >> > without down times.
> >> > --Suraj
> >> >
> >> >
> >> >
> >> > On Fri, Aug 29, 2014 at 3:16 PM, Jonathan Hsieh <jo...@cloudera.com>
> >> wrote:
> >> >
> >> > > I don't think we can or should guarantee a clean *rolling* upgrade
> from
> >> > > hbase 1.0 to 2.0. However, we absolutely should have a shutdown
> >> restart
> >> > > 1.0 -> 2.0 upgrade.
> >> > >
> >> > > The whole point of a major version is to allow for api and compat
> >> > breaking.
> >> > >
> >> > > There are a lot of things in flight that will likely make rolling
> >> upgrade
> >> > > hard to do -- for example removing zk and some of the proposals for
> >> > > consensus protocols that are trying to get in to 2.0 won't be
> >> compatible
> >> > > with older clients. Also, the deployment will likely be different
> due
> >> to
> >> > > the combined master/meta options and some of the proposals for
> having
> >> > meta
> >> > > splitting/sharded again will break a 1.0 client.
> >> > >
> >> > > Jon.
> >> > >
> >> > >
> >> > > On Fri, Aug 29, 2014 at 2:35 PM, Ted Yu <yu...@gmail.com>
> wrote:
> >> > >
> >> > > > bq. 1.0 to 2.0 we need to sure
> >> > > >
> >> > > > +1 on supporting rolling upgrade from 1.0 to 2.0
> >> > > >
> >> > > > Cheers
> >> > > >
> >> > > >
> >> > > > On Fri, Aug 29, 2014 at 11:24 AM, Jean-Marc Spaggiari <
> >> > > > jean-marc@spaggiari.org> wrote:
> >> > > >
> >> > > > > My opinion:
> >> > > > >
> >> > > > > If we have support for 0.98 to 1.00, support from 0.94 to 1.00
> >> might
> >> > be
> >> > > > > pretty the same thing.
> >> > > > >
> >> > > > > After that, 2,0 might be far in the future. So 0.94 to 2.0
> direct
> >> I'm
> >> > > not
> >> > > > > sure it's required. 1.0 to 2.0 we need to sure. People looking
> to
> >> > > upgrade
> >> > > > > from 0.94 to 2.0 might have to go to 1.0 first? I don't think it
> >> will
> >> > > be
> >> > > > a
> >> > > > > big constraint. But still, it all depends on the effort of work
> >> > > required
> >> > > > to
> >> > > > > implement upgrade from 0.94 to 2.0. If it's simple, let's to it.
> >> > Else,
> >> > > > > let's ask people to migrate to 1.0 first.
> >> > > > >
> >> > > > > Just my 2 ¢ ;)
> >> > > > >
> >> > > > >
> >> > > > > 2014-08-29 14:19 GMT-04:00 Esteban Gutierrez <
> esteban@cloudera.com
> >> >:
> >> > > > >
> >> > > > > > Hello all,
> >> > > > > >
> >> > > > > > Per suggestion of Sean in HBASE-11860 I'm sending this to the
> >> list
> >> > to
> >> > > > > > discuss this idea:
> >> > > > > >
> >> > > > > > I've been thinking that we should support upgrades from HBase
> >> > > clusters
> >> > > > > > running 0.94 to HBase 1.x initially. Do you guys concur that
> we
> >> > > should
> >> > > > > > support that upgrade path to HBase 1.x and depending the
> adoption
> >> > of
> >> > > > 1.x
> >> > > > > > consider to extend or deprecate the same functionality in
> HBase
> >> > 2.x?
> >> > > > > >
> >> > > > > > regards,
> >> > > > > > esteban.
> >> > > > > > --
> >> > > > > > Cloudera, Inc.
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > // Jonathan Hsieh (shay)
> >> > > // HBase Tech Lead, Software Engineer, Cloudera
> >> > > // jon@cloudera.com // @jmhsieh
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> // Jonathan Hsieh (shay)
> >> // HBase Tech Lead, Software Engineer, Cloudera
> >> // jon@cloudera.com // @jmhsieh
> >>
> >
> >
> >
> > --
> > Kevin O'Dell
> > Systems Engineer, Cloudera
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein (via Tom White)
>
--
Sean
Re: thinking about supporting upgrades to HBase 1.x and 2.x
Posted by Andrew Purtell <ap...@apache.org>.
0.98 was wire compatible with 0.96.
I think we will can manage 1.0 as wire compatible with 0.98 as long as
some new features remain off until the fleet is uniformly at 1.0.
For 2.0 there are architectural changes being considered which make
wire compatibility moot, I think this is what people are mostly
talking about.
On Wed, Sep 10, 2014 at 5:25 AM, Kevin O'dell <ke...@cloudera.com> wrote:
> I believe that Jon has it right. Previously we had been treating 9x as
> major upgrades and breaking compatibility between 90 -> 92 for example(I
> know it is old). Going forward from a supportability standpoint we should
> guarantee rolling upgrades to 1.0, 1.1, 1.2...1.9. Major version upgrades
> should always reserve the right to break compatibility, this will give us
> the ability to make more drastic HFile version changes, RPC changes, etc.
> That is not to say we should make all major version upgrades painful, but
> the guarantee should not be mandatory.
>
>
> On Wed, Sep 10, 2014 at 8:10 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
>> We can go down the 1.x path to 1.1, 1.9, all the way to 1.42.x and I'll
>> agree with you.
>>
>> But some of the things being proposed could change not just the rpcs (which
>> the protobufs would help with) but the communications patterns as well
>> (which protobuf cannot help with).
>>
>> Jon.
>>
>> On Fri, Aug 29, 2014 at 10:58 PM, Suraj Varma <sv...@gmail.com> wrote:
>>
>> > Wasn't all the effort to go to end to end "protobuf messaging" meant to
>> > support rolling upgrades across major versions?
>> >
>> > Perhaps I may be missing the point, but for us post-Singularity release,
>> > the assumption was that all upgrades, major & minor, could be done
>> > "rolling" as proto bufs would ensure backward compatibility.
>> >
>> > This was a pretty important feature to allow us to upgrade live clusters
>> > without down times.
>> > --Suraj
>> >
>> >
>> >
>> > On Fri, Aug 29, 2014 at 3:16 PM, Jonathan Hsieh <jo...@cloudera.com>
>> wrote:
>> >
>> > > I don't think we can or should guarantee a clean *rolling* upgrade from
>> > > hbase 1.0 to 2.0. However, we absolutely should have a shutdown
>> restart
>> > > 1.0 -> 2.0 upgrade.
>> > >
>> > > The whole point of a major version is to allow for api and compat
>> > breaking.
>> > >
>> > > There are a lot of things in flight that will likely make rolling
>> upgrade
>> > > hard to do -- for example removing zk and some of the proposals for
>> > > consensus protocols that are trying to get in to 2.0 won't be
>> compatible
>> > > with older clients. Also, the deployment will likely be different due
>> to
>> > > the combined master/meta options and some of the proposals for having
>> > meta
>> > > splitting/sharded again will break a 1.0 client.
>> > >
>> > > Jon.
>> > >
>> > >
>> > > On Fri, Aug 29, 2014 at 2:35 PM, Ted Yu <yu...@gmail.com> wrote:
>> > >
>> > > > bq. 1.0 to 2.0 we need to sure
>> > > >
>> > > > +1 on supporting rolling upgrade from 1.0 to 2.0
>> > > >
>> > > > Cheers
>> > > >
>> > > >
>> > > > On Fri, Aug 29, 2014 at 11:24 AM, Jean-Marc Spaggiari <
>> > > > jean-marc@spaggiari.org> wrote:
>> > > >
>> > > > > My opinion:
>> > > > >
>> > > > > If we have support for 0.98 to 1.00, support from 0.94 to 1.00
>> might
>> > be
>> > > > > pretty the same thing.
>> > > > >
>> > > > > After that, 2,0 might be far in the future. So 0.94 to 2.0 direct
>> I'm
>> > > not
>> > > > > sure it's required. 1.0 to 2.0 we need to sure. People looking to
>> > > upgrade
>> > > > > from 0.94 to 2.0 might have to go to 1.0 first? I don't think it
>> will
>> > > be
>> > > > a
>> > > > > big constraint. But still, it all depends on the effort of work
>> > > required
>> > > > to
>> > > > > implement upgrade from 0.94 to 2.0. If it's simple, let's to it.
>> > Else,
>> > > > > let's ask people to migrate to 1.0 first.
>> > > > >
>> > > > > Just my 2 ¢ ;)
>> > > > >
>> > > > >
>> > > > > 2014-08-29 14:19 GMT-04:00 Esteban Gutierrez <esteban@cloudera.com
>> >:
>> > > > >
>> > > > > > Hello all,
>> > > > > >
>> > > > > > Per suggestion of Sean in HBASE-11860 I'm sending this to the
>> list
>> > to
>> > > > > > discuss this idea:
>> > > > > >
>> > > > > > I've been thinking that we should support upgrades from HBase
>> > > clusters
>> > > > > > running 0.94 to HBase 1.x initially. Do you guys concur that we
>> > > should
>> > > > > > support that upgrade path to HBase 1.x and depending the adoption
>> > of
>> > > > 1.x
>> > > > > > consider to extend or deprecate the same functionality in HBase
>> > 2.x?
>> > > > > >
>> > > > > > regards,
>> > > > > > esteban.
>> > > > > > --
>> > > > > > Cloudera, Inc.
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > // Jonathan Hsieh (shay)
>> > > // HBase Tech Lead, Software Engineer, Cloudera
>> > > // jon@cloudera.com // @jmhsieh
>> > >
>> >
>>
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // HBase Tech Lead, Software Engineer, Cloudera
>> // jon@cloudera.com // @jmhsieh
>>
>
>
>
> --
> Kevin O'Dell
> Systems Engineer, Cloudera
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
Re: thinking about supporting upgrades to HBase 1.x and 2.x
Posted by Kevin O'dell <ke...@cloudera.com>.
I believe that Jon has it right. Previously we had been treating 9x as
major upgrades and breaking compatibility between 90 -> 92 for example(I
know it is old). Going forward from a supportability standpoint we should
guarantee rolling upgrades to 1.0, 1.1, 1.2...1.9. Major version upgrades
should always reserve the right to break compatibility, this will give us
the ability to make more drastic HFile version changes, RPC changes, etc.
That is not to say we should make all major version upgrades painful, but
the guarantee should not be mandatory.
On Wed, Sep 10, 2014 at 8:10 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> We can go down the 1.x path to 1.1, 1.9, all the way to 1.42.x and I'll
> agree with you.
>
> But some of the things being proposed could change not just the rpcs (which
> the protobufs would help with) but the communications patterns as well
> (which protobuf cannot help with).
>
> Jon.
>
> On Fri, Aug 29, 2014 at 10:58 PM, Suraj Varma <sv...@gmail.com> wrote:
>
> > Wasn't all the effort to go to end to end "protobuf messaging" meant to
> > support rolling upgrades across major versions?
> >
> > Perhaps I may be missing the point, but for us post-Singularity release,
> > the assumption was that all upgrades, major & minor, could be done
> > "rolling" as proto bufs would ensure backward compatibility.
> >
> > This was a pretty important feature to allow us to upgrade live clusters
> > without down times.
> > --Suraj
> >
> >
> >
> > On Fri, Aug 29, 2014 at 3:16 PM, Jonathan Hsieh <jo...@cloudera.com>
> wrote:
> >
> > > I don't think we can or should guarantee a clean *rolling* upgrade from
> > > hbase 1.0 to 2.0. However, we absolutely should have a shutdown
> restart
> > > 1.0 -> 2.0 upgrade.
> > >
> > > The whole point of a major version is to allow for api and compat
> > breaking.
> > >
> > > There are a lot of things in flight that will likely make rolling
> upgrade
> > > hard to do -- for example removing zk and some of the proposals for
> > > consensus protocols that are trying to get in to 2.0 won't be
> compatible
> > > with older clients. Also, the deployment will likely be different due
> to
> > > the combined master/meta options and some of the proposals for having
> > meta
> > > splitting/sharded again will break a 1.0 client.
> > >
> > > Jon.
> > >
> > >
> > > On Fri, Aug 29, 2014 at 2:35 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > bq. 1.0 to 2.0 we need to sure
> > > >
> > > > +1 on supporting rolling upgrade from 1.0 to 2.0
> > > >
> > > > Cheers
> > > >
> > > >
> > > > On Fri, Aug 29, 2014 at 11:24 AM, Jean-Marc Spaggiari <
> > > > jean-marc@spaggiari.org> wrote:
> > > >
> > > > > My opinion:
> > > > >
> > > > > If we have support for 0.98 to 1.00, support from 0.94 to 1.00
> might
> > be
> > > > > pretty the same thing.
> > > > >
> > > > > After that, 2,0 might be far in the future. So 0.94 to 2.0 direct
> I'm
> > > not
> > > > > sure it's required. 1.0 to 2.0 we need to sure. People looking to
> > > upgrade
> > > > > from 0.94 to 2.0 might have to go to 1.0 first? I don't think it
> will
> > > be
> > > > a
> > > > > big constraint. But still, it all depends on the effort of work
> > > required
> > > > to
> > > > > implement upgrade from 0.94 to 2.0. If it's simple, let's to it.
> > Else,
> > > > > let's ask people to migrate to 1.0 first.
> > > > >
> > > > > Just my 2 ¢ ;)
> > > > >
> > > > >
> > > > > 2014-08-29 14:19 GMT-04:00 Esteban Gutierrez <esteban@cloudera.com
> >:
> > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > Per suggestion of Sean in HBASE-11860 I'm sending this to the
> list
> > to
> > > > > > discuss this idea:
> > > > > >
> > > > > > I've been thinking that we should support upgrades from HBase
> > > clusters
> > > > > > running 0.94 to HBase 1.x initially. Do you guys concur that we
> > > should
> > > > > > support that upgrade path to HBase 1.x and depending the adoption
> > of
> > > > 1.x
> > > > > > consider to extend or deprecate the same functionality in HBase
> > 2.x?
> > > > > >
> > > > > > regards,
> > > > > > esteban.
> > > > > > --
> > > > > > Cloudera, Inc.
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > // Jonathan Hsieh (shay)
> > > // HBase Tech Lead, Software Engineer, Cloudera
> > > // jon@cloudera.com // @jmhsieh
> > >
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>
--
Kevin O'Dell
Systems Engineer, Cloudera