You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Stack <st...@duboce.net> on 2012/01/26 05:34:30 UTC
hbase 0.94.0
Lets branch end of february? No new features thereafter. Is this too
close in? Would be grand if 0.94.0 shipped before hbasecon. What
should 0.94.0 have in it? I don't mind if the list is short.
Unless someone else wants too, I don't mind being release manager
(will try to run a tighter ship this time around).
St.Ack
Re: hbase 0.94.0
Posted by Stack <st...@duboce.net>.
On Wed, Jan 25, 2012 at 9:48 PM, lars hofhansl <lh...@yahoo.com> wrote:
> ps. I'm happy to be the release manager if you're tired of it. :)
>
If you twist my arm like that .... (smile). Probably good to spread
the 'knowledge' (Ram is coming up to speed over on 0.90.6). You
probably have more of an iron fist than I too better able to fend off
committers (like me) trying to add the gold plating to the handles on
the kitchen sink.
St.Ack
Re: hbase 0.94.0
Posted by Ted Yu <yu...@gmail.com>.
+1 on 0.94 release.
+1 on Lars being the release manager.
Cheers
On Thu, Jan 26, 2012 at 6:36 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> +1. I like the list and the simple performance version theme, and seeing
> what the "Iron Fist of Lars H" is like. :)
>
> Can we start talking about goals for 0.96 goal as well? Wire compatibility
> is one that Jimmy and I are interested in tackling, or possibly adding it
> as an optional feature in 0.94 to ease the transition.
>
> Jon.
>
> On Wed, Jan 25, 2012 at 9:48 PM, lars hofhansl <lh...@yahoo.com>
> wrote:
>
> > +1! end of february sounds fine to me
> >
> >
> > My feature wish list:
> >
> > HBASE-2600
> > HBASE-4218 (done it seems)
> > HBASE-4608 (also getting close)
> >
> > HBASE-5128
> >
> > The usual set of race-condition-fixes...
> >
> >
> > I don't think we'll have tackled wire-compatibility by then.
> >
> >
> > We can call it the 0.94 performance release...
> >
> >
> > -- Lars
> >
> > ps. I'm happy to be the release manager if you're tired of it. :)
> >
> >
> >
> > ________________________________
> > From: Stack <st...@duboce.net>
> > To: HBase Dev List <de...@hbase.apache.org>
> > Sent: Wednesday, January 25, 2012 8:34 PM
> > Subject: hbase 0.94.0
> >
> > Lets branch end of february? No new features thereafter. Is this too
> > close in? Would be grand if 0.94.0 shipped before hbasecon. What
> > should 0.94.0 have in it? I don't mind if the list is short.
> >
> > Unless someone else wants too, I don't mind being release manager
> > (will try to run a tighter ship this time around).
> >
> > St.Ack
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
Re: hbase 0.94.0
Posted by lars hofhansl <lh...@yahoo.com>.
I think one of the appeals of HBase is that is chooses consistency over availability, which makes it easy to reason about the state of your data.
Throwing an eventually consistent index into the mix makes that reasoning more difficult again.
Every time I think about indexes in HBase I arrive at one of four scenarios:
1. Just handle it client side and write to two tables. Other clients maybe see the updates not atomically. Maybe combined with a WAL.
2. Have HBase support globally consistent indexes. Needs some form of minimal global transaction support.
3. Have HBase support local consistent indexes. I.e. keep the index with the region it indexes. Since the index cannot be addressed directly, a query would need to farm out to all regions (or at the very least regionservers).
4. Have an eventually consistent index with some form of WAL that is asynchronously applied. (what you described)
Granted I have not spent much time thinking about this, but none of these are really entirely appealing.
Found HBASE-2038 (#3) and HBASE-3340 (#4) (but you've seen those) :)
Alex Baranau and I were supposed to talk a while back, but I dropped the ball on that.
-- Lars
________________________________
From: Andrew Purtell <ap...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <lh...@yahoo.com>
Sent: Thursday, January 26, 2012 11:27 AM
Subject: Re: hbase 0.94.0
> Dare I mention secondary indexes for 0.96?
You can. :-) But what does that mean? Is there a specific JIRA that describes the scope of it?
I've been around the block a bit with this. Jon Gray and I talked about it last year, at the time the notion was "eventually consistent secondary indexing via WAL overloading". (About that fuzzy, alas.) I asked a Trend team to work on it. Unfortunately the result was not of satisfactory quality. Then I thought about doing the work myself. However if we are already "eventually consistent" then why not use something like ElasticSearch as an indexing service that can be shared between HBase based applications and others that have nothing to do with HBase? Something akin to what the Lily guys did but without the serialization. Anyway, that is my current thinking. If there were a native secondary indexing facility in HBase, we'd definitely use it, but if there isn't, that is ok too.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
> From: lars hofhansl <lh...@yahoo.com>
>To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>Sent: Thursday, January 26, 2012 9:10 AM
>Subject: Re: hbase 0.94.0
>
>"Iron Fist of Lars H"... Uh oh :)
>
>0.96 wire compatibility would be awesome. Seems rather ambitious for 0.94. I am happy to help with that too.
>Weren't the HortonWorks guys also interested in working on that?
>Once we have that, we can think about calling it 1.0.
>
>Dare I mention secondary indexes for 0.96?
>
>
>-- Lars
>
>________________________________
>From: Jonathan Hsieh <jo...@cloudera.com>
>To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
>Sent: Thursday, January 26, 2012 6:36 AM
>Subject: Re: hbase 0.94.0
>
>+1. I like the list and the simple performance version theme, and seeing
>what the "Iron Fist of Lars H" is like. :)
>
>Can we start talking about goals for 0.96 goal as well? Wire compatibility
>is one that Jimmy and I are interested in tackling, or possibly adding it
>as an optional feature in 0.94 to ease the transition.
>
>Jon.
>
>On Wed, Jan 25, 2012 at 9:48 PM, lars hofhansl <lh...@yahoo.com> wrote:
>
>> +1! end of february sounds fine to me
>>
>>
>> My feature wish list:
>>
>> HBASE-2600
>> HBASE-4218 (done it seems)
>> HBASE-4608 (also getting close)
>>
>> HBASE-5128
>>
>> The usual set of race-condition-fixes...
>>
>>
>> I don't think we'll have tackled wire-compatibility by then.
>>
>>
>> We can call it the 0.94 performance release...
>>
>>
>> -- Lars
>>
>> ps. I'm happy to be the release manager if you're tired of it. :)
>>
>>
>>
>> ________________________________
>> From: Stack <st...@duboce.net>
>> To: HBase Dev List <de...@hbase.apache.org>
>> Sent: Wednesday, January 25, 2012 8:34 PM
>> Subject: hbase 0.94.0
>>
>> Lets branch end of february? No new features thereafter. Is this too
>> close in? Would be grand if 0.94.0 shipped before hbasecon. What
>> should 0.94.0 have in it? I don't mind if the list is short.
>>
>> Unless someone else wants too, I don't mind being release manager
>> (will try to run a tighter ship this time around).
>>
>> St.Ack
>>
>
>
>
>--
>// Jonathan Hsieh (shay)
>// Software Engineer, Cloudera
>// jon@cloudera.com
>
>
>
Re: hbase 0.94.0
Posted by Andrew Purtell <ap...@apache.org>.
> Dare I mention secondary indexes for 0.96?
You can. :-) But what does that mean? Is there a specific JIRA that describes the scope of it?
I've been around the block a bit with this. Jon Gray and I talked about it last year, at the time the notion was "eventually consistent secondary indexing via WAL overloading". (About that fuzzy, alas.) I asked a Trend team to work on it. Unfortunately the result was not of satisfactory quality. Then I thought about doing the work myself. However if we are already "eventually consistent" then why not use something like ElasticSearch as an indexing service that can be shared between HBase based applications and others that have nothing to do with HBase? Something akin to what the Lily guys did but without the serialization. Anyway, that is my current thinking. If there were a native secondary indexing facility in HBase, we'd definitely use it, but if there isn't, that is ok too.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
> From: lars hofhansl <lh...@yahoo.com>
>To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>Sent: Thursday, January 26, 2012 9:10 AM
>Subject: Re: hbase 0.94.0
>
>"Iron Fist of Lars H"... Uh oh :)
>
>0.96 wire compatibility would be awesome. Seems rather ambitious for 0.94. I am happy to help with that too.
>Weren't the HortonWorks guys also interested in working on that?
>Once we have that, we can think about calling it 1.0.
>
>Dare I mention secondary indexes for 0.96?
>
>
>-- Lars
>
>________________________________
>From: Jonathan Hsieh <jo...@cloudera.com>
>To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
>Sent: Thursday, January 26, 2012 6:36 AM
>Subject: Re: hbase 0.94.0
>
>+1. I like the list and the simple performance version theme, and seeing
>what the "Iron Fist of Lars H" is like. :)
>
>Can we start talking about goals for 0.96 goal as well? Wire compatibility
>is one that Jimmy and I are interested in tackling, or possibly adding it
>as an optional feature in 0.94 to ease the transition.
>
>Jon.
>
>On Wed, Jan 25, 2012 at 9:48 PM, lars hofhansl <lh...@yahoo.com> wrote:
>
>> +1! end of february sounds fine to me
>>
>>
>> My feature wish list:
>>
>> HBASE-2600
>> HBASE-4218 (done it seems)
>> HBASE-4608 (also getting close)
>>
>> HBASE-5128
>>
>> The usual set of race-condition-fixes...
>>
>>
>> I don't think we'll have tackled wire-compatibility by then.
>>
>>
>> We can call it the 0.94 performance release...
>>
>>
>> -- Lars
>>
>> ps. I'm happy to be the release manager if you're tired of it. :)
>>
>>
>>
>> ________________________________
>> From: Stack <st...@duboce.net>
>> To: HBase Dev List <de...@hbase.apache.org>
>> Sent: Wednesday, January 25, 2012 8:34 PM
>> Subject: hbase 0.94.0
>>
>> Lets branch end of february? No new features thereafter. Is this too
>> close in? Would be grand if 0.94.0 shipped before hbasecon. What
>> should 0.94.0 have in it? I don't mind if the list is short.
>>
>> Unless someone else wants too, I don't mind being release manager
>> (will try to run a tighter ship this time around).
>>
>> St.Ack
>>
>
>
>
>--
>// Jonathan Hsieh (shay)
>// Software Engineer, Cloudera
>// jon@cloudera.com
>
>
>
Re: hbase 0.94.0
Posted by lars hofhansl <lh...@yahoo.com>.
"Iron Fist of Lars H"... Uh oh :)
0.96 wire compatibility would be awesome. Seems rather ambitious for 0.94. I am happy to help with that too.
Weren't the HortonWorks guys also interested in working on that?
Once we have that, we can think about calling it 1.0.
Dare I mention secondary indexes for 0.96?
-- Lars
________________________________
From: Jonathan Hsieh <jo...@cloudera.com>
To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
Sent: Thursday, January 26, 2012 6:36 AM
Subject: Re: hbase 0.94.0
+1. I like the list and the simple performance version theme, and seeing
what the "Iron Fist of Lars H" is like. :)
Can we start talking about goals for 0.96 goal as well? Wire compatibility
is one that Jimmy and I are interested in tackling, or possibly adding it
as an optional feature in 0.94 to ease the transition.
Jon.
On Wed, Jan 25, 2012 at 9:48 PM, lars hofhansl <lh...@yahoo.com> wrote:
> +1! end of february sounds fine to me
>
>
> My feature wish list:
>
> HBASE-2600
> HBASE-4218 (done it seems)
> HBASE-4608 (also getting close)
>
> HBASE-5128
>
> The usual set of race-condition-fixes...
>
>
> I don't think we'll have tackled wire-compatibility by then.
>
>
> We can call it the 0.94 performance release...
>
>
> -- Lars
>
> ps. I'm happy to be the release manager if you're tired of it. :)
>
>
>
> ________________________________
> From: Stack <st...@duboce.net>
> To: HBase Dev List <de...@hbase.apache.org>
> Sent: Wednesday, January 25, 2012 8:34 PM
> Subject: hbase 0.94.0
>
> Lets branch end of february? No new features thereafter. Is this too
> close in? Would be grand if 0.94.0 shipped before hbasecon. What
> should 0.94.0 have in it? I don't mind if the list is short.
>
> Unless someone else wants too, I don't mind being release manager
> (will try to run a tighter ship this time around).
>
> St.Ack
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: hbase 0.94.0
Posted by Jonathan Hsieh <jo...@cloudera.com>.
+1. I like the list and the simple performance version theme, and seeing
what the "Iron Fist of Lars H" is like. :)
Can we start talking about goals for 0.96 goal as well? Wire compatibility
is one that Jimmy and I are interested in tackling, or possibly adding it
as an optional feature in 0.94 to ease the transition.
Jon.
On Wed, Jan 25, 2012 at 9:48 PM, lars hofhansl <lh...@yahoo.com> wrote:
> +1! end of february sounds fine to me
>
>
> My feature wish list:
>
> HBASE-2600
> HBASE-4218 (done it seems)
> HBASE-4608 (also getting close)
>
> HBASE-5128
>
> The usual set of race-condition-fixes...
>
>
> I don't think we'll have tackled wire-compatibility by then.
>
>
> We can call it the 0.94 performance release...
>
>
> -- Lars
>
> ps. I'm happy to be the release manager if you're tired of it. :)
>
>
>
> ________________________________
> From: Stack <st...@duboce.net>
> To: HBase Dev List <de...@hbase.apache.org>
> Sent: Wednesday, January 25, 2012 8:34 PM
> Subject: hbase 0.94.0
>
> Lets branch end of february? No new features thereafter. Is this too
> close in? Would be grand if 0.94.0 shipped before hbasecon. What
> should 0.94.0 have in it? I don't mind if the list is short.
>
> Unless someone else wants too, I don't mind being release manager
> (will try to run a tighter ship this time around).
>
> St.Ack
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
RE: hbase 0.94.0
Posted by Ramakrishna s vasudevan <ra...@huawei.com>.
What ever may be the time i can support the release to my fullest :)
Regards
Ram
________________________________________
From: lars hofhansl [lhofhansl@yahoo.com]
Sent: Thursday, January 26, 2012 11:18 AM
To: dev@hbase.apache.org
Subject: Re: hbase 0.94.0
+1! end of february sounds fine to me
My feature wish list:
HBASE-2600
HBASE-4218 (done it seems)
HBASE-4608 (also getting close)
HBASE-5128
The usual set of race-condition-fixes...
I don't think we'll have tackled wire-compatibility by then.
We can call it the 0.94 performance release...
-- Lars
ps. I'm happy to be the release manager if you're tired of it. :)
________________________________
From: Stack <st...@duboce.net>
To: HBase Dev List <de...@hbase.apache.org>
Sent: Wednesday, January 25, 2012 8:34 PM
Subject: hbase 0.94.0
Lets branch end of february? No new features thereafter. Is this too
close in? Would be grand if 0.94.0 shipped before hbasecon. What
should 0.94.0 have in it? I don't mind if the list is short.
Unless someone else wants too, I don't mind being release manager
(will try to run a tighter ship this time around).
St.Ack
Re: hbase 0.94.0
Posted by lars hofhansl <lh...@yahoo.com>.
+1! end of february sounds fine to me
My feature wish list:
HBASE-2600
HBASE-4218 (done it seems)
HBASE-4608 (also getting close)
HBASE-5128
The usual set of race-condition-fixes...
I don't think we'll have tackled wire-compatibility by then.
We can call it the 0.94 performance release...
-- Lars
ps. I'm happy to be the release manager if you're tired of it. :)
________________________________
From: Stack <st...@duboce.net>
To: HBase Dev List <de...@hbase.apache.org>
Sent: Wednesday, January 25, 2012 8:34 PM
Subject: hbase 0.94.0
Lets branch end of february? No new features thereafter. Is this too
close in? Would be grand if 0.94.0 shipped before hbasecon. What
should 0.94.0 have in it? I don't mind if the list is short.
Unless someone else wants too, I don't mind being release manager
(will try to run a tighter ship this time around).
St.Ack
Re: hbase 0.94.0
Posted by lars hofhansl <lh...@yahoo.com>.
Good points.
I'd say we branch as soon as we identify a large feature that should not be in 0.94or by the end of February, whatever comes first.
That way we keep the committers work to a minimum as long as possible. Just my $0.02.
-- Lars
----- Original Message -----
From: Jonathan Hsieh <jo...@cloudera.com>
To: dev@hbase.apache.org
Cc:
Sent: Thursday, January 26, 2012 6:42 AM
Subject: Re: hbase 0.94.0
Can you help me understand the significance cutting the branch a month from
now? As a strawman -- now that 0.92 is out (and ideally just getting fixes
and supportability improvements instead of new features), why don't we cut
a branch even earlier and then just bring in the features we decide get
into 0.94 branch as they land?
Here's a rough pro/con, interested in learning about more opinions:
Pros:
It would allow the release manager to better manage scope and be more
selective on which features make it in sooner.
It would allow other features large features like the beginnings of
wire-compatiblity (maybe one of the major 0.96 goals) to start landing in
trunk sooner as 0.94 branch gets filled.
Cons:
More work for committers when code wants to land in multiple branches.
Possible divergence if the current wish list features take longer to land
than the 0.96 wishlist.
Jon
On Wed, Jan 25, 2012 at 8:34 PM, Stack <st...@duboce.net> wrote:
> Lets branch end of february? No new features thereafter. Is this too
> close in? Would be grand if 0.94.0 shipped before hbasecon. What
> should 0.94.0 have in it? I don't mind if the list is short.
>
> Unless someone else wants too, I don't mind being release manager
> (will try to run a tighter ship this time around).
>
> St.Ack
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: hbase 0.94.0
Posted by Jonathan Hsieh <jo...@cloudera.com>.
Can you help me understand the significance cutting the branch a month from
now? As a strawman -- now that 0.92 is out (and ideally just getting fixes
and supportability improvements instead of new features), why don't we cut
a branch even earlier and then just bring in the features we decide get
into 0.94 branch as they land?
Here's a rough pro/con, interested in learning about more opinions:
Pros:
It would allow the release manager to better manage scope and be more
selective on which features make it in sooner.
It would allow other features large features like the beginnings of
wire-compatiblity (maybe one of the major 0.96 goals) to start landing in
trunk sooner as 0.94 branch gets filled.
Cons:
More work for committers when code wants to land in multiple branches.
Possible divergence if the current wish list features take longer to land
than the 0.96 wishlist.
Jon
On Wed, Jan 25, 2012 at 8:34 PM, Stack <st...@duboce.net> wrote:
> Lets branch end of february? No new features thereafter. Is this too
> close in? Would be grand if 0.94.0 shipped before hbasecon. What
> should 0.94.0 have in it? I don't mind if the list is short.
>
> Unless someone else wants too, I don't mind being release manager
> (will try to run a tighter ship this time around).
>
> St.Ack
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: hbase 0.94.0
Posted by Gaojinchao <ga...@huawei.com>.
It is good news, but it is difficult to choose version. :)
Our current version is 0.90. we will choose our next version and start a large-scale test.
Which version will be HBase 1.0?
-----邮件原件-----
发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
发送时间: 2012年1月26日 12:35
收件人: HBase Dev List
主题: hbase 0.94.0
Lets branch end of february? No new features thereafter. Is this too
close in? Would be grand if 0.94.0 shipped before hbasecon. What
should 0.94.0 have in it? I don't mind if the list is short.
Unless someone else wants too, I don't mind being release manager
(will try to run a tighter ship this time around).
St.Ack
Re: hbase 0.94.0
Posted by Matt Corgan <mc...@hotpads.com>.
Maybe add a script to 0.92 (part of hbck?) that verifies that all of your
HFiles have been upgraded, and then you can trigger major compactions on
any that haven't been.
On a slightly different topic, is there anything in 0.94 that breaks wire
compatibility with 0.92? If not, then it seems like a waste of a rare
opportunity to break wire compatibility!
On Thu, Jan 26, 2012 at 12:25 PM, Jesse Yates <je...@gmail.com>wrote:
> +1 on removing it too, but maybe we should have some sort of upgrade
> script to help make the switch?
>
> I'm thinking something pluggable on both sides of the upgrade, so people
> can go from version X->Y, rather than having to upgrade first to an
> intermediate and then to he version they want (right it would be going from
> 0.20->.92->.96, IMO an excessive PITA).
>
> Just my two cents...
>
> - Jesse Yates
>
> Sent from my iPhone.
>
> On Jan 26, 2012, at 12:16 PM, lars hofhansl <lh...@yahoo.com> wrote:
>
> > Good point.
> > 0.94 is not branched, yet. And I think generally we do not support
> skipping releases for upgrades.
> > +1 on removing HFileV1 cruft for 0.94
> >
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Matt Corgan <mc...@hotpads.com>
> > To: dev@hbase.apache.org; Andrew Purtell <ap...@apache.org>
> > Sent: Thursday, January 26, 2012 11:51 AM
> > Subject: Re: hbase 0.94.0
> >
> > Are there any thoughts about when it is ok to stop being backwards
> > compatible? Mainly thinking of HFileV1... 0.92 will convert all
> HFileV1's
> > to HFileV2's, so it would probably have been ok to delete the code for
> > HFileV1 in 0.94 and force people to upgrade through 0.92. That didn't
> > actually happen, so it's looking like folks will be able to go straight
> > from 0.90 to 0.94. But, perhaps it should be deleted for 0.96?
> >
> >
> > On Thu, Jan 26, 2012 at 11:41 AM, Andrew Purtell <apurtell@apache.org
> >wrote:
> >
> >> Yeah, so
> >>
> >> - Security (basically another coprocessor for inclusion in mainline,
> >> like Constraints)
> >>
> >> if not in 0.92.1.
> >>
> >> Best regards,
> >>
> >> - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
> >>
> >>> ________________________________
> >>> From: Andrew Purtell <ap...@apache.org>
> >>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> >>> Sent: Thursday, January 26, 2012 11:28 AM
> >>> Subject: Re: hbase 0.94.0
> >>>
> >>> A limited set of changes so we can get it out on that kind of
> timeframe.
> >> :-)
> >>>
> >>> - Constraints (is ready to go, is a coprocessor, so is in the large
> >> just a new package to drop in)
> >>>
> >>> - Useful utilities for ops:
> >>>
> >>> - LoadTestTool (if Ted didn't end up backporting this into 0.92)
> >>>
> >>> - The store file locality thing I have mostly done, will finish it
> >>>
> >>> - Mikhail and crew's ongoing optimizations (HBASE-4218, etc.), the
> ones
> >> he considers fully baked
> >>>
> >>> I saw wire compatibility mentioned, for 0.96 but perhaps
> >> optional/transitional code in 0.94 as well. This is something we would
> try
> >> out and beat up upon in staging in earnest.
> >>>
> >>> Best regards,
> >>>
> >>> - Andy
> >>>
> >>> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> >> (via Tom White)
> >>>
> >>>
> >>>
> >>>> ________________________________
> >>>> From: Stack <st...@duboce.net>
> >>>> To: HBase Dev List <de...@hbase.apache.org>
> >>>> Sent: Wednesday, January 25, 2012 8:34 PM
> >>>> Subject: hbase 0.94.0
> >>>>
> >>>> Lets branch end of february? No new features thereafter. Is this too
> >>>> close in? Would be grand if 0.94.0 shipped before hbasecon. What
> >>>> should 0.94.0 have in it? I don't mind if the list is short.
> >>>>
> >>>> Unless someone else wants too, I don't mind being release manager
> >>>> (will try to run a tighter ship this time around).
> >>>>
> >>>> St.Ack
> >>>>
> >>>>
> >>>>
> >>>
> >>>
>
Re: hbase 0.94.0
Posted by Stack <st...@duboce.net>.
On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl <lh...@yahoo.com> wrote:
> Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the 0.90 -> 0.92 migration path
> make the migration code for HBASE-2600 (much) more complicated in 0.94?
>
hbase-2600 should go in when its ready. It might make 0.94. Its ok if
it doesn't I'd say. We need to tread carefully regards 2600 (Alex is
on it. He was here Tuesday. We talked of making a 2600 branch w/ his
current work and in which we'd work through bugs and migration).
If you want to facilitate CDH4 carrying 0.94, I'd say 2600 goes into
0.96 because it'll require migration effort by Clouderians hacking up
a migration script that runs the 0.90->0.92 .META. rewrite followed by
the 2600 .META. rewrite. I say 'by Clouderians' because as I say
above, I don't think we should be in the business of testing and
verifying what happens if you skip versions upgrading.
I'd be fine pushing out 2600. CDHs hang around a while. Having to
answer 0.92 questions when the rest of us are 0.94'ing or 0.96ing
would be a pain.
St.Ack
Re: hbase 0.94.0
Posted by lars hofhansl <lh...@yahoo.com>.
Thanks would be awesome! Let's do that in 0.96. :)
----- Original Message -----
From: Matt Corgan <mc...@hotpads.com>
To: lars hofhansl <lh...@yahoo.com>; dev <de...@hbase.apache.org>
Cc: Jonathan Hsieh <jo...@cloudera.com>
Sent: Friday, January 27, 2012 12:06 PM
Subject: Re: hbase 0.94.0
The reason I brought it up is now that Mikhail committed the data block
encoding I was going to take a stab at adding the prefix trie encoding I
was working on this past summer. My plan is to first make a minimally
invasive patch to prove correctness. But, after that there will probably
be some big performance gains to be had from reworking some of the things
like KeyValueScanner which I would not have the courage to get working with
v1.
So, that was why I asked, but all of that is still more hypothetical than
real, and I don't even know if the first part will be done before branching
.94 at the end of February. Makes sense to me to not delete v1 until
there's a good reason to, which it doesn't look like we have yet. If I get
to the point where v1 is halting progress then we can reevaluate based on
more specific issues. Maybe none of the prefix trie will even make .94...
..sent from my phone
On Jan 27, 2012 1:55 PM, "lars hofhansl" <lh...@yahoo.com> wrote:
> Hey Jon,
>
> understood. Makes 0.94 hard, though. If we decide now to have a 0.90 to
> 0.94 upgrade path and then timing does not work out and nobody signs up for
> the testing because it 0.92 is more convenient we'd have gone through this
> for nothing.
>
> So... Thinking about this more I am -1 on supporting an official upgrade
> path other then from one release to the next.
> That said, we do not have to break things intentionally.
>
>
> I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94... As long as
> we won't havethe same argument for 0.96 :)
> And I am not aware of any file compatibility issues.
>
>
> We can also leave the 0.92 migration code in, but not officially support
> it as 0.90 to 0.94 path.
> Then CDH4 can make sure (if needed) that it is all working.
>
>
> Does that work for you guys?
>
> -- Lars
>
>
>
> ________________________________
> From: Jonathan Hsieh <jo...@cloudera.com>
> To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
> Sent: Friday, January 27, 2012 10:10 AM
> Subject: Re: hbase 0.94.0
>
>
> Lars,
>
> The upcoming CDH4 Beta HBase will be based on the latest hotness, 0.92.0.
> A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. If that
> HBase is 0.92 based, we'll test that, and if timing works out and we decide
> 0.94, we'll have to have a path (0.90->0.94) for than and will test that.
>
> HBASE-2600 is a big change of encoding of meta, while my understanding is
> that 0.90->0,92 is a graceful HFile format conversion. Are there are
> things currently in trunk that further break compatiblity of the file
> format? (what jiras?)
>
> Jon.
>
>
> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl <lh...@yahoo.com>
> wrote:
>
> Not removing code for upgrade is fine.
> >
> >Todd, is Cloudera signing up for testing this path (0.90 to 0.94)?
> >
> >
> >Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the 0.90 ->
> 0.92 migration path
> >make the migration code for HBASE-2600 (much) more complicated in 0.94?
> >
> >
> >-- Lars
> >
> >
> >
> >----- Original Message -----
> >From: Stack <st...@duboce.net>
> >To: dev@hbase.apache.org
> >Cc:
> >Sent: Friday, January 27, 2012 9:02 AM
> >Subject: Re: hbase 0.94.0
> >
> >
> >On Fri, Jan 27, 2012 at 8:42 AM, Stack <st...@duboce.net> wrote:
> >> In this particular case, there was no explicit migration step needed
> >> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be
> >> running the 0.92 to 0.94 migration script (if there is one).
> >>
> >
> >Thinking more, the above only really holds if we keep the .META.
> >migration code that runs in 0.92 on startup on into 0.94 (I have a
> >patch here where I'm removing it... I should put this bit of code
> >back).
> >
> >I see Todd that you vote against removing hfile v1 in 0.94. To make
> >going from CDH3 to CDH4, we should not purge any migrating code or old
> >version support.
> >
> >I'd be fine w/ that. We'll need to work hard to ensure it taking it
> >on as a principal for 0.94. Ok w/ you "Iron Hand" Lars?
> >
> >St.Ack
> >
> >
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
>
> // jon@cloudera.com
Re: hbase 0.94.0
Posted by Nicolas Spiegelberg <ns...@fb.com>.
>>1) The trunk has more master work. Specifically 89-master related
>> features that we implemented for our grid architecture.
>
>You have a list of issues Nicolas?
It's at least 3 major features. We're more than willing to work with you
on porting. It sounds like eBay & Cloudera have some possible demand for
the functionality already...
>> 2) Client/Server compatibility. Even if we upgraded the servers to
>> 94/96/whatever, we still will have a lot of clients running 89 & we
>>can't
>> simply stop/start all of them at the same time. We need 89/trunk RPC
>> compat for client/server.
>
>An 0.89 client should be able to talk to a 0.94+ server?
That's what we need at least, to consider upgrading a system like
Messages. I understand that we'll have to do most of this work, but it
will definitely cause us a major inconvenience and severely delay our
upgrade possibilities. I would caution against putting other HBase
deployments based on later revisions in the same boat. Letting people add
a feature without a smooth upgrade strategy is like accepting code without
a peer review: there's a ton of hidden cost that you pretend not to notice.
Re: hbase 0.94.0
Posted by Nicolas Spiegelberg <ns...@fb.com>.
>>1) The trunk has more master work. Specifically 89-master related
>> features that we implemented for our grid architecture.
>
>You have a list of issues Nicolas?
We have a list, but it's non-trivial. You're more than welcome to help if
you want. :)
>> 2) Client/Server compatibility. Even if we upgraded the servers to
>> 94/96/whatever, we still will have a lot of clients running 89 & we
>>can't
>> simply stop/start all of them at the same time. We need 89/trunk RPC
>> compat for client/server.
>
>An 0.89 client should be able to talk to a 0.94+ server?
That's what we'll need. I'm assuming that we're the only ones who will
need this functionality, since we're the only ones who deployed
extensively on HBase
during that timeframe. However, it's still a major hindrance to upgrade.
>The talk was of removing old code if no longer needed; i.e you should
>not have the condition you describe above of having to have an hfilev1
>reader during upgrade if we had it that you couldn't start an upgrade
>w/o first purging hfilev1s.
Unfortunately, we do have production clusters running with HFileV1. This
is actually a result of us experimenting with non-89 versions (0.90)
instead of waiting to upgrade. The other problem is that upgrading a
production cluster will cause long downtime. We currently have a
5-min SLA agreement.
Re: hbase 0.94.0
Posted by Stack <st...@duboce.net>.
On Mon, Jan 30, 2012 at 11:21 AM, Nicolas Spiegelberg
<ns...@fb.com> wrote:
> 1) The trunk has more master work. Specifically 89-master related
> features that we implemented for our grid architecture. It sounds like
> most of those features are rising in prominence in the Open Source
> community as well (Ted from eBay & Todd from Cloudera, in particular).
> Hopefully, we can collaborate on porting/implementation.
You have a list of issues Nicolas?
> 2) Client/Server compatibility. Even if we upgraded the servers to
> 94/96/whatever, we still will have a lot of clients running 89 & we can't
> simply stop/start all of them at the same time. We need 89/trunk RPC
> compat for client/server.
An 0.89 client should be able to talk to a 0.94+ server?
> 3) Durable Server Upgrade. It does bother me that people are very quick
> to consider removing both RPC & persistent data compatibility from 90 to
> trunk (a branch that we're still actively releasing minor upgrades for).
> We will need the ability to mutate the HDFS & ZK persistent data from the
> 89 format to trunk. Specifically, work like HFileV1 reader is critical
> for analysis if the upgrade script fails and we need to debug.
>
The talk was of removing old code if no longer needed; i.e you should
not have the condition you describe above of having to have an hfilev1
reader during upgrade if we had it that you couldn't start an upgrade
w/o first purging hfilev1s.
The rpc incompatibility across major versions is to be undone in
future versions. I'm just trying to elicit if you think we need to be
able to make it work back into the past. If so, thats a new
requirement.
> Instead of discussing what classes we can throw away, can we instead think
> about a strategy for long-term, cross-version compatibility that minimally
> hurts trunk development? Is HFileV1's presence severely hindering another
> feature?
>
Unused code should be let go. Its a drag. Long term cross-version
compatibility, as Jon says later, is being started on.... proposal
coming.
St.Ack
Re: hbase 0.94.0
Posted by Jimmy Xiang <jx...@cloudera.com>.
Currently, Jon and me in Cloudera are trying to come up a proposal for
client server wire
compatibility and rolling upgrade support. We will share with the
community after we come
to some consensus internal at first. :)
Thanks,
Jimmy
On Mon, Jan 30, 2012 at 3:37 PM, Nicolas Spiegelberg <ns...@fb.com>wrote:
> >Traditionally there have been no guarantees of cross major version
> >compatibility. RPC especially. Never a rolling upgrade from an 0.90 to an
> >0.92, for example. For persistent data, there is a migration utility for
> >upping from one major release to the next.
>
> I'm advocating that RPC compatibility breakage is not acceptable for FB
> because this is a vital and highly-deployed infrastructure piece. I'm
> assuming this strategy may not be acceptable for other major contributors
> as well. I can't imagine that CDH customers don't need cross-version
> compat, which will most likely go from 92->96+. I think we need to have a
> client->server online migration strategy for currently active revisions.
> This is independent of whether we label the current build 1.0 or not. In
> fact, I would advocate that we want to be in the habit of cross-version
> compatibility and long-term thinking before we actually release a 1.0.
> With 1.0, we don't just want to have cross-version compat, we want to have
> the problem nailed or else it will cause major support problems.
>
> Note that I'm not advocating cross-service RPC compat at this time. I
> don't think we need to tackle online rolling upgrades of HBase from 92->94
> (e.g. Mixed RegionServer versions or mixed master-RS). Doing a start-stop
> of the entire HBase cluster is probably fine before 1.0. However, I think
> it's safe to say that there are multiple instances where the DBA team and
> the AppServer team are different people, especially with any group
> exploring multi-tenancy. For that use case, client & server compat is
> critical.
>
>
> >Regarding RPC, this state of affairs is not really acceptable to anyone
> >any more. Over in core there's work to move 99% of RPC to protobufs, with
> >only the thinnest Writable header. In this thread there seem several here
> >who want to tackle this for HBase now.
>
> Are the people working on this functionality are thinking about
> client-server compat? JIRA #s?
>
> >Regarding major version data migrations, the attitude I believe is pre
> >1.0 we can entertain design changes that break compatibility, in search
> >of something that works well enough to be 1.0. From that point forward,
> >compatibility is a requirement.
>
> What's the definition of working well enough to be 1.0? I thought having
> stable, durable PB+ online data storage would be considered working "well
> enough". Did we not declare that 0.90 was the "data durable" version of
> HBase that you could trust? Migrations should be a first-class priority
> after declaring the project "data durable". If you cannot reliably
> persist 100% of data after upgrade, then the version truly wasn't "data
> durable".
>
>
Re: hbase 0.94.0
Posted by tsuna <ts...@gmail.com>.
On Mon, Jan 30, 2012 at 3:37 PM, Nicolas Spiegelberg
<ns...@fb.com> wrote:
> I'm advocating that RPC compatibility breakage is not acceptable for FB
> because this is a vital and highly-deployed infrastructure piece. I'm
> assuming this strategy may not be acceptable for other major contributors
> as well. I can't imagine that CDH customers don't need cross-version
> compat, which will most likely go from 92->96+. I think we need to have a
> client->server online migration strategy for currently active revisions.
If you build your apps with asynchbase, not only you'll get better
performance out of HBase, but also you won't have these annoying
problems of RPC compatibility :)
> This is independent of whether we label the current build 1.0 or not. In
> fact, I would advocate that we want to be in the habit of cross-version
> compatibility and long-term thinking before we actually release a 1.0.
> With 1.0, we don't just want to have cross-version compat, we want to have
> the problem nailed or else it will cause major support problems.
>
> Note that I'm not advocating cross-service RPC compat at this time. I
> don't think we need to tackle online rolling upgrades of HBase from 92->94
> (e.g. Mixed RegionServer versions or mixed master-RS). Doing a start-stop
> of the entire HBase cluster is probably fine before 1.0. However, I think
> it's safe to say that there are multiple instances where the DBA team and
> the AppServer team are different people, especially with any group
> exploring multi-tenancy. For that use case, client & server compat is
> critical.
Agreed.
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
Re: hbase 0.94.0
Posted by Andrew Purtell <ap...@apache.org>.
Hi,
>>Traditionally there have been no guarantees of cross major version
>>compatibility. RPC especially. Never a rolling upgrade from an 0.90 to an
>>0.92, for example. For persistent data, there is a migration utility for
>>upping from one major release to the next.
>
>I'm advocating that RPC compatibility breakage is not acceptable for FB
>because this is a vital and highly-deployed infrastructure piece. I'm
>assuming this strategy may not be acceptable for other major contributors
>as well.
Of course. Traditionally or maybe "historically" HBase didn't made such guarantees because of the level of resources available to the project. We lived with Hadoop RPC and its limitations, there were other areas needing attention first. We supported a migration step only from one major version to the next, because file formats were (and still are) in evolution. Now that there are major contributors who can't accept a lack of cross major version compatibility, I'm sure this will change fast, there are resources enough now to make it happen.
> What's the definition of working well enough to be 1.0?
I believe if you search the mailing list archives for "hadoop 1.0" you'll find the discussion thread to which I refer.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
> From: Nicolas Spiegelberg <ns...@fb.com>
>To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>Sent: Monday, January 30, 2012 3:37 PM
>Subject: Re: hbase 0.94.0
>
>>Traditionally there have been no guarantees of cross major version
>>compatibility. RPC especially. Never a rolling upgrade from an 0.90 to an
>>0.92, for example. For persistent data, there is a migration utility for
>>upping from one major release to the next.
>
>I'm advocating that RPC compatibility breakage is not acceptable for FB
>because this is a vital and highly-deployed infrastructure piece. I'm
>assuming this strategy may not be acceptable for other major contributors
>as well. I can't imagine that CDH customers don't need cross-version
>compat, which will most likely go from 92->96+. I think we need to have a
>client->server online migration strategy for currently active revisions.
>This is independent of whether we label the current build 1.0 or not. In
>fact, I would advocate that we want to be in the habit of cross-version
>compatibility and long-term thinking before we actually release a 1.0.
>With 1.0, we don't just want to have cross-version compat, we want to have
>the problem nailed or else it will cause major support problems.
>
>Note that I'm not advocating cross-service RPC compat at this time. I
>don't think we need to tackle online rolling upgrades of HBase from 92->94
>(e.g. Mixed RegionServer versions or mixed master-RS). Doing a start-stop
>of the entire HBase cluster is probably fine before 1.0. However, I think
>it's safe to say that there are multiple instances where the DBA team and
>the AppServer team are different people, especially with any group
>exploring multi-tenancy. For that use case, client & server compat is
>critical.
>
>
>>Regarding RPC, this state of affairs is not really acceptable to anyone
>>any more. Over in core there's work to move 99% of RPC to protobufs, with
>>only the thinnest Writable header. In this thread there seem several here
>>who want to tackle this for HBase now.
>
>Are the people working on this functionality are thinking about
>client-server compat? JIRA #s?
>
>>Regarding major version data migrations, the attitude I believe is pre
>>1.0 we can entertain design changes that break compatibility, in search
>>of something that works well enough to be 1.0. From that point forward,
>>compatibility is a requirement.
>
>What's the definition of working well enough to be 1.0? I thought having
>stable, durable PB+ online data storage would be considered working "well
>enough". Did we not declare that 0.90 was the "data durable" version of
>HBase that you could trust? Migrations should be a first-class priority
>after declaring the project "data durable". If you cannot reliably
>persist 100% of data after upgrade, then the version truly wasn't "data
>durable".
>
>
>
>
Re: hbase 0.94.0
Posted by Stack <st...@duboce.net>.
On Mon, Jan 30, 2012 at 3:37 PM, Nicolas Spiegelberg
<ns...@fb.com> wrote:
> However, I think
> it's safe to say that there are multiple instances where the DBA team and
> the AppServer team are different people, especially with any group
> exploring multi-tenancy. For that use case, client & server compat is
> critical.
>
Sounds like we have a new requirement for the rpc compatibility work,
that rpc/zk must continue to work w/ clients as old as 0.89fb. I
could see how we might make it work for rpc switching on volunteered
header or running with multiple engines, one for old stuff and another
for the new, as Andrew suggests. I'm not sure how it'd work for zk
serialized data given it started to be versioned in 0.92 only (0.89fb
writes using RecoverableZK, 0.90 does not... so maybe we can skip
0.90?).
Trying to narrow the need to the client-server only communication buys
us near nought; the only interface particular to intra-servers
communication is the
http://hbase.apache.org/xref/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.html.
Its the other two, the admin interface
http://hbase.apache.org/xref/org/apache/hadoop/hbase/ipc/HMasterInterface.html,
and the client interface,
http://hbase.apache.org/xref/org/apache/hadoop/hbase/ipc/HRegionInterface.html,
that are fat.
St.Ack
Re: hbase 0.94.0
Posted by Nicolas Spiegelberg <ns...@fb.com>.
>Traditionally there have been no guarantees of cross major version
>compatibility. RPC especially. Never a rolling upgrade from an 0.90 to an
>0.92, for example. For persistent data, there is a migration utility for
>upping from one major release to the next.
I'm advocating that RPC compatibility breakage is not acceptable for FB
because this is a vital and highly-deployed infrastructure piece. I'm
assuming this strategy may not be acceptable for other major contributors
as well. I can't imagine that CDH customers don't need cross-version
compat, which will most likely go from 92->96+. I think we need to have a
client->server online migration strategy for currently active revisions.
This is independent of whether we label the current build 1.0 or not. In
fact, I would advocate that we want to be in the habit of cross-version
compatibility and long-term thinking before we actually release a 1.0.
With 1.0, we don't just want to have cross-version compat, we want to have
the problem nailed or else it will cause major support problems.
Note that I'm not advocating cross-service RPC compat at this time. I
don't think we need to tackle online rolling upgrades of HBase from 92->94
(e.g. Mixed RegionServer versions or mixed master-RS). Doing a start-stop
of the entire HBase cluster is probably fine before 1.0. However, I think
it's safe to say that there are multiple instances where the DBA team and
the AppServer team are different people, especially with any group
exploring multi-tenancy. For that use case, client & server compat is
critical.
>Regarding RPC, this state of affairs is not really acceptable to anyone
>any more. Over in core there's work to move 99% of RPC to protobufs, with
>only the thinnest Writable header. In this thread there seem several here
>who want to tackle this for HBase now.
Are the people working on this functionality are thinking about
client-server compat? JIRA #s?
>Regarding major version data migrations, the attitude I believe is pre
>1.0 we can entertain design changes that break compatibility, in search
>of something that works well enough to be 1.0. From that point forward,
>compatibility is a requirement.
What's the definition of working well enough to be 1.0? I thought having
stable, durable PB+ online data storage would be considered working "well
enough". Did we not declare that 0.90 was the "data durable" version of
HBase that you could trust? Migrations should be a first-class priority
after declaring the project "data durable". If you cannot reliably
persist 100% of data after upgrade, then the version truly wasn't "data
durable".
Re: hbase 0.94.0
Posted by Andrew Purtell <ap...@apache.org>.
> From: Nicolas Spiegelberg <ns...@fb.com>
> 2) Client/Server compatibility. Even if we upgraded the servers to
> 94/96/whatever, we still will have a lot of clients running 89 & we
> can't
0.92+ supports pluggable RPC engines. Can you manage 0.89 compat that way?
> 3) Durable Server Upgrade. It does bother me that people are very quick
> to consider removing both RPC & persistent data compatibility from 90 to
> trunk (a branch that we're still actively releasing minor upgrades for).
> We will need the ability to mutate the HDFS & ZK persistent data from the
> 89 format to trunk. Specifically, work like HFileV1 reader is critical
> for analysis if the upgrade script fails and we need to debug.
Traditionally there have been no guarantees of cross major version compatibility. RPC especially. Never a rolling upgrade from an 0.90 to an 0.92, for example. For persistent data, there is a migration utility for upping from one major release to the next.
Regarding RPC, this state of affairs is not really acceptable to anyone any more. Over in core there's work to move 99% of RPC to protobufs, with only the thinnest Writable header. In this thread there seem several here who want to tackle this for HBase now.
Regarding major version data migrations, the attitude I believe is pre 1.0 we can entertain design changes that break compatibility, in search of something that works well enough to be 1.0. From that point forward, compatibility is a requirement.
In practical terms, with long term scale deployments with compatibility requirements, I think we are quite near 1.0. Earlier I argued that we should adopt the label ("1.0") like Hadoop core did in concession to the realities of large scale production deploys, but there were good counter arguments against doing so, namely that HBase has a few rough edges yet.
> Instead of discussing what classes we can throw away, can we instead think
> about a strategy for long-term, cross-version compatibility that minimally
> hurts trunk development?
+1
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
----- Original Message -----
> From: Nicolas Spiegelberg <ns...@fb.com>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> Cc:
> Sent: Monday, January 30, 2012 11:21 AM
> Subject: Re: hbase 0.94.0
>
> Currently, we at Facebook are developing mostly on 89 but also doing some
> significant exploratory work on trunk. I think that most of our
> development will continue to be done on 89 in the near future. We plan to
> release some lower-risk projects on 94. However, we won't even entertain
> a wide-scale upgrade strategy until:
>
> 1) The trunk has more master work. Specifically 89-master related
> features that we implemented for our grid architecture. It sounds like
> most of those features are rising in prominence in the Open Source
> community as well (Ted from eBay & Todd from Cloudera, in particular).
> Hopefully, we can collaborate on porting/implementation.
> 2) Client/Server compatibility. Even if we upgraded the servers to
> 94/96/whatever, we still will have a lot of clients running 89 & we
> can't
> simply stop/start all of them at the same time. We need 89/trunk RPC
> compat for client/server.
> 3) Durable Server Upgrade. It does bother me that people are very quick
> to consider removing both RPC & persistent data compatibility from 90 to
> trunk (a branch that we're still actively releasing minor upgrades for).
> We will need the ability to mutate the HDFS & ZK persistent data from the
> 89 format to trunk. Specifically, work like HFileV1 reader is critical
> for analysis if the upgrade script fails and we need to debug.
>
> Instead of discussing what classes we can throw away, can we instead think
> about a strategy for long-term, cross-version compatibility that minimally
> hurts trunk development? Is HFileV1's presence severely hindering another
> feature?
>
> On 1/27/12 6:11 PM, "Jonathan Hsieh" <jo...@cloudera.com> wrote:
>
>> Lars, Matt,
>>
>> I like Matt's suggestion -- as long as it is not a burden let's keep
> it
>> in.
>> If it becomes one, let's do what is best for the project.
>>
>> If Cloudera needs to do the upgrade path, we'll provide one. If it
>> doesn't
>> go to the apache repo, we'll most likely open source it on github or
>> something, or "keep it on the jira" like some of the repair ruby
> scripts.
>>
>> Question -- to the Trend/SalesForce/FB folks can you talk about your plans
>> for upgrades? Are you guys moving from 0.89/0.90ish to 0.92 already? or
>> are you guys going to have to do the direct upgrade to 0.94 from 0.90 land
>> as well?
>>
>> Jon.
>>
>>
>> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan <mc...@hotpads.com>
> wrote:
>>
>>> The reason I brought it up is now that Mikhail committed the data block
>>> encoding I was going to take a stab at adding the prefix trie encoding
> I
>>> was working on this past summer. My plan is to first make a minimally
>>> invasive patch to prove correctness. But, after that there will
>>> probably
>>> be some big performance gains to be had from reworking some of the
>>> things
>>> like KeyValueScanner which I would not have the courage to get working
>>> with
>>> v1.
>>>
>>> So, that was why I asked, but all of that is still more hypothetical
>>> than
>>> real, and I don't even know if the first part will be done before
>>> branching
>>> .94 at the end of February. Makes sense to me to not delete v1 until
>>> there's a good reason to, which it doesn't look like we have
> yet. If I
>>> get
>>> to the point where v1 is halting progress then we can reevaluate based
>>> on
>>> more specific issues. Maybe none of the prefix trie will even make
>>> .94...
>>>
>>> ..sent from my phone
>>> On Jan 27, 2012 1:55 PM, "lars hofhansl"
> <lh...@yahoo.com> wrote:
>>>
>>>> Hey Jon,
>>>>
>>>> understood. Makes 0.94 hard, though. If we decide now to have a
> 0.90 to
>>>> 0.94 upgrade path and then timing does not work out and nobody
> signs
>>>> up for
>>>> the testing because it 0.92 is more convenient we'd have gone
> through
>>>> this
>>>> for nothing.
>>>>
>>>> So... Thinking about this more I am -1 on supporting an official
>>>> upgrade
>>>> path other then from one release to the next.
>>>> That said, we do not have to break things intentionally.
>>>>
>>>>
>>>> I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94...
> As long
>>>> as we won't havethe same argument for 0.96 :)
>>>>
>>>> And I am not aware of any file compatibility issues.
>>>>
>>>>
>>>> We can also leave the 0.92 migration code in, but not officially
>>>> support
>>>> it as 0.90 to 0.94 path.
>>>> Then CDH4 can make sure (if needed) that it is all working.
>>>>
>>>>
>>>> Does that work for you guys?
>>>>
>>>> -- Lars
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: Jonathan Hsieh <jo...@cloudera.com>
>>>> To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
>>>> Sent: Friday, January 27, 2012 10:10 AM
>>>> Subject: Re: hbase 0.94.0
>>>>
>>>>
>>>> Lars,
>>>>
>>>> The upcoming CDH4 Beta HBase will be based on the latest hotness,
>>>> 0.92.0.
>>>> A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase.
> If
>>>> that
>>>> HBase is 0.92 based, we'll test that, and if timing works out
> and we
>>>> decide
>>>> 0.94, we'll have to have a path (0.90->0.94) for than and
> will test
>>>> that.
>>>>
>>>> HBASE-2600 is a big change of encoding of meta, while my
> understanding
>>>> is
>>>> that 0.90->0,92 is a graceful HFile format conversion. Are
> there are
>>>> things currently in trunk that further break compatiblity of the
> file
>>>> format? (what jiras?)
>>>>
>>>> Jon.
>>>>
>>>>
>>>> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl
> <lh...@yahoo.com>
>>>> wrote:
>>>>
>>>> Not removing code for upgrade is fine.
>>>> >
>>>> >Todd, is Cloudera signing up for testing this path (0.90 to
> 0.94)?
>>>> >
>>>> >
>>>> >Stack, what's your feeling w.r.t. to HBASE-2600, will
> keeping the 0.90
>>>> -> 0.92 migration path
>>>> >make the migration code for HBASE-2600 (much) more complicated
> in
>>>> 0.94?
>>>> >
>>>> >
>>>> >-- Lars
>>>> >
>>>> >
>>>> >
>>>> >----- Original Message -----
>>>> >From: Stack <st...@duboce.net>
>>>> >To: dev@hbase.apache.org
>>>> >Cc:
>>>> >Sent: Friday, January 27, 2012 9:02 AM
>>>> >Subject: Re: hbase 0.94.0
>>>> >
>>>> >
>>>> >On Fri, Jan 27, 2012 at 8:42 AM, Stack <st...@duboce.net>
> wrote:
>>>> >> In this particular case, there was no explicit migration
> step needed
>>>> >> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should
> just be
>>>> >> running the 0.92 to 0.94 migration script (if there is
> one).
>>>> >>
>>>> >
>>>> >Thinking more, the above only really holds if we keep the
> .META.
>>>> >migration code that runs in 0.92 on startup on into 0.94 (I
> have a
>>>> >patch here where I'm removing it... I should put this bit
> of code
>>>> >back).
>>>> >
>>>> >I see Todd that you vote against removing hfile v1 in 0.94. To
> make
>>>> >going from CDH3 to CDH4, we should not purge any migrating code
> or old
>>>> >version support.
>>>> >
>>>> >I'd be fine w/ that. We'll need to work hard to ensure
> it taking it
>>>> >on as a principal for 0.94. Ok w/ you "Iron Hand"
> Lars?
>>>> >
>>>> >St.Ack
>>>> >
>>>> >
>>>>
>>>>
>>>> --
>>>> // Jonathan Hsieh (shay)
>>>> // Software Engineer, Cloudera
>>>>
>>>> // jon@cloudera.com
>>>>
>>>
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // Software Engineer, Cloudera
>> // jon@cloudera.com
>
Re: hbase 0.94.0
Posted by Devaraj Das <dd...@hortonworks.com>.
On Jan 30, 2012, at 11:39 AM, Jonathan Hsieh wrote:
> Nicolas,
>
> For point 2 and 3, there definitely is a group of us that have been
> informally discussing long term wire and format compatibility in
> conversations. There are some documents that should be coming out soon to
> the lists from these chats. The initial goal of this some of is to define a
> vocabulary, to try to scope and phase-in changes necessary to make this
> happen (possibly parts in 0.94, 0.96, 0.98...), and to define possible
> goals and non-goals.
>
> My hope is that a thought-out first cut will come from Jimmy and I in the
> next week or so so that we start having discussion to reach a consensus and
> then start implementing.
>
> Some folks from another group (I'll let them announce themselves) should be
> producing a doc focused more on the rpc portion of this soon as well.
>
Thanks for the hint, Jon (*smile*)
BTW I have opened - https://issues.apache.org/jira/browse/HBASE-5305 / HBASE-5306 to keep track of the issues..
We will upload some docs/discussion-points there soon.
> Jon.
>
> On Mon, Jan 30, 2012 at 11:21 AM, Nicolas Spiegelberg
> <ns...@fb.com>wrote:
>
>> Currently, we at Facebook are developing mostly on 89 but also doing some
>> significant exploratory work on trunk. I think that most of our
>> development will continue to be done on 89 in the near future. We plan to
>> release some lower-risk projects on 94. However, we won't even entertain
>> a wide-scale upgrade strategy until:
>>
>> 1) The trunk has more master work. Specifically 89-master related
>> features that we implemented for our grid architecture. It sounds like
>> most of those features are rising in prominence in the Open Source
>> community as well (Ted from eBay & Todd from Cloudera, in particular).
>> Hopefully, we can collaborate on porting/implementation.
>> 2) Client/Server compatibility. Even if we upgraded the servers to
>> 94/96/whatever, we still will have a lot of clients running 89 & we can't
>> simply stop/start all of them at the same time. We need 89/trunk RPC
>> compat for client/server.
>> 3) Durable Server Upgrade. It does bother me that people are very quick
>> to consider removing both RPC & persistent data compatibility from 90 to
>> trunk (a branch that we're still actively releasing minor upgrades for).
>> We will need the ability to mutate the HDFS & ZK persistent data from the
>> 89 format to trunk. Specifically, work like HFileV1 reader is critical
>> for analysis if the upgrade script fails and we need to debug.
>>
>> Instead of discussing what classes we can throw away, can we instead think
>> about a strategy for long-term, cross-version compatibility that minimally
>> hurts trunk development? Is HFileV1's presence severely hindering another
>> feature?
>>
>> On 1/27/12 6:11 PM, "Jonathan Hsieh" <jo...@cloudera.com> wrote:
>>
>>> Lars, Matt,
>>>
>>> I like Matt's suggestion -- as long as it is not a burden let's keep it
>>> in.
>>> If it becomes one, let's do what is best for the project.
>>>
>>> If Cloudera needs to do the upgrade path, we'll provide one. If it
>>> doesn't
>>> go to the apache repo, we'll most likely open source it on github or
>>> something, or "keep it on the jira" like some of the repair ruby scripts.
>>>
>>> Question -- to the Trend/SalesForce/FB folks can you talk about your plans
>>> for upgrades? Are you guys moving from 0.89/0.90ish to 0.92 already? or
>>> are you guys going to have to do the direct upgrade to 0.94 from 0.90 land
>>> as well?
>>>
>>> Jon.
>>>
>>>
>>> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan <mc...@hotpads.com>
>> wrote:
>>>
>>>> The reason I brought it up is now that Mikhail committed the data block
>>>> encoding I was going to take a stab at adding the prefix trie encoding I
>>>> was working on this past summer. My plan is to first make a minimally
>>>> invasive patch to prove correctness. But, after that there will
>>>> probably
>>>> be some big performance gains to be had from reworking some of the
>>>> things
>>>> like KeyValueScanner which I would not have the courage to get working
>>>> with
>>>> v1.
>>>>
>>>> So, that was why I asked, but all of that is still more hypothetical
>>>> than
>>>> real, and I don't even know if the first part will be done before
>>>> branching
>>>> .94 at the end of February. Makes sense to me to not delete v1 until
>>>> there's a good reason to, which it doesn't look like we have yet. If I
>>>> get
>>>> to the point where v1 is halting progress then we can reevaluate based
>>>> on
>>>> more specific issues. Maybe none of the prefix trie will even make
>>>> .94...
>>>>
>>>> ..sent from my phone
>>>> On Jan 27, 2012 1:55 PM, "lars hofhansl" <lh...@yahoo.com> wrote:
>>>>
>>>>> Hey Jon,
>>>>>
>>>>> understood. Makes 0.94 hard, though. If we decide now to have a 0.90 to
>>>>> 0.94 upgrade path and then timing does not work out and nobody signs
>>>>> up for
>>>>> the testing because it 0.92 is more convenient we'd have gone through
>>>>> this
>>>>> for nothing.
>>>>>
>>>>> So... Thinking about this more I am -1 on supporting an official
>>>>> upgrade
>>>>> path other then from one release to the next.
>>>>> That said, we do not have to break things intentionally.
>>>>>
>>>>>
>>>>> I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94... As long
>>>>> as we won't havethe same argument for 0.96 :)
>>>>>
>>>>> And I am not aware of any file compatibility issues.
>>>>>
>>>>>
>>>>> We can also leave the 0.92 migration code in, but not officially
>>>>> support
>>>>> it as 0.90 to 0.94 path.
>>>>> Then CDH4 can make sure (if needed) that it is all working.
>>>>>
>>>>>
>>>>> Does that work for you guys?
>>>>>
>>>>> -- Lars
>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>> From: Jonathan Hsieh <jo...@cloudera.com>
>>>>> To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
>>>>> Sent: Friday, January 27, 2012 10:10 AM
>>>>> Subject: Re: hbase 0.94.0
>>>>>
>>>>>
>>>>> Lars,
>>>>>
>>>>> The upcoming CDH4 Beta HBase will be based on the latest hotness,
>>>>> 0.92.0.
>>>>> A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. If
>>>>> that
>>>>> HBase is 0.92 based, we'll test that, and if timing works out and we
>>>>> decide
>>>>> 0.94, we'll have to have a path (0.90->0.94) for than and will test
>>>>> that.
>>>>>
>>>>> HBASE-2600 is a big change of encoding of meta, while my understanding
>>>>> is
>>>>> that 0.90->0,92 is a graceful HFile format conversion. Are there are
>>>>> things currently in trunk that further break compatiblity of the file
>>>>> format? (what jiras?)
>>>>>
>>>>> Jon.
>>>>>
>>>>>
>>>>> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl <lh...@yahoo.com>
>>>>> wrote:
>>>>>
>>>>> Not removing code for upgrade is fine.
>>>>>>
>>>>>> Todd, is Cloudera signing up for testing this path (0.90 to 0.94)?
>>>>>>
>>>>>>
>>>>>> Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the 0.90
>>>>> -> 0.92 migration path
>>>>>> make the migration code for HBASE-2600 (much) more complicated in
>>>>> 0.94?
>>>>>>
>>>>>>
>>>>>> -- Lars
>>>>>>
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: Stack <st...@duboce.net>
>>>>>> To: dev@hbase.apache.org
>>>>>> Cc:
>>>>>> Sent: Friday, January 27, 2012 9:02 AM
>>>>>> Subject: Re: hbase 0.94.0
>>>>>>
>>>>>>
>>>>>> On Fri, Jan 27, 2012 at 8:42 AM, Stack <st...@duboce.net> wrote:
>>>>>>> In this particular case, there was no explicit migration step needed
>>>>>>> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be
>>>>>>> running the 0.92 to 0.94 migration script (if there is one).
>>>>>>>
>>>>>>
>>>>>> Thinking more, the above only really holds if we keep the .META.
>>>>>> migration code that runs in 0.92 on startup on into 0.94 (I have a
>>>>>> patch here where I'm removing it... I should put this bit of code
>>>>>> back).
>>>>>>
>>>>>> I see Todd that you vote against removing hfile v1 in 0.94. To make
>>>>>> going from CDH3 to CDH4, we should not purge any migrating code or old
>>>>>> version support.
>>>>>>
>>>>>> I'd be fine w/ that. We'll need to work hard to ensure it taking it
>>>>>> on as a principal for 0.94. Ok w/ you "Iron Hand" Lars?
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> // Jonathan Hsieh (shay)
>>>>> // Software Engineer, Cloudera
>>>>>
>>>>> // jon@cloudera.com
>>>>>
>>>>
>>>
>>>
>>> --
>>> // Jonathan Hsieh (shay)
>>> // Software Engineer, Cloudera
>>> // jon@cloudera.com
>>
>>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
Re: hbase 0.94.0
Posted by Jonathan Hsieh <jo...@cloudera.com>.
Nicolas,
For point 2 and 3, there definitely is a group of us that have been
informally discussing long term wire and format compatibility in
conversations. There are some documents that should be coming out soon to
the lists from these chats. The initial goal of this some of is to define a
vocabulary, to try to scope and phase-in changes necessary to make this
happen (possibly parts in 0.94, 0.96, 0.98...), and to define possible
goals and non-goals.
My hope is that a thought-out first cut will come from Jimmy and I in the
next week or so so that we start having discussion to reach a consensus and
then start implementing.
Some folks from another group (I'll let them announce themselves) should be
producing a doc focused more on the rpc portion of this soon as well.
Jon.
On Mon, Jan 30, 2012 at 11:21 AM, Nicolas Spiegelberg
<ns...@fb.com>wrote:
> Currently, we at Facebook are developing mostly on 89 but also doing some
> significant exploratory work on trunk. I think that most of our
> development will continue to be done on 89 in the near future. We plan to
> release some lower-risk projects on 94. However, we won't even entertain
> a wide-scale upgrade strategy until:
>
> 1) The trunk has more master work. Specifically 89-master related
> features that we implemented for our grid architecture. It sounds like
> most of those features are rising in prominence in the Open Source
> community as well (Ted from eBay & Todd from Cloudera, in particular).
> Hopefully, we can collaborate on porting/implementation.
> 2) Client/Server compatibility. Even if we upgraded the servers to
> 94/96/whatever, we still will have a lot of clients running 89 & we can't
> simply stop/start all of them at the same time. We need 89/trunk RPC
> compat for client/server.
> 3) Durable Server Upgrade. It does bother me that people are very quick
> to consider removing both RPC & persistent data compatibility from 90 to
> trunk (a branch that we're still actively releasing minor upgrades for).
> We will need the ability to mutate the HDFS & ZK persistent data from the
> 89 format to trunk. Specifically, work like HFileV1 reader is critical
> for analysis if the upgrade script fails and we need to debug.
>
> Instead of discussing what classes we can throw away, can we instead think
> about a strategy for long-term, cross-version compatibility that minimally
> hurts trunk development? Is HFileV1's presence severely hindering another
> feature?
>
> On 1/27/12 6:11 PM, "Jonathan Hsieh" <jo...@cloudera.com> wrote:
>
> >Lars, Matt,
> >
> >I like Matt's suggestion -- as long as it is not a burden let's keep it
> >in.
> > If it becomes one, let's do what is best for the project.
> >
> >If Cloudera needs to do the upgrade path, we'll provide one. If it
> >doesn't
> >go to the apache repo, we'll most likely open source it on github or
> >something, or "keep it on the jira" like some of the repair ruby scripts.
> >
> >Question -- to the Trend/SalesForce/FB folks can you talk about your plans
> >for upgrades? Are you guys moving from 0.89/0.90ish to 0.92 already? or
> >are you guys going to have to do the direct upgrade to 0.94 from 0.90 land
> >as well?
> >
> >Jon.
> >
> >
> >On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan <mc...@hotpads.com>
> wrote:
> >
> >> The reason I brought it up is now that Mikhail committed the data block
> >> encoding I was going to take a stab at adding the prefix trie encoding I
> >> was working on this past summer. My plan is to first make a minimally
> >> invasive patch to prove correctness. But, after that there will
> >>probably
> >> be some big performance gains to be had from reworking some of the
> >>things
> >> like KeyValueScanner which I would not have the courage to get working
> >>with
> >> v1.
> >>
> >> So, that was why I asked, but all of that is still more hypothetical
> >>than
> >> real, and I don't even know if the first part will be done before
> >>branching
> >> .94 at the end of February. Makes sense to me to not delete v1 until
> >> there's a good reason to, which it doesn't look like we have yet. If I
> >>get
> >> to the point where v1 is halting progress then we can reevaluate based
> >>on
> >> more specific issues. Maybe none of the prefix trie will even make
> >>.94...
> >>
> >> ..sent from my phone
> >> On Jan 27, 2012 1:55 PM, "lars hofhansl" <lh...@yahoo.com> wrote:
> >>
> >>> Hey Jon,
> >>>
> >>> understood. Makes 0.94 hard, though. If we decide now to have a 0.90 to
> >>> 0.94 upgrade path and then timing does not work out and nobody signs
> >>>up for
> >>> the testing because it 0.92 is more convenient we'd have gone through
> >>>this
> >>> for nothing.
> >>>
> >>> So... Thinking about this more I am -1 on supporting an official
> >>>upgrade
> >>> path other then from one release to the next.
> >>> That said, we do not have to break things intentionally.
> >>>
> >>>
> >>> I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94... As long
> >>> as we won't havethe same argument for 0.96 :)
> >>>
> >>> And I am not aware of any file compatibility issues.
> >>>
> >>>
> >>> We can also leave the 0.92 migration code in, but not officially
> >>>support
> >>> it as 0.90 to 0.94 path.
> >>> Then CDH4 can make sure (if needed) that it is all working.
> >>>
> >>>
> >>> Does that work for you guys?
> >>>
> >>> -- Lars
> >>>
> >>>
> >>>
> >>> ________________________________
> >>> From: Jonathan Hsieh <jo...@cloudera.com>
> >>> To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
> >>> Sent: Friday, January 27, 2012 10:10 AM
> >>> Subject: Re: hbase 0.94.0
> >>>
> >>>
> >>> Lars,
> >>>
> >>> The upcoming CDH4 Beta HBase will be based on the latest hotness,
> >>>0.92.0.
> >>> A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. If
> >>>that
> >>> HBase is 0.92 based, we'll test that, and if timing works out and we
> >>>decide
> >>> 0.94, we'll have to have a path (0.90->0.94) for than and will test
> >>>that.
> >>>
> >>> HBASE-2600 is a big change of encoding of meta, while my understanding
> >>>is
> >>> that 0.90->0,92 is a graceful HFile format conversion. Are there are
> >>> things currently in trunk that further break compatiblity of the file
> >>> format? (what jiras?)
> >>>
> >>> Jon.
> >>>
> >>>
> >>> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl <lh...@yahoo.com>
> >>> wrote:
> >>>
> >>> Not removing code for upgrade is fine.
> >>> >
> >>> >Todd, is Cloudera signing up for testing this path (0.90 to 0.94)?
> >>> >
> >>> >
> >>> >Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the 0.90
> >>> -> 0.92 migration path
> >>> >make the migration code for HBASE-2600 (much) more complicated in
> >>>0.94?
> >>> >
> >>> >
> >>> >-- Lars
> >>> >
> >>> >
> >>> >
> >>> >----- Original Message -----
> >>> >From: Stack <st...@duboce.net>
> >>> >To: dev@hbase.apache.org
> >>> >Cc:
> >>> >Sent: Friday, January 27, 2012 9:02 AM
> >>> >Subject: Re: hbase 0.94.0
> >>> >
> >>> >
> >>> >On Fri, Jan 27, 2012 at 8:42 AM, Stack <st...@duboce.net> wrote:
> >>> >> In this particular case, there was no explicit migration step needed
> >>> >> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be
> >>> >> running the 0.92 to 0.94 migration script (if there is one).
> >>> >>
> >>> >
> >>> >Thinking more, the above only really holds if we keep the .META.
> >>> >migration code that runs in 0.92 on startup on into 0.94 (I have a
> >>> >patch here where I'm removing it... I should put this bit of code
> >>> >back).
> >>> >
> >>> >I see Todd that you vote against removing hfile v1 in 0.94. To make
> >>> >going from CDH3 to CDH4, we should not purge any migrating code or old
> >>> >version support.
> >>> >
> >>> >I'd be fine w/ that. We'll need to work hard to ensure it taking it
> >>> >on as a principal for 0.94. Ok w/ you "Iron Hand" Lars?
> >>> >
> >>> >St.Ack
> >>> >
> >>> >
> >>>
> >>>
> >>> --
> >>> // Jonathan Hsieh (shay)
> >>> // Software Engineer, Cloudera
> >>>
> >>> // jon@cloudera.com
> >>>
> >>
> >
> >
> >--
> >// Jonathan Hsieh (shay)
> >// Software Engineer, Cloudera
> >// jon@cloudera.com
>
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: hbase 0.94.0
Posted by Jonathan Hsieh <jo...@cloudera.com>.
I'm working on trunk/0.92 hbase-5128. There are things that have changed there that breaks/hangs tests and I need to get those good before I start cluster testing.
Sent from my iPhone
On Jan 27, 2012, at 18:27, lars hofhansl <lh...@yahoo.com> wrote:
> Salesforce will jumpstart with 0.94. :) Our dev cluster has the current trunk on it.
>
> OK let's just add these to 0.94:
> HBASE-4218
>
> HBASE-4608
> HBASE-5128
> script necessary to do HBASE-5293 in 0.96
>
>
> And defer the rest to 0.96, and branch 0.94 soon'ish.
> Do you think you can do a 0.92 and trunk version of HBASE-5128?
>
> -- Lars
>
>
> ________________________________
> From: Jonathan Hsieh <jo...@cloudera.com>
> To: Matt Corgan <mc...@hotpads.com>
> Cc: lars hofhansl <lh...@yahoo.com>; dev <de...@hbase.apache.org>
> Sent: Friday, January 27, 2012 6:11 PM
> Subject: Re: hbase 0.94.0
>
> Lars, Matt,
>
> I like Matt's suggestion -- as long as it is not a burden let's keep it in.
> If it becomes one, let's do what is best for the project.
>
> If Cloudera needs to do the upgrade path, we'll provide one. If it doesn't
> go to the apache repo, we'll most likely open source it on github or
> something, or "keep it on the jira" like some of the repair ruby scripts.
>
> Question -- to the Trend/SalesForce/FB folks can you talk about your plans
> for upgrades? Are you guys moving from 0.89/0.90ish to 0.92 already? or
> are you guys going to have to do the direct upgrade to 0.94 from 0.90 land
> as well?
>
> Jon.
>
>
> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan <mc...@hotpads.com> wrote:
>
>> The reason I brought it up is now that Mikhail committed the data block
>> encoding I was going to take a stab at adding the prefix trie encoding I
>> was working on this past summer. My plan is to first make a minimally
>> invasive patch to prove correctness. But, after that there will probably
>> be some big performance gains to be had from reworking some of the things
>> like KeyValueScanner which I would not have the courage to get working with
>> v1.
>>
>> So, that was why I asked, but all of that is still more hypothetical than
>> real, and I don't even know if the first part will be done before branching
>> .94 at the end of February. Makes sense to me to not delete v1 until
>> there's a good reason to, which it doesn't look like we have yet. If I get
>> to the point where v1 is halting progress then we can reevaluate based on
>> more specific issues. Maybe none of the prefix trie will even make .94...
>>
>> ..sent from my phone
>> On Jan 27, 2012 1:55 PM, "lars hofhansl" <lh...@yahoo.com> wrote:
>>
>>> Hey Jon,
>>>
>>> understood. Makes 0.94 hard, though. If we decide now to have a 0.90 to
>>> 0.94 upgrade path and then timing does not work out and nobody signs up for
>>> the testing because it 0.92 is more convenient we'd have gone through this
>>> for nothing.
>>>
>>> So... Thinking about this more I am -1 on supporting an official upgrade
>>> path other then from one release to the next.
>>> That said, we do not have to break things intentionally.
>>>
>>>
>>> I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94... As long
>>> as we won't havethe same argument for 0.96 :)
>>>
>>> And I am not aware of any file compatibility issues.
>>>
>>>
>>> We can also leave the 0.92 migration code in, but not officially support
>>> it as 0.90 to 0.94 path.
>>> Then CDH4 can make sure (if needed) that it is all working.
>>>
>>>
>>> Does that work for you guys?
>>>
>>> -- Lars
>>>
>>>
>>>
>>> ________________________________
>>> From: Jonathan Hsieh <jo...@cloudera.com>
>>> To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
>>> Sent: Friday, January 27, 2012 10:10 AM
>>> Subject: Re: hbase 0.94.0
>>>
>>>
>>> Lars,
>>>
>>> The upcoming CDH4 Beta HBase will be based on the latest hotness, 0.92.0.
>>> A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. If that
>>> HBase is 0.92 based, we'll test that, and if timing works out and we decide
>>> 0.94, we'll have to have a path (0.90->0.94) for than and will test that.
>>>
>>> HBASE-2600 is a big change of encoding of meta, while my understanding is
>>> that 0.90->0,92 is a graceful HFile format conversion. Are there are
>>> things currently in trunk that further break compatiblity of the file
>>> format? (what jiras?)
>>>
>>> Jon.
>>>
>>>
>>> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl <lh...@yahoo.com>
>>> wrote:
>>>
>>> Not removing code for upgrade is fine.
>>>>
>>>> Todd, is Cloudera signing up for testing this path (0.90 to 0.94)?
>>>>
>>>>
>>>> Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the 0.90
>>> -> 0.92 migration path
>>>> make the migration code for HBASE-2600 (much) more complicated in 0.94?
>>>>
>>>>
>>>> -- Lars
>>>>
>>>>
>>>>
>>>> ----- Original Message -----
>>>> From: Stack <st...@duboce.net>
>>>> To: dev@hbase.apache.org
>>>> Cc:
>>>> Sent: Friday, January 27, 2012 9:02 AM
>>>> Subject: Re: hbase 0.94.0
>>>>
>>>>
>>>> On Fri, Jan 27, 2012 at 8:42 AM, Stack <st...@duboce.net> wrote:
>>>>> In this particular case, there was no explicit migration step needed
>>>>> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be
>>>>> running the 0.92 to 0.94 migration script (if there is one).
>>>>>
>>>>
>>>> Thinking more, the above only really holds if we keep the .META.
>>>> migration code that runs in 0.92 on startup on into 0.94 (I have a
>>>> patch here where I'm removing it... I should put this bit of code
>>>> back).
>>>>
>>>> I see Todd that you vote against removing hfile v1 in 0.94. To make
>>>> going from CDH3 to CDH4, we should not purge any migrating code or old
>>>> version support.
>>>>
>>>> I'd be fine w/ that. We'll need to work hard to ensure it taking it
>>>> on as a principal for 0.94. Ok w/ you "Iron Hand" Lars?
>>>>
>>>> St.Ack
>>>>
>>>>
>>>
>>>
>>> --
>>> // Jonathan Hsieh (shay)
>>> // Software Engineer, Cloudera
>>>
>>> // jon@cloudera.com
>>>
>>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
Re: hbase 0.94.0
Posted by Stack <st...@duboce.net>.
On Mon, Jan 30, 2012 at 10:44 AM, Todd Lipcon <to...@cloudera.com> wrote:
> Maybe the discussion should be whether 0.94 should be a time-gated or
> feature-gated release? IMO we already have enough good stuff in there
> on the perf front. It will be great if the checksum improvement makes
> it, but if it doesn't, I'd rather have a release than delay for it.
>
+1 on time-based releases on a short cycle.
St.Ack
Re: hbase 0.94.0
Posted by Matt Corgan <mc...@hotpads.com>.
I had assumed facebook had integrated HFileV2 into their 0.89 branch a
while back and that pretty much all HFiles would have been converted to v2
by now. I guess that is not correct...
On Mon, Jan 30, 2012 at 11:53 AM, Nicolas Spiegelberg
<ns...@fb.com>wrote:
> I think it's also good to make a crucial distinction here: it's not like
> Facebook has a concrete wide-scale upgrade strategy, these 3 points are
> deal-breakers that won't allow us to even entertain an upgrade strategy.
> This is just as critical as HDFS data loss was in 0.90: it's something we
> can't entertain & is a mandatory hurdle to trunk being considered a
> realistic option. That said, I personally would like to upgrade our
> internal version to more closely align with other developers. It will
> reduce our porting overhead and increase the amount of benefit we can give
> the community & visa-versa.
>
> On 1/30/12 11:46 AM, "Ted Yu" <yu...@gmail.com> wrote:
>
> >Before long term wire and format compatibility is reached, I am -0 on time
> >based release.
> >
> >As Nicolas pointed out, FB wants to migrate from 0.89-fb to 0.94
> >If we do time-based releases without wire and format compatibility, it
> >would consume a lot of our energy satisfying such requirements from
> >different companies.
> >
> >I propose wire and format compatibility as the main theme for 0.96
> >
> >Cheers
> >
> >On Mon, Jan 30, 2012 at 11:40 AM, Andrew Purtell
> ><ap...@apache.org>wrote:
> >
> >> > Maybe the discussion should be whether 0.94 should be a time-gated or
> >> > feature-gated release? IMO we already have enough good stuff in there
> >> > on the perf front. It will be great if the checksum improvement makes
> >> > it, but if it doesn't, I'd rather have a release than delay for it.
> >>
> >>
> >> Time gated releases makes sense going forward for a few reasons, mark me
> >> down for that option.
> >>
> >>
> >> Best regards,
> >>
> >> - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
> >>
> >> ----- Original Message -----
> >> > From: Todd Lipcon <to...@cloudera.com>
> >> > To: dev@hbase.apache.org
> >> > Cc: lars hofhansl <lh...@yahoo.com>; Matt Corgan <
> >> mcorgan@hotpads.com>
> >> > Sent: Monday, January 30, 2012 10:44 AM
> >> > Subject: Re: hbase 0.94.0
> >> >
> >> > On Sun, Jan 29, 2012 at 7:30 AM, Ted Yu <yu...@gmail.com> wrote:
> >> >> Initial results from HBASE-5074, support checksums in HBase block
> >> cache,
> >> >> look promising.
> >> >>
> >> >> Shall we discuss whether 0.94 should include it (assuming Dhruba can
> >> finish
> >> >> the feature in Feb.) ?
> >> >
> >> > If it's a time-based release, then there's nothing to discuss. Either
> >> > it's done in time, in which case it's part of 94, or it's not, in
> >> > which case it's part of 96, right?
> >> >
> >> > Maybe the discussion should be whether 0.94 should be a time-gated or
> >> > feature-gated release? IMO we already have enough good stuff in there
> >> > on the perf front. It will be great if the checksum improvement makes
> >> > it, but if it doesn't, I'd rather have a release than delay for it.
> >> >
> >> > -Todd
> >> >
> >> >>
> >> >> Cheers
> >> >>
> >> >> On Fri, Jan 27, 2012 at 6:27 PM, lars hofhansl <lhofhansl@yahoo.com
> >
> >> > wrote:
> >> >>
> >> >>> Salesforce will jumpstart with 0.94. :) Our dev cluster has the
> >> current
> >> >>> trunk on it.
> >> >>>
> >> >>> OK let's just add these to 0.94:
> >> >>> HBASE-4218
> >> >>>
> >> >>> HBASE-4608
> >> >>> HBASE-5128
> >> >>> script necessary to do HBASE-5293 in 0.96
> >> >>>
> >> >>>
> >> >>> And defer the rest to 0.96, and branch 0.94 soon'ish.
> >> >>> Do you think you can do a 0.92 and trunk version of HBASE-5128?
> >> >>>
> >> >>> -- Lars
> >> >>>
> >> >>>
> >> >>> ________________________________
> >> >>> From: Jonathan Hsieh <jo...@cloudera.com>
> >> >>> To: Matt Corgan <mc...@hotpads.com>
> >> >>> Cc: lars hofhansl <lh...@yahoo.com>; dev
> >> > <de...@hbase.apache.org>
> >> >>> Sent: Friday, January 27, 2012 6:11 PM
> >> >>> Subject: Re: hbase 0.94.0
> >> >>>
> >> >>> Lars, Matt,
> >> >>>
> >> >>> I like Matt's suggestion -- as long as it is not a burden let's
> >> > keep it in.
> >> >>> If it becomes one, let's do what is best for the project.
> >> >>>
> >> >>> If Cloudera needs to do the upgrade path, we'll provide one. If it
> >> > doesn't
> >> >>> go to the apache repo, we'll most likely open source it on github
> >> > or
> >> >>> something, or "keep it on the jira" like some of the repair
> >> > ruby scripts.
> >> >>>
> >> >>> Question -- to the Trend/SalesForce/FB folks can you talk about
> >>your
> >> > plans
> >> >>> for upgrades? Are you guys moving from 0.89/0.90ish to 0.92
> >>already?
> >> > or
> >> >>> are you guys going to have to do the direct upgrade to 0.94 from
> >>0.90
> >> > land
> >> >>> as well?
> >> >>>
> >> >>> Jon.
> >> >>>
> >> >>>
> >> >>> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan
> >> > <mc...@hotpads.com> wrote:
> >> >>>
> >> >>> > The reason I brought it up is now that Mikhail committed the data
> >> > block
> >> >>> > encoding I was going to take a stab at adding the prefix trie
> >> > encoding I
> >> >>> > was working on this past summer. My plan is to first make a
> >> > minimally
> >> >>> > invasive patch to prove correctness. But, after that there will
> >> > probably
> >> >>> > be some big performance gains to be had from reworking some of
> >>the
> >> > things
> >> >>> > like KeyValueScanner which I would not have the courage to get
> >> > working
> >> >>> with
> >> >>> > v1.
> >> >>> >
> >> >>> > So, that was why I asked, but all of that is still more
> >> > hypothetical than
> >> >>> > real, and I don't even know if the first part will be done
> >> > before
> >> >>> branching
> >> >>> > .94 at the end of February. Makes sense to me to not delete v1
> >> > until
> >> >>> > there's a good reason to, which it doesn't look like we
> >> > have yet. If I
> >> >>> get
> >> >>> > to the point where v1 is halting progress then we can reevaluate
> >> > based on
> >> >>> > more specific issues. Maybe none of the prefix trie will even
> >> > make
> >> >>> .94...
> >> >>> >
> >> >>> > ..sent from my phone
> >> >>> > On Jan 27, 2012 1:55 PM, "lars hofhansl"
> >> > <lh...@yahoo.com> wrote:
> >> >>> >
> >> >>> >> Hey Jon,
> >> >>> >>
> >> >>> >> understood. Makes 0.94 hard, though. If we decide now to have
> >> > a 0.90 to
> >> >>> >> 0.94 upgrade path and then timing does not work out and nobody
> >> > signs up
> >> >>> for
> >> >>> >> the testing because it 0.92 is more convenient we'd have
> >> > gone through
> >> >>> this
> >> >>> >> for nothing.
> >> >>> >>
> >> >>> >> So... Thinking about this more I am -1 on supporting an
> >> > official upgrade
> >> >>> >> path other then from one release to the next.
> >> >>> >> That said, we do not have to break things intentionally.
> >> >>> >>
> >> >>> >>
> >> >>> >> I'm fine pushing HBASE-2600 and HFile v1 removal out of
> >> > 0.94... As long
> >> >>> >> as we won't havethe same argument for 0.96 :)
> >> >>> >>
> >> >>> >> And I am not aware of any file compatibility issues.
> >> >>> >>
> >> >>> >>
> >> >>> >> We can also leave the 0.92 migration code in, but not
> >> > officially support
> >> >>> >> it as 0.90 to 0.94 path.
> >> >>> >> Then CDH4 can make sure (if needed) that it is all working.
> >> >>> >>
> >> >>> >>
> >> >>> >> Does that work for you guys?
> >> >>> >>
> >> >>> >> -- Lars
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> ________________________________
> >> >>> >> From: Jonathan Hsieh <jo...@cloudera.com>
> >> >>> >> To: dev@hbase.apache.org; lars hofhansl
> >> > <lh...@yahoo.com>
> >> >>> >> Sent: Friday, January 27, 2012 10:10 AM
> >> >>> >> Subject: Re: hbase 0.94.0
> >> >>> >>
> >> >>> >>
> >> >>> >> Lars,
> >> >>> >>
> >> >>> >> The upcoming CDH4 Beta HBase will be based on the latest
> >> > hotness,
> >> >>> 0.92.0.
> >> >>> >> A CDH4 GA HBase will have to have an upgrade path from CDH3
> >> > HBase. If
> >> >>> that
> >> >>> >> HBase is 0.92 based, we'll test that, and if timing works
> >> > out and we
> >> >>> decide
> >> >>> >> 0.94, we'll have to have a path (0.90->0.94) for than
> >> > and will test
> >> >>> that.
> >> >>> >>
> >> >>> >> HBASE-2600 is a big change of encoding of meta, while my
> >> > understanding
> >> >>> is
> >> >>> >> that 0.90->0,92 is a graceful HFile format conversion. Are
> >> > there are
> >> >>> >> things currently in trunk that further break compatiblity of
> >> > the file
> >> >>> >> format? (what jiras?)
> >> >>> >>
> >> >>> >> Jon.
> >> >>> >>
> >> >>> >>
> >> >>> >> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl
> >> > <lh...@yahoo.com>
> >> >>> >> wrote:
> >> >>> >>
> >> >>> >> Not removing code for upgrade is fine.
> >> >>> >> >
> >> >>> >> >Todd, is Cloudera signing up for testing this path (0.90
> >> > to 0.94)?
> >> >>> >> >
> >> >>> >> >
> >> >>> >> >Stack, what's your feeling w.r.t. to HBASE-2600, will
> >> > keeping the 0.90
> >> >>> >> -> 0.92 migration path
> >> >>> >> >make the migration code for HBASE-2600 (much) more
> >> > complicated in 0.94?
> >> >>> >> >
> >> >>> >> >
> >> >>> >> >-- Lars
> >> >>> >> >
> >> >>> >> >
> >> >>> >> >
> >> >>> >> >----- Original Message -----
> >> >>> >> >From: Stack <st...@duboce.net>
> >> >>> >> >To: dev@hbase.apache.org
> >> >>> >> >Cc:
> >> >>> >> >Sent: Friday, January 27, 2012 9:02 AM
> >> >>> >> >Subject: Re: hbase 0.94.0
> >> >>> >> >
> >> >>> >> >
> >> >>> >> >On Fri, Jan 27, 2012 at 8:42 AM, Stack
> >> > <st...@duboce.net> wrote:
> >> >>> >> >> In this particular case, there was no explicit
> >> > migration step needed
> >> >>> >> >> going 0.90 to 0.92. Upgrading from 0.90 to 0.94
> >> > should just be
> >> >>> >> >> running the 0.92 to 0.94 migration script (if there
> >> > is one).
> >> >>> >> >>
> >> >>> >> >
> >> >>> >> >Thinking more, the above only really holds if we keep the
> >> > .META.
> >> >>> >> >migration code that runs in 0.92 on startup on into 0.94
> >> > (I have a
> >> >>> >> >patch here where I'm removing it... I should put this
> >> > bit of code
> >> >>> >> >back).
> >> >>> >> >
> >> >>> >> >I see Todd that you vote against removing hfile v1 in
> >> > 0.94. To make
> >> >>> >> >going from CDH3 to CDH4, we should not purge any migrating
> >> > code or old
> >> >>> >> >version support.
> >> >>> >> >
> >> >>> >> >I'd be fine w/ that. We'll need to work hard to
> >> > ensure it taking it
> >> >>> >> >on as a principal for 0.94. Ok w/ you "Iron
> >> > Hand" Lars?
> >> >>> >> >
> >> >>> >> >St.Ack
> >> >>> >> >
> >> >>> >> >
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >> // Jonathan Hsieh (shay)
> >> >>> >> // Software Engineer, Cloudera
> >> >>> >>
> >> >>> >> // jon@cloudera.com
> >> >>> >>
> >> >>> >
> >> >>>
> >> >>>
> >> >>> --
> >> >>> // Jonathan Hsieh (shay)
> >> >>> // Software Engineer, Cloudera
> >> >>> // jon@cloudera.com
> >> >>>
> >> >
> >> >
> >> >
> >> > --
> >> > Todd Lipcon
> >> > Software Engineer, Cloudera
> >> >
> >>
>
>
Re: hbase 0.94.0
Posted by Nicolas Spiegelberg <ns...@fb.com>.
I think it's also good to make a crucial distinction here: it's not like
Facebook has a concrete wide-scale upgrade strategy, these 3 points are
deal-breakers that won't allow us to even entertain an upgrade strategy.
This is just as critical as HDFS data loss was in 0.90: it's something we
can't entertain & is a mandatory hurdle to trunk being considered a
realistic option. That said, I personally would like to upgrade our
internal version to more closely align with other developers. It will
reduce our porting overhead and increase the amount of benefit we can give
the community & visa-versa.
On 1/30/12 11:46 AM, "Ted Yu" <yu...@gmail.com> wrote:
>Before long term wire and format compatibility is reached, I am -0 on time
>based release.
>
>As Nicolas pointed out, FB wants to migrate from 0.89-fb to 0.94
>If we do time-based releases without wire and format compatibility, it
>would consume a lot of our energy satisfying such requirements from
>different companies.
>
>I propose wire and format compatibility as the main theme for 0.96
>
>Cheers
>
>On Mon, Jan 30, 2012 at 11:40 AM, Andrew Purtell
><ap...@apache.org>wrote:
>
>> > Maybe the discussion should be whether 0.94 should be a time-gated or
>> > feature-gated release? IMO we already have enough good stuff in there
>> > on the perf front. It will be great if the checksum improvement makes
>> > it, but if it doesn't, I'd rather have a release than delay for it.
>>
>>
>> Time gated releases makes sense going forward for a few reasons, mark me
>> down for that option.
>>
>>
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
>>
>> ----- Original Message -----
>> > From: Todd Lipcon <to...@cloudera.com>
>> > To: dev@hbase.apache.org
>> > Cc: lars hofhansl <lh...@yahoo.com>; Matt Corgan <
>> mcorgan@hotpads.com>
>> > Sent: Monday, January 30, 2012 10:44 AM
>> > Subject: Re: hbase 0.94.0
>> >
>> > On Sun, Jan 29, 2012 at 7:30 AM, Ted Yu <yu...@gmail.com> wrote:
>> >> Initial results from HBASE-5074, support checksums in HBase block
>> cache,
>> >> look promising.
>> >>
>> >> Shall we discuss whether 0.94 should include it (assuming Dhruba can
>> finish
>> >> the feature in Feb.) ?
>> >
>> > If it's a time-based release, then there's nothing to discuss. Either
>> > it's done in time, in which case it's part of 94, or it's not, in
>> > which case it's part of 96, right?
>> >
>> > Maybe the discussion should be whether 0.94 should be a time-gated or
>> > feature-gated release? IMO we already have enough good stuff in there
>> > on the perf front. It will be great if the checksum improvement makes
>> > it, but if it doesn't, I'd rather have a release than delay for it.
>> >
>> > -Todd
>> >
>> >>
>> >> Cheers
>> >>
>> >> On Fri, Jan 27, 2012 at 6:27 PM, lars hofhansl <lh...@yahoo.com>
>> > wrote:
>> >>
>> >>> Salesforce will jumpstart with 0.94. :) Our dev cluster has the
>> current
>> >>> trunk on it.
>> >>>
>> >>> OK let's just add these to 0.94:
>> >>> HBASE-4218
>> >>>
>> >>> HBASE-4608
>> >>> HBASE-5128
>> >>> script necessary to do HBASE-5293 in 0.96
>> >>>
>> >>>
>> >>> And defer the rest to 0.96, and branch 0.94 soon'ish.
>> >>> Do you think you can do a 0.92 and trunk version of HBASE-5128?
>> >>>
>> >>> -- Lars
>> >>>
>> >>>
>> >>> ________________________________
>> >>> From: Jonathan Hsieh <jo...@cloudera.com>
>> >>> To: Matt Corgan <mc...@hotpads.com>
>> >>> Cc: lars hofhansl <lh...@yahoo.com>; dev
>> > <de...@hbase.apache.org>
>> >>> Sent: Friday, January 27, 2012 6:11 PM
>> >>> Subject: Re: hbase 0.94.0
>> >>>
>> >>> Lars, Matt,
>> >>>
>> >>> I like Matt's suggestion -- as long as it is not a burden let's
>> > keep it in.
>> >>> If it becomes one, let's do what is best for the project.
>> >>>
>> >>> If Cloudera needs to do the upgrade path, we'll provide one. If it
>> > doesn't
>> >>> go to the apache repo, we'll most likely open source it on github
>> > or
>> >>> something, or "keep it on the jira" like some of the repair
>> > ruby scripts.
>> >>>
>> >>> Question -- to the Trend/SalesForce/FB folks can you talk about
>>your
>> > plans
>> >>> for upgrades? Are you guys moving from 0.89/0.90ish to 0.92
>>already?
>> > or
>> >>> are you guys going to have to do the direct upgrade to 0.94 from
>>0.90
>> > land
>> >>> as well?
>> >>>
>> >>> Jon.
>> >>>
>> >>>
>> >>> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan
>> > <mc...@hotpads.com> wrote:
>> >>>
>> >>> > The reason I brought it up is now that Mikhail committed the data
>> > block
>> >>> > encoding I was going to take a stab at adding the prefix trie
>> > encoding I
>> >>> > was working on this past summer. My plan is to first make a
>> > minimally
>> >>> > invasive patch to prove correctness. But, after that there will
>> > probably
>> >>> > be some big performance gains to be had from reworking some of
>>the
>> > things
>> >>> > like KeyValueScanner which I would not have the courage to get
>> > working
>> >>> with
>> >>> > v1.
>> >>> >
>> >>> > So, that was why I asked, but all of that is still more
>> > hypothetical than
>> >>> > real, and I don't even know if the first part will be done
>> > before
>> >>> branching
>> >>> > .94 at the end of February. Makes sense to me to not delete v1
>> > until
>> >>> > there's a good reason to, which it doesn't look like we
>> > have yet. If I
>> >>> get
>> >>> > to the point where v1 is halting progress then we can reevaluate
>> > based on
>> >>> > more specific issues. Maybe none of the prefix trie will even
>> > make
>> >>> .94...
>> >>> >
>> >>> > ..sent from my phone
>> >>> > On Jan 27, 2012 1:55 PM, "lars hofhansl"
>> > <lh...@yahoo.com> wrote:
>> >>> >
>> >>> >> Hey Jon,
>> >>> >>
>> >>> >> understood. Makes 0.94 hard, though. If we decide now to have
>> > a 0.90 to
>> >>> >> 0.94 upgrade path and then timing does not work out and nobody
>> > signs up
>> >>> for
>> >>> >> the testing because it 0.92 is more convenient we'd have
>> > gone through
>> >>> this
>> >>> >> for nothing.
>> >>> >>
>> >>> >> So... Thinking about this more I am -1 on supporting an
>> > official upgrade
>> >>> >> path other then from one release to the next.
>> >>> >> That said, we do not have to break things intentionally.
>> >>> >>
>> >>> >>
>> >>> >> I'm fine pushing HBASE-2600 and HFile v1 removal out of
>> > 0.94... As long
>> >>> >> as we won't havethe same argument for 0.96 :)
>> >>> >>
>> >>> >> And I am not aware of any file compatibility issues.
>> >>> >>
>> >>> >>
>> >>> >> We can also leave the 0.92 migration code in, but not
>> > officially support
>> >>> >> it as 0.90 to 0.94 path.
>> >>> >> Then CDH4 can make sure (if needed) that it is all working.
>> >>> >>
>> >>> >>
>> >>> >> Does that work for you guys?
>> >>> >>
>> >>> >> -- Lars
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> ________________________________
>> >>> >> From: Jonathan Hsieh <jo...@cloudera.com>
>> >>> >> To: dev@hbase.apache.org; lars hofhansl
>> > <lh...@yahoo.com>
>> >>> >> Sent: Friday, January 27, 2012 10:10 AM
>> >>> >> Subject: Re: hbase 0.94.0
>> >>> >>
>> >>> >>
>> >>> >> Lars,
>> >>> >>
>> >>> >> The upcoming CDH4 Beta HBase will be based on the latest
>> > hotness,
>> >>> 0.92.0.
>> >>> >> A CDH4 GA HBase will have to have an upgrade path from CDH3
>> > HBase. If
>> >>> that
>> >>> >> HBase is 0.92 based, we'll test that, and if timing works
>> > out and we
>> >>> decide
>> >>> >> 0.94, we'll have to have a path (0.90->0.94) for than
>> > and will test
>> >>> that.
>> >>> >>
>> >>> >> HBASE-2600 is a big change of encoding of meta, while my
>> > understanding
>> >>> is
>> >>> >> that 0.90->0,92 is a graceful HFile format conversion. Are
>> > there are
>> >>> >> things currently in trunk that further break compatiblity of
>> > the file
>> >>> >> format? (what jiras?)
>> >>> >>
>> >>> >> Jon.
>> >>> >>
>> >>> >>
>> >>> >> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl
>> > <lh...@yahoo.com>
>> >>> >> wrote:
>> >>> >>
>> >>> >> Not removing code for upgrade is fine.
>> >>> >> >
>> >>> >> >Todd, is Cloudera signing up for testing this path (0.90
>> > to 0.94)?
>> >>> >> >
>> >>> >> >
>> >>> >> >Stack, what's your feeling w.r.t. to HBASE-2600, will
>> > keeping the 0.90
>> >>> >> -> 0.92 migration path
>> >>> >> >make the migration code for HBASE-2600 (much) more
>> > complicated in 0.94?
>> >>> >> >
>> >>> >> >
>> >>> >> >-- Lars
>> >>> >> >
>> >>> >> >
>> >>> >> >
>> >>> >> >----- Original Message -----
>> >>> >> >From: Stack <st...@duboce.net>
>> >>> >> >To: dev@hbase.apache.org
>> >>> >> >Cc:
>> >>> >> >Sent: Friday, January 27, 2012 9:02 AM
>> >>> >> >Subject: Re: hbase 0.94.0
>> >>> >> >
>> >>> >> >
>> >>> >> >On Fri, Jan 27, 2012 at 8:42 AM, Stack
>> > <st...@duboce.net> wrote:
>> >>> >> >> In this particular case, there was no explicit
>> > migration step needed
>> >>> >> >> going 0.90 to 0.92. Upgrading from 0.90 to 0.94
>> > should just be
>> >>> >> >> running the 0.92 to 0.94 migration script (if there
>> > is one).
>> >>> >> >>
>> >>> >> >
>> >>> >> >Thinking more, the above only really holds if we keep the
>> > .META.
>> >>> >> >migration code that runs in 0.92 on startup on into 0.94
>> > (I have a
>> >>> >> >patch here where I'm removing it... I should put this
>> > bit of code
>> >>> >> >back).
>> >>> >> >
>> >>> >> >I see Todd that you vote against removing hfile v1 in
>> > 0.94. To make
>> >>> >> >going from CDH3 to CDH4, we should not purge any migrating
>> > code or old
>> >>> >> >version support.
>> >>> >> >
>> >>> >> >I'd be fine w/ that. We'll need to work hard to
>> > ensure it taking it
>> >>> >> >on as a principal for 0.94. Ok w/ you "Iron
>> > Hand" Lars?
>> >>> >> >
>> >>> >> >St.Ack
>> >>> >> >
>> >>> >> >
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> // Jonathan Hsieh (shay)
>> >>> >> // Software Engineer, Cloudera
>> >>> >>
>> >>> >> // jon@cloudera.com
>> >>> >>
>> >>> >
>> >>>
>> >>>
>> >>> --
>> >>> // Jonathan Hsieh (shay)
>> >>> // Software Engineer, Cloudera
>> >>> // jon@cloudera.com
>> >>>
>> >
>> >
>> >
>> > --
>> > Todd Lipcon
>> > Software Engineer, Cloudera
>> >
>>
Re: hbase 0.94.0
Posted by Andrew Purtell <ap...@apache.org>.
> If we do time-based releases without wire and format compatibility, it
would consume a lot of our energy satisfying such requirements from
different companies.
Since currently we do not have guarantees of wire and format compatibility across major releases, only that of a migration step, this isn't really relevant to the question of time gated releasing or not.
Adding that requirement would have to happen first.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
> From: Ted Yu <yu...@gmail.com>
>To: dev@hbase.apache.org; Andrew Purtell <ap...@apache.org>
>Cc: lars hofhansl <lh...@yahoo.com>; Matt Corgan <mc...@hotpads.com>
>Sent: Monday, January 30, 2012 11:46 AM
>Subject: Re: hbase 0.94.0
>
>
>Before long term wire and format compatibility is reached, I am -0 on time based release.
>
>As Nicolas pointed out, FB wants to migrate from 0.89-fb to 0.94
>If we do time-based releases without wire and format compatibility, it would consume a lot of our energy satisfying such requirements from different companies.
>
>I propose wire and format compatibility as the main theme for 0.96
>
>Cheers
>
>
>On Mon, Jan 30, 2012 at 11:40 AM, Andrew Purtell <ap...@apache.org> wrote:
>
>> Maybe the discussion should be whether 0.94 should be a time-gated or
>>> feature-gated release? IMO we already have enough good stuff in there
>>> on the perf front. It will be great if the checksum improvement makes
>>> it, but if it doesn't, I'd rather have a release than delay for it.
>>
>>
>>Time gated releases makes sense going forward for a few reasons, mark me down for that option.
>>
>>
>>
>>Best regards,
>>
>> - Andy
>>
>>Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>>
>>
>>
>>----- Original Message -----
>>> From: Todd Lipcon <to...@cloudera.com>
>>> To: dev@hbase.apache.org
>>
>>> Cc: lars hofhansl <lh...@yahoo.com>; Matt Corgan <mc...@hotpads.com>
>>> Sent: Monday, January 30, 2012 10:44 AM
>>> Subject: Re: hbase 0.94.0
>>>
>>> On Sun, Jan 29, 2012 at 7:30 AM, Ted Yu <yu...@gmail.com> wrote:
>>>> Initial results from HBASE-5074, support checksums in HBase block cache,
>>>> look promising.
>>>>
>>>> Shall we discuss whether 0.94 should include it (assuming Dhruba can finish
>>>> the feature in Feb.) ?
>>>
>>> If it's a time-based release, then there's nothing to discuss. Either
>>> it's done in time, in which case it's part of 94, or it's not, in
>>> which case it's part of 96, right?
>>>
>>> Maybe the discussion should be whether 0.94 should be a time-gated or
>>> feature-gated release? IMO we already have enough good stuff in there
>>> on the perf front. It will be great if the checksum improvement makes
>>> it, but if it doesn't, I'd rather have a release than delay for it.
>>>
>>> -Todd
>>>
>>>>
>>>> Cheers
>>>>
>>>> On Fri, Jan 27, 2012 at 6:27 PM, lars hofhansl <lh...@yahoo.com>
>>> wrote:
>>>>
>>>>> Salesforce will jumpstart with 0.94. :) Our dev cluster has the current
>>>>> trunk on it.
>>>>>
>>>>> OK let's just add these to 0.94:
>>>>> HBASE-4218
>>>>>
>>>>> HBASE-4608
>>>>> HBASE-5128
>>>>> script necessary to do HBASE-5293 in 0.96
>>>>>
>>>>>
>>>>> And defer the rest to 0.96, and branch 0.94 soon'ish.
>>>>> Do you think you can do a 0.92 and trunk version of HBASE-5128?
>>>>>
>>>>> -- Lars
>>>>>
>>>>>
>>>>> ________________________________
>>>>> From: Jonathan Hsieh <jo...@cloudera.com>
>>>>> To: Matt Corgan <mc...@hotpads.com>
>>>>> Cc: lars hofhansl <lh...@yahoo.com>; dev
>>> <de...@hbase.apache.org>
>>>>> Sent: Friday, January 27, 2012 6:11 PM
>>>>> Subject: Re: hbase 0.94.0
>>>>>
>>>>> Lars, Matt,
>>>>>
>>>>> I like Matt's suggestion -- as long as it is not a burden let's
>>> keep it in.
>>>>> If it becomes one, let's do what is best for the project.
>>>>>
>>>>> If Cloudera needs to do the upgrade path, we'll provide one. If it
>>> doesn't
>>>>> go to the apache repo, we'll most likely open source it on github
>>> or
>>>>> something, or "keep it on the jira" like some of the repair
>>> ruby scripts.
>>>>>
>>>>> Question -- to the Trend/SalesForce/FB folks can you talk about your
>>> plans
>>>>> for upgrades? Are you guys moving from 0.89/0.90ish to 0.92 already?
>>> or
>>>>> are you guys going to have to do the direct upgrade to 0.94 from 0.90
>>> land
>>>>> as well?
>>>>>
>>>>> Jon.
>>>>>
>>>>>
>>>>> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan
>>> <mc...@hotpads.com> wrote:
>>>>>
>>>>> > The reason I brought it up is now that Mikhail committed the data
>>> block
>>>>> > encoding I was going to take a stab at adding the prefix trie
>>> encoding I
>>>>> > was working on this past summer. My plan is to first make a
>>> minimally
>>>>> > invasive patch to prove correctness. But, after that there will
>>> probably
>>>>> > be some big performance gains to be had from reworking some of the
>>> things
>>>>> > like KeyValueScanner which I would not have the courage to get
>>> working
>>>>> with
>>>>> > v1.
>>>>> >
>>>>> > So, that was why I asked, but all of that is still more
>>> hypothetical than
>>>>> > real, and I don't even know if the first part will be done
>>> before
>>>>> branching
>>>>> > .94 at the end of February. Makes sense to me to not delete v1
>>> until
>>>>> > there's a good reason to, which it doesn't look like we
>>> have yet. If I
>>>>> get
>>>>> > to the point where v1 is halting progress then we can reevaluate
>>> based on
>>>>> > more specific issues. Maybe none of the prefix trie will even
>>> make
>>>>> .94...
>>>>> >
>>>>> > ..sent from my phone
>>>>> > On Jan 27, 2012 1:55 PM, "lars hofhansl"
>>> <lh...@yahoo.com> wrote:
>>>>> >
>>>>> >> Hey Jon,
>>>>> >>
>>>>> >> understood. Makes 0.94 hard, though. If we decide now to have
>>> a 0.90 to
>>>>> >> 0.94 upgrade path and then timing does not work out and nobody
>>> signs up
>>>>> for
>>>>> >> the testing because it 0.92 is more convenient we'd have
>>> gone through
>>>>> this
>>>>> >> for nothing.
>>>>> >>
>>>>> >> So... Thinking about this more I am -1 on supporting an
>>> official upgrade
>>>>> >> path other then from one release to the next.
>>>>> >> That said, we do not have to break things intentionally.
>>>>> >>
>>>>> >>
>>>>> >> I'm fine pushing HBASE-2600 and HFile v1 removal out of
>>> 0.94... As long
>>>>> >> as we won't havethe same argument for 0.96 :)
>>>>> >>
>>>>> >> And I am not aware of any file compatibility issues.
>>>>> >>
>>>>> >>
>>>>> >> We can also leave the 0.92 migration code in, but not
>>> officially support
>>>>> >> it as 0.90 to 0.94 path.
>>>>> >> Then CDH4 can make sure (if needed) that it is all working.
>>>>> >>
>>>>> >>
>>>>> >> Does that work for you guys?
>>>>> >>
>>>>> >> -- Lars
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> ________________________________
>>>>> >> From: Jonathan Hsieh <jo...@cloudera.com>
>>>>> >> To: dev@hbase.apache.org; lars hofhansl
>>> <lh...@yahoo.com>
>>>>> >> Sent: Friday, January 27, 2012 10:10 AM
>>>>> >> Subject: Re: hbase 0.94.0
>>>>> >>
>>>>> >>
>>>>> >> Lars,
>>>>> >>
>>>>> >> The upcoming CDH4 Beta HBase will be based on the latest
>>> hotness,
>>>>> 0.92.0.
>>>>> >> A CDH4 GA HBase will have to have an upgrade path from CDH3
>>> HBase. If
>>>>> that
>>>>> >> HBase is 0.92 based, we'll test that, and if timing works
>>> out and we
>>>>> decide
>>>>> >> 0.94, we'll have to have a path (0.90->0.94) for than
>>> and will test
>>>>> that.
>>>>> >>
>>>>> >> HBASE-2600 is a big change of encoding of meta, while my
>>> understanding
>>>>> is
>>>>> >> that 0.90->0,92 is a graceful HFile format conversion. Are
>>> there are
>>>>> >> things currently in trunk that further break compatiblity of
>>> the file
>>>>> >> format? (what jiras?)
>>>>> >>
>>>>> >> Jon.
>>>>> >>
>>>>> >>
>>>>> >> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl
>>> <lh...@yahoo.com>
>>>>> >> wrote:
>>>>> >>
>>>>> >> Not removing code for upgrade is fine.
>>>>> >> >
>>>>> >> >Todd, is Cloudera signing up for testing this path (0.90
>>> to 0.94)?
>>>>> >> >
>>>>> >> >
>>>>> >> >Stack, what's your feeling w.r.t. to HBASE-2600, will
>>> keeping the 0.90
>>>>> >> -> 0.92 migration path
>>>>> >> >make the migration code for HBASE-2600 (much) more
>>> complicated in 0.94?
>>>>> >> >
>>>>> >> >
>>>>> >> >-- Lars
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >----- Original Message -----
>>>>> >> >From: Stack <st...@duboce.net>
>>>>> >> >To: dev@hbase.apache.org
>>>>> >> >Cc:
>>>>> >> >Sent: Friday, January 27, 2012 9:02 AM
>>>>> >> >Subject: Re: hbase 0.94.0
>>>>> >> >
>>>>> >> >
>>>>> >> >On Fri, Jan 27, 2012 at 8:42 AM, Stack
>>> <st...@duboce.net> wrote:
>>>>> >> >> In this particular case, there was no explicit
>>> migration step needed
>>>>> >> >> going 0.90 to 0.92. Upgrading from 0.90 to 0.94
>>> should just be
>>>>> >> >> running the 0.92 to 0.94 migration script (if there
>>> is one).
>>>>> >> >>
>>>>> >> >
>>>>> >> >Thinking more, the above only really holds if we keep the
>>> .META.
>>>>> >> >migration code that runs in 0.92 on startup on into 0.94
>>> (I have a
>>>>> >> >patch here where I'm removing it... I should put this
>>> bit of code
>>>>> >> >back).
>>>>> >> >
>>>>> >> >I see Todd that you vote against removing hfile v1 in
>>> 0.94. To make
>>>>> >> >going from CDH3 to CDH4, we should not purge any migrating
>>> code or old
>>>>> >> >version support.
>>>>> >> >
>>>>> >> >I'd be fine w/ that. We'll need to work hard to
>>> ensure it taking it
>>>>> >> >on as a principal for 0.94. Ok w/ you "Iron
>>> Hand" Lars?
>>>>> >> >
>>>>> >> >St.Ack
>>>>> >> >
>>>>> >> >
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> // Jonathan Hsieh (shay)
>>>>> >> // Software Engineer, Cloudera
>>>>> >>
>>>>> >> // jon@cloudera.com
>>>>> >>
>>>>> >
>>>>>
>>>>>
>>>>> --
>>>>> // Jonathan Hsieh (shay)
>>>>> // Software Engineer, Cloudera
>>>>> // jon@cloudera.com
>>>>>
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>
>
>
>
Re: hbase 0.94.0
Posted by Ted Yu <yu...@gmail.com>.
Before long term wire and format compatibility is reached, I am -0 on time
based release.
As Nicolas pointed out, FB wants to migrate from 0.89-fb to 0.94
If we do time-based releases without wire and format compatibility, it
would consume a lot of our energy satisfying such requirements from
different companies.
I propose wire and format compatibility as the main theme for 0.96
Cheers
On Mon, Jan 30, 2012 at 11:40 AM, Andrew Purtell <ap...@apache.org>wrote:
> > Maybe the discussion should be whether 0.94 should be a time-gated or
> > feature-gated release? IMO we already have enough good stuff in there
> > on the perf front. It will be great if the checksum improvement makes
> > it, but if it doesn't, I'd rather have a release than delay for it.
>
>
> Time gated releases makes sense going forward for a few reasons, mark me
> down for that option.
>
>
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
>
> ----- Original Message -----
> > From: Todd Lipcon <to...@cloudera.com>
> > To: dev@hbase.apache.org
> > Cc: lars hofhansl <lh...@yahoo.com>; Matt Corgan <
> mcorgan@hotpads.com>
> > Sent: Monday, January 30, 2012 10:44 AM
> > Subject: Re: hbase 0.94.0
> >
> > On Sun, Jan 29, 2012 at 7:30 AM, Ted Yu <yu...@gmail.com> wrote:
> >> Initial results from HBASE-5074, support checksums in HBase block
> cache,
> >> look promising.
> >>
> >> Shall we discuss whether 0.94 should include it (assuming Dhruba can
> finish
> >> the feature in Feb.) ?
> >
> > If it's a time-based release, then there's nothing to discuss. Either
> > it's done in time, in which case it's part of 94, or it's not, in
> > which case it's part of 96, right?
> >
> > Maybe the discussion should be whether 0.94 should be a time-gated or
> > feature-gated release? IMO we already have enough good stuff in there
> > on the perf front. It will be great if the checksum improvement makes
> > it, but if it doesn't, I'd rather have a release than delay for it.
> >
> > -Todd
> >
> >>
> >> Cheers
> >>
> >> On Fri, Jan 27, 2012 at 6:27 PM, lars hofhansl <lh...@yahoo.com>
> > wrote:
> >>
> >>> Salesforce will jumpstart with 0.94. :) Our dev cluster has the
> current
> >>> trunk on it.
> >>>
> >>> OK let's just add these to 0.94:
> >>> HBASE-4218
> >>>
> >>> HBASE-4608
> >>> HBASE-5128
> >>> script necessary to do HBASE-5293 in 0.96
> >>>
> >>>
> >>> And defer the rest to 0.96, and branch 0.94 soon'ish.
> >>> Do you think you can do a 0.92 and trunk version of HBASE-5128?
> >>>
> >>> -- Lars
> >>>
> >>>
> >>> ________________________________
> >>> From: Jonathan Hsieh <jo...@cloudera.com>
> >>> To: Matt Corgan <mc...@hotpads.com>
> >>> Cc: lars hofhansl <lh...@yahoo.com>; dev
> > <de...@hbase.apache.org>
> >>> Sent: Friday, January 27, 2012 6:11 PM
> >>> Subject: Re: hbase 0.94.0
> >>>
> >>> Lars, Matt,
> >>>
> >>> I like Matt's suggestion -- as long as it is not a burden let's
> > keep it in.
> >>> If it becomes one, let's do what is best for the project.
> >>>
> >>> If Cloudera needs to do the upgrade path, we'll provide one. If it
> > doesn't
> >>> go to the apache repo, we'll most likely open source it on github
> > or
> >>> something, or "keep it on the jira" like some of the repair
> > ruby scripts.
> >>>
> >>> Question -- to the Trend/SalesForce/FB folks can you talk about your
> > plans
> >>> for upgrades? Are you guys moving from 0.89/0.90ish to 0.92 already?
> > or
> >>> are you guys going to have to do the direct upgrade to 0.94 from 0.90
> > land
> >>> as well?
> >>>
> >>> Jon.
> >>>
> >>>
> >>> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan
> > <mc...@hotpads.com> wrote:
> >>>
> >>> > The reason I brought it up is now that Mikhail committed the data
> > block
> >>> > encoding I was going to take a stab at adding the prefix trie
> > encoding I
> >>> > was working on this past summer. My plan is to first make a
> > minimally
> >>> > invasive patch to prove correctness. But, after that there will
> > probably
> >>> > be some big performance gains to be had from reworking some of the
> > things
> >>> > like KeyValueScanner which I would not have the courage to get
> > working
> >>> with
> >>> > v1.
> >>> >
> >>> > So, that was why I asked, but all of that is still more
> > hypothetical than
> >>> > real, and I don't even know if the first part will be done
> > before
> >>> branching
> >>> > .94 at the end of February. Makes sense to me to not delete v1
> > until
> >>> > there's a good reason to, which it doesn't look like we
> > have yet. If I
> >>> get
> >>> > to the point where v1 is halting progress then we can reevaluate
> > based on
> >>> > more specific issues. Maybe none of the prefix trie will even
> > make
> >>> .94...
> >>> >
> >>> > ..sent from my phone
> >>> > On Jan 27, 2012 1:55 PM, "lars hofhansl"
> > <lh...@yahoo.com> wrote:
> >>> >
> >>> >> Hey Jon,
> >>> >>
> >>> >> understood. Makes 0.94 hard, though. If we decide now to have
> > a 0.90 to
> >>> >> 0.94 upgrade path and then timing does not work out and nobody
> > signs up
> >>> for
> >>> >> the testing because it 0.92 is more convenient we'd have
> > gone through
> >>> this
> >>> >> for nothing.
> >>> >>
> >>> >> So... Thinking about this more I am -1 on supporting an
> > official upgrade
> >>> >> path other then from one release to the next.
> >>> >> That said, we do not have to break things intentionally.
> >>> >>
> >>> >>
> >>> >> I'm fine pushing HBASE-2600 and HFile v1 removal out of
> > 0.94... As long
> >>> >> as we won't havethe same argument for 0.96 :)
> >>> >>
> >>> >> And I am not aware of any file compatibility issues.
> >>> >>
> >>> >>
> >>> >> We can also leave the 0.92 migration code in, but not
> > officially support
> >>> >> it as 0.90 to 0.94 path.
> >>> >> Then CDH4 can make sure (if needed) that it is all working.
> >>> >>
> >>> >>
> >>> >> Does that work for you guys?
> >>> >>
> >>> >> -- Lars
> >>> >>
> >>> >>
> >>> >>
> >>> >> ________________________________
> >>> >> From: Jonathan Hsieh <jo...@cloudera.com>
> >>> >> To: dev@hbase.apache.org; lars hofhansl
> > <lh...@yahoo.com>
> >>> >> Sent: Friday, January 27, 2012 10:10 AM
> >>> >> Subject: Re: hbase 0.94.0
> >>> >>
> >>> >>
> >>> >> Lars,
> >>> >>
> >>> >> The upcoming CDH4 Beta HBase will be based on the latest
> > hotness,
> >>> 0.92.0.
> >>> >> A CDH4 GA HBase will have to have an upgrade path from CDH3
> > HBase. If
> >>> that
> >>> >> HBase is 0.92 based, we'll test that, and if timing works
> > out and we
> >>> decide
> >>> >> 0.94, we'll have to have a path (0.90->0.94) for than
> > and will test
> >>> that.
> >>> >>
> >>> >> HBASE-2600 is a big change of encoding of meta, while my
> > understanding
> >>> is
> >>> >> that 0.90->0,92 is a graceful HFile format conversion. Are
> > there are
> >>> >> things currently in trunk that further break compatiblity of
> > the file
> >>> >> format? (what jiras?)
> >>> >>
> >>> >> Jon.
> >>> >>
> >>> >>
> >>> >> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl
> > <lh...@yahoo.com>
> >>> >> wrote:
> >>> >>
> >>> >> Not removing code for upgrade is fine.
> >>> >> >
> >>> >> >Todd, is Cloudera signing up for testing this path (0.90
> > to 0.94)?
> >>> >> >
> >>> >> >
> >>> >> >Stack, what's your feeling w.r.t. to HBASE-2600, will
> > keeping the 0.90
> >>> >> -> 0.92 migration path
> >>> >> >make the migration code for HBASE-2600 (much) more
> > complicated in 0.94?
> >>> >> >
> >>> >> >
> >>> >> >-- Lars
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> >----- Original Message -----
> >>> >> >From: Stack <st...@duboce.net>
> >>> >> >To: dev@hbase.apache.org
> >>> >> >Cc:
> >>> >> >Sent: Friday, January 27, 2012 9:02 AM
> >>> >> >Subject: Re: hbase 0.94.0
> >>> >> >
> >>> >> >
> >>> >> >On Fri, Jan 27, 2012 at 8:42 AM, Stack
> > <st...@duboce.net> wrote:
> >>> >> >> In this particular case, there was no explicit
> > migration step needed
> >>> >> >> going 0.90 to 0.92. Upgrading from 0.90 to 0.94
> > should just be
> >>> >> >> running the 0.92 to 0.94 migration script (if there
> > is one).
> >>> >> >>
> >>> >> >
> >>> >> >Thinking more, the above only really holds if we keep the
> > .META.
> >>> >> >migration code that runs in 0.92 on startup on into 0.94
> > (I have a
> >>> >> >patch here where I'm removing it... I should put this
> > bit of code
> >>> >> >back).
> >>> >> >
> >>> >> >I see Todd that you vote against removing hfile v1 in
> > 0.94. To make
> >>> >> >going from CDH3 to CDH4, we should not purge any migrating
> > code or old
> >>> >> >version support.
> >>> >> >
> >>> >> >I'd be fine w/ that. We'll need to work hard to
> > ensure it taking it
> >>> >> >on as a principal for 0.94. Ok w/ you "Iron
> > Hand" Lars?
> >>> >> >
> >>> >> >St.Ack
> >>> >> >
> >>> >> >
> >>> >>
> >>> >>
> >>> >> --
> >>> >> // Jonathan Hsieh (shay)
> >>> >> // Software Engineer, Cloudera
> >>> >>
> >>> >> // jon@cloudera.com
> >>> >>
> >>> >
> >>>
> >>>
> >>> --
> >>> // Jonathan Hsieh (shay)
> >>> // Software Engineer, Cloudera
> >>> // jon@cloudera.com
> >>>
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>
Re: hbase 0.94.0
Posted by Andrew Purtell <ap...@apache.org>.
> Maybe the discussion should be whether 0.94 should be a time-gated or
> feature-gated release? IMO we already have enough good stuff in there
> on the perf front. It will be great if the checksum improvement makes
> it, but if it doesn't, I'd rather have a release than delay for it.
Time gated releases makes sense going forward for a few reasons, mark me down for that option.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
----- Original Message -----
> From: Todd Lipcon <to...@cloudera.com>
> To: dev@hbase.apache.org
> Cc: lars hofhansl <lh...@yahoo.com>; Matt Corgan <mc...@hotpads.com>
> Sent: Monday, January 30, 2012 10:44 AM
> Subject: Re: hbase 0.94.0
>
> On Sun, Jan 29, 2012 at 7:30 AM, Ted Yu <yu...@gmail.com> wrote:
>> Initial results from HBASE-5074, support checksums in HBase block cache,
>> look promising.
>>
>> Shall we discuss whether 0.94 should include it (assuming Dhruba can finish
>> the feature in Feb.) ?
>
> If it's a time-based release, then there's nothing to discuss. Either
> it's done in time, in which case it's part of 94, or it's not, in
> which case it's part of 96, right?
>
> Maybe the discussion should be whether 0.94 should be a time-gated or
> feature-gated release? IMO we already have enough good stuff in there
> on the perf front. It will be great if the checksum improvement makes
> it, but if it doesn't, I'd rather have a release than delay for it.
>
> -Todd
>
>>
>> Cheers
>>
>> On Fri, Jan 27, 2012 at 6:27 PM, lars hofhansl <lh...@yahoo.com>
> wrote:
>>
>>> Salesforce will jumpstart with 0.94. :) Our dev cluster has the current
>>> trunk on it.
>>>
>>> OK let's just add these to 0.94:
>>> HBASE-4218
>>>
>>> HBASE-4608
>>> HBASE-5128
>>> script necessary to do HBASE-5293 in 0.96
>>>
>>>
>>> And defer the rest to 0.96, and branch 0.94 soon'ish.
>>> Do you think you can do a 0.92 and trunk version of HBASE-5128?
>>>
>>> -- Lars
>>>
>>>
>>> ________________________________
>>> From: Jonathan Hsieh <jo...@cloudera.com>
>>> To: Matt Corgan <mc...@hotpads.com>
>>> Cc: lars hofhansl <lh...@yahoo.com>; dev
> <de...@hbase.apache.org>
>>> Sent: Friday, January 27, 2012 6:11 PM
>>> Subject: Re: hbase 0.94.0
>>>
>>> Lars, Matt,
>>>
>>> I like Matt's suggestion -- as long as it is not a burden let's
> keep it in.
>>> If it becomes one, let's do what is best for the project.
>>>
>>> If Cloudera needs to do the upgrade path, we'll provide one. If it
> doesn't
>>> go to the apache repo, we'll most likely open source it on github
> or
>>> something, or "keep it on the jira" like some of the repair
> ruby scripts.
>>>
>>> Question -- to the Trend/SalesForce/FB folks can you talk about your
> plans
>>> for upgrades? Are you guys moving from 0.89/0.90ish to 0.92 already?
> or
>>> are you guys going to have to do the direct upgrade to 0.94 from 0.90
> land
>>> as well?
>>>
>>> Jon.
>>>
>>>
>>> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan
> <mc...@hotpads.com> wrote:
>>>
>>> > The reason I brought it up is now that Mikhail committed the data
> block
>>> > encoding I was going to take a stab at adding the prefix trie
> encoding I
>>> > was working on this past summer. My plan is to first make a
> minimally
>>> > invasive patch to prove correctness. But, after that there will
> probably
>>> > be some big performance gains to be had from reworking some of the
> things
>>> > like KeyValueScanner which I would not have the courage to get
> working
>>> with
>>> > v1.
>>> >
>>> > So, that was why I asked, but all of that is still more
> hypothetical than
>>> > real, and I don't even know if the first part will be done
> before
>>> branching
>>> > .94 at the end of February. Makes sense to me to not delete v1
> until
>>> > there's a good reason to, which it doesn't look like we
> have yet. If I
>>> get
>>> > to the point where v1 is halting progress then we can reevaluate
> based on
>>> > more specific issues. Maybe none of the prefix trie will even
> make
>>> .94...
>>> >
>>> > ..sent from my phone
>>> > On Jan 27, 2012 1:55 PM, "lars hofhansl"
> <lh...@yahoo.com> wrote:
>>> >
>>> >> Hey Jon,
>>> >>
>>> >> understood. Makes 0.94 hard, though. If we decide now to have
> a 0.90 to
>>> >> 0.94 upgrade path and then timing does not work out and nobody
> signs up
>>> for
>>> >> the testing because it 0.92 is more convenient we'd have
> gone through
>>> this
>>> >> for nothing.
>>> >>
>>> >> So... Thinking about this more I am -1 on supporting an
> official upgrade
>>> >> path other then from one release to the next.
>>> >> That said, we do not have to break things intentionally.
>>> >>
>>> >>
>>> >> I'm fine pushing HBASE-2600 and HFile v1 removal out of
> 0.94... As long
>>> >> as we won't havethe same argument for 0.96 :)
>>> >>
>>> >> And I am not aware of any file compatibility issues.
>>> >>
>>> >>
>>> >> We can also leave the 0.92 migration code in, but not
> officially support
>>> >> it as 0.90 to 0.94 path.
>>> >> Then CDH4 can make sure (if needed) that it is all working.
>>> >>
>>> >>
>>> >> Does that work for you guys?
>>> >>
>>> >> -- Lars
>>> >>
>>> >>
>>> >>
>>> >> ________________________________
>>> >> From: Jonathan Hsieh <jo...@cloudera.com>
>>> >> To: dev@hbase.apache.org; lars hofhansl
> <lh...@yahoo.com>
>>> >> Sent: Friday, January 27, 2012 10:10 AM
>>> >> Subject: Re: hbase 0.94.0
>>> >>
>>> >>
>>> >> Lars,
>>> >>
>>> >> The upcoming CDH4 Beta HBase will be based on the latest
> hotness,
>>> 0.92.0.
>>> >> A CDH4 GA HBase will have to have an upgrade path from CDH3
> HBase. If
>>> that
>>> >> HBase is 0.92 based, we'll test that, and if timing works
> out and we
>>> decide
>>> >> 0.94, we'll have to have a path (0.90->0.94) for than
> and will test
>>> that.
>>> >>
>>> >> HBASE-2600 is a big change of encoding of meta, while my
> understanding
>>> is
>>> >> that 0.90->0,92 is a graceful HFile format conversion. Are
> there are
>>> >> things currently in trunk that further break compatiblity of
> the file
>>> >> format? (what jiras?)
>>> >>
>>> >> Jon.
>>> >>
>>> >>
>>> >> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl
> <lh...@yahoo.com>
>>> >> wrote:
>>> >>
>>> >> Not removing code for upgrade is fine.
>>> >> >
>>> >> >Todd, is Cloudera signing up for testing this path (0.90
> to 0.94)?
>>> >> >
>>> >> >
>>> >> >Stack, what's your feeling w.r.t. to HBASE-2600, will
> keeping the 0.90
>>> >> -> 0.92 migration path
>>> >> >make the migration code for HBASE-2600 (much) more
> complicated in 0.94?
>>> >> >
>>> >> >
>>> >> >-- Lars
>>> >> >
>>> >> >
>>> >> >
>>> >> >----- Original Message -----
>>> >> >From: Stack <st...@duboce.net>
>>> >> >To: dev@hbase.apache.org
>>> >> >Cc:
>>> >> >Sent: Friday, January 27, 2012 9:02 AM
>>> >> >Subject: Re: hbase 0.94.0
>>> >> >
>>> >> >
>>> >> >On Fri, Jan 27, 2012 at 8:42 AM, Stack
> <st...@duboce.net> wrote:
>>> >> >> In this particular case, there was no explicit
> migration step needed
>>> >> >> going 0.90 to 0.92. Upgrading from 0.90 to 0.94
> should just be
>>> >> >> running the 0.92 to 0.94 migration script (if there
> is one).
>>> >> >>
>>> >> >
>>> >> >Thinking more, the above only really holds if we keep the
> .META.
>>> >> >migration code that runs in 0.92 on startup on into 0.94
> (I have a
>>> >> >patch here where I'm removing it... I should put this
> bit of code
>>> >> >back).
>>> >> >
>>> >> >I see Todd that you vote against removing hfile v1 in
> 0.94. To make
>>> >> >going from CDH3 to CDH4, we should not purge any migrating
> code or old
>>> >> >version support.
>>> >> >
>>> >> >I'd be fine w/ that. We'll need to work hard to
> ensure it taking it
>>> >> >on as a principal for 0.94. Ok w/ you "Iron
> Hand" Lars?
>>> >> >
>>> >> >St.Ack
>>> >> >
>>> >> >
>>> >>
>>> >>
>>> >> --
>>> >> // Jonathan Hsieh (shay)
>>> >> // Software Engineer, Cloudera
>>> >>
>>> >> // jon@cloudera.com
>>> >>
>>> >
>>>
>>>
>>> --
>>> // Jonathan Hsieh (shay)
>>> // Software Engineer, Cloudera
>>> // jon@cloudera.com
>>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: hbase 0.94.0
Posted by Todd Lipcon <to...@cloudera.com>.
On Sun, Jan 29, 2012 at 7:30 AM, Ted Yu <yu...@gmail.com> wrote:
> Initial results from HBASE-5074, support checksums in HBase block cache,
> look promising.
>
> Shall we discuss whether 0.94 should include it (assuming Dhruba can finish
> the feature in Feb.) ?
If it's a time-based release, then there's nothing to discuss. Either
it's done in time, in which case it's part of 94, or it's not, in
which case it's part of 96, right?
Maybe the discussion should be whether 0.94 should be a time-gated or
feature-gated release? IMO we already have enough good stuff in there
on the perf front. It will be great if the checksum improvement makes
it, but if it doesn't, I'd rather have a release than delay for it.
-Todd
>
> Cheers
>
> On Fri, Jan 27, 2012 at 6:27 PM, lars hofhansl <lh...@yahoo.com> wrote:
>
>> Salesforce will jumpstart with 0.94. :) Our dev cluster has the current
>> trunk on it.
>>
>> OK let's just add these to 0.94:
>> HBASE-4218
>>
>> HBASE-4608
>> HBASE-5128
>> script necessary to do HBASE-5293 in 0.96
>>
>>
>> And defer the rest to 0.96, and branch 0.94 soon'ish.
>> Do you think you can do a 0.92 and trunk version of HBASE-5128?
>>
>> -- Lars
>>
>>
>> ________________________________
>> From: Jonathan Hsieh <jo...@cloudera.com>
>> To: Matt Corgan <mc...@hotpads.com>
>> Cc: lars hofhansl <lh...@yahoo.com>; dev <de...@hbase.apache.org>
>> Sent: Friday, January 27, 2012 6:11 PM
>> Subject: Re: hbase 0.94.0
>>
>> Lars, Matt,
>>
>> I like Matt's suggestion -- as long as it is not a burden let's keep it in.
>> If it becomes one, let's do what is best for the project.
>>
>> If Cloudera needs to do the upgrade path, we'll provide one. If it doesn't
>> go to the apache repo, we'll most likely open source it on github or
>> something, or "keep it on the jira" like some of the repair ruby scripts.
>>
>> Question -- to the Trend/SalesForce/FB folks can you talk about your plans
>> for upgrades? Are you guys moving from 0.89/0.90ish to 0.92 already? or
>> are you guys going to have to do the direct upgrade to 0.94 from 0.90 land
>> as well?
>>
>> Jon.
>>
>>
>> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan <mc...@hotpads.com> wrote:
>>
>> > The reason I brought it up is now that Mikhail committed the data block
>> > encoding I was going to take a stab at adding the prefix trie encoding I
>> > was working on this past summer. My plan is to first make a minimally
>> > invasive patch to prove correctness. But, after that there will probably
>> > be some big performance gains to be had from reworking some of the things
>> > like KeyValueScanner which I would not have the courage to get working
>> with
>> > v1.
>> >
>> > So, that was why I asked, but all of that is still more hypothetical than
>> > real, and I don't even know if the first part will be done before
>> branching
>> > .94 at the end of February. Makes sense to me to not delete v1 until
>> > there's a good reason to, which it doesn't look like we have yet. If I
>> get
>> > to the point where v1 is halting progress then we can reevaluate based on
>> > more specific issues. Maybe none of the prefix trie will even make
>> .94...
>> >
>> > ..sent from my phone
>> > On Jan 27, 2012 1:55 PM, "lars hofhansl" <lh...@yahoo.com> wrote:
>> >
>> >> Hey Jon,
>> >>
>> >> understood. Makes 0.94 hard, though. If we decide now to have a 0.90 to
>> >> 0.94 upgrade path and then timing does not work out and nobody signs up
>> for
>> >> the testing because it 0.92 is more convenient we'd have gone through
>> this
>> >> for nothing.
>> >>
>> >> So... Thinking about this more I am -1 on supporting an official upgrade
>> >> path other then from one release to the next.
>> >> That said, we do not have to break things intentionally.
>> >>
>> >>
>> >> I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94... As long
>> >> as we won't havethe same argument for 0.96 :)
>> >>
>> >> And I am not aware of any file compatibility issues.
>> >>
>> >>
>> >> We can also leave the 0.92 migration code in, but not officially support
>> >> it as 0.90 to 0.94 path.
>> >> Then CDH4 can make sure (if needed) that it is all working.
>> >>
>> >>
>> >> Does that work for you guys?
>> >>
>> >> -- Lars
>> >>
>> >>
>> >>
>> >> ________________________________
>> >> From: Jonathan Hsieh <jo...@cloudera.com>
>> >> To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
>> >> Sent: Friday, January 27, 2012 10:10 AM
>> >> Subject: Re: hbase 0.94.0
>> >>
>> >>
>> >> Lars,
>> >>
>> >> The upcoming CDH4 Beta HBase will be based on the latest hotness,
>> 0.92.0.
>> >> A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. If
>> that
>> >> HBase is 0.92 based, we'll test that, and if timing works out and we
>> decide
>> >> 0.94, we'll have to have a path (0.90->0.94) for than and will test
>> that.
>> >>
>> >> HBASE-2600 is a big change of encoding of meta, while my understanding
>> is
>> >> that 0.90->0,92 is a graceful HFile format conversion. Are there are
>> >> things currently in trunk that further break compatiblity of the file
>> >> format? (what jiras?)
>> >>
>> >> Jon.
>> >>
>> >>
>> >> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl <lh...@yahoo.com>
>> >> wrote:
>> >>
>> >> Not removing code for upgrade is fine.
>> >> >
>> >> >Todd, is Cloudera signing up for testing this path (0.90 to 0.94)?
>> >> >
>> >> >
>> >> >Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the 0.90
>> >> -> 0.92 migration path
>> >> >make the migration code for HBASE-2600 (much) more complicated in 0.94?
>> >> >
>> >> >
>> >> >-- Lars
>> >> >
>> >> >
>> >> >
>> >> >----- Original Message -----
>> >> >From: Stack <st...@duboce.net>
>> >> >To: dev@hbase.apache.org
>> >> >Cc:
>> >> >Sent: Friday, January 27, 2012 9:02 AM
>> >> >Subject: Re: hbase 0.94.0
>> >> >
>> >> >
>> >> >On Fri, Jan 27, 2012 at 8:42 AM, Stack <st...@duboce.net> wrote:
>> >> >> In this particular case, there was no explicit migration step needed
>> >> >> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be
>> >> >> running the 0.92 to 0.94 migration script (if there is one).
>> >> >>
>> >> >
>> >> >Thinking more, the above only really holds if we keep the .META.
>> >> >migration code that runs in 0.92 on startup on into 0.94 (I have a
>> >> >patch here where I'm removing it... I should put this bit of code
>> >> >back).
>> >> >
>> >> >I see Todd that you vote against removing hfile v1 in 0.94. To make
>> >> >going from CDH3 to CDH4, we should not purge any migrating code or old
>> >> >version support.
>> >> >
>> >> >I'd be fine w/ that. We'll need to work hard to ensure it taking it
>> >> >on as a principal for 0.94. Ok w/ you "Iron Hand" Lars?
>> >> >
>> >> >St.Ack
>> >> >
>> >> >
>> >>
>> >>
>> >> --
>> >> // Jonathan Hsieh (shay)
>> >> // Software Engineer, Cloudera
>> >>
>> >> // jon@cloudera.com
>> >>
>> >
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // Software Engineer, Cloudera
>> // jon@cloudera.com
>>
--
Todd Lipcon
Software Engineer, Cloudera
Re: hbase 0.94.0
Posted by Ted Yu <yu...@gmail.com>.
Initial results from HBASE-5074, support checksums in HBase block cache,
look promising.
Shall we discuss whether 0.94 should include it (assuming Dhruba can finish
the feature in Feb.) ?
Cheers
On Fri, Jan 27, 2012 at 6:27 PM, lars hofhansl <lh...@yahoo.com> wrote:
> Salesforce will jumpstart with 0.94. :) Our dev cluster has the current
> trunk on it.
>
> OK let's just add these to 0.94:
> HBASE-4218
>
> HBASE-4608
> HBASE-5128
> script necessary to do HBASE-5293 in 0.96
>
>
> And defer the rest to 0.96, and branch 0.94 soon'ish.
> Do you think you can do a 0.92 and trunk version of HBASE-5128?
>
> -- Lars
>
>
> ________________________________
> From: Jonathan Hsieh <jo...@cloudera.com>
> To: Matt Corgan <mc...@hotpads.com>
> Cc: lars hofhansl <lh...@yahoo.com>; dev <de...@hbase.apache.org>
> Sent: Friday, January 27, 2012 6:11 PM
> Subject: Re: hbase 0.94.0
>
> Lars, Matt,
>
> I like Matt's suggestion -- as long as it is not a burden let's keep it in.
> If it becomes one, let's do what is best for the project.
>
> If Cloudera needs to do the upgrade path, we'll provide one. If it doesn't
> go to the apache repo, we'll most likely open source it on github or
> something, or "keep it on the jira" like some of the repair ruby scripts.
>
> Question -- to the Trend/SalesForce/FB folks can you talk about your plans
> for upgrades? Are you guys moving from 0.89/0.90ish to 0.92 already? or
> are you guys going to have to do the direct upgrade to 0.94 from 0.90 land
> as well?
>
> Jon.
>
>
> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan <mc...@hotpads.com> wrote:
>
> > The reason I brought it up is now that Mikhail committed the data block
> > encoding I was going to take a stab at adding the prefix trie encoding I
> > was working on this past summer. My plan is to first make a minimally
> > invasive patch to prove correctness. But, after that there will probably
> > be some big performance gains to be had from reworking some of the things
> > like KeyValueScanner which I would not have the courage to get working
> with
> > v1.
> >
> > So, that was why I asked, but all of that is still more hypothetical than
> > real, and I don't even know if the first part will be done before
> branching
> > .94 at the end of February. Makes sense to me to not delete v1 until
> > there's a good reason to, which it doesn't look like we have yet. If I
> get
> > to the point where v1 is halting progress then we can reevaluate based on
> > more specific issues. Maybe none of the prefix trie will even make
> .94...
> >
> > ..sent from my phone
> > On Jan 27, 2012 1:55 PM, "lars hofhansl" <lh...@yahoo.com> wrote:
> >
> >> Hey Jon,
> >>
> >> understood. Makes 0.94 hard, though. If we decide now to have a 0.90 to
> >> 0.94 upgrade path and then timing does not work out and nobody signs up
> for
> >> the testing because it 0.92 is more convenient we'd have gone through
> this
> >> for nothing.
> >>
> >> So... Thinking about this more I am -1 on supporting an official upgrade
> >> path other then from one release to the next.
> >> That said, we do not have to break things intentionally.
> >>
> >>
> >> I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94... As long
> >> as we won't havethe same argument for 0.96 :)
> >>
> >> And I am not aware of any file compatibility issues.
> >>
> >>
> >> We can also leave the 0.92 migration code in, but not officially support
> >> it as 0.90 to 0.94 path.
> >> Then CDH4 can make sure (if needed) that it is all working.
> >>
> >>
> >> Does that work for you guys?
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ________________________________
> >> From: Jonathan Hsieh <jo...@cloudera.com>
> >> To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
> >> Sent: Friday, January 27, 2012 10:10 AM
> >> Subject: Re: hbase 0.94.0
> >>
> >>
> >> Lars,
> >>
> >> The upcoming CDH4 Beta HBase will be based on the latest hotness,
> 0.92.0.
> >> A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. If
> that
> >> HBase is 0.92 based, we'll test that, and if timing works out and we
> decide
> >> 0.94, we'll have to have a path (0.90->0.94) for than and will test
> that.
> >>
> >> HBASE-2600 is a big change of encoding of meta, while my understanding
> is
> >> that 0.90->0,92 is a graceful HFile format conversion. Are there are
> >> things currently in trunk that further break compatiblity of the file
> >> format? (what jiras?)
> >>
> >> Jon.
> >>
> >>
> >> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl <lh...@yahoo.com>
> >> wrote:
> >>
> >> Not removing code for upgrade is fine.
> >> >
> >> >Todd, is Cloudera signing up for testing this path (0.90 to 0.94)?
> >> >
> >> >
> >> >Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the 0.90
> >> -> 0.92 migration path
> >> >make the migration code for HBASE-2600 (much) more complicated in 0.94?
> >> >
> >> >
> >> >-- Lars
> >> >
> >> >
> >> >
> >> >----- Original Message -----
> >> >From: Stack <st...@duboce.net>
> >> >To: dev@hbase.apache.org
> >> >Cc:
> >> >Sent: Friday, January 27, 2012 9:02 AM
> >> >Subject: Re: hbase 0.94.0
> >> >
> >> >
> >> >On Fri, Jan 27, 2012 at 8:42 AM, Stack <st...@duboce.net> wrote:
> >> >> In this particular case, there was no explicit migration step needed
> >> >> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be
> >> >> running the 0.92 to 0.94 migration script (if there is one).
> >> >>
> >> >
> >> >Thinking more, the above only really holds if we keep the .META.
> >> >migration code that runs in 0.92 on startup on into 0.94 (I have a
> >> >patch here where I'm removing it... I should put this bit of code
> >> >back).
> >> >
> >> >I see Todd that you vote against removing hfile v1 in 0.94. To make
> >> >going from CDH3 to CDH4, we should not purge any migrating code or old
> >> >version support.
> >> >
> >> >I'd be fine w/ that. We'll need to work hard to ensure it taking it
> >> >on as a principal for 0.94. Ok w/ you "Iron Hand" Lars?
> >> >
> >> >St.Ack
> >> >
> >> >
> >>
> >>
> >> --
> >> // Jonathan Hsieh (shay)
> >> // Software Engineer, Cloudera
> >>
> >> // jon@cloudera.com
> >>
> >
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
Re: hbase 0.94.0
Posted by lars hofhansl <lh...@yahoo.com>.
Salesforce will jumpstart with 0.94. :) Our dev cluster has the current trunk on it.
OK let's just add these to 0.94:
HBASE-4218
HBASE-4608
HBASE-5128
script necessary to do HBASE-5293 in 0.96
And defer the rest to 0.96, and branch 0.94 soon'ish.
Do you think you can do a 0.92 and trunk version of HBASE-5128?
-- Lars
________________________________
From: Jonathan Hsieh <jo...@cloudera.com>
To: Matt Corgan <mc...@hotpads.com>
Cc: lars hofhansl <lh...@yahoo.com>; dev <de...@hbase.apache.org>
Sent: Friday, January 27, 2012 6:11 PM
Subject: Re: hbase 0.94.0
Lars, Matt,
I like Matt's suggestion -- as long as it is not a burden let's keep it in.
If it becomes one, let's do what is best for the project.
If Cloudera needs to do the upgrade path, we'll provide one. If it doesn't
go to the apache repo, we'll most likely open source it on github or
something, or "keep it on the jira" like some of the repair ruby scripts.
Question -- to the Trend/SalesForce/FB folks can you talk about your plans
for upgrades? Are you guys moving from 0.89/0.90ish to 0.92 already? or
are you guys going to have to do the direct upgrade to 0.94 from 0.90 land
as well?
Jon.
On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan <mc...@hotpads.com> wrote:
> The reason I brought it up is now that Mikhail committed the data block
> encoding I was going to take a stab at adding the prefix trie encoding I
> was working on this past summer. My plan is to first make a minimally
> invasive patch to prove correctness. But, after that there will probably
> be some big performance gains to be had from reworking some of the things
> like KeyValueScanner which I would not have the courage to get working with
> v1.
>
> So, that was why I asked, but all of that is still more hypothetical than
> real, and I don't even know if the first part will be done before branching
> .94 at the end of February. Makes sense to me to not delete v1 until
> there's a good reason to, which it doesn't look like we have yet. If I get
> to the point where v1 is halting progress then we can reevaluate based on
> more specific issues. Maybe none of the prefix trie will even make .94...
>
> ..sent from my phone
> On Jan 27, 2012 1:55 PM, "lars hofhansl" <lh...@yahoo.com> wrote:
>
>> Hey Jon,
>>
>> understood. Makes 0.94 hard, though. If we decide now to have a 0.90 to
>> 0.94 upgrade path and then timing does not work out and nobody signs up for
>> the testing because it 0.92 is more convenient we'd have gone through this
>> for nothing.
>>
>> So... Thinking about this more I am -1 on supporting an official upgrade
>> path other then from one release to the next.
>> That said, we do not have to break things intentionally.
>>
>>
>> I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94... As long
>> as we won't havethe same argument for 0.96 :)
>>
>> And I am not aware of any file compatibility issues.
>>
>>
>> We can also leave the 0.92 migration code in, but not officially support
>> it as 0.90 to 0.94 path.
>> Then CDH4 can make sure (if needed) that it is all working.
>>
>>
>> Does that work for you guys?
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Jonathan Hsieh <jo...@cloudera.com>
>> To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
>> Sent: Friday, January 27, 2012 10:10 AM
>> Subject: Re: hbase 0.94.0
>>
>>
>> Lars,
>>
>> The upcoming CDH4 Beta HBase will be based on the latest hotness, 0.92.0.
>> A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. If that
>> HBase is 0.92 based, we'll test that, and if timing works out and we decide
>> 0.94, we'll have to have a path (0.90->0.94) for than and will test that.
>>
>> HBASE-2600 is a big change of encoding of meta, while my understanding is
>> that 0.90->0,92 is a graceful HFile format conversion. Are there are
>> things currently in trunk that further break compatiblity of the file
>> format? (what jiras?)
>>
>> Jon.
>>
>>
>> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl <lh...@yahoo.com>
>> wrote:
>>
>> Not removing code for upgrade is fine.
>> >
>> >Todd, is Cloudera signing up for testing this path (0.90 to 0.94)?
>> >
>> >
>> >Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the 0.90
>> -> 0.92 migration path
>> >make the migration code for HBASE-2600 (much) more complicated in 0.94?
>> >
>> >
>> >-- Lars
>> >
>> >
>> >
>> >----- Original Message -----
>> >From: Stack <st...@duboce.net>
>> >To: dev@hbase.apache.org
>> >Cc:
>> >Sent: Friday, January 27, 2012 9:02 AM
>> >Subject: Re: hbase 0.94.0
>> >
>> >
>> >On Fri, Jan 27, 2012 at 8:42 AM, Stack <st...@duboce.net> wrote:
>> >> In this particular case, there was no explicit migration step needed
>> >> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be
>> >> running the 0.92 to 0.94 migration script (if there is one).
>> >>
>> >
>> >Thinking more, the above only really holds if we keep the .META.
>> >migration code that runs in 0.92 on startup on into 0.94 (I have a
>> >patch here where I'm removing it... I should put this bit of code
>> >back).
>> >
>> >I see Todd that you vote against removing hfile v1 in 0.94. To make
>> >going from CDH3 to CDH4, we should not purge any migrating code or old
>> >version support.
>> >
>> >I'd be fine w/ that. We'll need to work hard to ensure it taking it
>> >on as a principal for 0.94. Ok w/ you "Iron Hand" Lars?
>> >
>> >St.Ack
>> >
>> >
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // Software Engineer, Cloudera
>>
>> // jon@cloudera.com
>>
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: hbase 0.94.0
Posted by Nicolas Spiegelberg <ns...@fb.com>.
Currently, we at Facebook are developing mostly on 89 but also doing some
significant exploratory work on trunk. I think that most of our
development will continue to be done on 89 in the near future. We plan to
release some lower-risk projects on 94. However, we won't even entertain
a wide-scale upgrade strategy until:
1) The trunk has more master work. Specifically 89-master related
features that we implemented for our grid architecture. It sounds like
most of those features are rising in prominence in the Open Source
community as well (Ted from eBay & Todd from Cloudera, in particular).
Hopefully, we can collaborate on porting/implementation.
2) Client/Server compatibility. Even if we upgraded the servers to
94/96/whatever, we still will have a lot of clients running 89 & we can't
simply stop/start all of them at the same time. We need 89/trunk RPC
compat for client/server.
3) Durable Server Upgrade. It does bother me that people are very quick
to consider removing both RPC & persistent data compatibility from 90 to
trunk (a branch that we're still actively releasing minor upgrades for).
We will need the ability to mutate the HDFS & ZK persistent data from the
89 format to trunk. Specifically, work like HFileV1 reader is critical
for analysis if the upgrade script fails and we need to debug.
Instead of discussing what classes we can throw away, can we instead think
about a strategy for long-term, cross-version compatibility that minimally
hurts trunk development? Is HFileV1's presence severely hindering another
feature?
On 1/27/12 6:11 PM, "Jonathan Hsieh" <jo...@cloudera.com> wrote:
>Lars, Matt,
>
>I like Matt's suggestion -- as long as it is not a burden let's keep it
>in.
> If it becomes one, let's do what is best for the project.
>
>If Cloudera needs to do the upgrade path, we'll provide one. If it
>doesn't
>go to the apache repo, we'll most likely open source it on github or
>something, or "keep it on the jira" like some of the repair ruby scripts.
>
>Question -- to the Trend/SalesForce/FB folks can you talk about your plans
>for upgrades? Are you guys moving from 0.89/0.90ish to 0.92 already? or
>are you guys going to have to do the direct upgrade to 0.94 from 0.90 land
>as well?
>
>Jon.
>
>
>On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan <mc...@hotpads.com> wrote:
>
>> The reason I brought it up is now that Mikhail committed the data block
>> encoding I was going to take a stab at adding the prefix trie encoding I
>> was working on this past summer. My plan is to first make a minimally
>> invasive patch to prove correctness. But, after that there will
>>probably
>> be some big performance gains to be had from reworking some of the
>>things
>> like KeyValueScanner which I would not have the courage to get working
>>with
>> v1.
>>
>> So, that was why I asked, but all of that is still more hypothetical
>>than
>> real, and I don't even know if the first part will be done before
>>branching
>> .94 at the end of February. Makes sense to me to not delete v1 until
>> there's a good reason to, which it doesn't look like we have yet. If I
>>get
>> to the point where v1 is halting progress then we can reevaluate based
>>on
>> more specific issues. Maybe none of the prefix trie will even make
>>.94...
>>
>> ..sent from my phone
>> On Jan 27, 2012 1:55 PM, "lars hofhansl" <lh...@yahoo.com> wrote:
>>
>>> Hey Jon,
>>>
>>> understood. Makes 0.94 hard, though. If we decide now to have a 0.90 to
>>> 0.94 upgrade path and then timing does not work out and nobody signs
>>>up for
>>> the testing because it 0.92 is more convenient we'd have gone through
>>>this
>>> for nothing.
>>>
>>> So... Thinking about this more I am -1 on supporting an official
>>>upgrade
>>> path other then from one release to the next.
>>> That said, we do not have to break things intentionally.
>>>
>>>
>>> I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94... As long
>>> as we won't havethe same argument for 0.96 :)
>>>
>>> And I am not aware of any file compatibility issues.
>>>
>>>
>>> We can also leave the 0.92 migration code in, but not officially
>>>support
>>> it as 0.90 to 0.94 path.
>>> Then CDH4 can make sure (if needed) that it is all working.
>>>
>>>
>>> Does that work for you guys?
>>>
>>> -- Lars
>>>
>>>
>>>
>>> ________________________________
>>> From: Jonathan Hsieh <jo...@cloudera.com>
>>> To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
>>> Sent: Friday, January 27, 2012 10:10 AM
>>> Subject: Re: hbase 0.94.0
>>>
>>>
>>> Lars,
>>>
>>> The upcoming CDH4 Beta HBase will be based on the latest hotness,
>>>0.92.0.
>>> A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. If
>>>that
>>> HBase is 0.92 based, we'll test that, and if timing works out and we
>>>decide
>>> 0.94, we'll have to have a path (0.90->0.94) for than and will test
>>>that.
>>>
>>> HBASE-2600 is a big change of encoding of meta, while my understanding
>>>is
>>> that 0.90->0,92 is a graceful HFile format conversion. Are there are
>>> things currently in trunk that further break compatiblity of the file
>>> format? (what jiras?)
>>>
>>> Jon.
>>>
>>>
>>> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl <lh...@yahoo.com>
>>> wrote:
>>>
>>> Not removing code for upgrade is fine.
>>> >
>>> >Todd, is Cloudera signing up for testing this path (0.90 to 0.94)?
>>> >
>>> >
>>> >Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the 0.90
>>> -> 0.92 migration path
>>> >make the migration code for HBASE-2600 (much) more complicated in
>>>0.94?
>>> >
>>> >
>>> >-- Lars
>>> >
>>> >
>>> >
>>> >----- Original Message -----
>>> >From: Stack <st...@duboce.net>
>>> >To: dev@hbase.apache.org
>>> >Cc:
>>> >Sent: Friday, January 27, 2012 9:02 AM
>>> >Subject: Re: hbase 0.94.0
>>> >
>>> >
>>> >On Fri, Jan 27, 2012 at 8:42 AM, Stack <st...@duboce.net> wrote:
>>> >> In this particular case, there was no explicit migration step needed
>>> >> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be
>>> >> running the 0.92 to 0.94 migration script (if there is one).
>>> >>
>>> >
>>> >Thinking more, the above only really holds if we keep the .META.
>>> >migration code that runs in 0.92 on startup on into 0.94 (I have a
>>> >patch here where I'm removing it... I should put this bit of code
>>> >back).
>>> >
>>> >I see Todd that you vote against removing hfile v1 in 0.94. To make
>>> >going from CDH3 to CDH4, we should not purge any migrating code or old
>>> >version support.
>>> >
>>> >I'd be fine w/ that. We'll need to work hard to ensure it taking it
>>> >on as a principal for 0.94. Ok w/ you "Iron Hand" Lars?
>>> >
>>> >St.Ack
>>> >
>>> >
>>>
>>>
>>> --
>>> // Jonathan Hsieh (shay)
>>> // Software Engineer, Cloudera
>>>
>>> // jon@cloudera.com
>>>
>>
>
>
>--
>// Jonathan Hsieh (shay)
>// Software Engineer, Cloudera
>// jon@cloudera.com
Re: hbase 0.94.0
Posted by Jonathan Hsieh <jo...@cloudera.com>.
Lars, Matt,
I like Matt's suggestion -- as long as it is not a burden let's keep it in.
If it becomes one, let's do what is best for the project.
If Cloudera needs to do the upgrade path, we'll provide one. If it doesn't
go to the apache repo, we'll most likely open source it on github or
something, or "keep it on the jira" like some of the repair ruby scripts.
Question -- to the Trend/SalesForce/FB folks can you talk about your plans
for upgrades? Are you guys moving from 0.89/0.90ish to 0.92 already? or
are you guys going to have to do the direct upgrade to 0.94 from 0.90 land
as well?
Jon.
On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan <mc...@hotpads.com> wrote:
> The reason I brought it up is now that Mikhail committed the data block
> encoding I was going to take a stab at adding the prefix trie encoding I
> was working on this past summer. My plan is to first make a minimally
> invasive patch to prove correctness. But, after that there will probably
> be some big performance gains to be had from reworking some of the things
> like KeyValueScanner which I would not have the courage to get working with
> v1.
>
> So, that was why I asked, but all of that is still more hypothetical than
> real, and I don't even know if the first part will be done before branching
> .94 at the end of February. Makes sense to me to not delete v1 until
> there's a good reason to, which it doesn't look like we have yet. If I get
> to the point where v1 is halting progress then we can reevaluate based on
> more specific issues. Maybe none of the prefix trie will even make .94...
>
> ..sent from my phone
> On Jan 27, 2012 1:55 PM, "lars hofhansl" <lh...@yahoo.com> wrote:
>
>> Hey Jon,
>>
>> understood. Makes 0.94 hard, though. If we decide now to have a 0.90 to
>> 0.94 upgrade path and then timing does not work out and nobody signs up for
>> the testing because it 0.92 is more convenient we'd have gone through this
>> for nothing.
>>
>> So... Thinking about this more I am -1 on supporting an official upgrade
>> path other then from one release to the next.
>> That said, we do not have to break things intentionally.
>>
>>
>> I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94... As long
>> as we won't havethe same argument for 0.96 :)
>>
>> And I am not aware of any file compatibility issues.
>>
>>
>> We can also leave the 0.92 migration code in, but not officially support
>> it as 0.90 to 0.94 path.
>> Then CDH4 can make sure (if needed) that it is all working.
>>
>>
>> Does that work for you guys?
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Jonathan Hsieh <jo...@cloudera.com>
>> To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
>> Sent: Friday, January 27, 2012 10:10 AM
>> Subject: Re: hbase 0.94.0
>>
>>
>> Lars,
>>
>> The upcoming CDH4 Beta HBase will be based on the latest hotness, 0.92.0.
>> A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. If that
>> HBase is 0.92 based, we'll test that, and if timing works out and we decide
>> 0.94, we'll have to have a path (0.90->0.94) for than and will test that.
>>
>> HBASE-2600 is a big change of encoding of meta, while my understanding is
>> that 0.90->0,92 is a graceful HFile format conversion. Are there are
>> things currently in trunk that further break compatiblity of the file
>> format? (what jiras?)
>>
>> Jon.
>>
>>
>> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl <lh...@yahoo.com>
>> wrote:
>>
>> Not removing code for upgrade is fine.
>> >
>> >Todd, is Cloudera signing up for testing this path (0.90 to 0.94)?
>> >
>> >
>> >Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the 0.90
>> -> 0.92 migration path
>> >make the migration code for HBASE-2600 (much) more complicated in 0.94?
>> >
>> >
>> >-- Lars
>> >
>> >
>> >
>> >----- Original Message -----
>> >From: Stack <st...@duboce.net>
>> >To: dev@hbase.apache.org
>> >Cc:
>> >Sent: Friday, January 27, 2012 9:02 AM
>> >Subject: Re: hbase 0.94.0
>> >
>> >
>> >On Fri, Jan 27, 2012 at 8:42 AM, Stack <st...@duboce.net> wrote:
>> >> In this particular case, there was no explicit migration step needed
>> >> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be
>> >> running the 0.92 to 0.94 migration script (if there is one).
>> >>
>> >
>> >Thinking more, the above only really holds if we keep the .META.
>> >migration code that runs in 0.92 on startup on into 0.94 (I have a
>> >patch here where I'm removing it... I should put this bit of code
>> >back).
>> >
>> >I see Todd that you vote against removing hfile v1 in 0.94. To make
>> >going from CDH3 to CDH4, we should not purge any migrating code or old
>> >version support.
>> >
>> >I'd be fine w/ that. We'll need to work hard to ensure it taking it
>> >on as a principal for 0.94. Ok w/ you "Iron Hand" Lars?
>> >
>> >St.Ack
>> >
>> >
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // Software Engineer, Cloudera
>>
>> // jon@cloudera.com
>>
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: hbase 0.94.0
Posted by Matt Corgan <mc...@hotpads.com>.
The reason I brought it up is now that Mikhail committed the data block
encoding I was going to take a stab at adding the prefix trie encoding I
was working on this past summer. My plan is to first make a minimally
invasive patch to prove correctness. But, after that there will probably
be some big performance gains to be had from reworking some of the things
like KeyValueScanner which I would not have the courage to get working with
v1.
So, that was why I asked, but all of that is still more hypothetical than
real, and I don't even know if the first part will be done before branching
.94 at the end of February. Makes sense to me to not delete v1 until
there's a good reason to, which it doesn't look like we have yet. If I get
to the point where v1 is halting progress then we can reevaluate based on
more specific issues. Maybe none of the prefix trie will even make .94...
..sent from my phone
On Jan 27, 2012 1:55 PM, "lars hofhansl" <lh...@yahoo.com> wrote:
> Hey Jon,
>
> understood. Makes 0.94 hard, though. If we decide now to have a 0.90 to
> 0.94 upgrade path and then timing does not work out and nobody signs up for
> the testing because it 0.92 is more convenient we'd have gone through this
> for nothing.
>
> So... Thinking about this more I am -1 on supporting an official upgrade
> path other then from one release to the next.
> That said, we do not have to break things intentionally.
>
>
> I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94... As long as
> we won't havethe same argument for 0.96 :)
> And I am not aware of any file compatibility issues.
>
>
> We can also leave the 0.92 migration code in, but not officially support
> it as 0.90 to 0.94 path.
> Then CDH4 can make sure (if needed) that it is all working.
>
>
> Does that work for you guys?
>
> -- Lars
>
>
>
> ________________________________
> From: Jonathan Hsieh <jo...@cloudera.com>
> To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
> Sent: Friday, January 27, 2012 10:10 AM
> Subject: Re: hbase 0.94.0
>
>
> Lars,
>
> The upcoming CDH4 Beta HBase will be based on the latest hotness, 0.92.0.
> A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. If that
> HBase is 0.92 based, we'll test that, and if timing works out and we decide
> 0.94, we'll have to have a path (0.90->0.94) for than and will test that.
>
> HBASE-2600 is a big change of encoding of meta, while my understanding is
> that 0.90->0,92 is a graceful HFile format conversion. Are there are
> things currently in trunk that further break compatiblity of the file
> format? (what jiras?)
>
> Jon.
>
>
> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl <lh...@yahoo.com>
> wrote:
>
> Not removing code for upgrade is fine.
> >
> >Todd, is Cloudera signing up for testing this path (0.90 to 0.94)?
> >
> >
> >Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the 0.90 ->
> 0.92 migration path
> >make the migration code for HBASE-2600 (much) more complicated in 0.94?
> >
> >
> >-- Lars
> >
> >
> >
> >----- Original Message -----
> >From: Stack <st...@duboce.net>
> >To: dev@hbase.apache.org
> >Cc:
> >Sent: Friday, January 27, 2012 9:02 AM
> >Subject: Re: hbase 0.94.0
> >
> >
> >On Fri, Jan 27, 2012 at 8:42 AM, Stack <st...@duboce.net> wrote:
> >> In this particular case, there was no explicit migration step needed
> >> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be
> >> running the 0.92 to 0.94 migration script (if there is one).
> >>
> >
> >Thinking more, the above only really holds if we keep the .META.
> >migration code that runs in 0.92 on startup on into 0.94 (I have a
> >patch here where I'm removing it... I should put this bit of code
> >back).
> >
> >I see Todd that you vote against removing hfile v1 in 0.94. To make
> >going from CDH3 to CDH4, we should not purge any migrating code or old
> >version support.
> >
> >I'd be fine w/ that. We'll need to work hard to ensure it taking it
> >on as a principal for 0.94. Ok w/ you "Iron Hand" Lars?
> >
> >St.Ack
> >
> >
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
>
> // jon@cloudera.com
Re: hbase 0.94.0
Posted by lars hofhansl <lh...@yahoo.com>.
Hey Jon,
understood. Makes 0.94 hard, though. If we decide now to have a 0.90 to 0.94 upgrade path and then timing does not work out and nobody signs up for the testing because it 0.92 is more convenient we'd have gone through this for nothing.
So... Thinking about this more I am -1 on supporting an official upgrade path other then from one release to the next.
That said, we do not have to break things intentionally.
I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94... As long as we won't havethe same argument for 0.96 :)
And I am not aware of any file compatibility issues.
We can also leave the 0.92 migration code in, but not officially support it as 0.90 to 0.94 path.
Then CDH4 can make sure (if needed) that it is all working.
Does that work for you guys?
-- Lars
________________________________
From: Jonathan Hsieh <jo...@cloudera.com>
To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
Sent: Friday, January 27, 2012 10:10 AM
Subject: Re: hbase 0.94.0
Lars,
The upcoming CDH4 Beta HBase will be based on the latest hotness, 0.92.0. A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. If that HBase is 0.92 based, we'll test that, and if timing works out and we decide 0.94, we'll have to have a path (0.90->0.94) for than and will test that.
HBASE-2600 is a big change of encoding of meta, while my understanding is that 0.90->0,92 is a graceful HFile format conversion. Are there are things currently in trunk that further break compatiblity of the file format? (what jiras?)
Jon.
On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl <lh...@yahoo.com> wrote:
Not removing code for upgrade is fine.
>
>Todd, is Cloudera signing up for testing this path (0.90 to 0.94)?
>
>
>Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the 0.90 -> 0.92 migration path
>make the migration code for HBASE-2600 (much) more complicated in 0.94?
>
>
>-- Lars
>
>
>
>----- Original Message -----
>From: Stack <st...@duboce.net>
>To: dev@hbase.apache.org
>Cc:
>Sent: Friday, January 27, 2012 9:02 AM
>Subject: Re: hbase 0.94.0
>
>
>On Fri, Jan 27, 2012 at 8:42 AM, Stack <st...@duboce.net> wrote:
>> In this particular case, there was no explicit migration step needed
>> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be
>> running the 0.92 to 0.94 migration script (if there is one).
>>
>
>Thinking more, the above only really holds if we keep the .META.
>migration code that runs in 0.92 on startup on into 0.94 (I have a
>patch here where I'm removing it... I should put this bit of code
>back).
>
>I see Todd that you vote against removing hfile v1 in 0.94. To make
>going from CDH3 to CDH4, we should not purge any migrating code or old
>version support.
>
>I'd be fine w/ that. We'll need to work hard to ensure it taking it
>on as a principal for 0.94. Ok w/ you "Iron Hand" Lars?
>
>St.Ack
>
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: hbase 0.94.0
Posted by Jonathan Hsieh <jo...@cloudera.com>.
Lars,
The upcoming CDH4 Beta HBase will be based on the latest hotness, 0.92.0. A
CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. If that
HBase is 0.92 based, we'll test that, and if timing works out and we decide
0.94, we'll have to have a path (0.90->0.94) for than and will test that.
HBASE-2600 is a big change of encoding of meta, while my understanding is
that 0.90->0,92 is a graceful HFile format conversion. Are there are
things currently in trunk that further break compatiblity of the file
format? (what jiras?)
Jon.
On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl <lh...@yahoo.com> wrote:
> Not removing code for upgrade is fine.
>
> Todd, is Cloudera signing up for testing this path (0.90 to 0.94)?
>
>
> Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the 0.90 ->
> 0.92 migration path
> make the migration code for HBASE-2600 (much) more complicated in 0.94?
>
>
> -- Lars
>
>
> ----- Original Message -----
> From: Stack <st...@duboce.net>
> To: dev@hbase.apache.org
> Cc:
> Sent: Friday, January 27, 2012 9:02 AM
> Subject: Re: hbase 0.94.0
>
> On Fri, Jan 27, 2012 at 8:42 AM, Stack <st...@duboce.net> wrote:
> > In this particular case, there was no explicit migration step needed
> > going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be
> > running the 0.92 to 0.94 migration script (if there is one).
> >
>
> Thinking more, the above only really holds if we keep the .META.
> migration code that runs in 0.92 on startup on into 0.94 (I have a
> patch here where I'm removing it... I should put this bit of code
> back).
>
> I see Todd that you vote against removing hfile v1 in 0.94. To make
> going from CDH3 to CDH4, we should not purge any migrating code or old
> version support.
>
> I'd be fine w/ that. We'll need to work hard to ensure it taking it
> on as a principal for 0.94. Ok w/ you "Iron Hand" Lars?
>
> St.Ack
>
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: hbase 0.94.0
Posted by lars hofhansl <lh...@yahoo.com>.
Not removing code for upgrade is fine.
Todd, is Cloudera signing up for testing this path (0.90 to 0.94)?
Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the 0.90 -> 0.92 migration path
make the migration code for HBASE-2600 (much) more complicated in 0.94?
-- Lars
----- Original Message -----
From: Stack <st...@duboce.net>
To: dev@hbase.apache.org
Cc:
Sent: Friday, January 27, 2012 9:02 AM
Subject: Re: hbase 0.94.0
On Fri, Jan 27, 2012 at 8:42 AM, Stack <st...@duboce.net> wrote:
> In this particular case, there was no explicit migration step needed
> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be
> running the 0.92 to 0.94 migration script (if there is one).
>
Thinking more, the above only really holds if we keep the .META.
migration code that runs in 0.92 on startup on into 0.94 (I have a
patch here where I'm removing it... I should put this bit of code
back).
I see Todd that you vote against removing hfile v1 in 0.94. To make
going from CDH3 to CDH4, we should not purge any migrating code or old
version support.
I'd be fine w/ that. We'll need to work hard to ensure it taking it
on as a principal for 0.94. Ok w/ you "Iron Hand" Lars?
St.Ack
Re: hbase 0.94.0
Posted by Stack <st...@duboce.net>.
On Fri, Jan 27, 2012 at 8:42 AM, Stack <st...@duboce.net> wrote:
> In this particular case, there was no explicit migration step needed
> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be
> running the 0.92 to 0.94 migration script (if there is one).
>
Thinking more, the above only really holds if we keep the .META.
migration code that runs in 0.92 on startup on into 0.94 (I have a
patch here where I'm removing it... I should put this bit of code
back).
I see Todd that you vote against removing hfile v1 in 0.94. To make
going from CDH3 to CDH4, we should not purge any migrating code or old
version support.
I'd be fine w/ that. We'll need to work hard to ensure it taking it
on as a principal for 0.94. Ok w/ you "Iron Hand" Lars?
St.Ack
Re: hbase 0.94.0
Posted by Stack <st...@duboce.net>.
On Fri, Jan 27, 2012 at 7:20 AM, Todd Lipcon <to...@cloudera.com> wrote:
> On Thu, Jan 26, 2012 at 9:13 PM, Stack <st...@duboce.net> wrote:
>>
>> -1 on our working on stuff to make it easier to skip intermediary
>> versions. Migrations are hard enough going between particular
>> versions already. Support for skipping versions seems like a waste of
>> our time (For the historians in the audience, see our migration doc.
>> 'General Migration Notes' [1] where we are explicit that you must step
>> up through the versions -- you can't skip versions upgrading).
>
> Yes and no... from our CDH perspective, if 0.94.x is out by the time
> we need to get CDH4 "GA", then we'll skip from 0.90.x in CDH3 to
> 0.94.x in CDH4, and hence we need an upgrade path. So if upstream
> doesn't support it, we'll have to hack it in as a CDH special. That's
> fine -- but if there's no change that forces us to break
> compatibility, why break it?
>
I'm not advocating breaking anything.
I'm suggesting we not spend time ensuring upgrades can skip versions
citing in justification a principal of ours regards upgrades (We can
discuss changing the principal....).
In this particular case, there was no explicit migration step needed
going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be
running the 0.92 to 0.94 migration script (if there is one).
St.Ack
Re: hbase 0.94.0
Posted by lars hofhansl <lh...@yahoo.com>.
Won't we then find another reason next release? I agree with Stack, that we should only provide a guaranteed and tested upgrade path from
one release to the next.
In this specific case it's fine to to remove that code later since (as you say) it is not causing problems and is not holding up other development.
Let's target this for 0.96 then? I.e. the script goes into 0.94 and in 0.96 we remove HFile v1.
Are you against incorporation of HBASE-2600? Someone needs to verify that this is not causing upgrade problems from 0.90.
-- Lars
----- Original Message -----
From: Todd Lipcon <to...@cloudera.com>
To: dev@hbase.apache.org
Cc: lars hofhansl <lh...@yahoo.com>
Sent: Friday, January 27, 2012 7:20 AM
Subject: Re: hbase 0.94.0
On Thu, Jan 26, 2012 at 9:13 PM, Stack <st...@duboce.net> wrote:
>
> -1 on our working on stuff to make it easier to skip intermediary
> versions. Migrations are hard enough going between particular
> versions already. Support for skipping versions seems like a waste of
> our time (For the historians in the audience, see our migration doc.
> 'General Migration Notes' [1] where we are explicit that you must step
> up through the versions -- you can't skip versions upgrading).
Yes and no... from our CDH perspective, if 0.94.x is out by the time
we need to get CDH4 "GA", then we'll skip from 0.90.x in CDH3 to
0.94.x in CDH4, and hence we need an upgrade path. So if upstream
doesn't support it, we'll have to hack it in as a CDH special. That's
fine -- but if there's no change that forces us to break
compatibility, why break it?
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: hbase 0.94.0
Posted by Todd Lipcon <to...@cloudera.com>.
On Thu, Jan 26, 2012 at 9:13 PM, Stack <st...@duboce.net> wrote:
>
> -1 on our working on stuff to make it easier to skip intermediary
> versions. Migrations are hard enough going between particular
> versions already. Support for skipping versions seems like a waste of
> our time (For the historians in the audience, see our migration doc.
> 'General Migration Notes' [1] where we are explicit that you must step
> up through the versions -- you can't skip versions upgrading).
Yes and no... from our CDH perspective, if 0.94.x is out by the time
we need to get CDH4 "GA", then we'll skip from 0.90.x in CDH3 to
0.94.x in CDH4, and hence we need an upgrade path. So if upstream
doesn't support it, we'll have to hack it in as a CDH special. That's
fine -- but if there's no change that forces us to break
compatibility, why break it?
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: hbase 0.94.0
Posted by Ted Yu <yu...@gmail.com>.
I think we should have 3 consecutive successful builds on Apache Jenkins
for TRUNK before branching.
Cheers
On Thu, Jan 26, 2012 at 9:13 PM, Stack <st...@duboce.net> wrote:
> (Replying in one mail to a couple of points raised above in this thread)
>
> On Thu, Jan 26, 2012 at 9:37 AM, lars hofhansl <lh...@yahoo.com>
> wrote:
> > I'd say we branch as soon as we identify a large feature that should not
> be in 0.94or by the end of February, whatever comes first.
> > That way we keep the committers work to a minimum as long as possible.
> Just my $0.02.
> >
>
> Above sounds good to me (Would suggest its how to how the RM wants to run
> it).
>
> On Thu, Jan 26, 2012 at 12:25 PM, Jesse Yates <je...@gmail.com>
> wrote:
> >
> > I'm thinking something pluggable on both sides of the upgrade, so people
> can go from version X->Y, rather than having to upgrade first to an
> intermediate and then to he version they want (right it would be going from
> 0.20->.92->.96, IMO an excessive PITA).
> >
>
> -1 on our working on stuff to make it easier to skip intermediary
> versions. Migrations are hard enough going between particular
> versions already. Support for skipping versions seems like a waste of
> our time (For the historians in the audience, see our migration doc.
> 'General Migration Notes' [1] where we are explicit that you must step
> up through the versions -- you can't skip versions upgrading).
>
> I agree that v1 hfile should go (and all the other stuff deprecated in
> 0.90.x/0.92.x while we're at it) but Matt brings up critical point
> that we'd need to check there are no v1 files before we could start
> 0.94. That means a migration tool/step needs to be written as you lot
> speculate above (in the past we had ./bin/hbase migrate
> [--check|--execute]). We could bump our filesystem version in
> hbase.version when done so we don't start a 0.94 on a datadir that
> accidently has v1 hfiles around.
>
> On Thu, Jan 26, 2012 at 12:31 PM, Matt Corgan <mc...@hotpads.com> wrote:
> > On a slightly different topic, is there anything in 0.94 that breaks wire
> > compatibility with 0.92? If not, then it seems like a waste of a rare
> > opportunity to break wire compatibility!
>
> Smile. Agreed.
>
> St.Ack
> 1. http://wiki.apache.org/hadoop/Hbase/HowToMigrate
>
Re: hbase 0.94.0
Posted by Stack <st...@duboce.net>.
(Replying in one mail to a couple of points raised above in this thread)
On Thu, Jan 26, 2012 at 9:37 AM, lars hofhansl <lh...@yahoo.com> wrote:
> I'd say we branch as soon as we identify a large feature that should not be in 0.94or by the end of February, whatever comes first.
> That way we keep the committers work to a minimum as long as possible. Just my $0.02.
>
Above sounds good to me (Would suggest its how to how the RM wants to run it).
On Thu, Jan 26, 2012 at 12:25 PM, Jesse Yates <je...@gmail.com> wrote:
>
> I'm thinking something pluggable on both sides of the upgrade, so people can go from version X->Y, rather than having to upgrade first to an intermediate and then to he version they want (right it would be going from 0.20->.92->.96, IMO an excessive PITA).
>
-1 on our working on stuff to make it easier to skip intermediary
versions. Migrations are hard enough going between particular
versions already. Support for skipping versions seems like a waste of
our time (For the historians in the audience, see our migration doc.
'General Migration Notes' [1] where we are explicit that you must step
up through the versions -- you can't skip versions upgrading).
I agree that v1 hfile should go (and all the other stuff deprecated in
0.90.x/0.92.x while we're at it) but Matt brings up critical point
that we'd need to check there are no v1 files before we could start
0.94. That means a migration tool/step needs to be written as you lot
speculate above (in the past we had ./bin/hbase migrate
[--check|--execute]). We could bump our filesystem version in
hbase.version when done so we don't start a 0.94 on a datadir that
accidently has v1 hfiles around.
On Thu, Jan 26, 2012 at 12:31 PM, Matt Corgan <mc...@hotpads.com> wrote:
> On a slightly different topic, is there anything in 0.94 that breaks wire
> compatibility with 0.92? If not, then it seems like a waste of a rare
> opportunity to break wire compatibility!
Smile. Agreed.
St.Ack
1. http://wiki.apache.org/hadoop/Hbase/HowToMigrate
Re: hbase 0.94.0
Posted by lars hofhansl <lh...@yahoo.com>.
Cool... Does this script still exist somewhere, can we reuse it?
----- Original Message -----
From: Jean-Daniel Cryans <jd...@apache.org>
To: dev@hbase.apache.org
Cc:
Sent: Thursday, January 26, 2012 5:36 PM
Subject: Re: hbase 0.94.0
The upgrade from 0.20 to 0.89 was kinda like that; it required having
all your regions major compacted including catalog tables.
Unfortunately the last part was hard to do since shutting down the
cluster could do a flush on META or ROOT so we added a script in the
0.20 branch (so it was never even released) to major compact an
offline cluster (it could also check if everything was major
compacted).
Having to upgrade to a point release sounds easy in comparison.
J-D
On Thu, Jan 26, 2012 at 5:30 PM, lars hofhansl <lh...@yahoo.com> wrote:
> Seem weird to force folks to install specific point releases to do an upgrade. But I am certainly not opposed to have this in 0.92.1.
> Do you have time to work on this script? Could probably be a ruby script to be run by the HBase shell.
>
>
>
> ----- Original Message -----
> From: Matt Corgan <mc...@hotpads.com>
> To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
> Cc:
> Sent: Thursday, January 26, 2012 5:21 PM
> Subject: Re: hbase 0.94.0
>
> It would be a simple, read-only script that iterates through your hbase
> directory on hdfs, looks at each file, and prints out any regions or tables
> that contain v1 files. Either you've been running 0.92 for a while are are
> not likely to have many v1 files, or you are using 0.92 as the "script" to
> migrate your hfiles in which case all your regions will need major
> compacting. The user can copy/paste the identified regions (or whole
> tables) over into the shell to major compact them. After major compacting
> all of them, run the script again to verify there are no v1 files. Then
> you are safe to upgrade from 0.92 to 0.94.
>
> Lars - did you mean it would be too late to add a script like that to
> 0.92.1 release? If it were included in the next release, then seems like
> it would be safe to drop v1 support from 0.94.0.
>
>
> On Thu, Jan 26, 2012 at 4:53 PM, lars hofhansl <lh...@yahoo.com> wrote:
>
>> I fear it will be close to impossible to have an upgrade path from any
>> version of HBase to every future version.
>> At some point we have to make the cut, or the code will littered with old
>> cruft and upgrade logic, not even to speak of the testing overhead
>> to verify that all old versions can be upgraded to the latest one.
>>
>>
>> If we only support upgrade from one version to the next we can make sure
>> that it is rock solid and think through all the corner cases.
>>
>> And then we can stop maintaining old code and focus on fixing bugs and
>> adding features.
>>
>>
>> I like Matt's idea being able to check that all HFiles did in fact upgrade
>> to V2 (that falls into the "rock solid" part).
>> And maybe that means it is too late to remove HFileV1 in 0.94.
>>
>>
>>
>> ----- Original Message -----
>> From: Jonathan Hsieh <jo...@cloudera.com>
>> To: dev@hbase.apache.org
>> Cc:
>> Sent: Thursday, January 26, 2012 4:22 PM
>> Subject: Re: hbase 0.94.0
>>
>> +1 to having some sort of migration mechanism especially for the files
>> side. I say this out of personal interest -- I'm fairly certain that at
>> some point I'm going to have to support these upgrades.
>>
>> Jon.
>>
>> On Thu, Jan 26, 2012 at 12:25 PM, Jesse Yates <jesse.k.yates@gmail.com
>> >wrote:
>>
>> > +1 on removing it too, but maybe we should have some sort of upgrade
>> > script to help make the switch?
>> >
>> > I'm thinking something pluggable on both sides of the upgrade, so people
>> > can go from version X->Y, rather than having to upgrade first to an
>> > intermediate and then to he version they want (right it would be going
>> from
>> > 0.20->.92->.96, IMO an excessive PITA).
>> >
>> > Just my two cents...
>> >
>> > - Jesse Yates
>> >
>> > Sent from my iPhone.
>> >
>> > On Jan 26, 2012, at 12:16 PM, lars hofhansl <lh...@yahoo.com> wrote:
>> >
>> > > Good point.
>> > > 0.94 is not branched, yet. And I think generally we do not support
>> > skipping releases for upgrades.
>> > > +1 on removing HFileV1 cruft for 0.94
>> > >
>> > >
>> > > -- Lars
>> > >
>> > >
>> > >
>> > > ________________________________
>> > > From: Matt Corgan <mc...@hotpads.com>
>> > > To: dev@hbase.apache.org; Andrew Purtell <ap...@apache.org>
>> > > Sent: Thursday, January 26, 2012 11:51 AM
>> > > Subject: Re: hbase 0.94.0
>> > >
>> > > Are there any thoughts about when it is ok to stop being backwards
>> > > compatible? Mainly thinking of HFileV1... 0.92 will convert all
>> > HFileV1's
>> > > to HFileV2's, so it would probably have been ok to delete the code for
>> > > HFileV1 in 0.94 and force people to upgrade through 0.92. That didn't
>> > > actually happen, so it's looking like folks will be able to go straight
>> > > from 0.90 to 0.94. But, perhaps it should be deleted for 0.96?
>> > >
>> > >
>> > > On Thu, Jan 26, 2012 at 11:41 AM, Andrew Purtell <apurtell@apache.org
>> > >wrote:
>> > >
>> > >> Yeah, so
>> > >>
>> > >> - Security (basically another coprocessor for inclusion in
>> mainline,
>> > >> like Constraints)
>> > >>
>> > >> if not in 0.92.1.
>> > >>
>> > >> Best regards,
>> > >>
>> > >> - Andy
>> > >>
>> > >> Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein
>> > >> (via Tom White)
>> > >>
>> > >>
>> > >>> ________________________________
>> > >>> From: Andrew Purtell <ap...@apache.org>
>> > >>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>> > >>> Sent: Thursday, January 26, 2012 11:28 AM
>> > >>> Subject: Re: hbase 0.94.0
>> > >>>
>> > >>> A limited set of changes so we can get it out on that kind of
>> > timeframe.
>> > >> :-)
>> > >>>
>> > >>> - Constraints (is ready to go, is a coprocessor, so is in the large
>> > >> just a new package to drop in)
>> > >>>
>> > >>> - Useful utilities for ops:
>> > >>>
>> > >>> - LoadTestTool (if Ted didn't end up backporting this into 0.92)
>> > >>>
>> > >>> - The store file locality thing I have mostly done, will finish
>> it
>> > >>>
>> > >>> - Mikhail and crew's ongoing optimizations (HBASE-4218, etc.), the
>> > ones
>> > >> he considers fully baked
>> > >>>
>> > >>> I saw wire compatibility mentioned, for 0.96 but perhaps
>> > >> optional/transitional code in 0.94 as well. This is something we would
>> > try
>> > >> out and beat up upon in staging in earnest.
>> > >>>
>> > >>> Best regards,
>> > >>>
>> > >>> - Andy
>> > >>>
>> > >>> Problems worthy of attack prove their worth by hitting back. - Piet
>> > Hein
>> > >> (via Tom White)
>> > >>>
>> > >>>
>> > >>>
>> > >>>> ________________________________
>> > >>>> From: Stack <st...@duboce.net>
>> > >>>> To: HBase Dev List <de...@hbase.apache.org>
>> > >>>> Sent: Wednesday, January 25, 2012 8:34 PM
>> > >>>> Subject: hbase 0.94.0
>> > >>>>
>> > >>>> Lets branch end of february? No new features thereafter. Is this
>> too
>> > >>>> close in? Would be grand if 0.94.0 shipped before hbasecon. What
>> > >>>> should 0.94.0 have in it? I don't mind if the list is short.
>> > >>>>
>> > >>>> Unless someone else wants too, I don't mind being release manager
>> > >>>> (will try to run a tighter ship this time around).
>> > >>>>
>> > >>>> St.Ack
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>
>> > >>>
>> >
>>
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // Software Engineer, Cloudera
>> // jon@cloudera.com
>>
>>
>
Re: hbase 0.94.0
Posted by Jean-Daniel Cryans <jd...@apache.org>.
The upgrade from 0.20 to 0.89 was kinda like that; it required having
all your regions major compacted including catalog tables.
Unfortunately the last part was hard to do since shutting down the
cluster could do a flush on META or ROOT so we added a script in the
0.20 branch (so it was never even released) to major compact an
offline cluster (it could also check if everything was major
compacted).
Having to upgrade to a point release sounds easy in comparison.
J-D
On Thu, Jan 26, 2012 at 5:30 PM, lars hofhansl <lh...@yahoo.com> wrote:
> Seem weird to force folks to install specific point releases to do an upgrade. But I am certainly not opposed to have this in 0.92.1.
> Do you have time to work on this script? Could probably be a ruby script to be run by the HBase shell.
>
>
>
> ----- Original Message -----
> From: Matt Corgan <mc...@hotpads.com>
> To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
> Cc:
> Sent: Thursday, January 26, 2012 5:21 PM
> Subject: Re: hbase 0.94.0
>
> It would be a simple, read-only script that iterates through your hbase
> directory on hdfs, looks at each file, and prints out any regions or tables
> that contain v1 files. Either you've been running 0.92 for a while are are
> not likely to have many v1 files, or you are using 0.92 as the "script" to
> migrate your hfiles in which case all your regions will need major
> compacting. The user can copy/paste the identified regions (or whole
> tables) over into the shell to major compact them. After major compacting
> all of them, run the script again to verify there are no v1 files. Then
> you are safe to upgrade from 0.92 to 0.94.
>
> Lars - did you mean it would be too late to add a script like that to
> 0.92.1 release? If it were included in the next release, then seems like
> it would be safe to drop v1 support from 0.94.0.
>
>
> On Thu, Jan 26, 2012 at 4:53 PM, lars hofhansl <lh...@yahoo.com> wrote:
>
>> I fear it will be close to impossible to have an upgrade path from any
>> version of HBase to every future version.
>> At some point we have to make the cut, or the code will littered with old
>> cruft and upgrade logic, not even to speak of the testing overhead
>> to verify that all old versions can be upgraded to the latest one.
>>
>>
>> If we only support upgrade from one version to the next we can make sure
>> that it is rock solid and think through all the corner cases.
>>
>> And then we can stop maintaining old code and focus on fixing bugs and
>> adding features.
>>
>>
>> I like Matt's idea being able to check that all HFiles did in fact upgrade
>> to V2 (that falls into the "rock solid" part).
>> And maybe that means it is too late to remove HFileV1 in 0.94.
>>
>>
>>
>> ----- Original Message -----
>> From: Jonathan Hsieh <jo...@cloudera.com>
>> To: dev@hbase.apache.org
>> Cc:
>> Sent: Thursday, January 26, 2012 4:22 PM
>> Subject: Re: hbase 0.94.0
>>
>> +1 to having some sort of migration mechanism especially for the files
>> side. I say this out of personal interest -- I'm fairly certain that at
>> some point I'm going to have to support these upgrades.
>>
>> Jon.
>>
>> On Thu, Jan 26, 2012 at 12:25 PM, Jesse Yates <jesse.k.yates@gmail.com
>> >wrote:
>>
>> > +1 on removing it too, but maybe we should have some sort of upgrade
>> > script to help make the switch?
>> >
>> > I'm thinking something pluggable on both sides of the upgrade, so people
>> > can go from version X->Y, rather than having to upgrade first to an
>> > intermediate and then to he version they want (right it would be going
>> from
>> > 0.20->.92->.96, IMO an excessive PITA).
>> >
>> > Just my two cents...
>> >
>> > - Jesse Yates
>> >
>> > Sent from my iPhone.
>> >
>> > On Jan 26, 2012, at 12:16 PM, lars hofhansl <lh...@yahoo.com> wrote:
>> >
>> > > Good point.
>> > > 0.94 is not branched, yet. And I think generally we do not support
>> > skipping releases for upgrades.
>> > > +1 on removing HFileV1 cruft for 0.94
>> > >
>> > >
>> > > -- Lars
>> > >
>> > >
>> > >
>> > > ________________________________
>> > > From: Matt Corgan <mc...@hotpads.com>
>> > > To: dev@hbase.apache.org; Andrew Purtell <ap...@apache.org>
>> > > Sent: Thursday, January 26, 2012 11:51 AM
>> > > Subject: Re: hbase 0.94.0
>> > >
>> > > Are there any thoughts about when it is ok to stop being backwards
>> > > compatible? Mainly thinking of HFileV1... 0.92 will convert all
>> > HFileV1's
>> > > to HFileV2's, so it would probably have been ok to delete the code for
>> > > HFileV1 in 0.94 and force people to upgrade through 0.92. That didn't
>> > > actually happen, so it's looking like folks will be able to go straight
>> > > from 0.90 to 0.94. But, perhaps it should be deleted for 0.96?
>> > >
>> > >
>> > > On Thu, Jan 26, 2012 at 11:41 AM, Andrew Purtell <apurtell@apache.org
>> > >wrote:
>> > >
>> > >> Yeah, so
>> > >>
>> > >> - Security (basically another coprocessor for inclusion in
>> mainline,
>> > >> like Constraints)
>> > >>
>> > >> if not in 0.92.1.
>> > >>
>> > >> Best regards,
>> > >>
>> > >> - Andy
>> > >>
>> > >> Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein
>> > >> (via Tom White)
>> > >>
>> > >>
>> > >>> ________________________________
>> > >>> From: Andrew Purtell <ap...@apache.org>
>> > >>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>> > >>> Sent: Thursday, January 26, 2012 11:28 AM
>> > >>> Subject: Re: hbase 0.94.0
>> > >>>
>> > >>> A limited set of changes so we can get it out on that kind of
>> > timeframe.
>> > >> :-)
>> > >>>
>> > >>> - Constraints (is ready to go, is a coprocessor, so is in the large
>> > >> just a new package to drop in)
>> > >>>
>> > >>> - Useful utilities for ops:
>> > >>>
>> > >>> - LoadTestTool (if Ted didn't end up backporting this into 0.92)
>> > >>>
>> > >>> - The store file locality thing I have mostly done, will finish
>> it
>> > >>>
>> > >>> - Mikhail and crew's ongoing optimizations (HBASE-4218, etc.), the
>> > ones
>> > >> he considers fully baked
>> > >>>
>> > >>> I saw wire compatibility mentioned, for 0.96 but perhaps
>> > >> optional/transitional code in 0.94 as well. This is something we would
>> > try
>> > >> out and beat up upon in staging in earnest.
>> > >>>
>> > >>> Best regards,
>> > >>>
>> > >>> - Andy
>> > >>>
>> > >>> Problems worthy of attack prove their worth by hitting back. - Piet
>> > Hein
>> > >> (via Tom White)
>> > >>>
>> > >>>
>> > >>>
>> > >>>> ________________________________
>> > >>>> From: Stack <st...@duboce.net>
>> > >>>> To: HBase Dev List <de...@hbase.apache.org>
>> > >>>> Sent: Wednesday, January 25, 2012 8:34 PM
>> > >>>> Subject: hbase 0.94.0
>> > >>>>
>> > >>>> Lets branch end of february? No new features thereafter. Is this
>> too
>> > >>>> close in? Would be grand if 0.94.0 shipped before hbasecon. What
>> > >>>> should 0.94.0 have in it? I don't mind if the list is short.
>> > >>>>
>> > >>>> Unless someone else wants too, I don't mind being release manager
>> > >>>> (will try to run a tighter ship this time around).
>> > >>>>
>> > >>>> St.Ack
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>
>> > >>>
>> >
>>
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // Software Engineer, Cloudera
>> // jon@cloudera.com
>>
>>
>
Re: hbase 0.94.0
Posted by lars hofhansl <lh...@yahoo.com>.
Seem weird to force folks to install specific point releases to do an upgrade. But I am certainly not opposed to have this in 0.92.1.
Do you have time to work on this script? Could probably be a ruby script to be run by the HBase shell.
----- Original Message -----
From: Matt Corgan <mc...@hotpads.com>
To: dev@hbase.apache.org; lars hofhansl <lh...@yahoo.com>
Cc:
Sent: Thursday, January 26, 2012 5:21 PM
Subject: Re: hbase 0.94.0
It would be a simple, read-only script that iterates through your hbase
directory on hdfs, looks at each file, and prints out any regions or tables
that contain v1 files. Either you've been running 0.92 for a while are are
not likely to have many v1 files, or you are using 0.92 as the "script" to
migrate your hfiles in which case all your regions will need major
compacting. The user can copy/paste the identified regions (or whole
tables) over into the shell to major compact them. After major compacting
all of them, run the script again to verify there are no v1 files. Then
you are safe to upgrade from 0.92 to 0.94.
Lars - did you mean it would be too late to add a script like that to
0.92.1 release? If it were included in the next release, then seems like
it would be safe to drop v1 support from 0.94.0.
On Thu, Jan 26, 2012 at 4:53 PM, lars hofhansl <lh...@yahoo.com> wrote:
> I fear it will be close to impossible to have an upgrade path from any
> version of HBase to every future version.
> At some point we have to make the cut, or the code will littered with old
> cruft and upgrade logic, not even to speak of the testing overhead
> to verify that all old versions can be upgraded to the latest one.
>
>
> If we only support upgrade from one version to the next we can make sure
> that it is rock solid and think through all the corner cases.
>
> And then we can stop maintaining old code and focus on fixing bugs and
> adding features.
>
>
> I like Matt's idea being able to check that all HFiles did in fact upgrade
> to V2 (that falls into the "rock solid" part).
> And maybe that means it is too late to remove HFileV1 in 0.94.
>
>
>
> ----- Original Message -----
> From: Jonathan Hsieh <jo...@cloudera.com>
> To: dev@hbase.apache.org
> Cc:
> Sent: Thursday, January 26, 2012 4:22 PM
> Subject: Re: hbase 0.94.0
>
> +1 to having some sort of migration mechanism especially for the files
> side. I say this out of personal interest -- I'm fairly certain that at
> some point I'm going to have to support these upgrades.
>
> Jon.
>
> On Thu, Jan 26, 2012 at 12:25 PM, Jesse Yates <jesse.k.yates@gmail.com
> >wrote:
>
> > +1 on removing it too, but maybe we should have some sort of upgrade
> > script to help make the switch?
> >
> > I'm thinking something pluggable on both sides of the upgrade, so people
> > can go from version X->Y, rather than having to upgrade first to an
> > intermediate and then to he version they want (right it would be going
> from
> > 0.20->.92->.96, IMO an excessive PITA).
> >
> > Just my two cents...
> >
> > - Jesse Yates
> >
> > Sent from my iPhone.
> >
> > On Jan 26, 2012, at 12:16 PM, lars hofhansl <lh...@yahoo.com> wrote:
> >
> > > Good point.
> > > 0.94 is not branched, yet. And I think generally we do not support
> > skipping releases for upgrades.
> > > +1 on removing HFileV1 cruft for 0.94
> > >
> > >
> > > -- Lars
> > >
> > >
> > >
> > > ________________________________
> > > From: Matt Corgan <mc...@hotpads.com>
> > > To: dev@hbase.apache.org; Andrew Purtell <ap...@apache.org>
> > > Sent: Thursday, January 26, 2012 11:51 AM
> > > Subject: Re: hbase 0.94.0
> > >
> > > Are there any thoughts about when it is ok to stop being backwards
> > > compatible? Mainly thinking of HFileV1... 0.92 will convert all
> > HFileV1's
> > > to HFileV2's, so it would probably have been ok to delete the code for
> > > HFileV1 in 0.94 and force people to upgrade through 0.92. That didn't
> > > actually happen, so it's looking like folks will be able to go straight
> > > from 0.90 to 0.94. But, perhaps it should be deleted for 0.96?
> > >
> > >
> > > On Thu, Jan 26, 2012 at 11:41 AM, Andrew Purtell <apurtell@apache.org
> > >wrote:
> > >
> > >> Yeah, so
> > >>
> > >> - Security (basically another coprocessor for inclusion in
> mainline,
> > >> like Constraints)
> > >>
> > >> if not in 0.92.1.
> > >>
> > >> Best regards,
> > >>
> > >> - Andy
> > >>
> > >> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > >> (via Tom White)
> > >>
> > >>
> > >>> ________________________________
> > >>> From: Andrew Purtell <ap...@apache.org>
> > >>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> > >>> Sent: Thursday, January 26, 2012 11:28 AM
> > >>> Subject: Re: hbase 0.94.0
> > >>>
> > >>> A limited set of changes so we can get it out on that kind of
> > timeframe.
> > >> :-)
> > >>>
> > >>> - Constraints (is ready to go, is a coprocessor, so is in the large
> > >> just a new package to drop in)
> > >>>
> > >>> - Useful utilities for ops:
> > >>>
> > >>> - LoadTestTool (if Ted didn't end up backporting this into 0.92)
> > >>>
> > >>> - The store file locality thing I have mostly done, will finish
> it
> > >>>
> > >>> - Mikhail and crew's ongoing optimizations (HBASE-4218, etc.), the
> > ones
> > >> he considers fully baked
> > >>>
> > >>> I saw wire compatibility mentioned, for 0.96 but perhaps
> > >> optional/transitional code in 0.94 as well. This is something we would
> > try
> > >> out and beat up upon in staging in earnest.
> > >>>
> > >>> Best regards,
> > >>>
> > >>> - Andy
> > >>>
> > >>> Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > >> (via Tom White)
> > >>>
> > >>>
> > >>>
> > >>>> ________________________________
> > >>>> From: Stack <st...@duboce.net>
> > >>>> To: HBase Dev List <de...@hbase.apache.org>
> > >>>> Sent: Wednesday, January 25, 2012 8:34 PM
> > >>>> Subject: hbase 0.94.0
> > >>>>
> > >>>> Lets branch end of february? No new features thereafter. Is this
> too
> > >>>> close in? Would be grand if 0.94.0 shipped before hbasecon. What
> > >>>> should 0.94.0 have in it? I don't mind if the list is short.
> > >>>>
> > >>>> Unless someone else wants too, I don't mind being release manager
> > >>>> (will try to run a tighter ship this time around).
> > >>>>
> > >>>> St.Ack
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>>
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
>
Re: hbase 0.94.0
Posted by Matt Corgan <mc...@hotpads.com>.
It would be a simple, read-only script that iterates through your hbase
directory on hdfs, looks at each file, and prints out any regions or tables
that contain v1 files. Either you've been running 0.92 for a while are are
not likely to have many v1 files, or you are using 0.92 as the "script" to
migrate your hfiles in which case all your regions will need major
compacting. The user can copy/paste the identified regions (or whole
tables) over into the shell to major compact them. After major compacting
all of them, run the script again to verify there are no v1 files. Then
you are safe to upgrade from 0.92 to 0.94.
Lars - did you mean it would be too late to add a script like that to
0.92.1 release? If it were included in the next release, then seems like
it would be safe to drop v1 support from 0.94.0.
On Thu, Jan 26, 2012 at 4:53 PM, lars hofhansl <lh...@yahoo.com> wrote:
> I fear it will be close to impossible to have an upgrade path from any
> version of HBase to every future version.
> At some point we have to make the cut, or the code will littered with old
> cruft and upgrade logic, not even to speak of the testing overhead
> to verify that all old versions can be upgraded to the latest one.
>
>
> If we only support upgrade from one version to the next we can make sure
> that it is rock solid and think through all the corner cases.
>
> And then we can stop maintaining old code and focus on fixing bugs and
> adding features.
>
>
> I like Matt's idea being able to check that all HFiles did in fact upgrade
> to V2 (that falls into the "rock solid" part).
> And maybe that means it is too late to remove HFileV1 in 0.94.
>
>
>
> ----- Original Message -----
> From: Jonathan Hsieh <jo...@cloudera.com>
> To: dev@hbase.apache.org
> Cc:
> Sent: Thursday, January 26, 2012 4:22 PM
> Subject: Re: hbase 0.94.0
>
> +1 to having some sort of migration mechanism especially for the files
> side. I say this out of personal interest -- I'm fairly certain that at
> some point I'm going to have to support these upgrades.
>
> Jon.
>
> On Thu, Jan 26, 2012 at 12:25 PM, Jesse Yates <jesse.k.yates@gmail.com
> >wrote:
>
> > +1 on removing it too, but maybe we should have some sort of upgrade
> > script to help make the switch?
> >
> > I'm thinking something pluggable on both sides of the upgrade, so people
> > can go from version X->Y, rather than having to upgrade first to an
> > intermediate and then to he version they want (right it would be going
> from
> > 0.20->.92->.96, IMO an excessive PITA).
> >
> > Just my two cents...
> >
> > - Jesse Yates
> >
> > Sent from my iPhone.
> >
> > On Jan 26, 2012, at 12:16 PM, lars hofhansl <lh...@yahoo.com> wrote:
> >
> > > Good point.
> > > 0.94 is not branched, yet. And I think generally we do not support
> > skipping releases for upgrades.
> > > +1 on removing HFileV1 cruft for 0.94
> > >
> > >
> > > -- Lars
> > >
> > >
> > >
> > > ________________________________
> > > From: Matt Corgan <mc...@hotpads.com>
> > > To: dev@hbase.apache.org; Andrew Purtell <ap...@apache.org>
> > > Sent: Thursday, January 26, 2012 11:51 AM
> > > Subject: Re: hbase 0.94.0
> > >
> > > Are there any thoughts about when it is ok to stop being backwards
> > > compatible? Mainly thinking of HFileV1... 0.92 will convert all
> > HFileV1's
> > > to HFileV2's, so it would probably have been ok to delete the code for
> > > HFileV1 in 0.94 and force people to upgrade through 0.92. That didn't
> > > actually happen, so it's looking like folks will be able to go straight
> > > from 0.90 to 0.94. But, perhaps it should be deleted for 0.96?
> > >
> > >
> > > On Thu, Jan 26, 2012 at 11:41 AM, Andrew Purtell <apurtell@apache.org
> > >wrote:
> > >
> > >> Yeah, so
> > >>
> > >> - Security (basically another coprocessor for inclusion in
> mainline,
> > >> like Constraints)
> > >>
> > >> if not in 0.92.1.
> > >>
> > >> Best regards,
> > >>
> > >> - Andy
> > >>
> > >> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > >> (via Tom White)
> > >>
> > >>
> > >>> ________________________________
> > >>> From: Andrew Purtell <ap...@apache.org>
> > >>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> > >>> Sent: Thursday, January 26, 2012 11:28 AM
> > >>> Subject: Re: hbase 0.94.0
> > >>>
> > >>> A limited set of changes so we can get it out on that kind of
> > timeframe.
> > >> :-)
> > >>>
> > >>> - Constraints (is ready to go, is a coprocessor, so is in the large
> > >> just a new package to drop in)
> > >>>
> > >>> - Useful utilities for ops:
> > >>>
> > >>> - LoadTestTool (if Ted didn't end up backporting this into 0.92)
> > >>>
> > >>> - The store file locality thing I have mostly done, will finish
> it
> > >>>
> > >>> - Mikhail and crew's ongoing optimizations (HBASE-4218, etc.), the
> > ones
> > >> he considers fully baked
> > >>>
> > >>> I saw wire compatibility mentioned, for 0.96 but perhaps
> > >> optional/transitional code in 0.94 as well. This is something we would
> > try
> > >> out and beat up upon in staging in earnest.
> > >>>
> > >>> Best regards,
> > >>>
> > >>> - Andy
> > >>>
> > >>> Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > >> (via Tom White)
> > >>>
> > >>>
> > >>>
> > >>>> ________________________________
> > >>>> From: Stack <st...@duboce.net>
> > >>>> To: HBase Dev List <de...@hbase.apache.org>
> > >>>> Sent: Wednesday, January 25, 2012 8:34 PM
> > >>>> Subject: hbase 0.94.0
> > >>>>
> > >>>> Lets branch end of february? No new features thereafter. Is this
> too
> > >>>> close in? Would be grand if 0.94.0 shipped before hbasecon. What
> > >>>> should 0.94.0 have in it? I don't mind if the list is short.
> > >>>>
> > >>>> Unless someone else wants too, I don't mind being release manager
> > >>>> (will try to run a tighter ship this time around).
> > >>>>
> > >>>> St.Ack
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>>
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
>
Re: hbase 0.94.0
Posted by Andrew Purtell <ap...@apache.org>.
> I fear it will be close to impossible to have an upgrade path from any version of HBase to every future version.
> [...]
> If we only support upgrade from one version to the next we can make sure that it is rock solid and think through all the corner cases.
+1
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
----- Original Message -----
> From: lars hofhansl <lh...@yahoo.com>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> Cc:
> Sent: Thursday, January 26, 2012 4:53 PM
> Subject: Re: hbase 0.94.0
>
> I fear it will be close to impossible to have an upgrade path from any version
> of HBase to every future version.
> At some point we have to make the cut, or the code will littered with old cruft
> and upgrade logic, not even to speak of the testing overhead
> to verify that all old versions can be upgraded to the latest one.
>
>
> If we only support upgrade from one version to the next we can make sure that it
> is rock solid and think through all the corner cases.
>
> And then we can stop maintaining old code and focus on fixing bugs and adding
> features.
>
>
> I like Matt's idea being able to check that all HFiles did in fact upgrade
> to V2 (that falls into the "rock solid" part).
> And maybe that means it is too late to remove HFileV1 in 0.94.
>
>
>
> ----- Original Message -----
> From: Jonathan Hsieh <jo...@cloudera.com>
> To: dev@hbase.apache.org
> Cc:
> Sent: Thursday, January 26, 2012 4:22 PM
> Subject: Re: hbase 0.94.0
>
> +1 to having some sort of migration mechanism especially for the files
> side. I say this out of personal interest -- I'm fairly certain that at
> some point I'm going to have to support these upgrades.
>
> Jon.
>
> On Thu, Jan 26, 2012 at 12:25 PM, Jesse Yates
> <je...@gmail.com>wrote:
>
>> +1 on removing it too, but maybe we should have some sort of upgrade
>> script to help make the switch?
>>
>> I'm thinking something pluggable on both sides of the upgrade, so
> people
>> can go from version X->Y, rather than having to upgrade first to an
>> intermediate and then to he version they want (right it would be going from
>> 0.20->.92->.96, IMO an excessive PITA).
>>
>> Just my two cents...
>>
>> - Jesse Yates
>>
>> Sent from my iPhone.
>>
>> On Jan 26, 2012, at 12:16 PM, lars hofhansl <lh...@yahoo.com>
> wrote:
>>
>> > Good point.
>> > 0.94 is not branched, yet. And I think generally we do not support
>> skipping releases for upgrades.
>> > +1 on removing HFileV1 cruft for 0.94
>> >
>> >
>> > -- Lars
>> >
>> >
>> >
>> > ________________________________
>> > From: Matt Corgan <mc...@hotpads.com>
>> > To: dev@hbase.apache.org; Andrew Purtell <ap...@apache.org>
>> > Sent: Thursday, January 26, 2012 11:51 AM
>> > Subject: Re: hbase 0.94.0
>> >
>> > Are there any thoughts about when it is ok to stop being backwards
>> > compatible? Mainly thinking of HFileV1... 0.92 will convert all
>> HFileV1's
>> > to HFileV2's, so it would probably have been ok to delete the code
> for
>> > HFileV1 in 0.94 and force people to upgrade through 0.92. That
> didn't
>> > actually happen, so it's looking like folks will be able to go
> straight
>> > from 0.90 to 0.94. But, perhaps it should be deleted for 0.96?
>> >
>> >
>> > On Thu, Jan 26, 2012 at 11:41 AM, Andrew Purtell
> <apurtell@apache.org
>> >wrote:
>> >
>> >> Yeah, so
>> >>
>> >> - Security (basically another coprocessor for inclusion in
> mainline,
>> >> like Constraints)
>> >>
>> >> if not in 0.92.1.
>> >>
>> >> Best regards,
>> >>
>> >> - Andy
>> >>
>> >> Problems worthy of attack prove their worth by hitting back. -
> Piet Hein
>> >> (via Tom White)
>> >>
>> >>
>> >>> ________________________________
>> >>> From: Andrew Purtell <ap...@apache.org>
>> >>> To: "dev@hbase.apache.org"
> <de...@hbase.apache.org>
>> >>> Sent: Thursday, January 26, 2012 11:28 AM
>> >>> Subject: Re: hbase 0.94.0
>> >>>
>> >>> A limited set of changes so we can get it out on that kind of
>> timeframe.
>> >> :-)
>> >>>
>> >>> - Constraints (is ready to go, is a coprocessor, so is in
> the large
>> >> just a new package to drop in)
>> >>>
>> >>> - Useful utilities for ops:
>> >>>
>> >>> - LoadTestTool (if Ted didn't end up backporting this
> into 0.92)
>> >>>
>> >>> - The store file locality thing I have mostly done, will
> finish it
>> >>>
>> >>> - Mikhail and crew's ongoing optimizations (HBASE-4218,
> etc.), the
>> ones
>> >> he considers fully baked
>> >>>
>> >>> I saw wire compatibility mentioned, for 0.96 but perhaps
>> >> optional/transitional code in 0.94 as well. This is something we
> would
>> try
>> >> out and beat up upon in staging in earnest.
>> >>>
>> >>> Best regards,
>> >>>
>> >>> - Andy
>> >>>
>> >>> Problems worthy of attack prove their worth by hitting back. -
> Piet
>> Hein
>> >> (via Tom White)
>> >>>
>> >>>
>> >>>
>> >>>> ________________________________
>> >>>> From: Stack <st...@duboce.net>
>> >>>> To: HBase Dev List <de...@hbase.apache.org>
>> >>>> Sent: Wednesday, January 25, 2012 8:34 PM
>> >>>> Subject: hbase 0.94.0
>> >>>>
>> >>>> Lets branch end of february? No new features thereafter.
> Is this too
>> >>>> close in? Would be grand if 0.94.0 shipped before
> hbasecon. What
>> >>>> should 0.94.0 have in it? I don't mind if the list is
> short.
>> >>>>
>> >>>> Unless someone else wants too, I don't mind being
> release manager
>> >>>> (will try to run a tighter ship this time around).
>> >>>>
>> >>>> St.Ack
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>>
>>
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
Re: hbase 0.94.0
Posted by lars hofhansl <lh...@yahoo.com>.
I fear it will be close to impossible to have an upgrade path from any version of HBase to every future version.
At some point we have to make the cut, or the code will littered with old cruft and upgrade logic, not even to speak of the testing overhead
to verify that all old versions can be upgraded to the latest one.
If we only support upgrade from one version to the next we can make sure that it is rock solid and think through all the corner cases.
And then we can stop maintaining old code and focus on fixing bugs and adding features.
I like Matt's idea being able to check that all HFiles did in fact upgrade to V2 (that falls into the "rock solid" part).
And maybe that means it is too late to remove HFileV1 in 0.94.
----- Original Message -----
From: Jonathan Hsieh <jo...@cloudera.com>
To: dev@hbase.apache.org
Cc:
Sent: Thursday, January 26, 2012 4:22 PM
Subject: Re: hbase 0.94.0
+1 to having some sort of migration mechanism especially for the files
side. I say this out of personal interest -- I'm fairly certain that at
some point I'm going to have to support these upgrades.
Jon.
On Thu, Jan 26, 2012 at 12:25 PM, Jesse Yates <je...@gmail.com>wrote:
> +1 on removing it too, but maybe we should have some sort of upgrade
> script to help make the switch?
>
> I'm thinking something pluggable on both sides of the upgrade, so people
> can go from version X->Y, rather than having to upgrade first to an
> intermediate and then to he version they want (right it would be going from
> 0.20->.92->.96, IMO an excessive PITA).
>
> Just my two cents...
>
> - Jesse Yates
>
> Sent from my iPhone.
>
> On Jan 26, 2012, at 12:16 PM, lars hofhansl <lh...@yahoo.com> wrote:
>
> > Good point.
> > 0.94 is not branched, yet. And I think generally we do not support
> skipping releases for upgrades.
> > +1 on removing HFileV1 cruft for 0.94
> >
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Matt Corgan <mc...@hotpads.com>
> > To: dev@hbase.apache.org; Andrew Purtell <ap...@apache.org>
> > Sent: Thursday, January 26, 2012 11:51 AM
> > Subject: Re: hbase 0.94.0
> >
> > Are there any thoughts about when it is ok to stop being backwards
> > compatible? Mainly thinking of HFileV1... 0.92 will convert all
> HFileV1's
> > to HFileV2's, so it would probably have been ok to delete the code for
> > HFileV1 in 0.94 and force people to upgrade through 0.92. That didn't
> > actually happen, so it's looking like folks will be able to go straight
> > from 0.90 to 0.94. But, perhaps it should be deleted for 0.96?
> >
> >
> > On Thu, Jan 26, 2012 at 11:41 AM, Andrew Purtell <apurtell@apache.org
> >wrote:
> >
> >> Yeah, so
> >>
> >> - Security (basically another coprocessor for inclusion in mainline,
> >> like Constraints)
> >>
> >> if not in 0.92.1.
> >>
> >> Best regards,
> >>
> >> - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
> >>
> >>> ________________________________
> >>> From: Andrew Purtell <ap...@apache.org>
> >>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> >>> Sent: Thursday, January 26, 2012 11:28 AM
> >>> Subject: Re: hbase 0.94.0
> >>>
> >>> A limited set of changes so we can get it out on that kind of
> timeframe.
> >> :-)
> >>>
> >>> - Constraints (is ready to go, is a coprocessor, so is in the large
> >> just a new package to drop in)
> >>>
> >>> - Useful utilities for ops:
> >>>
> >>> - LoadTestTool (if Ted didn't end up backporting this into 0.92)
> >>>
> >>> - The store file locality thing I have mostly done, will finish it
> >>>
> >>> - Mikhail and crew's ongoing optimizations (HBASE-4218, etc.), the
> ones
> >> he considers fully baked
> >>>
> >>> I saw wire compatibility mentioned, for 0.96 but perhaps
> >> optional/transitional code in 0.94 as well. This is something we would
> try
> >> out and beat up upon in staging in earnest.
> >>>
> >>> Best regards,
> >>>
> >>> - Andy
> >>>
> >>> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> >> (via Tom White)
> >>>
> >>>
> >>>
> >>>> ________________________________
> >>>> From: Stack <st...@duboce.net>
> >>>> To: HBase Dev List <de...@hbase.apache.org>
> >>>> Sent: Wednesday, January 25, 2012 8:34 PM
> >>>> Subject: hbase 0.94.0
> >>>>
> >>>> Lets branch end of february? No new features thereafter. Is this too
> >>>> close in? Would be grand if 0.94.0 shipped before hbasecon. What
> >>>> should 0.94.0 have in it? I don't mind if the list is short.
> >>>>
> >>>> Unless someone else wants too, I don't mind being release manager
> >>>> (will try to run a tighter ship this time around).
> >>>>
> >>>> St.Ack
> >>>>
> >>>>
> >>>>
> >>>
> >>>
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: hbase 0.94.0
Posted by Jonathan Hsieh <jo...@cloudera.com>.
+1 to having some sort of migration mechanism especially for the files
side. I say this out of personal interest -- I'm fairly certain that at
some point I'm going to have to support these upgrades.
Jon.
On Thu, Jan 26, 2012 at 12:25 PM, Jesse Yates <je...@gmail.com>wrote:
> +1 on removing it too, but maybe we should have some sort of upgrade
> script to help make the switch?
>
> I'm thinking something pluggable on both sides of the upgrade, so people
> can go from version X->Y, rather than having to upgrade first to an
> intermediate and then to he version they want (right it would be going from
> 0.20->.92->.96, IMO an excessive PITA).
>
> Just my two cents...
>
> - Jesse Yates
>
> Sent from my iPhone.
>
> On Jan 26, 2012, at 12:16 PM, lars hofhansl <lh...@yahoo.com> wrote:
>
> > Good point.
> > 0.94 is not branched, yet. And I think generally we do not support
> skipping releases for upgrades.
> > +1 on removing HFileV1 cruft for 0.94
> >
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Matt Corgan <mc...@hotpads.com>
> > To: dev@hbase.apache.org; Andrew Purtell <ap...@apache.org>
> > Sent: Thursday, January 26, 2012 11:51 AM
> > Subject: Re: hbase 0.94.0
> >
> > Are there any thoughts about when it is ok to stop being backwards
> > compatible? Mainly thinking of HFileV1... 0.92 will convert all
> HFileV1's
> > to HFileV2's, so it would probably have been ok to delete the code for
> > HFileV1 in 0.94 and force people to upgrade through 0.92. That didn't
> > actually happen, so it's looking like folks will be able to go straight
> > from 0.90 to 0.94. But, perhaps it should be deleted for 0.96?
> >
> >
> > On Thu, Jan 26, 2012 at 11:41 AM, Andrew Purtell <apurtell@apache.org
> >wrote:
> >
> >> Yeah, so
> >>
> >> - Security (basically another coprocessor for inclusion in mainline,
> >> like Constraints)
> >>
> >> if not in 0.92.1.
> >>
> >> Best regards,
> >>
> >> - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
> >>
> >>> ________________________________
> >>> From: Andrew Purtell <ap...@apache.org>
> >>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> >>> Sent: Thursday, January 26, 2012 11:28 AM
> >>> Subject: Re: hbase 0.94.0
> >>>
> >>> A limited set of changes so we can get it out on that kind of
> timeframe.
> >> :-)
> >>>
> >>> - Constraints (is ready to go, is a coprocessor, so is in the large
> >> just a new package to drop in)
> >>>
> >>> - Useful utilities for ops:
> >>>
> >>> - LoadTestTool (if Ted didn't end up backporting this into 0.92)
> >>>
> >>> - The store file locality thing I have mostly done, will finish it
> >>>
> >>> - Mikhail and crew's ongoing optimizations (HBASE-4218, etc.), the
> ones
> >> he considers fully baked
> >>>
> >>> I saw wire compatibility mentioned, for 0.96 but perhaps
> >> optional/transitional code in 0.94 as well. This is something we would
> try
> >> out and beat up upon in staging in earnest.
> >>>
> >>> Best regards,
> >>>
> >>> - Andy
> >>>
> >>> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> >> (via Tom White)
> >>>
> >>>
> >>>
> >>>> ________________________________
> >>>> From: Stack <st...@duboce.net>
> >>>> To: HBase Dev List <de...@hbase.apache.org>
> >>>> Sent: Wednesday, January 25, 2012 8:34 PM
> >>>> Subject: hbase 0.94.0
> >>>>
> >>>> Lets branch end of february? No new features thereafter. Is this too
> >>>> close in? Would be grand if 0.94.0 shipped before hbasecon. What
> >>>> should 0.94.0 have in it? I don't mind if the list is short.
> >>>>
> >>>> Unless someone else wants too, I don't mind being release manager
> >>>> (will try to run a tighter ship this time around).
> >>>>
> >>>> St.Ack
> >>>>
> >>>>
> >>>>
> >>>
> >>>
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: hbase 0.94.0
Posted by Jesse Yates <je...@gmail.com>.
+1 on removing it too, but maybe we should have some sort of upgrade script to help make the switch?
I'm thinking something pluggable on both sides of the upgrade, so people can go from version X->Y, rather than having to upgrade first to an intermediate and then to he version they want (right it would be going from 0.20->.92->.96, IMO an excessive PITA).
Just my two cents...
- Jesse Yates
Sent from my iPhone.
On Jan 26, 2012, at 12:16 PM, lars hofhansl <lh...@yahoo.com> wrote:
> Good point.
> 0.94 is not branched, yet. And I think generally we do not support skipping releases for upgrades.
> +1 on removing HFileV1 cruft for 0.94
>
>
> -- Lars
>
>
>
> ________________________________
> From: Matt Corgan <mc...@hotpads.com>
> To: dev@hbase.apache.org; Andrew Purtell <ap...@apache.org>
> Sent: Thursday, January 26, 2012 11:51 AM
> Subject: Re: hbase 0.94.0
>
> Are there any thoughts about when it is ok to stop being backwards
> compatible? Mainly thinking of HFileV1... 0.92 will convert all HFileV1's
> to HFileV2's, so it would probably have been ok to delete the code for
> HFileV1 in 0.94 and force people to upgrade through 0.92. That didn't
> actually happen, so it's looking like folks will be able to go straight
> from 0.90 to 0.94. But, perhaps it should be deleted for 0.96?
>
>
> On Thu, Jan 26, 2012 at 11:41 AM, Andrew Purtell <ap...@apache.org>wrote:
>
>> Yeah, so
>>
>> - Security (basically another coprocessor for inclusion in mainline,
>> like Constraints)
>>
>> if not in 0.92.1.
>>
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
>>
>>> ________________________________
>>> From: Andrew Purtell <ap...@apache.org>
>>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>>> Sent: Thursday, January 26, 2012 11:28 AM
>>> Subject: Re: hbase 0.94.0
>>>
>>> A limited set of changes so we can get it out on that kind of timeframe.
>> :-)
>>>
>>> - Constraints (is ready to go, is a coprocessor, so is in the large
>> just a new package to drop in)
>>>
>>> - Useful utilities for ops:
>>>
>>> - LoadTestTool (if Ted didn't end up backporting this into 0.92)
>>>
>>> - The store file locality thing I have mostly done, will finish it
>>>
>>> - Mikhail and crew's ongoing optimizations (HBASE-4218, etc.), the ones
>> he considers fully baked
>>>
>>> I saw wire compatibility mentioned, for 0.96 but perhaps
>> optional/transitional code in 0.94 as well. This is something we would try
>> out and beat up upon in staging in earnest.
>>>
>>> Best regards,
>>>
>>> - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>>
>>>
>>>
>>>> ________________________________
>>>> From: Stack <st...@duboce.net>
>>>> To: HBase Dev List <de...@hbase.apache.org>
>>>> Sent: Wednesday, January 25, 2012 8:34 PM
>>>> Subject: hbase 0.94.0
>>>>
>>>> Lets branch end of february? No new features thereafter. Is this too
>>>> close in? Would be grand if 0.94.0 shipped before hbasecon. What
>>>> should 0.94.0 have in it? I don't mind if the list is short.
>>>>
>>>> Unless someone else wants too, I don't mind being release manager
>>>> (will try to run a tighter ship this time around).
>>>>
>>>> St.Ack
>>>>
>>>>
>>>>
>>>
>>>
Re: hbase 0.94.0
Posted by lars hofhansl <lh...@yahoo.com>.
Good point.
0.94 is not branched, yet. And I think generally we do not support skipping releases for upgrades.
+1 on removing HFileV1 cruft for 0.94
-- Lars
________________________________
From: Matt Corgan <mc...@hotpads.com>
To: dev@hbase.apache.org; Andrew Purtell <ap...@apache.org>
Sent: Thursday, January 26, 2012 11:51 AM
Subject: Re: hbase 0.94.0
Are there any thoughts about when it is ok to stop being backwards
compatible? Mainly thinking of HFileV1... 0.92 will convert all HFileV1's
to HFileV2's, so it would probably have been ok to delete the code for
HFileV1 in 0.94 and force people to upgrade through 0.92. That didn't
actually happen, so it's looking like folks will be able to go straight
from 0.90 to 0.94. But, perhaps it should be deleted for 0.96?
On Thu, Jan 26, 2012 at 11:41 AM, Andrew Purtell <ap...@apache.org>wrote:
> Yeah, so
>
> - Security (basically another coprocessor for inclusion in mainline,
> like Constraints)
>
> if not in 0.92.1.
>
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
>
> >________________________________
> > From: Andrew Purtell <ap...@apache.org>
> >To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> >Sent: Thursday, January 26, 2012 11:28 AM
> >Subject: Re: hbase 0.94.0
> >
> >A limited set of changes so we can get it out on that kind of timeframe.
> :-)
> >
> > - Constraints (is ready to go, is a coprocessor, so is in the large
> just a new package to drop in)
> >
> > - Useful utilities for ops:
> >
> > - LoadTestTool (if Ted didn't end up backporting this into 0.92)
> >
> > - The store file locality thing I have mostly done, will finish it
> >
> > - Mikhail and crew's ongoing optimizations (HBASE-4218, etc.), the ones
> he considers fully baked
> >
> >I saw wire compatibility mentioned, for 0.96 but perhaps
> optional/transitional code in 0.94 as well. This is something we would try
> out and beat up upon in staging in earnest.
> >
> >Best regards,
> >
> > - Andy
> >
> >Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
> >
> >
> >
> >>________________________________
> >> From: Stack <st...@duboce.net>
> >>To: HBase Dev List <de...@hbase.apache.org>
> >>Sent: Wednesday, January 25, 2012 8:34 PM
> >>Subject: hbase 0.94.0
> >>
> >>Lets branch end of february? No new features thereafter. Is this too
> >>close in? Would be grand if 0.94.0 shipped before hbasecon. What
> >>should 0.94.0 have in it? I don't mind if the list is short.
> >>
> >>Unless someone else wants too, I don't mind being release manager
> >>(will try to run a tighter ship this time around).
> >>
> >>St.Ack
> >>
> >>
> >>
> >
> >
>
Re: hbase 0.94.0
Posted by Matt Corgan <mc...@hotpads.com>.
Are there any thoughts about when it is ok to stop being backwards
compatible? Mainly thinking of HFileV1... 0.92 will convert all HFileV1's
to HFileV2's, so it would probably have been ok to delete the code for
HFileV1 in 0.94 and force people to upgrade through 0.92. That didn't
actually happen, so it's looking like folks will be able to go straight
from 0.90 to 0.94. But, perhaps it should be deleted for 0.96?
On Thu, Jan 26, 2012 at 11:41 AM, Andrew Purtell <ap...@apache.org>wrote:
> Yeah, so
>
> - Security (basically another coprocessor for inclusion in mainline,
> like Constraints)
>
> if not in 0.92.1.
>
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
>
> >________________________________
> > From: Andrew Purtell <ap...@apache.org>
> >To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> >Sent: Thursday, January 26, 2012 11:28 AM
> >Subject: Re: hbase 0.94.0
> >
> >A limited set of changes so we can get it out on that kind of timeframe.
> :-)
> >
> > - Constraints (is ready to go, is a coprocessor, so is in the large
> just a new package to drop in)
> >
> > - Useful utilities for ops:
> >
> > - LoadTestTool (if Ted didn't end up backporting this into 0.92)
> >
> > - The store file locality thing I have mostly done, will finish it
> >
> > - Mikhail and crew's ongoing optimizations (HBASE-4218, etc.), the ones
> he considers fully baked
> >
> >I saw wire compatibility mentioned, for 0.96 but perhaps
> optional/transitional code in 0.94 as well. This is something we would try
> out and beat up upon in staging in earnest.
> >
> >Best regards,
> >
> > - Andy
> >
> >Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
> >
> >
> >
> >>________________________________
> >> From: Stack <st...@duboce.net>
> >>To: HBase Dev List <de...@hbase.apache.org>
> >>Sent: Wednesday, January 25, 2012 8:34 PM
> >>Subject: hbase 0.94.0
> >>
> >>Lets branch end of february? No new features thereafter. Is this too
> >>close in? Would be grand if 0.94.0 shipped before hbasecon. What
> >>should 0.94.0 have in it? I don't mind if the list is short.
> >>
> >>Unless someone else wants too, I don't mind being release manager
> >>(will try to run a tighter ship this time around).
> >>
> >>St.Ack
> >>
> >>
> >>
> >
> >
>
Re: hbase 0.94.0
Posted by Andrew Purtell <ap...@apache.org>.
Yeah, so
- Security (basically another coprocessor for inclusion in mainline, like Constraints)
if not in 0.92.1.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
> From: Andrew Purtell <ap...@apache.org>
>To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>Sent: Thursday, January 26, 2012 11:28 AM
>Subject: Re: hbase 0.94.0
>
>A limited set of changes so we can get it out on that kind of timeframe. :-)
>
> - Constraints (is ready to go, is a coprocessor, so is in the large just a new package to drop in)
>
> - Useful utilities for ops:
>
> - LoadTestTool (if Ted didn't end up backporting this into 0.92)
>
> - The store file locality thing I have mostly done, will finish it
>
> - Mikhail and crew's ongoing optimizations (HBASE-4218, etc.), the ones he considers fully baked
>
>I saw wire compatibility mentioned, for 0.96 but perhaps optional/transitional code in 0.94 as well. This is something we would try out and beat up upon in staging in earnest.
>
>Best regards,
>
> - Andy
>
>Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>
>
>
>>________________________________
>> From: Stack <st...@duboce.net>
>>To: HBase Dev List <de...@hbase.apache.org>
>>Sent: Wednesday, January 25, 2012 8:34 PM
>>Subject: hbase 0.94.0
>>
>>Lets branch end of february? No new features thereafter. Is this too
>>close in? Would be grand if 0.94.0 shipped before hbasecon. What
>>should 0.94.0 have in it? I don't mind if the list is short.
>>
>>Unless someone else wants too, I don't mind being release manager
>>(will try to run a tighter ship this time around).
>>
>>St.Ack
>>
>>
>>
>
>
Re: hbase 0.94.0
Posted by Andrew Purtell <ap...@apache.org>.
A limited set of changes so we can get it out on that kind of timeframe. :-)
- Constraints (is ready to go, is a coprocessor, so is in the large just a new package to drop in)
- Useful utilities for ops:
- LoadTestTool (if Ted didn't end up backporting this into 0.92)
- The store file locality thing I have mostly done, will finish it
- Mikhail and crew's ongoing optimizations (HBASE-4218, etc.), the ones he considers fully baked
I saw wire compatibility mentioned, for 0.96 but perhaps optional/transitional code in 0.94 as well. This is something we would try out and beat up upon in staging in earnest.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>________________________________
> From: Stack <st...@duboce.net>
>To: HBase Dev List <de...@hbase.apache.org>
>Sent: Wednesday, January 25, 2012 8:34 PM
>Subject: hbase 0.94.0
>
>Lets branch end of february? No new features thereafter. Is this too
>close in? Would be grand if 0.94.0 shipped before hbasecon. What
>should 0.94.0 have in it? I don't mind if the list is short.
>
>Unless someone else wants too, I don't mind being release manager
>(will try to run a tighter ship this time around).
>
>St.Ack
>
>
>