You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Devaraj Das <dd...@hortonworks.com> on 2013/02/06 22:53:08 UTC
Re: [DISCUSSION] Sorting out issues for 0.96 for (eventual) release
Stack, did you lose the bet to JD yet :-)
But seriously, should we consider branching 0.96 now? I know there are
quite a few blockers/criticals but they can be fixed/merged even after
the branching. Given that we want to release sooner a 0.96, I think it
makes sense to branch sooner as well and lock the scope somewhat (that
will also hopefully avoid creating more new blockers).
What is the absolute list of things we must wait for before branching?
Should it be only Snapshots? Should it be RPC, KeyValue serialization
& Snapshots. There are many other things being developed in trunk and
many of them are probably okay for 0.96 as well if they are completed
on time. But I think we should put a stake on the ground and consider
a date for branching and work towards achieving that. After that it'd
be up to the RM to consider something for 0.96 or not.
Should we aim for Feb 28 as the date for branching. Then we can have
an RC in approximately 1.5 - 2 months (April mid-end), and it would
probably be in line with what Andrew was thinking about when he
started this thread. Does it sound too aggressive?
Thanks,
Devaraj.
On Thu, Jan 10, 2013 at 4:02 PM, Stack <st...@duboce.net> wrote:
> We tried to doc our versioning up on wiki:
> http://wiki.apache.org/hadoop/Hbase/HBaseVersions?action=edit&editor=text
> Its out of date and needs moving into the ref guide but there is a
> section
> we could reuse that Todd did for 89, the "Development Release", explaining
> 0.95.
>
> (Chatting with Todd, he suggested not waiting for 0.96 to branch but just
> branch first 0.95 from trunk -- we could do that too though I wouldn't mind
> the rpc stabilizing first...)
>
> St.Ack
>
>
> On Thu, Jan 10, 2013 at 3:38 PM, Andrew Purtell <ap...@apache.org> wrote:
>
>> On Thu, Jan 10, 2013 at 3:34 PM, Enis Söztutar <en...@gmail.com> wrote:
>>
>> > I dont see why odd/even is better than adding -dev/-beta suffix,
>> > [...] If historically, 0.89/0.90 worked well, we
>> > might as well go with it.
>> >
>>
>> Right, the notion is to just do what we did before, having had this
>> discussion previously, to avoid going over this again.
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
Re: [DISCUSSION] Sorting out issues for 0.96 for (eventual) release
Posted by Stack <st...@duboce.net>.
On Wed, Feb 6, 2013 at 2:11 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> Can someone summarize where we are at with the RPC currently? The KV
> stuff is to make sure we don't regress from core get/put perf
> characteristics of 0.94 right?
>
On RPC/KV Serialiization:
A patch that specifies new pb-based rpc format is up here:
https://issues.apache.org/jira/browse/HBASE-7533 Still TODO is reedit of
rpc spec and implementation of how we will pass blocks of keyvalues. Its
been spec'd, but as you'd expect, after you've done an implementation, its
funny how the spec needs to change (for example, I thought we could pass
EncodedDataBlocks originally). I need to finish implementation.
Passing blocks of KVs everywhere will not be done for 0.96. This would
require reaching up into base Mutation objects client-side and ditto on the
server-side refactoring so throughout the pipeline we do not have
dependencies on the current serialization format; rather we'd rely on the
Cell Interface that is out in hbase-common instead. This refactoring is a
bunch of work and won't happen anytime soon. I was just going to do it a
few places to prove the passing of blocks-of-kvs mechanism works.
We have to do blocks of KeyValues rather than passing of pb'd KVs because
pb marshalling/unmarshalling is 10x slower than our current dumb passing of
the kv backing byte array. We can't keep passing the current kv backing
byte array because KVs need to evolve going forward [1].
St.Ack
1.
https://docs.google.com/document/d/1WEtrq-JTIUhlnlnvA0oYRLp0F8MKpEBeBSCFcQiacdw/edit#heading=h.su4gd8tohaza
Re: [DISCUSSION] Sorting out issues for 0.96 for (eventual) release
Posted by Jonathan Hsieh <jo...@cloudera.com>.
I'm am in general +1 for this scope lock and branching soon. There
are a whole lot of non-trivial refactors/features that are close to
commit but I think will cause pain as we try to stabilize. And with
some of the internals changes that are being proposed (new fs layout)
and have been prototyped (removing root, etc) it looks like 0.96 will
be the first singularity.. to be followed by another singularity
called 0.98 (or 2.0)
Can someone summarize where we are at with the RPC currently? The KV
stuff is to make sure we don't regress from core get/put perf
characteristics of 0.94 right?
Snapshot work is currently bogged down trying to get tests to pass on
a snapshot+trunk merge (see HBASE-7290 for issues). :(. Once I get a
clean run there I plan on formally posting the branch for review and
doing the real cluster testing on the merge snapshot+trunk branch
instead of just the snapshot branch. On this front, we've got the
failures categorized and at the moment have hacks (some unacceptable,
but pointing towards root cause) to fix all but one of these failures.
Jon.
On Wed, Feb 6, 2013 at 1:53 PM, Devaraj Das <dd...@hortonworks.com> wrote:
> Stack, did you lose the bet to JD yet :-)
>
> But seriously, should we consider branching 0.96 now? I know there are
> quite a few blockers/criticals but they can be fixed/merged even after
> the branching. Given that we want to release sooner a 0.96, I think it
> makes sense to branch sooner as well and lock the scope somewhat (that
> will also hopefully avoid creating more new blockers).
>
> What is the absolute list of things we must wait for before branching?
> Should it be only Snapshots? Should it be RPC, KeyValue serialization
> & Snapshots. There are many other things being developed in trunk and
> many of them are probably okay for 0.96 as well if they are completed
> on time. But I think we should put a stake on the ground and consider
> a date for branching and work towards achieving that. After that it'd
> be up to the RM to consider something for 0.96 or not.
>
> Should we aim for Feb 28 as the date for branching. Then we can have
> an RC in approximately 1.5 - 2 months (April mid-end), and it would
> probably be in line with what Andrew was thinking about when he
> started this thread. Does it sound too aggressive?
>
> Thanks,
> Devaraj.
>
> On Thu, Jan 10, 2013 at 4:02 PM, Stack <st...@duboce.net> wrote:
>> We tried to doc our versioning up on wiki:
>> http://wiki.apache.org/hadoop/Hbase/HBaseVersions?action=edit&editor=text
>> Its out of date and needs moving into the ref guide but there is a
>> section
>> we could reuse that Todd did for 89, the "Development Release", explaining
>> 0.95.
>>
>> (Chatting with Todd, he suggested not waiting for 0.96 to branch but just
>> branch first 0.95 from trunk -- we could do that too though I wouldn't mind
>> the rpc stabilizing first...)
>>
>> St.Ack
>>
>>
>> On Thu, Jan 10, 2013 at 3:38 PM, Andrew Purtell <ap...@apache.org> wrote:
>>
>>> On Thu, Jan 10, 2013 at 3:34 PM, Enis Söztutar <en...@gmail.com> wrote:
>>>
>>> > I dont see why odd/even is better than adding -dev/-beta suffix,
>>> > [...] If historically, 0.89/0.90 worked well, we
>>> > might as well go with it.
>>> >
>>>
>>> Right, the notion is to just do what we did before, having had this
>>> discussion previously, to avoid going over this again.
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>> (via Tom White)
>>>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: [DISCUSSION] Sorting out issues for 0.96 for (eventual) release
Posted by Stack <st...@duboce.net>.
No feedback on whether February 15th is ok to branch? Lets say 22nd then.
If no complaint, will put it up as branch day.
Thanks all,
St.Ack
On Thu, Feb 7, 2013 at 12:41 AM, Nicolas Liochon <nk...@gmail.com> wrote:
> +1 for the maven style snapshots: at the beginning, we can create them
> without branching.
>
>
> On Thu, Feb 7, 2013 at 12:29 AM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > Sounds good to me, and you're the RM anyway. :-)
> >
> >
> > On Wed, Feb 6, 2013 at 3:26 PM, Stack <st...@duboce.net> wrote:
> >
> > > On Wed, Feb 6, 2013 at 3:11 PM, Andrew Purtell <ap...@apache.org>
> > > wrote:
> > >
> > > > However, we could branch and start releasing 0.95.<date> developer
> > > preview
> > > > releases pretty soon, right?, to start testing the relative stability
> > as
> > > > these things land.
> > > >
> > > >
> > > Rather than 0.95, I was thinking we could just do maven-style snapshot
> > > releases: i.e. 0.96.0.MAVEN_SUPPLIED_DATE
> > >
> > > St.Ack
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
Re: [DISCUSSION] Sorting out issues for 0.96 for (eventual) release
Posted by Nicolas Liochon <nk...@gmail.com>.
+1 for the maven style snapshots: at the beginning, we can create them
without branching.
On Thu, Feb 7, 2013 at 12:29 AM, Andrew Purtell <ap...@apache.org> wrote:
> Sounds good to me, and you're the RM anyway. :-)
>
>
> On Wed, Feb 6, 2013 at 3:26 PM, Stack <st...@duboce.net> wrote:
>
> > On Wed, Feb 6, 2013 at 3:11 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > However, we could branch and start releasing 0.95.<date> developer
> > preview
> > > releases pretty soon, right?, to start testing the relative stability
> as
> > > these things land.
> > >
> > >
> > Rather than 0.95, I was thinking we could just do maven-style snapshot
> > releases: i.e. 0.96.0.MAVEN_SUPPLIED_DATE
> >
> > St.Ack
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: [DISCUSSION] Sorting out issues for 0.96 for (eventual) release
Posted by Andrew Purtell <ap...@apache.org>.
Sounds good to me, and you're the RM anyway. :-)
On Wed, Feb 6, 2013 at 3:26 PM, Stack <st...@duboce.net> wrote:
> On Wed, Feb 6, 2013 at 3:11 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > However, we could branch and start releasing 0.95.<date> developer
> preview
> > releases pretty soon, right?, to start testing the relative stability as
> > these things land.
> >
> >
> Rather than 0.95, I was thinking we could just do maven-style snapshot
> releases: i.e. 0.96.0.MAVEN_SUPPLIED_DATE
>
> St.Ack
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [DISCUSSION] Sorting out issues for 0.96 for (eventual) release
Posted by Stack <st...@duboce.net>.
On Wed, Feb 6, 2013 at 3:11 PM, Andrew Purtell <ap...@apache.org> wrote:
> However, we could branch and start releasing 0.95.<date> developer preview
> releases pretty soon, right?, to start testing the relative stability as
> these things land.
>
>
Rather than 0.95, I was thinking we could just do maven-style snapshot
releases: i.e. 0.96.0.MAVEN_SUPPLIED_DATE
St.Ack
Re: [DISCUSSION] Sorting out issues for 0.96 for (eventual) release
Posted by Andrew Purtell <ap...@apache.org>.
+1 these need to happen before release:
On Wed, Feb 6, 2013 at 2:29 PM, Stack <st...@duboce.net> wrote:
> Was waiting on the outstanding big code movements -- snapshots coming in,
> prefix-tree, client module moves -- and nailing rpc format /keyvalue
> serialization going forward for 0.96 so we don't have to do it in two
> places.
However, we could branch and start releasing 0.95.<date> developer preview
releases pretty soon, right?, to start testing the relative stability as
these things land.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [DISCUSSION] Sorting out issues for 0.96 for (eventual) release
Posted by Stack <st...@duboce.net>.
On Wed, Feb 6, 2013 at 1:53 PM, Devaraj Das <dd...@hortonworks.com> wrote:
> Stack, did you lose the bet to JD yet :-)
>
>
It is not looking good.
> But seriously, should we consider branching 0.96 now? I know there are
> quite a few blockers/criticals but they can be fixed/merged even after
> the branching. Given that we want to release sooner a 0.96, I think it
> makes sense to branch sooner as well and lock the scope somewhat (that
> will also hopefully avoid creating more new blockers).
>
>
Was waiting on the outstanding big code movements -- snapshots coming in,
prefix-tree, client module moves -- and nailing rpc format /keyvalue
serialization going forward for 0.96 so we don't have to do it in two
places.
> Should it be only Snapshots? Should it be RPC, KeyValue serialization
> & Snapshots. There are many other things being developed in trunk and
> many of them are probably okay for 0.96 as well if they are completed
> on time. But I think we should put a stake on the ground and consider
> a date for branching and work towards achieving that. After that it'd
> be up to the RM to consider something for 0.96 or not.
>
>
Agreed.
> Should we aim for Feb 28 as the date for branching. Then we can have
> an RC in approximately 1.5 - 2 months (April mid-end), and it would
> probably be in line with what Andrew was thinking about when he
> started this thread. Does it sound too aggressive?
>
How about Feb15th?
Thanks for revivifying this thread DD.
St.Ack