You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Stack <st...@duboce.net> on 2013/07/27 23:52:40 UTC
[UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
The end of July is upon us. I intend to cut a 0.95.2 next weekend and
0.95.2 will be 0.96.0 but for bug fixes and migration polish. No more
'features' will be allowed after next weekend.
I have done some moving around of hbase 0.95.2 issues [1] to reflect what
the priorities are for this week (to my mind). It is all about polish, bad
bugs, unit test failures, and packaging/publishing/build issues.
I'll be working on the blockers this week [2]. A few are in need of
reviews; e.g. "HBASE-3787 Increment is non-idempotent but client retries
RPC". Feel free to take over any blockers if you'd like to help out: e.g.
"HBASE-7386 Investigate providing some supervisor support for znode
deletion"
Criticals [3] are mostly just unit test issues that are probably fixed by
now and another few that are patches in need of test/review: e.g.
"HBASE-8874 PutCombiner is skipping KeyValues while combining puts of same
row during bulkload" and "HBASE-8778 Region assigments scan table directory
making them slow for huge tables".
"HBASE-6127 TestAtomicOperation#testMultiRowMutationMultiThreads
occasionally fails" is a bad one in need of attention. Again, if up for
helping out, be our guest.
There are some great patches hanging out in the priority major issues
section that are patch available that could be committed but for want of
review: e.g. "HBASE-8369 MapReduce over snapshot files".
Regards the big features that are racing to make the 0.96 cutoff -- namely
namespaces, tags, and serialization lib -- as I see it, Francis needs
reviews if namespaces are to make it, tags ditto, and the serialization
libs are nice-to-have auxillaries that can come in at any time. If not
done by the end of this week, then as I see it, these features do not make
the cut.
Unit tests are mostly passing. The problematics are being worked on (e.g.
JD is on the replication set -- it likely a real problem rather than a
flakey test).
How's the above sound?
St.Ack
P.S. It is late but if folks want to meet this week to hack in patches
together, just say. I could organize an afternoon or day if you all think
it would be a good idea.
1.
https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel
2.
https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%20%220.95.2%22%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Blocker%20ORDER%20BY%20key%20DESC
3.
https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%20%220.95.2%22%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Critical%20ORDER%20BY%20key%20DESC
On Tue, Jul 9, 2013 at 1:14 PM, Stack <st...@duboce.net> wrote:
> I am shooting for end of July for 0.96 being 'complete'. I would like to
> make a 0.96 release in August. We have some criticals outstanding but I
> think we could ship even if these are not fixed in time (excepting
> migration polish and of course remaining build fixes). See [1.] for the
> current list of issues. Please re-prioritize issues as you see fit (or
> better, move issues out of 0.95.2 if you do not think they will be done in
> time).
>
> What to do with namespaces -- the last 0.96 'feature' -- given the above
> timeline? Currently it is a massive patch out on a branch. It is still
> not done, in want of review, and the author is going on holidays for a few
> weeks soon. My thinking as of now, going by the rate of change over the
> last few weeks and estimating what is yet to be done, is that namespaces
> will not make it. I am willing to be convinced otherwise but that is how
> it looks to me currently.
>
> I am going to start just disabling flakey unit tests in 0.95 from here on
> out. When folks get the itch, they can fix at leisure first on trunk and
> then over in 0.95.
>
> What else?
>
> Thanks,
> St.Ack
>
> 1.
> https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel
>
>
>
> On Mon, Jun 24, 2013 at 8:45 PM, Stack <st...@duboce.net> wrote:
>
>> (Changed the subject)
>>
>> On Mon, Jun 24, 2013 at 4:04 PM, Nick Dimiduk <nd...@gmail.com> wrote:
>>
>>> I want to see initial data type APIs ship out with 0.95.2. A patch for
>>> ordered byte serialization is up (HBASE-8201) and is nearing
>>> steady-state.
>>> However, sershe is the only person who's left feedback. I just posted an
>>> early patch for the data type API itself (HBASE-8693). It should get some
>>> eyes from all manor of interested parties, but I'll settle for folk from
>>> Phoenix for now.
>>>
>>>
>> It would be cool if Phoenix and Kiji fellows and any one else interested
>> would weigh in and take a look see.
>>
>> This does not strike me as something we should hold up the release for
>> though. It looks like something that could go in at any time?
>>
>>
>>
>>> Should these tasks be escalated to criticals in order to grab attention?
>>>
>>>
>> I don't think that works going by past experience (and I don't think this
>> a blocker on 0.96)
>>
>>
>>
>>> Additional comments inline.
>>>
>>>
>> ...
>>
>>
>>
>>> Namespaces is the long pole and progress seems slow. Do we hold up the
>>> > release for them? How can we hurry this effort along? Swat team
>>> descends
>>> > on Y!?
>>> >
>>>
>>> It would be a shame to not get a decision on this in for 0.96.
>>>
>>>
>> Agree. We need to get 0.96 out though. It has been too long.
>>
>>
>>
>>> + Is anyone testing? Integration tests fail on ec2 build from time to
>>> time
>>> > [2]. Our Elliott dug in on one of the failures a few days back and
>>> found
>>> > legit issue w/ no retry on admin tasks (I heart hbase-it tests). Our
>>> unit
>>> > test story is better [3] but there are still the odd failures.
>>> >
>>>
>>> With the creation of the new list, noticing these issues is going to push
>>> further back-burner. Nannying this stuff should retain focus. I'll
>>> volunteer to track on these issues as I see them.
>>>
>>>
>> Thank you Nick.
>>
>>
>>
>>> Notice though that some of the more recent failures are caused by lack of
>>> disk space on the Jenkins build host.
>>>
>>>
>> Oh. Missed that. Let me dig in.
>>
>>
>> St.Ack
>>
>
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
On Tue, Aug 13, 2013 at 5:58 PM, Elliott Clark <ec...@apache.org> wrote:
> This change can't be done on a later date since it changes HTrace's
> public api that users of HBase would be calling.
>
Makes sense, I would concede this point. As long as there's an artifact up
in Maven central (or another public repo) before the commit goes in it's
fine by me technically.
In terms of process, the remainder of our disagreement isn't settled, your
explanation does not match up with your earlier reasoning in the case of
our work and moves the goalposts, as I have outlined, but this is moot now.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Elliott Clark <ec...@apache.org>.
I have explained why this patch is very different than a new HFile +
Tags, but I'm more than willing to re-iterate those reasons.
* It's at least an order of magnitude smaller than HFile v3.
* It's not a new feature.
* It's a minor tweak on one that's already there
* It's much less risky.
* It's in a production ready state now.
* It's already got a plus one.
* It's been running on integration test clusters for a while (as
witnessed by it being used to find actual bugs in 0.95).
* This change can't be done on a later date since it changes HTrace's
public api that users of HBase would be calling.
On Tue, Aug 13, 2013 at 5:40 PM, Andrew Purtell <ap...@apache.org> wrote:
> Elliott,
>
> It would be good if you could address the specific points that I have
> called out. I want to be sure that at least our decision making is
> consistent and without the appearance of a double standard.
>
>
> On Tue, Aug 13, 2013 at 5:24 PM, Elliott Clark <ec...@apache.org> wrote:
>
>> Any thoughts on HBASE-9121. At it's heart it's just a version bump of
>> HTrace to 2.0 (fully released into maven central and available on
>> Cloudera's github). The new version of htrace is modularized so that
>> hbase-client doesn't drag in any new dependencies on the client side.
>>
>> Along with that modularization we would get the ability for 0.95/0.96
>> to relay to Zipkin producing some pretty useful info for stabilizing
>> the release(tracing is what found that scan pre-fetching was broken)
>> and very useful for our users.
>>
>> For me I think that it's small enough that it would meet a higher bar
>> to get things into 0.95. But since there were some thoughts on here
>> I've held off on comitting.
>>
>> On Mon, Aug 5, 2013 at 10:01 AM, Andrew Purtell <ap...@apache.org>
>> wrote:
>> > Clarification would be great. I read Elliot's veto of our work as based
>> on
>> > both process and maturity concerns. Let's see how they stack up.
>> >
>> >> HFileV3:
>> >> * Showed up very late in the merge window
>> >
>> > Same
>> >
>> >> * Still needs revisions
>> >
>> > Has not even been reviewed yet. Depends on code only existing in a
>> > developer's private GitHub.
>> >
>> >> * Hasn't been put on large clusters publicly.
>> >
>> > Same
>> >
>> >> * Is not the green field HFileV3, and doesn't have time for a complete
>> > re-do
>> >
>> > Not applicable
>> >
>> >> * Can easily be done without out downtime
>> >
>> > Same
>> >
>> >> * Will 100% have perf impacts.
>> >
>> > See my other comment about perf impacts if V3 is used without tags.
>> >
>> >
>> > I have no objection to the trace work on technical grounds.
>> >
>> >
>> > On Mon, Aug 5, 2013 at 8:31 AM, Stack <st...@duboce.net> wrote:
>> >
>> >> On Sat, Aug 3, 2013 at 3:10 PM, Andrew Purtell <ap...@apache.org>
>> >> wrote:
>> >>
>> >> > JIRA says that was created August 3, ie today, is that right?
>> >> >
>> >> >
>> >> Yes.
>> >>
>> >> (Elliott can clarify later when he gets in) My understanding is that
>> it is
>> >> an update to our bundled htrace jar -- pushing a new release -- and
>> adding
>> >> some trace extra trace spans to expose where time is being spent during
>> >> MTTR.
>> >>
>> >> St.Ack
>> >>
>> >
>> >
>> >
>> > --
>> > Best regards,
>> >
>> > - Andy
>> >
>> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> > (via Tom White)
>>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
Elliott,
It would be good if you could address the specific points that I have
called out. I want to be sure that at least our decision making is
consistent and without the appearance of a double standard.
On Tue, Aug 13, 2013 at 5:24 PM, Elliott Clark <ec...@apache.org> wrote:
> Any thoughts on HBASE-9121. At it's heart it's just a version bump of
> HTrace to 2.0 (fully released into maven central and available on
> Cloudera's github). The new version of htrace is modularized so that
> hbase-client doesn't drag in any new dependencies on the client side.
>
> Along with that modularization we would get the ability for 0.95/0.96
> to relay to Zipkin producing some pretty useful info for stabilizing
> the release(tracing is what found that scan pre-fetching was broken)
> and very useful for our users.
>
> For me I think that it's small enough that it would meet a higher bar
> to get things into 0.95. But since there were some thoughts on here
> I've held off on comitting.
>
> On Mon, Aug 5, 2013 at 10:01 AM, Andrew Purtell <ap...@apache.org>
> wrote:
> > Clarification would be great. I read Elliot's veto of our work as based
> on
> > both process and maturity concerns. Let's see how they stack up.
> >
> >> HFileV3:
> >> * Showed up very late in the merge window
> >
> > Same
> >
> >> * Still needs revisions
> >
> > Has not even been reviewed yet. Depends on code only existing in a
> > developer's private GitHub.
> >
> >> * Hasn't been put on large clusters publicly.
> >
> > Same
> >
> >> * Is not the green field HFileV3, and doesn't have time for a complete
> > re-do
> >
> > Not applicable
> >
> >> * Can easily be done without out downtime
> >
> > Same
> >
> >> * Will 100% have perf impacts.
> >
> > See my other comment about perf impacts if V3 is used without tags.
> >
> >
> > I have no objection to the trace work on technical grounds.
> >
> >
> > On Mon, Aug 5, 2013 at 8:31 AM, Stack <st...@duboce.net> wrote:
> >
> >> On Sat, Aug 3, 2013 at 3:10 PM, Andrew Purtell <ap...@apache.org>
> >> wrote:
> >>
> >> > JIRA says that was created August 3, ie today, is that right?
> >> >
> >> >
> >> Yes.
> >>
> >> (Elliott can clarify later when he gets in) My understanding is that
> it is
> >> an update to our bundled htrace jar -- pushing a new release -- and
> adding
> >> some trace extra trace spans to expose where time is being spent during
> >> MTTR.
> >>
> >> St.Ack
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Elliott Clark <ec...@apache.org>.
Any thoughts on HBASE-9121. At it's heart it's just a version bump of
HTrace to 2.0 (fully released into maven central and available on
Cloudera's github). The new version of htrace is modularized so that
hbase-client doesn't drag in any new dependencies on the client side.
Along with that modularization we would get the ability for 0.95/0.96
to relay to Zipkin producing some pretty useful info for stabilizing
the release(tracing is what found that scan pre-fetching was broken)
and very useful for our users.
For me I think that it's small enough that it would meet a higher bar
to get things into 0.95. But since there were some thoughts on here
I've held off on comitting.
On Mon, Aug 5, 2013 at 10:01 AM, Andrew Purtell <ap...@apache.org> wrote:
> Clarification would be great. I read Elliot's veto of our work as based on
> both process and maturity concerns. Let's see how they stack up.
>
>> HFileV3:
>> * Showed up very late in the merge window
>
> Same
>
>> * Still needs revisions
>
> Has not even been reviewed yet. Depends on code only existing in a
> developer's private GitHub.
>
>> * Hasn't been put on large clusters publicly.
>
> Same
>
>> * Is not the green field HFileV3, and doesn't have time for a complete
> re-do
>
> Not applicable
>
>> * Can easily be done without out downtime
>
> Same
>
>> * Will 100% have perf impacts.
>
> See my other comment about perf impacts if V3 is used without tags.
>
>
> I have no objection to the trace work on technical grounds.
>
>
> On Mon, Aug 5, 2013 at 8:31 AM, Stack <st...@duboce.net> wrote:
>
>> On Sat, Aug 3, 2013 at 3:10 PM, Andrew Purtell <ap...@apache.org>
>> wrote:
>>
>> > JIRA says that was created August 3, ie today, is that right?
>> >
>> >
>> Yes.
>>
>> (Elliott can clarify later when he gets in) My understanding is that it is
>> an update to our bundled htrace jar -- pushing a new release -- and adding
>> some trace extra trace spans to expose where time is being spent during
>> MTTR.
>>
>> St.Ack
>>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
Clarification would be great. I read Elliot's veto of our work as based on
both process and maturity concerns. Let's see how they stack up.
> HFileV3:
> * Showed up very late in the merge window
Same
> * Still needs revisions
Has not even been reviewed yet. Depends on code only existing in a
developer's private GitHub.
> * Hasn't been put on large clusters publicly.
Same
> * Is not the green field HFileV3, and doesn't have time for a complete
re-do
Not applicable
> * Can easily be done without out downtime
Same
> * Will 100% have perf impacts.
See my other comment about perf impacts if V3 is used without tags.
I have no objection to the trace work on technical grounds.
On Mon, Aug 5, 2013 at 8:31 AM, Stack <st...@duboce.net> wrote:
> On Sat, Aug 3, 2013 at 3:10 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > JIRA says that was created August 3, ie today, is that right?
> >
> >
> Yes.
>
> (Elliott can clarify later when he gets in) My understanding is that it is
> an update to our bundled htrace jar -- pushing a new release -- and adding
> some trace extra trace spans to expose where time is being spent during
> MTTR.
>
> St.Ack
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Stack <st...@duboce.net>.
On Sat, Aug 3, 2013 at 3:10 PM, Andrew Purtell <ap...@apache.org> wrote:
> JIRA says that was created August 3, ie today, is that right?
>
>
Yes.
(Elliott can clarify later when he gets in) My understanding is that it is
an update to our bundled htrace jar -- pushing a new release -- and adding
some trace extra trace spans to expose where time is being spent during
MTTR.
St.Ack
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
JIRA says that was created August 3, ie today, is that right?
On Saturday, August 3, 2013, Jean-Marc Spaggiari wrote:
> Is it HBASE-9121 ?
>
> 2013/8/3 Andrew Purtell <apurtell@apache.org <javascript:;>>
>
> > Is there a patch available for the tracing work mentioned? I don't think
> > I've seen anything about it.
> >
> >
> > On Saturday, August 3, 2013, Stack wrote:
> >
> > > On Sat, Jul 27, 2013 at 2:52 PM, Stack <stack@duboce.net<javascript:;><javascript:;>>
> > > wrote:
> > >
> > > > The end of July is upon us. I intend to cut a 0.95.2 next weekend
> and
> > > > 0.95.2 will be 0.96.0 but for bug fixes and migration polish. No
> more
> > > > 'features' will be allowed after next weekend.
> > > >
> > > >
> > > >
> > > > I have done some moving around of hbase 0.95.2 issues [1] to reflect
> > what
> > > > the priorities are for this week (to my mind). It is all about
> polish,
> > > bad
> > > > bugs, unit test failures, and packaging/publishing/build issues.
> > > >
> > > > I'll be working on the blockers this week [2]. A few are in need of
> > > > reviews; e.g. "HBASE-3787 Increment is non-idempotent but client
> > retries
> > > > RPC". Feel free to take over any blockers if you'd like to help out:
> > > e.g.
> > > > "HBASE-7386 Investigate providing some supervisor support for znode
> > > > deletion"
> > > >
> > > > Criticals [3] are mostly just unit test issues that are probably
> fixed
> > by
> > > > now and another few that are patches in need of test/review: e.g.
> > > > "HBASE-8874 PutCombiner is skipping KeyValues while combining puts of
> > > same
> > > > row during bulkload" and "HBASE-8778 Region assigments scan table
> > > directory
> > > > making them slow for huge tables".
> > > > "HBASE-6127 TestAtomicOperation#testMultiRowMutationMultiThreads
> > > > occasionally fails" is a bad one in need of attention. Again, if up
> > for
> > > > helping out, be our guest.
> > > >
> > > > There are some great patches hanging out in the priority major issues
> > > > section that are patch available that could be committed but for want
> > of
> > > > review: e.g. "HBASE-8369 MapReduce over snapshot files".
> > > >
> > > > Regards the big features that are racing to make the 0.96 cutoff --
> > > namely
> > > > namespaces, tags, and serialization lib -- as I see it, Francis needs
> > > > reviews if namespaces are to make it, tags ditto, and the
> serialization
> > > > libs are nice-to-have auxillaries that can come in at any time. If
> not
> > > > done by the end of this week, then as I see it, these features do not
> > > make
> > > > the cut.
> > > >
> > > > Unit tests are mostly passing. The problematics are being worked on
> > > (e.g.
> > > > JD is on the replication set -- it likely a real problem rather than
> a
> > > > flakey test).
> > > >
> > > > How's the above sound?
> > > > St.Ack
> > > >
> > > > P.S. It is late but if folks want to meet this week to hack in
> patches
> > > > together, just say. I could organize an afternoon or day if you all
> > > think
> > > > it would be a good idea.
> > > >
> > > > 1.
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel
> > > > 2.
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%20%220.95.2%22%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Blocker%20ORDER%20BY%20key%20DESC
> > > > 3.
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%20%220.95.2%22%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Critical%20ORDER%20BY%20key%20DESC
> > > >
> > > >
> > > >
> > > UPDATE:
> > >
> > > I was to cut 0
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Is it HBASE-9121 ?
2013/8/3 Andrew Purtell <ap...@apache.org>
> Is there a patch available for the tracing work mentioned? I don't think
> I've seen anything about it.
>
>
> On Saturday, August 3, 2013, Stack wrote:
>
> > On Sat, Jul 27, 2013 at 2:52 PM, Stack <stack@duboce.net <javascript:;>>
> > wrote:
> >
> > > The end of July is upon us. I intend to cut a 0.95.2 next weekend and
> > > 0.95.2 will be 0.96.0 but for bug fixes and migration polish. No more
> > > 'features' will be allowed after next weekend.
> > >
> > >
> > >
> > > I have done some moving around of hbase 0.95.2 issues [1] to reflect
> what
> > > the priorities are for this week (to my mind). It is all about polish,
> > bad
> > > bugs, unit test failures, and packaging/publishing/build issues.
> > >
> > > I'll be working on the blockers this week [2]. A few are in need of
> > > reviews; e.g. "HBASE-3787 Increment is non-idempotent but client
> retries
> > > RPC". Feel free to take over any blockers if you'd like to help out:
> > e.g.
> > > "HBASE-7386 Investigate providing some supervisor support for znode
> > > deletion"
> > >
> > > Criticals [3] are mostly just unit test issues that are probably fixed
> by
> > > now and another few that are patches in need of test/review: e.g.
> > > "HBASE-8874 PutCombiner is skipping KeyValues while combining puts of
> > same
> > > row during bulkload" and "HBASE-8778 Region assigments scan table
> > directory
> > > making them slow for huge tables".
> > > "HBASE-6127 TestAtomicOperation#testMultiRowMutationMultiThreads
> > > occasionally fails" is a bad one in need of attention. Again, if up
> for
> > > helping out, be our guest.
> > >
> > > There are some great patches hanging out in the priority major issues
> > > section that are patch available that could be committed but for want
> of
> > > review: e.g. "HBASE-8369 MapReduce over snapshot files".
> > >
> > > Regards the big features that are racing to make the 0.96 cutoff --
> > namely
> > > namespaces, tags, and serialization lib -- as I see it, Francis needs
> > > reviews if namespaces are to make it, tags ditto, and the serialization
> > > libs are nice-to-have auxillaries that can come in at any time. If not
> > > done by the end of this week, then as I see it, these features do not
> > make
> > > the cut.
> > >
> > > Unit tests are mostly passing. The problematics are being worked on
> > (e.g.
> > > JD is on the replication set -- it likely a real problem rather than a
> > > flakey test).
> > >
> > > How's the above sound?
> > > St.Ack
> > >
> > > P.S. It is late but if folks want to meet this week to hack in patches
> > > together, just say. I could organize an afternoon or day if you all
> > think
> > > it would be a good idea.
> > >
> > > 1.
> > >
> >
> https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel
> > > 2.
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%20%220.95.2%22%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Blocker%20ORDER%20BY%20key%20DESC
> > > 3.
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%20%220.95.2%22%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Critical%20ORDER%20BY%20key%20DESC
> > >
> > >
> > >
> > UPDATE:
> >
> > I was to cut 0.95.2 this weekend. I am pushing out the cut-date to Weds
> or
> > so of next week. The gating factor is namespaces. It needs a few more
> > days of patch cycling. I'll cut 0.95.2 after it goes in. It'll be the
> > last feature on the 0.95/0.96 branch. Thereafter, only bug fixes,
> > migration cleanup, and doc additions will be allowed (0.96.0RC will
> follow
> > soon after the 0.95.2 developer release goes out).
> >
> > It looks like the serialization ilb will make the cut, API+Ordering; Nick
> > has signed-on some consumers it seems.
> >
> > On KV tags, we have an outstanding -1 but -1s can change; lets see what
> the
> > new posted patch looks like.
> >
> > Elliott has fancy-pants tracing that he should be able to get in before
> > 0.95.2.
> >
> > On the dodgy-looking outstanding blockers:
> >
> > "HBASE-3787 Increment is non-idempotent but client retries RPC" still
> needs
> > reviews. It is a difficult problem well-researched by our Sergey;
> perhaps
> > this does not make it?
> >
> > On "HBASE-7386 Investigate providing some supervisor support for znode
> > deletion" we could doc. the "ugly" wrapper/watcher process with why it
> > exists and suggest it should be supervise instead (with a template) but
> > this could be post release?
> >
> > Criticals seem to be all in good hands wanting a bit of testing or a last
> > review. Lets get them in.
> >
> > Reviewing the major issues, I do not see anything we should hold up the
> > release; please speak up if you think different (How about HBASE-7667,
> the
> > stripe compaction work or the fsync work in HBASE-5954?.
> >
> > On unit tests, we are mostly passing now as reported a few days ago.
> There
> > are a few flakies that would be sweet to purge; e.g:
> >
> > HBASE-7980 TestZKInterProcessReadWriteLock fails occasionally in QA test
> > run
> > HBASE-9023 TestIOFencing.testFencingAroundCompactionAfterWALSync
> >
> > TestDistributedLogSplitting can go 'invisibile' on occasion; i.e. tests
> > fail it is only one absent from list of completed tests.
> >
> > But unit tests are looking good on 0.95 and trunk:
> > https://builds.apache.org/view/H-L/view/HBase/
> >
> > Your RM,
> > St.Ack
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > >
> > > On Tue, Jul 9, 2013 at 1:14 PM, Stack <st...@duboce.net> wrote:
> > >
> > >> I am shooting for end of July for 0.96 being 'complete'. I would
> like
> > >> to make a 0.96 release in August. We have some criticals outstanding
> > but I
> > >> think we could ship even if these are not fixed in time (excepting
> > >> migration polish and of course remaining build fixes). See [1.] for
> the
> > >> current list of issues. Please re-prioritize issues as you see fit
> (or
> > >> better, move issues out of 0.95.2 if you do not think they will be
> done
> > in
> > >> time).
> > >>
> > >> What to do with namespaces -- the last 0.96 'feature' -- given the
> above
> > >> timeline? Currently it is a massive patch out on a branch. It is
> still
> > >> not done, in want of review, and the author is going on holidays for a
> > few
> > >> weeks soon. My thinking as of now, going by the rate of change over
> the
> > >> last few weeks and estimating what is yet to be done, is that
> namespaces
> > >> will not make it. I am willing to be convinced otherwise but that is
> > how
> > >> it looks to me currently.
> > >>
> > >> I am going to start just disabling flakey unit tests in 0.95 from here
> > on
> > >> out. When folks get the itch, they can fix at leisure first on trunk
> > and
> > >> then over in 0.95.
> > >>
> > >> What else?
> > >>
> > >> Thanks,
> > >> St.Ack
> > >>
> > >> 1.
> > >>
> >
> https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel
> > >>
> > >>
> > >>
> > >> On Mon, Jun 24, 2013 at 8:45 PM, Stack <st...@duboce.net> wrote:
> > >>
> > >>> (Changed the subject)
> > >>>
> > >>> On Mon, Jun 24, 2013 at 4:04 PM, Nick Dimiduk <ndimiduk@gmail.com
> > >wrote:
> > >>>
> > >>>> I want to see initial data type APIs ship out with 0.95.2. A patch
> for
> > >>>> ordered byte serialization is up (HBASE-8201) and is nearing
> > >>>> steady-state.
> > >>>> However, sershe is the only person who's left feedback. I just
> posted
> > an
> > >>>> early patch for the data type API itself (HBASE-8693). It should get
> > >>>> some
> > >>>> eyes from all manor of interested parties, but I'll settle for folk
> > from
> > >>>> Phoenix for now.
> > >>>>
> > >>>>
> > >>> It would be cool if Phoenix and Kiji fellows and any one else
> > interested
> > >>> would weigh in and take a look see.
> > >>>
> > >>> This does not strike me as something we should hold up the release
> for
> > >>> though. It looks like something that could go in at any time?
> > >>>
> > >>>
> > >>>
> > >>>> Should these tasks be escalated to criticals in order to grab
> > attention?
> > >>>>
> > >>>>
> > >>> I don't think that works going by past experience (and I don't think
> > >>> this a blocker on 0.96)
> > >>>
> > >>>
> > >>>
> > >>>> Additional comments inline.
> > >>>>
> > >>>>
> > >>> ...
> > >>>
> > >>>
> > >>>
> > >>>> Namespaces is the long pole and progress seems slow. Do we hold up
> > the
> > >>>> > release for them? How can we hurry this effort along? Swat team
> > >>>> descends
> > >>>> > on Y!?
> > >>>> >
> > >>>>
> > >>>> It would be a shame to not get a decision on this in for 0.96.
> > >>>>
> > >>>>
> > >>> Agree. We need to get 0.96 out though. It has been too long.
> > >>>
> > >>>
> > >>>
> > >>>> + Is anyone testing? Integration tests fail on ec2 build from time
> > to
> > >>>> time
> > >>>> > [2]. Our Elliott dug in on one of the failures a few days back
> and
> > >>>> found
> > >>>> > legit issue w/ no retry on admin tasks (I heart hbase-it tests).
> > Our
> >
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
Is there a patch available for the tracing work mentioned? I don't think
I've seen anything about it.
On Saturday, August 3, 2013, Stack wrote:
> On Sat, Jul 27, 2013 at 2:52 PM, Stack <stack@duboce.net <javascript:;>>
> wrote:
>
> > The end of July is upon us. I intend to cut a 0.95.2 next weekend and
> > 0.95.2 will be 0.96.0 but for bug fixes and migration polish. No more
> > 'features' will be allowed after next weekend.
> >
> >
> >
> > I have done some moving around of hbase 0.95.2 issues [1] to reflect what
> > the priorities are for this week (to my mind). It is all about polish,
> bad
> > bugs, unit test failures, and packaging/publishing/build issues.
> >
> > I'll be working on the blockers this week [2]. A few are in need of
> > reviews; e.g. "HBASE-3787 Increment is non-idempotent but client retries
> > RPC". Feel free to take over any blockers if you'd like to help out:
> e.g.
> > "HBASE-7386 Investigate providing some supervisor support for znode
> > deletion"
> >
> > Criticals [3] are mostly just unit test issues that are probably fixed by
> > now and another few that are patches in need of test/review: e.g.
> > "HBASE-8874 PutCombiner is skipping KeyValues while combining puts of
> same
> > row during bulkload" and "HBASE-8778 Region assigments scan table
> directory
> > making them slow for huge tables".
> > "HBASE-6127 TestAtomicOperation#testMultiRowMutationMultiThreads
> > occasionally fails" is a bad one in need of attention. Again, if up for
> > helping out, be our guest.
> >
> > There are some great patches hanging out in the priority major issues
> > section that are patch available that could be committed but for want of
> > review: e.g. "HBASE-8369 MapReduce over snapshot files".
> >
> > Regards the big features that are racing to make the 0.96 cutoff --
> namely
> > namespaces, tags, and serialization lib -- as I see it, Francis needs
> > reviews if namespaces are to make it, tags ditto, and the serialization
> > libs are nice-to-have auxillaries that can come in at any time. If not
> > done by the end of this week, then as I see it, these features do not
> make
> > the cut.
> >
> > Unit tests are mostly passing. The problematics are being worked on
> (e.g.
> > JD is on the replication set -- it likely a real problem rather than a
> > flakey test).
> >
> > How's the above sound?
> > St.Ack
> >
> > P.S. It is late but if folks want to meet this week to hack in patches
> > together, just say. I could organize an afternoon or day if you all
> think
> > it would be a good idea.
> >
> > 1.
> >
> https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel
> > 2.
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%20%220.95.2%22%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Blocker%20ORDER%20BY%20key%20DESC
> > 3.
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%20%220.95.2%22%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Critical%20ORDER%20BY%20key%20DESC
> >
> >
> >
> UPDATE:
>
> I was to cut 0.95.2 this weekend. I am pushing out the cut-date to Weds or
> so of next week. The gating factor is namespaces. It needs a few more
> days of patch cycling. I'll cut 0.95.2 after it goes in. It'll be the
> last feature on the 0.95/0.96 branch. Thereafter, only bug fixes,
> migration cleanup, and doc additions will be allowed (0.96.0RC will follow
> soon after the 0.95.2 developer release goes out).
>
> It looks like the serialization ilb will make the cut, API+Ordering; Nick
> has signed-on some consumers it seems.
>
> On KV tags, we have an outstanding -1 but -1s can change; lets see what the
> new posted patch looks like.
>
> Elliott has fancy-pants tracing that he should be able to get in before
> 0.95.2.
>
> On the dodgy-looking outstanding blockers:
>
> "HBASE-3787 Increment is non-idempotent but client retries RPC" still needs
> reviews. It is a difficult problem well-researched by our Sergey; perhaps
> this does not make it?
>
> On "HBASE-7386 Investigate providing some supervisor support for znode
> deletion" we could doc. the "ugly" wrapper/watcher process with why it
> exists and suggest it should be supervise instead (with a template) but
> this could be post release?
>
> Criticals seem to be all in good hands wanting a bit of testing or a last
> review. Lets get them in.
>
> Reviewing the major issues, I do not see anything we should hold up the
> release; please speak up if you think different (How about HBASE-7667, the
> stripe compaction work or the fsync work in HBASE-5954?.
>
> On unit tests, we are mostly passing now as reported a few days ago. There
> are a few flakies that would be sweet to purge; e.g:
>
> HBASE-7980 TestZKInterProcessReadWriteLock fails occasionally in QA test
> run
> HBASE-9023 TestIOFencing.testFencingAroundCompactionAfterWALSync
>
> TestDistributedLogSplitting can go 'invisibile' on occasion; i.e. tests
> fail it is only one absent from list of completed tests.
>
> But unit tests are looking good on 0.95 and trunk:
> https://builds.apache.org/view/H-L/view/HBase/
>
> Your RM,
> St.Ack
>
>
>
>
>
>
>
>
>
> >
> > On Tue, Jul 9, 2013 at 1:14 PM, Stack <st...@duboce.net> wrote:
> >
> >> I am shooting for end of July for 0.96 being 'complete'. I would like
> >> to make a 0.96 release in August. We have some criticals outstanding
> but I
> >> think we could ship even if these are not fixed in time (excepting
> >> migration polish and of course remaining build fixes). See [1.] for the
> >> current list of issues. Please re-prioritize issues as you see fit (or
> >> better, move issues out of 0.95.2 if you do not think they will be done
> in
> >> time).
> >>
> >> What to do with namespaces -- the last 0.96 'feature' -- given the above
> >> timeline? Currently it is a massive patch out on a branch. It is still
> >> not done, in want of review, and the author is going on holidays for a
> few
> >> weeks soon. My thinking as of now, going by the rate of change over the
> >> last few weeks and estimating what is yet to be done, is that namespaces
> >> will not make it. I am willing to be convinced otherwise but that is
> how
> >> it looks to me currently.
> >>
> >> I am going to start just disabling flakey unit tests in 0.95 from here
> on
> >> out. When folks get the itch, they can fix at leisure first on trunk
> and
> >> then over in 0.95.
> >>
> >> What else?
> >>
> >> Thanks,
> >> St.Ack
> >>
> >> 1.
> >>
> https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel
> >>
> >>
> >>
> >> On Mon, Jun 24, 2013 at 8:45 PM, Stack <st...@duboce.net> wrote:
> >>
> >>> (Changed the subject)
> >>>
> >>> On Mon, Jun 24, 2013 at 4:04 PM, Nick Dimiduk <ndimiduk@gmail.com
> >wrote:
> >>>
> >>>> I want to see initial data type APIs ship out with 0.95.2. A patch for
> >>>> ordered byte serialization is up (HBASE-8201) and is nearing
> >>>> steady-state.
> >>>> However, sershe is the only person who's left feedback. I just posted
> an
> >>>> early patch for the data type API itself (HBASE-8693). It should get
> >>>> some
> >>>> eyes from all manor of interested parties, but I'll settle for folk
> from
> >>>> Phoenix for now.
> >>>>
> >>>>
> >>> It would be cool if Phoenix and Kiji fellows and any one else
> interested
> >>> would weigh in and take a look see.
> >>>
> >>> This does not strike me as something we should hold up the release for
> >>> though. It looks like something that could go in at any time?
> >>>
> >>>
> >>>
> >>>> Should these tasks be escalated to criticals in order to grab
> attention?
> >>>>
> >>>>
> >>> I don't think that works going by past experience (and I don't think
> >>> this a blocker on 0.96)
> >>>
> >>>
> >>>
> >>>> Additional comments inline.
> >>>>
> >>>>
> >>> ...
> >>>
> >>>
> >>>
> >>>> Namespaces is the long pole and progress seems slow. Do we hold up
> the
> >>>> > release for them? How can we hurry this effort along? Swat team
> >>>> descends
> >>>> > on Y!?
> >>>> >
> >>>>
> >>>> It would be a shame to not get a decision on this in for 0.96.
> >>>>
> >>>>
> >>> Agree. We need to get 0.96 out though. It has been too long.
> >>>
> >>>
> >>>
> >>>> + Is anyone testing? Integration tests fail on ec2 build from time
> to
> >>>> time
> >>>> > [2]. Our Elliott dug in on one of the failures a few days back and
> >>>> found
> >>>> > legit issue w/ no retry on admin tasks (I heart hbase-it tests).
> Our
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Stack <st...@duboce.net>.
On Wed, Aug 14, 2013 at 1:26 PM, Andrew Purtell <ap...@apache.org> wrote:
> On Wed, Aug 14, 2013 at 12:57 PM, Stack <st...@duboce.net> wrote:
>
> > I see no reason that 0.98 cannot come out a week or a month after 0.96;
>
> tags is close and justification enough for a major release.
> > I propose Andrew as RM for 0.98/1.0.0 if he is up for taking it on.
>
>
> I would be pleased to volunteer to RM 0.98.
>
>
Sounds good to me. Would suggest announcing your taking it up in a new
thread rather than down here in the middle of this one -- perhaps
soliciting if any objection? -- and maybe as part of a message announcing
our starting up the 0.98 cycle. Good on you Andrew.
> If I were to be your RM for 0.98, then I would suggest a .0 release in the
> beginning of October. There are 42 open issues against 0.98 specifically
> and I would like to also provide everyone some time for post 0.96 release
> thinking.
>
>
Be aware that issues have been moved here just to get them out of 0.96.
Feel free to punt them on again if not being worked on or not appropriate.
We might want to put out a call for what folks think should be in a release
that is slated for October -- or what they are working on and think they
can finish inside the October constraint.
> I am not sure we should move from 0.98 to 1.0 without another interim
> release, that would be a call for a later time perhaps, maybe a 1.0 release
> at the start of January 2014.
>
>
Ok. Something to discuss.
Thanks A,
St.Ack
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Stack <st...@duboce.net>.
On Wed, Aug 14, 2013 at 6:39 PM, Devaraj Das <dd...@hortonworks.com> wrote:
> A late breaking one- https://issues.apache.org/jira/browse/HBASE-9227.
> Kindly consider for the upcoming RC.
>
>
np
There is another blocker outstanding still. Hopefully we can get these
both resolved in the morning.
St.Ack
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Stack <st...@duboce.net>.
On Thu, Aug 15, 2013 at 8:30 AM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> >>Lets then make a tags-only release on the
> heels of 0.96. As said above, tags is cause enough for a major hbase
> release.
> Okie. Let me see if anything is needed so that we can avoid compatability
> issues later. I will consider your suggestion also.
>
If you need "placeholders" -- as per Todd suggestion -- then lets get them
in (they can come in after 0.95.2 and before 0.96.0RC0 np).
St.Ack
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by ramkrishna vasudevan <ra...@gmail.com>.
>>Lets then make a tags-only release on the
heels of 0.96. As said above, tags is cause enough for a major hbase
release.
Okie. Let me see if anything is needed so that we can avoid compatability
issues later. I will consider your suggestion also.
Thanks Stack.
Regards
Ram
On Thu, Aug 15, 2013 at 8:51 PM, Stack <st...@duboce.net> wrote:
> On Wed, Aug 14, 2013 at 11:44 PM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
> > Hi Stack
> >
> > First I would like to thank you for the reviews and understand the
> > decision.
> > I would just like to highlight few facts if this does not make into 0.96
> > then we may have to
> > -> Handle backward compatability issues with WAL. Currently inorder to
> > retrieve tags we write a taglength thought it is 0. This would help us
> to
> > replay WAL with Tags.
> >
>
> Is there not a version in WALEdit or on the WAL file that would allow us
> distinguish a WALEdit written w/ tags from one w/o?
>
>
>
> > -> Codecs like PRefixTreeCodec needs to be working for versions without
> > Tags. Currently as PrefixTreeCodec is not out in any release these are
> not
> > a problem for now.
> >
>
> Could we have two codecs, one that supports tags and another that does not?
>
>
> >
> > So to avoid these problems we can get these Tags on 0.96 release. May
> be a
> > few more days of time. Is it possible? I can try getting people for
> more
> > reviews too.
>
>
> My sense going by recent experience is that landing big patches, what w/
> review cycles and then stabilization of build post commit, is that the
> process always takes way more time than I expect. You fellas are almost
> there -- tags is shaping up nicely -- and you and Anoop are super receptive
> and reactive to criticism so it would all likely run faster than usual but
> I predict at least another week before it lands -- perhaps too weeks -- and
> that is too long for a 0.96 that is already way too late.
>
> Lets get tags into trunk. I can help after this development release and the
> release candidate gets posted. Lets then make a tags-only release on the
> heels of 0.96. As said above, tags is cause enough for a major hbase
> release.
>
> Good on you Ram,
> St.Ack
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Stack <st...@duboce.net>.
On Wed, Aug 14, 2013 at 11:44 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> Hi Stack
>
> First I would like to thank you for the reviews and understand the
> decision.
> I would just like to highlight few facts if this does not make into 0.96
> then we may have to
> -> Handle backward compatability issues with WAL. Currently inorder to
> retrieve tags we write a taglength thought it is 0. This would help us to
> replay WAL with Tags.
>
Is there not a version in WALEdit or on the WAL file that would allow us
distinguish a WALEdit written w/ tags from one w/o?
> -> Codecs like PRefixTreeCodec needs to be working for versions without
> Tags. Currently as PrefixTreeCodec is not out in any release these are not
> a problem for now.
>
Could we have two codecs, one that supports tags and another that does not?
>
> So to avoid these problems we can get these Tags on 0.96 release. May be a
> few more days of time. Is it possible? I can try getting people for more
> reviews too.
My sense going by recent experience is that landing big patches, what w/
review cycles and then stabilization of build post commit, is that the
process always takes way more time than I expect. You fellas are almost
there -- tags is shaping up nicely -- and you and Anoop are super receptive
and reactive to criticism so it would all likely run faster than usual but
I predict at least another week before it lands -- perhaps too weeks -- and
that is too long for a 0.96 that is already way too late.
Lets get tags into trunk. I can help after this development release and the
release candidate gets posted. Lets then make a tags-only release on the
heels of 0.96. As said above, tags is cause enough for a major hbase
release.
Good on you Ram,
St.Ack
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Okie.. Let me try to do that . You wanna take a look at HFileV3 with Tags?
Would always help in getting more reviews. Thanks Todd.
Regards
Ram
On Thu, Aug 15, 2013 at 1:45 PM, Todd Lipcon <to...@cloudera.com> wrote:
> Hey Ram,
>
> Just a thought: could you make the backward-incompatible changes in the
> data format ahead of actually having support for using them? EG if you need
> a new field, put it unused in the WAL now to reserve space for it?
>
> -Todd
>
> On Wed, Aug 14, 2013 at 11:44 PM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
> > Hi Stack
> >
> > First I would like to thank you for the reviews and understand the
> > decision.
> > I would just like to highlight few facts if this does not make into 0.96
> > then we may have to
> > -> Handle backward compatability issues with WAL. Currently inorder to
> > retrieve tags we write a taglength thought it is 0. This would help us
> to
> > replay WAL with Tags.
> > -> Codecs like PRefixTreeCodec needs to be working for versions without
> > Tags. Currently as PrefixTreeCodec is not out in any release these are
> not
> > a problem for now.
> >
> > So to avoid these problems we can get these Tags on 0.96 release. May
> be a
> > few more days of time. Is it possible? I can try getting people for
> more
> > reviews too.
> > Thanks Stack. Nice of you.
> >
> > Regards
> > Ram
> >
> >
> > On Thu, Aug 15, 2013 at 7:09 AM, Devaraj Das <dd...@hortonworks.com>
> wrote:
> >
> > > A late breaking one- https://issues.apache.org/jira/browse/HBASE-9227.
> > > Kindly consider for the upcoming RC.
> > >
> > >
> > > On Wed, Aug 14, 2013 at 1:26 PM, Andrew Purtell <ap...@apache.org>
> > > wrote:
> > >
> > > > On Wed, Aug 14, 2013 at 12:57 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > I see no reason that 0.98 cannot come out a week or a month after
> > 0.96;
> > > >
> > > > tags is close and justification enough for a major release.
> > > > > I propose Andrew as RM for 0.98/1.0.0 if he is up for taking it on.
> > > >
> > > >
> > > > I would be pleased to volunteer to RM 0.98.
> > > >
> > > > It will be a challenge for anyone to live up to your outstanding
> > example
> > > > for 0.96.
> > > >
> > > > If I were to be your RM for 0.98, then I would suggest a .0 release
> in
> > > the
> > > > beginning of October. There are 42 open issues against 0.98
> > specifically
> > > > and I would like to also provide everyone some time for post 0.96
> > release
> > > > thinking.
> > > >
> > > > I am not sure we should move from 0.98 to 1.0 without another interim
> > > > release, that would be a call for a later time perhaps, maybe a 1.0
> > > release
> > > > at the start of January 2014.
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > > - Andy
> > > >
> > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > > > (via Tom White)
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Todd Lipcon <to...@cloudera.com>.
Hey Ram,
Just a thought: could you make the backward-incompatible changes in the
data format ahead of actually having support for using them? EG if you need
a new field, put it unused in the WAL now to reserve space for it?
-Todd
On Wed, Aug 14, 2013 at 11:44 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> Hi Stack
>
> First I would like to thank you for the reviews and understand the
> decision.
> I would just like to highlight few facts if this does not make into 0.96
> then we may have to
> -> Handle backward compatability issues with WAL. Currently inorder to
> retrieve tags we write a taglength thought it is 0. This would help us to
> replay WAL with Tags.
> -> Codecs like PRefixTreeCodec needs to be working for versions without
> Tags. Currently as PrefixTreeCodec is not out in any release these are not
> a problem for now.
>
> So to avoid these problems we can get these Tags on 0.96 release. May be a
> few more days of time. Is it possible? I can try getting people for more
> reviews too.
> Thanks Stack. Nice of you.
>
> Regards
> Ram
>
>
> On Thu, Aug 15, 2013 at 7:09 AM, Devaraj Das <dd...@hortonworks.com> wrote:
>
> > A late breaking one- https://issues.apache.org/jira/browse/HBASE-9227.
> > Kindly consider for the upcoming RC.
> >
> >
> > On Wed, Aug 14, 2013 at 1:26 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > On Wed, Aug 14, 2013 at 12:57 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > I see no reason that 0.98 cannot come out a week or a month after
> 0.96;
> > >
> > > tags is close and justification enough for a major release.
> > > > I propose Andrew as RM for 0.98/1.0.0 if he is up for taking it on.
> > >
> > >
> > > I would be pleased to volunteer to RM 0.98.
> > >
> > > It will be a challenge for anyone to live up to your outstanding
> example
> > > for 0.96.
> > >
> > > If I were to be your RM for 0.98, then I would suggest a .0 release in
> > the
> > > beginning of October. There are 42 open issues against 0.98
> specifically
> > > and I would like to also provide everyone some time for post 0.96
> release
> > > thinking.
> > >
> > > I am not sure we should move from 0.98 to 1.0 without another interim
> > > release, that would be a call for a later time perhaps, maybe a 1.0
> > release
> > > at the start of January 2014.
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Hi Stack
First I would like to thank you for the reviews and understand the
decision.
I would just like to highlight few facts if this does not make into 0.96
then we may have to
-> Handle backward compatability issues with WAL. Currently inorder to
retrieve tags we write a taglength thought it is 0. This would help us to
replay WAL with Tags.
-> Codecs like PRefixTreeCodec needs to be working for versions without
Tags. Currently as PrefixTreeCodec is not out in any release these are not
a problem for now.
So to avoid these problems we can get these Tags on 0.96 release. May be a
few more days of time. Is it possible? I can try getting people for more
reviews too.
Thanks Stack. Nice of you.
Regards
Ram
On Thu, Aug 15, 2013 at 7:09 AM, Devaraj Das <dd...@hortonworks.com> wrote:
> A late breaking one- https://issues.apache.org/jira/browse/HBASE-9227.
> Kindly consider for the upcoming RC.
>
>
> On Wed, Aug 14, 2013 at 1:26 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > On Wed, Aug 14, 2013 at 12:57 PM, Stack <st...@duboce.net> wrote:
> >
> > > I see no reason that 0.98 cannot come out a week or a month after 0.96;
> >
> > tags is close and justification enough for a major release.
> > > I propose Andrew as RM for 0.98/1.0.0 if he is up for taking it on.
> >
> >
> > I would be pleased to volunteer to RM 0.98.
> >
> > It will be a challenge for anyone to live up to your outstanding example
> > for 0.96.
> >
> > If I were to be your RM for 0.98, then I would suggest a .0 release in
> the
> > beginning of October. There are 42 open issues against 0.98 specifically
> > and I would like to also provide everyone some time for post 0.96 release
> > thinking.
> >
> > I am not sure we should move from 0.98 to 1.0 without another interim
> > release, that would be a call for a later time perhaps, maybe a 1.0
> release
> > at the start of January 2014.
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Devaraj Das <dd...@hortonworks.com>.
A late breaking one- https://issues.apache.org/jira/browse/HBASE-9227.
Kindly consider for the upcoming RC.
On Wed, Aug 14, 2013 at 1:26 PM, Andrew Purtell <ap...@apache.org> wrote:
> On Wed, Aug 14, 2013 at 12:57 PM, Stack <st...@duboce.net> wrote:
>
> > I see no reason that 0.98 cannot come out a week or a month after 0.96;
>
> tags is close and justification enough for a major release.
> > I propose Andrew as RM for 0.98/1.0.0 if he is up for taking it on.
>
>
> I would be pleased to volunteer to RM 0.98.
>
> It will be a challenge for anyone to live up to your outstanding example
> for 0.96.
>
> If I were to be your RM for 0.98, then I would suggest a .0 release in the
> beginning of October. There are 42 open issues against 0.98 specifically
> and I would like to also provide everyone some time for post 0.96 release
> thinking.
>
> I am not sure we should move from 0.98 to 1.0 without another interim
> release, that would be a call for a later time perhaps, maybe a 1.0 release
> at the start of January 2014.
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
On Wed, Aug 14, 2013 at 12:57 PM, Stack <st...@duboce.net> wrote:
> I see no reason that 0.98 cannot come out a week or a month after 0.96;
tags is close and justification enough for a major release.
> I propose Andrew as RM for 0.98/1.0.0 if he is up for taking it on.
I would be pleased to volunteer to RM 0.98.
It will be a challenge for anyone to live up to your outstanding example
for 0.96.
If I were to be your RM for 0.98, then I would suggest a .0 release in the
beginning of October. There are 42 open issues against 0.98 specifically
and I would like to also provide everyone some time for post 0.96 release
thinking.
I am not sure we should move from 0.98 to 1.0 without another interim
release, that would be a call for a later time perhaps, maybe a 1.0 release
at the start of January 2014.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Stack <st...@duboce.net>.
Its 11 days! since the last update. I'm going to cut 0.95.2 this evening
after the last blocker goes in. This means only bug fixes, doc., and
polish are allowed on the 0.95 branch from here on out. I'll put up a
0.96.0RC0 soon after, probably next week.
Namespaces are in. It destabilized build for a while which is only to be
expected for a patch of near 3M I suppose but all seems back to normal
again.
Nick's client-side serialization additions made it in too.
I just took a quick look at the tags work and it is much improved and
getting close w/ a couple of receptive developers on the other end fast
accommodating review comments -- but IMO it still needs to make a few more
rounds of review which will put off the 0.95.2/0.96.0 even more. I am
making the call that the tags work does not make the 0.96 cut (It also has
a -1 against it).
St.Ack
PS: I see no reason that 0.98 cannot come out a week or a month after 0.96;
tags is close and justification enough for a major release. Or as has been
suggested already, just skip 0.98 and just go to 1.0.0 altogether once tags
are in? I propose Andrew as RM for 0.98/1.0.0 if he is up for taking it on.
On Sat, Aug 3, 2013 at 1:35 PM, Stack <st...@duboce.net> wrote:
> On Sat, Jul 27, 2013 at 2:52 PM, Stack <st...@duboce.net> wrote:
>
>> The end of July is upon us. I intend to cut a 0.95.2 next weekend and
>> 0.95.2 will be 0.96.0 but for bug fixes and migration polish. No more
>> 'features' will be allowed after next weekend.
>>
>>
>>
>> I have done some moving around of hbase 0.95.2 issues [1] to reflect what
>> the priorities are for this week (to my mind). It is all about polish, bad
>> bugs, unit test failures, and packaging/publishing/build issues.
>>
>> I'll be working on the blockers this week [2]. A few are in need of
>> reviews; e.g. "HBASE-3787 Increment is non-idempotent but client retries
>> RPC". Feel free to take over any blockers if you'd like to help out: e.g.
>> "HBASE-7386 Investigate providing some supervisor support for znode
>> deletion"
>>
>> Criticals [3] are mostly just unit test issues that are probably fixed by
>> now and another few that are patches in need of test/review: e.g.
>> "HBASE-8874 PutCombiner is skipping KeyValues while combining puts of same
>> row during bulkload" and "HBASE-8778 Region assigments scan table directory
>> making them slow for huge tables".
>> "HBASE-6127 TestAtomicOperation#testMultiRowMutationMultiThreads
>> occasionally fails" is a bad one in need of attention. Again, if up for
>> helping out, be our guest.
>>
>> There are some great patches hanging out in the priority major issues
>> section that are patch available that could be committed but for want of
>> review: e.g. "HBASE-8369 MapReduce over snapshot files".
>>
>> Regards the big features that are racing to make the 0.96 cutoff --
>> namely namespaces, tags, and serialization lib -- as I see it, Francis
>> needs reviews if namespaces are to make it, tags ditto, and the
>> serialization libs are nice-to-have auxillaries that can come in at any
>> time. If not done by the end of this week, then as I see it, these
>> features do not make the cut.
>>
>> Unit tests are mostly passing. The problematics are being worked on
>> (e.g. JD is on the replication set -- it likely a real problem rather than
>> a flakey test).
>>
>> How's the above sound?
>> St.Ack
>>
>> P.S. It is late but if folks want to meet this week to hack in patches
>> together, just say. I could organize an afternoon or day if you all think
>> it would be a good idea.
>>
>> 1.
>> https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel
>> 2.
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%20%220.95.2%22%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Blocker%20ORDER%20BY%20key%20DESC
>> 3.
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%20%220.95.2%22%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Critical%20ORDER%20BY%20key%20DESC
>>
>>
>>
> UPDATE:
>
> I was to cut 0.95.2 this weekend. I am pushing out the cut-date to Weds
> or so of next week. The gating factor is namespaces. It needs a few more
> days of patch cycling. I'll cut 0.95.2 after it goes in. It'll be the
> last feature on the 0.95/0.96 branch. Thereafter, only bug fixes,
> migration cleanup, and doc additions will be allowed (0.96.0RC will follow
> soon after the 0.95.2 developer release goes out).
>
> It looks like the serialization ilb will make the cut, API+Ordering; Nick
> has signed-on some consumers it seems.
>
> On KV tags, we have an outstanding -1 but -1s can change; lets see what
> the new posted patch looks like.
>
> Elliott has fancy-pants tracing that he should be able to get in before
> 0.95.2.
>
> On the dodgy-looking outstanding blockers:
>
> "HBASE-3787 Increment is non-idempotent but client retries RPC" still
> needs reviews. It is a difficult problem well-researched by our Sergey;
> perhaps this does not make it?
>
> On "HBASE-7386 Investigate providing some supervisor support for znode
> deletion" we could doc. the "ugly" wrapper/watcher process with why it
> exists and suggest it should be supervise instead (with a template) but
> this could be post release?
>
> Criticals seem to be all in good hands wanting a bit of testing or a last
> review. Lets get them in.
>
> Reviewing the major issues, I do not see anything we should hold up the
> release; please speak up if you think different (How about HBASE-7667, the
> stripe compaction work or the fsync work in HBASE-5954?.
>
> On unit tests, we are mostly passing now as reported a few days ago.
> There are a few flakies that would be sweet to purge; e.g:
>
> HBASE-7980 TestZKInterProcessReadWriteLock fails occasionally in QA test
> run
> HBASE-9023 TestIOFencing.testFencingAroundCompactionAfterWALSync
>
> TestDistributedLogSplitting can go 'invisibile' on occasion; i.e. tests
> fail it is only one absent from list of completed tests.
>
> But unit tests are looking good on 0.95 and trunk:
> https://builds.apache.org/view/H-L/view/HBase/
>
> Your RM,
> St.Ack
>
>
>
>
>
>
>
>
>
>>
>> On Tue, Jul 9, 2013 at 1:14 PM, Stack <st...@duboce.net> wrote:
>>
>>> I am shooting for end of July for 0.96 being 'complete'. I would like
>>> to make a 0.96 release in August. We have some criticals outstanding but I
>>> think we could ship even if these are not fixed in time (excepting
>>> migration polish and of course remaining build fixes). See [1.] for the
>>> current list of issues. Please re-prioritize issues as you see fit (or
>>> better, move issues out of 0.95.2 if you do not think they will be done in
>>> time).
>>>
>>> What to do with namespaces -- the last 0.96 'feature' -- given the above
>>> timeline? Currently it is a massive patch out on a branch. It is still
>>> not done, in want of review, and the author is going on holidays for a few
>>> weeks soon. My thinking as of now, going by the rate of change over the
>>> last few weeks and estimating what is yet to be done, is that namespaces
>>> will not make it. I am willing to be convinced otherwise but that is how
>>> it looks to me currently.
>>>
>>> I am going to start just disabling flakey unit tests in 0.95 from here
>>> on out. When folks get the itch, they can fix at leisure first on trunk
>>> and then over in 0.95.
>>>
>>> What else?
>>>
>>> Thanks,
>>> St.Ack
>>>
>>> 1.
>>> https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel
>>>
>>>
>>>
>>> On Mon, Jun 24, 2013 at 8:45 PM, Stack <st...@duboce.net> wrote:
>>>
>>>> (Changed the subject)
>>>>
>>>> On Mon, Jun 24, 2013 at 4:04 PM, Nick Dimiduk <nd...@gmail.com>wrote:
>>>>
>>>>> I want to see initial data type APIs ship out with 0.95.2. A patch for
>>>>> ordered byte serialization is up (HBASE-8201) and is nearing
>>>>> steady-state.
>>>>> However, sershe is the only person who's left feedback. I just posted
>>>>> an
>>>>> early patch for the data type API itself (HBASE-8693). It should get
>>>>> some
>>>>> eyes from all manor of interested parties, but I'll settle for folk
>>>>> from
>>>>> Phoenix for now.
>>>>>
>>>>>
>>>> It would be cool if Phoenix and Kiji fellows and any one else
>>>> interested would weigh in and take a look see.
>>>>
>>>> This does not strike me as something we should hold up the release for
>>>> though. It looks like something that could go in at any time?
>>>>
>>>>
>>>>
>>>>> Should these tasks be escalated to criticals in order to grab
>>>>> attention?
>>>>>
>>>>>
>>>> I don't think that works going by past experience (and I don't think
>>>> this a blocker on 0.96)
>>>>
>>>>
>>>>
>>>>> Additional comments inline.
>>>>>
>>>>>
>>>> ...
>>>>
>>>>
>>>>
>>>>> Namespaces is the long pole and progress seems slow. Do we hold up
>>>>> the
>>>>> > release for them? How can we hurry this effort along? Swat team
>>>>> descends
>>>>> > on Y!?
>>>>> >
>>>>>
>>>>> It would be a shame to not get a decision on this in for 0.96.
>>>>>
>>>>>
>>>> Agree. We need to get 0.96 out though. It has been too long.
>>>>
>>>>
>>>>
>>>>> + Is anyone testing? Integration tests fail on ec2 build from time
>>>>> to time
>>>>> > [2]. Our Elliott dug in on one of the failures a few days back and
>>>>> found
>>>>> > legit issue w/ no retry on admin tasks (I heart hbase-it tests).
>>>>> Our unit
>>>>> > test story is better [3] but there are still the odd failures.
>>>>> >
>>>>>
>>>>> With the creation of the new list, noticing these issues is going to
>>>>> push
>>>>> further back-burner. Nannying this stuff should retain focus. I'll
>>>>> volunteer to track on these issues as I see them.
>>>>>
>>>>>
>>>> Thank you Nick.
>>>>
>>>>
>>>>
>>>>> Notice though that some of the more recent failures are caused by lack
>>>>> of
>>>>> disk space on the Jenkins build host.
>>>>>
>>>>>
>>>> Oh. Missed that. Let me dig in.
>>>>
>>>>
>>>> St.Ack
>>>>
>>>
>>>
>>
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Stack <st...@duboce.net>.
On Sat, Jul 27, 2013 at 2:52 PM, Stack <st...@duboce.net> wrote:
> The end of July is upon us. I intend to cut a 0.95.2 next weekend and
> 0.95.2 will be 0.96.0 but for bug fixes and migration polish. No more
> 'features' will be allowed after next weekend.
>
>
>
> I have done some moving around of hbase 0.95.2 issues [1] to reflect what
> the priorities are for this week (to my mind). It is all about polish, bad
> bugs, unit test failures, and packaging/publishing/build issues.
>
> I'll be working on the blockers this week [2]. A few are in need of
> reviews; e.g. "HBASE-3787 Increment is non-idempotent but client retries
> RPC". Feel free to take over any blockers if you'd like to help out: e.g.
> "HBASE-7386 Investigate providing some supervisor support for znode
> deletion"
>
> Criticals [3] are mostly just unit test issues that are probably fixed by
> now and another few that are patches in need of test/review: e.g.
> "HBASE-8874 PutCombiner is skipping KeyValues while combining puts of same
> row during bulkload" and "HBASE-8778 Region assigments scan table directory
> making them slow for huge tables".
> "HBASE-6127 TestAtomicOperation#testMultiRowMutationMultiThreads
> occasionally fails" is a bad one in need of attention. Again, if up for
> helping out, be our guest.
>
> There are some great patches hanging out in the priority major issues
> section that are patch available that could be committed but for want of
> review: e.g. "HBASE-8369 MapReduce over snapshot files".
>
> Regards the big features that are racing to make the 0.96 cutoff -- namely
> namespaces, tags, and serialization lib -- as I see it, Francis needs
> reviews if namespaces are to make it, tags ditto, and the serialization
> libs are nice-to-have auxillaries that can come in at any time. If not
> done by the end of this week, then as I see it, these features do not make
> the cut.
>
> Unit tests are mostly passing. The problematics are being worked on (e.g.
> JD is on the replication set -- it likely a real problem rather than a
> flakey test).
>
> How's the above sound?
> St.Ack
>
> P.S. It is late but if folks want to meet this week to hack in patches
> together, just say. I could organize an afternoon or day if you all think
> it would be a good idea.
>
> 1.
> https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel
> 2.
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%20%220.95.2%22%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Blocker%20ORDER%20BY%20key%20DESC
> 3.
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20fixVersion%20%3D%20%220.95.2%22%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Critical%20ORDER%20BY%20key%20DESC
>
>
>
UPDATE:
I was to cut 0.95.2 this weekend. I am pushing out the cut-date to Weds or
so of next week. The gating factor is namespaces. It needs a few more
days of patch cycling. I'll cut 0.95.2 after it goes in. It'll be the
last feature on the 0.95/0.96 branch. Thereafter, only bug fixes,
migration cleanup, and doc additions will be allowed (0.96.0RC will follow
soon after the 0.95.2 developer release goes out).
It looks like the serialization ilb will make the cut, API+Ordering; Nick
has signed-on some consumers it seems.
On KV tags, we have an outstanding -1 but -1s can change; lets see what the
new posted patch looks like.
Elliott has fancy-pants tracing that he should be able to get in before
0.95.2.
On the dodgy-looking outstanding blockers:
"HBASE-3787 Increment is non-idempotent but client retries RPC" still needs
reviews. It is a difficult problem well-researched by our Sergey; perhaps
this does not make it?
On "HBASE-7386 Investigate providing some supervisor support for znode
deletion" we could doc. the "ugly" wrapper/watcher process with why it
exists and suggest it should be supervise instead (with a template) but
this could be post release?
Criticals seem to be all in good hands wanting a bit of testing or a last
review. Lets get them in.
Reviewing the major issues, I do not see anything we should hold up the
release; please speak up if you think different (How about HBASE-7667, the
stripe compaction work or the fsync work in HBASE-5954?.
On unit tests, we are mostly passing now as reported a few days ago. There
are a few flakies that would be sweet to purge; e.g:
HBASE-7980 TestZKInterProcessReadWriteLock fails occasionally in QA test run
HBASE-9023 TestIOFencing.testFencingAroundCompactionAfterWALSync
TestDistributedLogSplitting can go 'invisibile' on occasion; i.e. tests
fail it is only one absent from list of completed tests.
But unit tests are looking good on 0.95 and trunk:
https://builds.apache.org/view/H-L/view/HBase/
Your RM,
St.Ack
>
> On Tue, Jul 9, 2013 at 1:14 PM, Stack <st...@duboce.net> wrote:
>
>> I am shooting for end of July for 0.96 being 'complete'. I would like
>> to make a 0.96 release in August. We have some criticals outstanding but I
>> think we could ship even if these are not fixed in time (excepting
>> migration polish and of course remaining build fixes). See [1.] for the
>> current list of issues. Please re-prioritize issues as you see fit (or
>> better, move issues out of 0.95.2 if you do not think they will be done in
>> time).
>>
>> What to do with namespaces -- the last 0.96 'feature' -- given the above
>> timeline? Currently it is a massive patch out on a branch. It is still
>> not done, in want of review, and the author is going on holidays for a few
>> weeks soon. My thinking as of now, going by the rate of change over the
>> last few weeks and estimating what is yet to be done, is that namespaces
>> will not make it. I am willing to be convinced otherwise but that is how
>> it looks to me currently.
>>
>> I am going to start just disabling flakey unit tests in 0.95 from here on
>> out. When folks get the itch, they can fix at leisure first on trunk and
>> then over in 0.95.
>>
>> What else?
>>
>> Thanks,
>> St.Ack
>>
>> 1.
>> https://issues.apache.org/jira/browse/HBASE/fixforversion/12320040#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel
>>
>>
>>
>> On Mon, Jun 24, 2013 at 8:45 PM, Stack <st...@duboce.net> wrote:
>>
>>> (Changed the subject)
>>>
>>> On Mon, Jun 24, 2013 at 4:04 PM, Nick Dimiduk <nd...@gmail.com>wrote:
>>>
>>>> I want to see initial data type APIs ship out with 0.95.2. A patch for
>>>> ordered byte serialization is up (HBASE-8201) and is nearing
>>>> steady-state.
>>>> However, sershe is the only person who's left feedback. I just posted an
>>>> early patch for the data type API itself (HBASE-8693). It should get
>>>> some
>>>> eyes from all manor of interested parties, but I'll settle for folk from
>>>> Phoenix for now.
>>>>
>>>>
>>> It would be cool if Phoenix and Kiji fellows and any one else interested
>>> would weigh in and take a look see.
>>>
>>> This does not strike me as something we should hold up the release for
>>> though. It looks like something that could go in at any time?
>>>
>>>
>>>
>>>> Should these tasks be escalated to criticals in order to grab attention?
>>>>
>>>>
>>> I don't think that works going by past experience (and I don't think
>>> this a blocker on 0.96)
>>>
>>>
>>>
>>>> Additional comments inline.
>>>>
>>>>
>>> ...
>>>
>>>
>>>
>>>> Namespaces is the long pole and progress seems slow. Do we hold up the
>>>> > release for them? How can we hurry this effort along? Swat team
>>>> descends
>>>> > on Y!?
>>>> >
>>>>
>>>> It would be a shame to not get a decision on this in for 0.96.
>>>>
>>>>
>>> Agree. We need to get 0.96 out though. It has been too long.
>>>
>>>
>>>
>>>> + Is anyone testing? Integration tests fail on ec2 build from time to
>>>> time
>>>> > [2]. Our Elliott dug in on one of the failures a few days back and
>>>> found
>>>> > legit issue w/ no retry on admin tasks (I heart hbase-it tests). Our
>>>> unit
>>>> > test story is better [3] but there are still the odd failures.
>>>> >
>>>>
>>>> With the creation of the new list, noticing these issues is going to
>>>> push
>>>> further back-burner. Nannying this stuff should retain focus. I'll
>>>> volunteer to track on these issues as I see them.
>>>>
>>>>
>>> Thank you Nick.
>>>
>>>
>>>
>>>> Notice though that some of the more recent failures are caused by lack
>>>> of
>>>> disk space on the Jenkins build host.
>>>>
>>>>
>>> Oh. Missed that. Let me dig in.
>>>
>>>
>>> St.Ack
>>>
>>
>>
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by ramkrishna vasudevan <ra...@gmail.com>.
>>(My objection is the nomenclature -- if you have to do the
overloads to get in your tags, go for it)
We are trying to avoid this. Will review internally and try to arrive at
an approach sooner. And also at other comments. Thanks Stack.
Regards
Ram
On Mon, Jul 29, 2013 at 11:21 PM, Elliott Clark <ec...@apache.org> wrote:
> On Mon, Jul 29, 2013 at 10:26 AM, Stack <st...@duboce.net> wrote:
> > Main objection is that the included v3 is not a 'green field'/redo so
> lets
> > not call it that
>
> We added minor version into hfile versioning (Broke things while doing
> it), so maybe it's better to call it 2.5 ?
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Stack <st...@duboce.net>.
On Mon, Jul 29, 2013 at 9:23 PM, Ted Yu <yu...@gmail.com> wrote:
> I ran test suite using
> https://github.com/francisliu/hbase_namespace/tree/core_8408_rebase_1 last
> week and got several test failures.
> See tail of this email.
>
>
So, the namespaces branch which is based off an older version of trunk has
some failing tests in it? Was it namespaces that caused the failures? Did
you look?
The set of tests posted look to belong to tests that have had work
subsequently done on them to make them more reliable in trunk. Others have
been turned off in trunk and 0.95 because they were flakey: e.g.
TestRSKilledWhenMasterInitializing.
The above observation would therefore add zero to this discussion?
> I think namespace feature still needs some polishing. Francis and people
> interested in this feature can continue to work on the git branch so that
> all major issues (including migration) are ironed out.
>
Did you review the patch?
The ns branch comes in now or not at all because it redoes the fs layout.
It can't come in at a later time as you suggest above.
St.Ack
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Ted Yu <yu...@gmail.com>.
I ran test suite using
https://github.com/francisliu/hbase_namespace/tree/core_8408_rebase_1 last
week and got several test failures.
See tail of this email.
I think namespace feature still needs some polishing. Francis and people
interested in this feature can continue to work on the git branch so that
all major issues (including migration) are ironed out.
Cheers
Failed tests:
testCompactionRecordDoesntBlockRolling(org.apache.hadoop.hbase.regionserver.wal.TestLogRolling):
Should have WAL; one table is not flushed expected:<1> but was:<2>
testLogRollOnDatanodeDeath(org.apache.hadoop.hbase.regionserver.wal.TestLogRolling):
New log file should have the default replication instead of 1
testFavoredNodesPresentForRoundRobinAssignment(org.apache.hadoop.hbase.master.TestRegionPlacement)
Tests in error:
org.apache.hadoop.hbase.catalog.TestMetaMigrationConvertingToPB:
java.net.URISyntaxException: Relative path in absolute URI: hbase:meta
testCorrectnessWhenMasterFailOver(org.apache.hadoop.hbase.regionserver.TestRSKilledWhenMasterInitializing):
test timed out after 120000 milliseconds
testRunShellTests(org.apache.hadoop.hbase.client.TestShell):
(RuntimeError) Shell unit tests failed. Check output file for details.
On Mon, Jul 29, 2013 at 6:46 PM, Enis Söztutar <en...@gmail.com> wrote:
> For namespaces, I was sidetracked for most of last couple of weeks and this
> week as well. I did play with the code a bit and did a couple of
> iterations, but did not get the chance to look into the latest patch. I
> know Stack also looked at it. Unless there are other reviewers, I guess it
> won't make into 0.95.2. This has been said many times before though that it
> is very hard to bring this patch as it is after 0.96 goes out, because
> rolling restarts will be broken.
>
> I am not sure why there is not much interest in this though. It adds
> namespaces, and fixes the system tables problem. Let us reach a decision on
> this either way and move on I guess.
>
> Enis
>
>
> On Mon, Jul 29, 2013 at 11:50 AM, Andrew Purtell <apurtell@apache.org
> >wrote:
>
> > On Mon, Jul 29, 2013 at 11:43 AM, Andrew Purtell <apurtell@apache.org
> > >wrote:
> >
> > > On Mon, Jul 29, 2013 at 10:51 AM, Elliott Clark <ec...@apache.org>
> > wrote:
> > >
> > >> On Mon, Jul 29, 2013 at 10:26 AM, Stack <st...@duboce.net> wrote:
> > >> > Main objection is that the included v3 is not a 'green field'/redo
> so
> > >> lets
> > >> > not call it that
> > >>
> > >> We added minor version into hfile versioning (Broke things while doing
> > >> it), so maybe it's better to call it 2.5 ?
> > >>
> > >
> > > Works for me if that's the preference. Key is being able to switch on
> > > something that might introduce new overheads only if tags are desired
> on
> > > account of some use case that depends on them.
> > >
> >
> > The consequence is we will have less freedom to make new classes for tag
> > specific handling. Granted the work to date has bled through, but that
> > could change. So another option is copy / extend more code until v3
> meets a
> > consensus standard of greenfield.
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Enis Söztutar <en...@gmail.com>.
For namespaces, I was sidetracked for most of last couple of weeks and this
week as well. I did play with the code a bit and did a couple of
iterations, but did not get the chance to look into the latest patch. I
know Stack also looked at it. Unless there are other reviewers, I guess it
won't make into 0.95.2. This has been said many times before though that it
is very hard to bring this patch as it is after 0.96 goes out, because
rolling restarts will be broken.
I am not sure why there is not much interest in this though. It adds
namespaces, and fixes the system tables problem. Let us reach a decision on
this either way and move on I guess.
Enis
On Mon, Jul 29, 2013 at 11:50 AM, Andrew Purtell <ap...@apache.org>wrote:
> On Mon, Jul 29, 2013 at 11:43 AM, Andrew Purtell <apurtell@apache.org
> >wrote:
>
> > On Mon, Jul 29, 2013 at 10:51 AM, Elliott Clark <ec...@apache.org>
> wrote:
> >
> >> On Mon, Jul 29, 2013 at 10:26 AM, Stack <st...@duboce.net> wrote:
> >> > Main objection is that the included v3 is not a 'green field'/redo so
> >> lets
> >> > not call it that
> >>
> >> We added minor version into hfile versioning (Broke things while doing
> >> it), so maybe it's better to call it 2.5 ?
> >>
> >
> > Works for me if that's the preference. Key is being able to switch on
> > something that might introduce new overheads only if tags are desired on
> > account of some use case that depends on them.
> >
>
> The consequence is we will have less freedom to make new classes for tag
> specific handling. Granted the work to date has bled through, but that
> could change. So another option is copy / extend more code until v3 meets a
> consensus standard of greenfield.
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
On Mon, Jul 29, 2013 at 11:43 AM, Andrew Purtell <ap...@apache.org>wrote:
> On Mon, Jul 29, 2013 at 10:51 AM, Elliott Clark <ec...@apache.org> wrote:
>
>> On Mon, Jul 29, 2013 at 10:26 AM, Stack <st...@duboce.net> wrote:
>> > Main objection is that the included v3 is not a 'green field'/redo so
>> lets
>> > not call it that
>>
>> We added minor version into hfile versioning (Broke things while doing
>> it), so maybe it's better to call it 2.5 ?
>>
>
> Works for me if that's the preference. Key is being able to switch on
> something that might introduce new overheads only if tags are desired on
> account of some use case that depends on them.
>
The consequence is we will have less freedom to make new classes for tag
specific handling. Granted the work to date has bled through, but that
could change. So another option is copy / extend more code until v3 meets a
consensus standard of greenfield.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
On Mon, Jul 29, 2013 at 10:51 AM, Elliott Clark <ec...@apache.org> wrote:
> On Mon, Jul 29, 2013 at 10:26 AM, Stack <st...@duboce.net> wrote:
> > Main objection is that the included v3 is not a 'green field'/redo so
> lets
> > not call it that
>
> We added minor version into hfile versioning (Broke things while doing
> it), so maybe it's better to call it 2.5 ?
>
Works for me if that's the preference. Key is being able to switch on
something that might introduce new overheads only if tags are desired on
account of some use case that depends on them.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Elliott Clark <ec...@apache.org>.
On Mon, Jul 29, 2013 at 10:26 AM, Stack <st...@duboce.net> wrote:
> Main objection is that the included v3 is not a 'green field'/redo so lets
> not call it that
We added minor version into hfile versioning (Broke things while doing
it), so maybe it's better to call it 2.5 ?
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Stack <st...@duboce.net>.
On Mon, Jul 29, 2013 at 8:45 AM, Andrew Purtell <ap...@apache.org> wrote:
> On Mon, Jul 29, 2013 at 8:16 AM, Stack <st...@duboce.net> wrote:
>
> > In the patch, it looks to me that v3 is not a true new type but just an
> h2
> > w/ tags shoehorned in. Going forward v3 will not be extensible as done.
> >
>
> I think that's a case of only extending what is needed at this time. More
> V2 classes can be subclassed or replaced as required?
>
>
It looks like methods are being overloaded to add tags. Overloading to add
mvcc was ugly. Tag-effort is compounding this approach.
I'd have thought a v3 would do tags, mvcc, etc., natively.
Main objection is that the included v3 is not a 'green field'/redo so lets
not call it that (My objection is the nomenclature -- if you have to do the
overloads to get in your tags, go for it).
St.Ack
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
On Mon, Jul 29, 2013 at 8:16 AM, Stack <st...@duboce.net> wrote:
> In the patch, it looks to me that v3 is not a true new type but just an h2
> w/ tags shoehorned in. Going forward v3 will not be extensible as done.
>
I think that's a case of only extending what is needed at this time. More
V2 classes can be subclassed or replaced as required?
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Stack <st...@duboce.net>.
On Sun, Jul 28, 2013 at 5:43 PM, Andrew Purtell <ap...@apache.org> wrote:
> On Sun, Jul 28, 2013 at 4:01 PM, Stack <st...@duboce.net> wrote:
>
> > I just did a review of tags. They look like they need a good bit of
> > work yet.
> >
>
> Yes that patch I put up on RB is the one that was cluster tested and
> profiled. Let me work with Ram about getting the latest up there. We also
> needed to get a sense of what can work. For that, please correct me if I am
> mistaken about any of the following:
>
> - An HFileV3 is fine in concept.
>
>
Yes.
In the patch, it looks to me that v3 is not a true new type but just an h2
w/ tags shoehorned in. Going forward v3 will not be extensible as done.
It might as well be called h2+tags?
> - We subclass KeyValue (as TaggedKeyValue) to avoid changing KeyValue's
> serialization code and to avoid heap size inflation for the no tags
> case.
> This is fine modulo issues with encapsulation.
>
>
Yes.
> - Not plumbing tags all the way through to the client as to minimize
> risk in 0.96 is ok. (And this benefits security as a bonus and is also
> fine
> for strictly server-side tag use cases.) This ties in to the above as to
> why a subclass is needed to carry around tags as in memory state, like
> memstoreTS.
>
>
Yes
> - Changing the encoders is fine, since they are shared by all HFile
> implementations. Of course as few changes as possible. Tag specific
> changes
> there - method signature changes mostly - are currently workable but
> ugly,
> but we can handle that in another pass by subclassing more HFile
> classes to
> carry V3 specific state.
>
>
Yes.
Thanks.
St.Ack
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
On Sun, Jul 28, 2013 at 4:01 PM, Stack <st...@duboce.net> wrote:
> I just did a review of tags. They look like they need a good bit of
> work yet.
>
Yes that patch I put up on RB is the one that was cluster tested and
profiled. Let me work with Ram about getting the latest up there. We also
needed to get a sense of what can work. For that, please correct me if I am
mistaken about any of the following:
- An HFileV3 is fine in concept.
- We subclass KeyValue (as TaggedKeyValue) to avoid changing KeyValue's
serialization code and to avoid heap size inflation for the no tags case.
This is fine modulo issues with encapsulation.
- Not plumbing tags all the way through to the client as to minimize
risk in 0.96 is ok. (And this benefits security as a bonus and is also fine
for strictly server-side tag use cases.) This ties in to the above as to
why a subclass is needed to carry around tags as in memory state, like
memstoreTS.
- Changing the encoders is fine, since they are shared by all HFile
implementations. Of course as few changes as possible. Tag specific changes
there - method signature changes mostly - are currently workable but ugly,
but we can handle that in another pass by subclassing more HFile classes to
carry V3 specific state.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Stack <st...@duboce.net>.
On Sun, Jul 28, 2013 at 11:14 AM, Andrew Purtell <ap...@apache.org>wrote:
> On Sat, Jul 27, 2013 at 2:52 PM, Stack <st...@duboce.net> wrote:
>
> > Regards the big features that are racing to make the 0.96 cutoff --
> namely
> > namespaces, tags, and serialization lib -- as I see it, Francis needs
> > reviews if namespaces are to make it, tags ditto, and the serialization
> > libs are nice-to-have auxillaries that can come in at any time. If not
> > done by the end of this week, then as I see it, these features do not
> make
> > the cut.
> >
>
> I think it would be great if namespaces, tags, and the serialization
> work are all in the RC so we can proceed to beat the shit out of it (and
> them). I don't see a clamor for more so this seems to me a place to draw
> the line where everyone is happy. Reviews are important but so is test and
> validation as part of an RC. If there's a problem with any of them, they
> can be dropped for the next.
>
>
I just did a review of tags. They look like they need a good bit of work
yet.
St.Ack
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
Thanks, I appreciate you taking the time to do that. I promise I won't make
this a debate but I would like to respond to a couple of points, for your
consideration.
> Showed up very late in the merge window
Yes, for that I apologize, we have our respective timetables and sometimes
they coincide and sometimes the interaction is not optimal. Our focuses
have been different and then an arbitrary line was drawn in the sand. We
would like to work with you here. We have made a few dramatic changes to
our approach already to try and accommodate it.
> * Is not the green field HFileV3, and doesn't have time for a complete
re-do
I think it's better actually if a V3 shares as much code as possible with
the V2 implementation except for what is actually different, copy-on-write
as it were. That said, I may be getting ahead of the latest patch so will
wait.
> * Can easily be done without out downtime
> ** (changing the encoders could be done without needing downtime. meaning
no singularity is required).
Therefore indeed we can split the work into pre and post singularity. I
have not seen the latest patch but the set of pre-singularity change was
small before and will be the same or smaller, and post changes that are
optional and can be rolled into seems fine, to me, for 0.96.x where x > 0,
so there's no conflict here.
I wouldn't broach the subject of 1.0 because that is a premature
discussion, though I probably don't agree with how you have structured it,
at least there is one point of disagreement.
> Will 100% have perf impacts.
To be pedantic, small cluster (granted) benchmarking does not show a 100%
impact. Rather if V2 is used instead of V3 the impact of the changes are
not distinguishable from background platform variability. I agree wider
testing and probably tuning is necessary before production. There will be
other experimental features in 0.96.
On Tue, Jul 30, 2013 at 4:08 PM, Elliott Clark <ec...@apache.org> wrote:
> On Tue, Jul 30, 2013 at 3:29 PM, Andrew Purtell <ap...@apache.org>
> wrote:
> > Perhaps you can elaborate on the difference you see between HFileV3 and
> > namespaces?
>
> Sure. From my perspective,
>
> Namespaces:
> * arrived earlier in the merge window.
> * Has been run publicly on a large cluster
> * Has had more public reviews.
> * Was re-worked to get closer to a greenfield best implementation
> * Is closer to being merged
> * Is less prone to have unforeseen perf impacts.
> * Is almost impossible to create without needing downtime
> ** (hence it should go in the singularity if possible).
>
>
> HFileV3:
> * Showed up very late in the merge window
> * Still needs revisions
> * Hasn't been put on large clusters publicly.
> * Is not the green field HFileV3, and doesn't have time for a complete
> re-do
> * Can easily be done without out downtime
> ** (changing the encoders could be done without needing downtime.
> meaning no singularity is required).
> * Will 100% have perf impacts.
>
> None of that actually has anything to do with what I think is a more
> exciting feature. It's all about stability of the project and trying
> to find a way move us towards the stable and externally explainable
> release cycles that other real databases have.
>
> I'm actually really excited about tagging. It could allow a lot of
> new and cool features without adding too much core code.
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Elliott Clark <ec...@apache.org>.
On Tue, Jul 30, 2013 at 3:29 PM, Andrew Purtell <ap...@apache.org> wrote:
> Perhaps you can elaborate on the difference you see between HFileV3 and
> namespaces?
Sure. From my perspective,
Namespaces:
* arrived earlier in the merge window.
* Has been run publicly on a large cluster
* Has had more public reviews.
* Was re-worked to get closer to a greenfield best implementation
* Is closer to being merged
* Is less prone to have unforeseen perf impacts.
* Is almost impossible to create without needing downtime
** (hence it should go in the singularity if possible).
HFileV3:
* Showed up very late in the merge window
* Still needs revisions
* Hasn't been put on large clusters publicly.
* Is not the green field HFileV3, and doesn't have time for a complete re-do
* Can easily be done without out downtime
** (changing the encoders could be done without needing downtime.
meaning no singularity is required).
* Will 100% have perf impacts.
None of that actually has anything to do with what I think is a more
exciting feature. It's all about stability of the project and trying
to find a way move us towards the stable and externally explainable
release cycles that other real databases have.
I'm actually really excited about tagging. It could allow a lot of
new and cool features without adding too much core code.
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
On Tue, Jul 30, 2013 at 2:37 PM, Elliott Clark <ec...@apache.org> wrote:
> And to say that HFileV3 can go in and still go 1.0.
>
Perhaps you can elaborate on the difference you see between HFileV3 and
namespaces?
Also, what if HFileV3 were not essential to what we are after ?
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Elliott Clark <ec...@apache.org>.
On Tue, Jul 30, 2013 at 3:28 PM, Enis Söztutar <en...@gmail.com> wrote:
> but I fear that if we prematurely call this 1.0, then we
> would be asking for trouble.
That's fine by me. I was just trying to find a middle ground that
could work for Andrew.
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
On Wed, Jul 31, 2013 at 1:44 PM, James Taylor <jt...@salesforce.com>wrote:
> Given that nothing depends on the order preserving types (i.e. no risk of
> breaking stuff), why not just include them?
>
I was wondering that too but haven't been following closely.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by James Taylor <jt...@salesforce.com>.
IMHO, the value is agreeing on a serialization format across multiple
products. Without a common serialization format, we won't have a good
interop story. Phoenix needs the serialization format to be order
preserving.
Given that nothing depends on the order preserving types (i.e. no risk of
breaking stuff), why not just include them?
On Wed, Jul 31, 2013 at 1:34 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> On Wed, Jul 31, 2013 at 1:19 PM, James Taylor <jtaylor@salesforce.com
> >wrote:
>
> > But the value in your patch is fixing the serialization format such that
> it
> > is order preserving. Unfortunately, without this, Phoenix can't adopt it.
> > It's existing type system and query processing is predicated on this.
> >
>
> Two patches, two value propositions. Providing a data type api with some
> pre-made implementations that users can use and that external projects can
> standardize on is value of itself. Phoenix can extend this API to provide
> it's own encodings, but I agree it provides in HBase something Phoenix has
> already worked out for itself. The biggest win here is that two consumers
> of HBase can agree on precisely what they mean when they say they encode a
> value.
>
> The second piece is the order-preserving encoding scheme. Having HBase ship
> a single scheme that can be used across the board has much wider utility.
> Delivering it through the API described previously is practical. Lacking
> this, Phoenix can still plug it's existing encoding code into the data type
> API, as I described in another email.
>
> I want to see them both shipped. Breaking it down like this was a way to
> allow for prudent concessions considering the timelines.
>
> On Wed, Jul 31, 2013 at 12:04 PM, Nick Dimiduk <nd...@gmail.com>
> wrote:
> >
> > > On Wed, Jul 31, 2013 at 10:31 AM, Stack <st...@duboce.net> wrote:
> > >
> > > > So what would be the incentive using the new API be?
> > > >
> > >
> > > Hopefully the new API is nicer than managing byte[]'s on manually. The
> > only
> > > incentive for users would be keeping up with progress, giving users the
> > > chance to start migrating their applications. For the external tools,
> I'm
> > > looking forward to using this to make defining Hive tables over HBase
> > > nicer. The current column mapping stuff is clunky and this API gives a
> > much
> > > improved mechanism for declaring column types. I can't do that without
> an
> > > API shipping with HBase. Maybe Elliott can weigh in on the Imapala
> side,
> > > James on Phoenix, Bueller from Kiji?
> > >
> > > And then when the implementation changes -- it serializes in sort-order
> > --
> > > > will it confuse?
> > > >
> > >
> > > Let's continue my Hive example. Assuming DataType (9091 + 8694) ships
> in
> > > 0.96, Hive gets plumbed, and users get to start defining their tables
> in
> > > terms of LegacyInteger, LegacyBytesFixedWidth, and Struct. When the
> > > OrderedBytes patch (8201) comes in with it's type implementations,
> users
> > at
> > > their leisure can drop the new types in when they're ready to
> transition.
> > > The Ordered* types don't replace the Legacy* types, they augment the
> > > catalog of types that HBase provides.
> > >
> >
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Nick Dimiduk <nd...@gmail.com>.
On Wed, Jul 31, 2013 at 1:19 PM, James Taylor <jt...@salesforce.com>wrote:
> But the value in your patch is fixing the serialization format such that it
> is order preserving. Unfortunately, without this, Phoenix can't adopt it.
> It's existing type system and query processing is predicated on this.
>
Two patches, two value propositions. Providing a data type api with some
pre-made implementations that users can use and that external projects can
standardize on is value of itself. Phoenix can extend this API to provide
it's own encodings, but I agree it provides in HBase something Phoenix has
already worked out for itself. The biggest win here is that two consumers
of HBase can agree on precisely what they mean when they say they encode a
value.
The second piece is the order-preserving encoding scheme. Having HBase ship
a single scheme that can be used across the board has much wider utility.
Delivering it through the API described previously is practical. Lacking
this, Phoenix can still plug it's existing encoding code into the data type
API, as I described in another email.
I want to see them both shipped. Breaking it down like this was a way to
allow for prudent concessions considering the timelines.
On Wed, Jul 31, 2013 at 12:04 PM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> > On Wed, Jul 31, 2013 at 10:31 AM, Stack <st...@duboce.net> wrote:
> >
> > > So what would be the incentive using the new API be?
> > >
> >
> > Hopefully the new API is nicer than managing byte[]'s on manually. The
> only
> > incentive for users would be keeping up with progress, giving users the
> > chance to start migrating their applications. For the external tools, I'm
> > looking forward to using this to make defining Hive tables over HBase
> > nicer. The current column mapping stuff is clunky and this API gives a
> much
> > improved mechanism for declaring column types. I can't do that without an
> > API shipping with HBase. Maybe Elliott can weigh in on the Imapala side,
> > James on Phoenix, Bueller from Kiji?
> >
> > And then when the implementation changes -- it serializes in sort-order
> --
> > > will it confuse?
> > >
> >
> > Let's continue my Hive example. Assuming DataType (9091 + 8694) ships in
> > 0.96, Hive gets plumbed, and users get to start defining their tables in
> > terms of LegacyInteger, LegacyBytesFixedWidth, and Struct. When the
> > OrderedBytes patch (8201) comes in with it's type implementations, users
> at
> > their leisure can drop the new types in when they're ready to transition.
> > The Ordered* types don't replace the Legacy* types, they augment the
> > catalog of types that HBase provides.
> >
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by James Taylor <jt...@salesforce.com>.
But the value in your patch is fixing the serialization format such that it
is order preserving. Unfortunately, without this, Phoenix can't adopt it.
It's existing type system and query processing is predicated on this.
On Wed, Jul 31, 2013 at 12:04 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> On Wed, Jul 31, 2013 at 10:31 AM, Stack <st...@duboce.net> wrote:
>
> > So what would be the incentive using the new API be?
> >
>
> Hopefully the new API is nicer than managing byte[]'s on manually. The only
> incentive for users would be keeping up with progress, giving users the
> chance to start migrating their applications. For the external tools, I'm
> looking forward to using this to make defining Hive tables over HBase
> nicer. The current column mapping stuff is clunky and this API gives a much
> improved mechanism for declaring column types. I can't do that without an
> API shipping with HBase. Maybe Elliott can weigh in on the Imapala side,
> James on Phoenix, Bueller from Kiji?
>
> And then when the implementation changes -- it serializes in sort-order --
> > will it confuse?
> >
>
> Let's continue my Hive example. Assuming DataType (9091 + 8694) ships in
> 0.96, Hive gets plumbed, and users get to start defining their tables in
> terms of LegacyInteger, LegacyBytesFixedWidth, and Struct. When the
> OrderedBytes patch (8201) comes in with it's type implementations, users at
> their leisure can drop the new types in when they're ready to transition.
> The Ordered* types don't replace the Legacy* types, they augment the
> catalog of types that HBase provides.
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Nick Dimiduk <nd...@gmail.com>.
On Wed, Jul 31, 2013 at 10:31 AM, Stack <st...@duboce.net> wrote:
> So what would be the incentive using the new API be?
>
Hopefully the new API is nicer than managing byte[]'s on manually. The only
incentive for users would be keeping up with progress, giving users the
chance to start migrating their applications. For the external tools, I'm
looking forward to using this to make defining Hive tables over HBase
nicer. The current column mapping stuff is clunky and this API gives a much
improved mechanism for declaring column types. I can't do that without an
API shipping with HBase. Maybe Elliott can weigh in on the Imapala side,
James on Phoenix, Bueller from Kiji?
And then when the implementation changes -- it serializes in sort-order --
> will it confuse?
>
Let's continue my Hive example. Assuming DataType (9091 + 8694) ships in
0.96, Hive gets plumbed, and users get to start defining their tables in
terms of LegacyInteger, LegacyBytesFixedWidth, and Struct. When the
OrderedBytes patch (8201) comes in with it's type implementations, users at
their leisure can drop the new types in when they're ready to transition.
The Ordered* types don't replace the Legacy* types, they augment the
catalog of types that HBase provides.
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Stack <st...@duboce.net>.
On Wed, Jul 31, 2013 at 9:55 AM, Nick Dimiduk <nd...@gmail.com> wrote:
> On Tue, Jul 30, 2013 at 9:38 PM, James Taylor <jtaylor@salesforce.com
> >wrote:
>
> > So row key order won't match the natural sort order?
> >
>
> With the "Legacy" types that are based on Bytes, you get whatever you get.
> Strings and pass-through byte[]'s work like normal; positive integers will
> work, but not negatives; &c. HBase would not ship out-of-the-box with
> general purpose order-preserving types, but you'd have the API and be able
> to implement your own.
>
>
So what would be the incentive using the new API be?
And then when the implementation changes -- it serializes in sort-order --
will it confuse?
Thanks Nick,
St.Ack
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Nick Dimiduk <nd...@gmail.com>.
On Tue, Jul 30, 2013 at 9:38 PM, James Taylor <jt...@salesforce.com>wrote:
> So row key order won't match the natural sort order?
>
With the "Legacy" types that are based on Bytes, you get whatever you get.
Strings and pass-through byte[]'s work like normal; positive integers will
work, but not negatives; &c. HBase would not ship out-of-the-box with
general purpose order-preserving types, but you'd have the API and be able
to implement your own.
On Tue, Jul 30, 2013 at 9:11 PM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> > On Tue, Jul 30, 2013 at 8:20 PM, James Taylor <jtaylor@salesforce.com
> > >wrote:
> >
> > > What's the functionality that we'll lose without the order-preserving
> > part
> > > being included?
> > >
> >
> > Well, order preservation ;) Lacking 8201, we'd get all the existing Bytes
> > goodness but wrapped up in 8693's API. The framework is laid for other
> > HBase components, user applications, and downstream projects to start
> > building on it immediately. I'd like to start pluming it into some new
> > Filters, a couple of the MapReduce tools, Hive's interop layer, maybe
> even
> > replace PDataType with DataType if you're open to such a patch. The
> > OrderedBytes stuff will require applications to have a plan for data
> > migration when they decide to transition over to it, but they can get a
> > head-start on the boiler-plate code.
> >
> > On Tue, Jul 30, 2013 at 5:39 PM, Nick Dimiduk <nd...@gmail.com>
> wrote:
> > >
> > > > On Tue, Jul 30, 2013 at 3:28 PM, Enis Söztutar <en...@gmail.com>
> > > wrote:
> > > >
> > > > > Let me elaborate. There are at least new RPC PB, PB structures in
> > > HFiles
> > > > /
> > > > > hlogs, and zk, table locks, bucket cache, online merge, stochastic
> > LB,
> > > > > hbase on windows, *new data types*, AM changes, favorite node
> > > assignment,
> > > > > dist log replay, and tons of MTTR changes that are not run in
> > > production
> > > > so
> > > > > far.
> > > >
> > > >
> > > > Re: new data types, allow me to throw my hat into the ring as well. I
> > > spoke
> > > > with Stack this morning about a plan that I hope will allow the new
> > data
> > > > type API to squeeze in without requiring the order-preserving
> encoding.
> > > I'd
> > > > like to get this API out with 0.96 so that users interested in
> adopting
> > > > this feature can start migrating their applications sooner than
> later.
> > > > Jealously, I want to start work on the Hive/HBase plumbing using this
> > API
> > > > and the Hive guys are only willing to accept code that is built
> > against a
> > > > labeled HBase release. I'm hoping other projects (*cough* Phoenix,
> > > Impala,
> > > > Kiji *cough*) are keen to follow suit. It also allows for the feature
> > to
> > > > fan out internally -- Filters, Coprocessors, ImportTsv, &c. Using
> this
> > > > approach, DataType and Legacy* friends come in for 0.96.0 and
> > > OrderedBytes
> > > > can join the party in a 0.96.x when it's ready for adoption.
> > > >
> > > > Thanks,
> > > > Nick
> > > >
> > >
> >
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by James Taylor <jt...@salesforce.com>.
So row key order won't match the natural sort order?
On Tue, Jul 30, 2013 at 9:11 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> On Tue, Jul 30, 2013 at 8:20 PM, James Taylor <jtaylor@salesforce.com
> >wrote:
>
> > What's the functionality that we'll lose without the order-preserving
> part
> > being included?
> >
>
> Well, order preservation ;) Lacking 8201, we'd get all the existing Bytes
> goodness but wrapped up in 8693's API. The framework is laid for other
> HBase components, user applications, and downstream projects to start
> building on it immediately. I'd like to start pluming it into some new
> Filters, a couple of the MapReduce tools, Hive's interop layer, maybe even
> replace PDataType with DataType if you're open to such a patch. The
> OrderedBytes stuff will require applications to have a plan for data
> migration when they decide to transition over to it, but they can get a
> head-start on the boiler-plate code.
>
> On Tue, Jul 30, 2013 at 5:39 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> >
> > > On Tue, Jul 30, 2013 at 3:28 PM, Enis Söztutar <en...@gmail.com>
> > wrote:
> > >
> > > > Let me elaborate. There are at least new RPC PB, PB structures in
> > HFiles
> > > /
> > > > hlogs, and zk, table locks, bucket cache, online merge, stochastic
> LB,
> > > > hbase on windows, *new data types*, AM changes, favorite node
> > assignment,
> > > > dist log replay, and tons of MTTR changes that are not run in
> > production
> > > so
> > > > far.
> > >
> > >
> > > Re: new data types, allow me to throw my hat into the ring as well. I
> > spoke
> > > with Stack this morning about a plan that I hope will allow the new
> data
> > > type API to squeeze in without requiring the order-preserving encoding.
> > I'd
> > > like to get this API out with 0.96 so that users interested in adopting
> > > this feature can start migrating their applications sooner than later.
> > > Jealously, I want to start work on the Hive/HBase plumbing using this
> API
> > > and the Hive guys are only willing to accept code that is built
> against a
> > > labeled HBase release. I'm hoping other projects (*cough* Phoenix,
> > Impala,
> > > Kiji *cough*) are keen to follow suit. It also allows for the feature
> to
> > > fan out internally -- Filters, Coprocessors, ImportTsv, &c. Using this
> > > approach, DataType and Legacy* friends come in for 0.96.0 and
> > OrderedBytes
> > > can join the party in a 0.96.x when it's ready for adoption.
> > >
> > > Thanks,
> > > Nick
> > >
> >
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Nick Dimiduk <nd...@gmail.com>.
On Tue, Jul 30, 2013 at 8:20 PM, James Taylor <jt...@salesforce.com>wrote:
> What's the functionality that we'll lose without the order-preserving part
> being included?
>
Well, order preservation ;) Lacking 8201, we'd get all the existing Bytes
goodness but wrapped up in 8693's API. The framework is laid for other
HBase components, user applications, and downstream projects to start
building on it immediately. I'd like to start pluming it into some new
Filters, a couple of the MapReduce tools, Hive's interop layer, maybe even
replace PDataType with DataType if you're open to such a patch. The
OrderedBytes stuff will require applications to have a plan for data
migration when they decide to transition over to it, but they can get a
head-start on the boiler-plate code.
On Tue, Jul 30, 2013 at 5:39 PM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> > On Tue, Jul 30, 2013 at 3:28 PM, Enis Söztutar <en...@gmail.com>
> wrote:
> >
> > > Let me elaborate. There are at least new RPC PB, PB structures in
> HFiles
> > /
> > > hlogs, and zk, table locks, bucket cache, online merge, stochastic LB,
> > > hbase on windows, *new data types*, AM changes, favorite node
> assignment,
> > > dist log replay, and tons of MTTR changes that are not run in
> production
> > so
> > > far.
> >
> >
> > Re: new data types, allow me to throw my hat into the ring as well. I
> spoke
> > with Stack this morning about a plan that I hope will allow the new data
> > type API to squeeze in without requiring the order-preserving encoding.
> I'd
> > like to get this API out with 0.96 so that users interested in adopting
> > this feature can start migrating their applications sooner than later.
> > Jealously, I want to start work on the Hive/HBase plumbing using this API
> > and the Hive guys are only willing to accept code that is built against a
> > labeled HBase release. I'm hoping other projects (*cough* Phoenix,
> Impala,
> > Kiji *cough*) are keen to follow suit. It also allows for the feature to
> > fan out internally -- Filters, Coprocessors, ImportTsv, &c. Using this
> > approach, DataType and Legacy* friends come in for 0.96.0 and
> OrderedBytes
> > can join the party in a 0.96.x when it's ready for adoption.
> >
> > Thanks,
> > Nick
> >
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by James Taylor <jt...@salesforce.com>.
Hey Nick,
What's the functionality that we'll lose without the order-preserving part
being included?
Thanks,
James
On Tue, Jul 30, 2013 at 5:39 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> On Tue, Jul 30, 2013 at 3:28 PM, Enis Söztutar <en...@gmail.com> wrote:
>
> > Let me elaborate. There are at least new RPC PB, PB structures in HFiles
> /
> > hlogs, and zk, table locks, bucket cache, online merge, stochastic LB,
> > hbase on windows, *new data types*, AM changes, favorite node assignment,
> > dist log replay, and tons of MTTR changes that are not run in production
> so
> > far.
>
>
> Re: new data types, allow me to throw my hat into the ring as well. I spoke
> with Stack this morning about a plan that I hope will allow the new data
> type API to squeeze in without requiring the order-preserving encoding. I'd
> like to get this API out with 0.96 so that users interested in adopting
> this feature can start migrating their applications sooner than later.
> Jealously, I want to start work on the Hive/HBase plumbing using this API
> and the Hive guys are only willing to accept code that is built against a
> labeled HBase release. I'm hoping other projects (*cough* Phoenix, Impala,
> Kiji *cough*) are keen to follow suit. It also allows for the feature to
> fan out internally -- Filters, Coprocessors, ImportTsv, &c. Using this
> approach, DataType and Legacy* friends come in for 0.96.0 and OrderedBytes
> can join the party in a 0.96.x when it's ready for adoption.
>
> Thanks,
> Nick
>
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Nick Dimiduk <nd...@gmail.com>.
On Tue, Jul 30, 2013 at 3:28 PM, Enis Söztutar <en...@gmail.com> wrote:
> Let me elaborate. There are at least new RPC PB, PB structures in HFiles /
> hlogs, and zk, table locks, bucket cache, online merge, stochastic LB,
> hbase on windows, *new data types*, AM changes, favorite node assignment,
> dist log replay, and tons of MTTR changes that are not run in production so
> far.
Re: new data types, allow me to throw my hat into the ring as well. I spoke
with Stack this morning about a plan that I hope will allow the new data
type API to squeeze in without requiring the order-preserving encoding. I'd
like to get this API out with 0.96 so that users interested in adopting
this feature can start migrating their applications sooner than later.
Jealously, I want to start work on the Hive/HBase plumbing using this API
and the Hive guys are only willing to accept code that is built against a
labeled HBase release. I'm hoping other projects (*cough* Phoenix, Impala,
Kiji *cough*) are keen to follow suit. It also allows for the feature to
fan out internally -- Filters, Coprocessors, ImportTsv, &c. Using this
approach, DataType and Legacy* friends come in for 0.96.0 and OrderedBytes
can join the party in a 0.96.x when it's ready for adoption.
Thanks,
Nick
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Enis Söztutar <en...@gmail.com>.
> It seems incongruous to say things aren't tested at scale is what's
> holding us back from 1.0. And to say that HFileV3 can go in and still
> go 1.0.
>
Let me elaborate. There are at least new RPC PB, PB structures in HFiles /
hlogs, and zk, table locks, bucket cache, online merge, stochastic LB,
hbase on windows, new data types, AM changes, favorite node assignment,
dist log replay, and tons of MTTR changes that are not run in production so
far. The amount of new stuff is scary. Yes, we and other groups are doing
extensive testing, but I fear that if we prematurely call this 1.0, then we
would be asking for trouble.
HFile v3 on the other hand, can be baked in 0.96.x releases if time allows
it. I am expecting that to also go a few minor releases, and not be
committed just before 1.0 comes out.
Enis
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Elliott Clark <ec...@apache.org>.
On Tue, Jul 30, 2013 at 1:55 PM, Enis Söztutar <en...@gmail.com> wrote:
> There are some features that were not tested at scale. I think we can
> do 0.96, and mark later releases of 0.96.x to be 1.0.0.
It seems incongruous to say things aren't tested at scale is what's
holding us back from 1.0. And to say that HFileV3 can go in and still
go 1.0.
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Enis Söztutar <en...@gmail.com>.
I don't think we should release what is scheduled to be 0.96 as 1.0.0,
since I think we should give the new features time for one more release to
bake. There are some features that were not tested at scale. I think we can
do 0.96, and mark later releases of 0.96.x to be 1.0.0.
For HFile v3, I would not object to bring this in for later releases of
0.96.x if we cannot do timely releases. Ideally, we should continue on the
major release train, and have 0.98/1.1.0 (or whatever is next) in 3-4
months or so. Then we won't have non-ending discussions of backporting and
all.
Enis
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
On Mon, Jul 29, 2013 at 9:51 PM, Elliott Clark <ec...@apache.org> wrote:
> As things currently stand, I'm -1 (binding) any new features other
> than namespaces in 0.96.x.
>
I am not happy with any 0.96 release that does not allow forward progress
with security features.
What about the compromise I proposed mere hours ago?
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Elliott Clark <ec...@apache.org>.
As things currently stand, I'm -1 (binding) any new features other
than namespaces in 0.96.x.
I think we can all agree that 0.96 has taken way too long to get out.
0.96 will be a monster release with bunches of new features, tons of
new code paths, and lots of uncertainty. To combat that lots of people
and companies have put tons of effort into testing and stabilizing the
current 0.95 branch. The best way I see forward is to start releasing
much faster.
I'd like to propose:
* we stabilize 0.95 branch completely. Make that branch rock solid.
* When we are happy with it, release it as 1.0.0.
* Then (a couple of months) after release 1.1.0 with hfile v3 and any
other feature that haven't made the cut for 1.0.0 .
* really start using semver(http://semver.org/) so that users can know
for sure that a patch version doesn't bring much risk at all.
That will make sure that any new features contributed aren't stuck
waiting to be released forever, and we will always be able to have a
very stable patch version that risk adverse people can use.
Additionally having faster minor release trains will mean that we can
be much more rigid in our back-porting policies.
On Mon, Jul 29, 2013 at 9:11 PM, Andrew Purtell <ap...@apache.org> wrote:
> On Mon, Jul 29, 2013 at 8:44 PM, Stack <st...@duboce.net> wrote:
>
>> > Would we be able to get an HFile V3 into 0.96.1 if 0.96.0 goes out
>> without?
>>
>> It'd be a feature I suppose? 0.98? What you thinking Andrew?
>>
>
> Waiting for tags in 0.98 would not be good because we have dependent issues
> that would have to wait too. If 0.96.1 instead of 0.96.0 is doable for the
> file format we can focus on API changes for 0.96.0 that are a must and so
> narrow the scope under consideration today.
>
>> Redo what ye have but it'll take too long? Could ye get core parts in?
>
> We are addressing feedback. Updated patch on RB forthcoming.
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
On Mon, Jul 29, 2013 at 8:44 PM, Stack <st...@duboce.net> wrote:
> > Would we be able to get an HFile V3 into 0.96.1 if 0.96.0 goes out
> without?
>
> It'd be a feature I suppose? 0.98? What you thinking Andrew?
>
Waiting for tags in 0.98 would not be good because we have dependent issues
that would have to wait too. If 0.96.1 instead of 0.96.0 is doable for the
file format we can focus on API changes for 0.96.0 that are a must and so
narrow the scope under consideration today.
> Redo what ye have but it'll take too long? Could ye get core parts in?
We are addressing feedback. Updated patch on RB forthcoming.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Stack <st...@duboce.net>.
On Mon, Jul 29, 2013 at 7:02 PM, Andrew Purtell <ap...@apache.org> wrote:
> On Sun, Jul 28, 2013 at 11:14 AM, Andrew Purtell <apurtell@apache.org
> >wrote:
>
> > On Sat, Jul 27, 2013 at 2:52 PM, Stack <st...@duboce.net> wrote:
> >
> >> Regards the big features that are racing to make the 0.96 cutoff --
> namely
> >> namespaces, tags, and serialization lib -- as I see it, Francis needs
> >> reviews if namespaces are to make it, tags ditto, and the serialization
> >> libs are nice-to-have auxillaries that can come in at any time. If not
> >> done by the end of this week, then as I see it, these features do not
> make
> >> the cut.
> >>
> >
> > I think it would be great if namespaces, tags, and the serialization
> > work are all in the RC
>
>
> Would we be able to get an HFile V3 into 0.96.1 if 0.96.0 goes out without?
>
>
It'd be a feature I suppose? 0.98? What you thinking Andrew? Redo what
ye have but it'll take too long? Could ye get core parts in?
St.Ack
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
On Sun, Jul 28, 2013 at 11:14 AM, Andrew Purtell <ap...@apache.org>wrote:
> On Sat, Jul 27, 2013 at 2:52 PM, Stack <st...@duboce.net> wrote:
>
>> Regards the big features that are racing to make the 0.96 cutoff -- namely
>> namespaces, tags, and serialization lib -- as I see it, Francis needs
>> reviews if namespaces are to make it, tags ditto, and the serialization
>> libs are nice-to-have auxillaries that can come in at any time. If not
>> done by the end of this week, then as I see it, these features do not make
>> the cut.
>>
>
> I think it would be great if namespaces, tags, and the serialization
> work are all in the RC
Would we be able to get an HFile V3 into 0.96.1 if 0.96.0 goes out without?
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [UPDATE] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues
Posted by Andrew Purtell <ap...@apache.org>.
On Sat, Jul 27, 2013 at 2:52 PM, Stack <st...@duboce.net> wrote:
> Regards the big features that are racing to make the 0.96 cutoff -- namely
> namespaces, tags, and serialization lib -- as I see it, Francis needs
> reviews if namespaces are to make it, tags ditto, and the serialization
> libs are nice-to-have auxillaries that can come in at any time. If not
> done by the end of this week, then as I see it, these features do not make
> the cut.
>
I think it would be great if namespaces, tags, and the serialization
work are all in the RC so we can proceed to beat the shit out of it (and
them). I don't see a clamor for more so this seems to me a place to draw
the line where everyone is happy. Reviews are important but so is test and
validation as part of an RC. If there's a problem with any of them, they
can be dropped for the next.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)