You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Stack <st...@duboce.net> on 2013/11/03 05:44:58 UTC
Minutes from Developer Meetup at HWX October 24th
Below are rough minutes from the developer meetup last Thursday week down
at HortonWorks: http://www.meetup.com/hackathon/events/144366512/
The agenda was fast moving and the notes I kept were sparse (pardon me).
Hopefully the below at least conveys some flavor of what transpired.
Below is proposed agenda with discussion filled in in between in italics.
Git/Gerrit<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.bi2n1jnf8wr9>
Lieutenants<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ts3zsiollp5s>
MTTR<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ngm4d0ee18u2>
Distributed log
replay<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.5nxjf2azyr63>
Region online for
writes<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.7h4u62q0yu1t>
Client<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.dxyzoaydy7up>
0.98<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.jny6o6f3j3x8>
1.0<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.4sjuoukolha1>
Release passes IT tests for time period before
release?<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ijcofohq0rk1>
Sequenceid<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.14eyjkb3glf1>
MVCC<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.hbv260gur4ui>
Multiwal<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.tab1edfjmpgq>
Speculative Read<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.3xr7mrnpmpby>
Favored node finish
up<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.2s9y1srcix0z>
Compactions<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.23pauxcj0ux8>
Stripe?<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.xsncn8yoed07>
Master Redo<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ngm4d0ee18u2>
Secondary indices from
phoenix<https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.mk56oao2p3vj>
Git/Gerrit
Move to this? Or do Git without gerrit ?
All were +1 on going to GIT. A gentleman Accumulo fellow present
volunteered to pass us his script for how they made the transition.
DISCUSSION already started on dev list.
Lieutenants/Component Owners
Review. How is this going? What can we do to improve? Not enough reviews
going on.
The list of Component Owners needs an edit. It is stale. We need to
revive/renew/refresh this initiative.
More Friction Committing
Not enough reviews of stuff going in and this is a db after all? +1
(eclark) +1 from Dave
Generally agreed. Pointed at Lieutenants/Component Owners role. Was
thought there should be some sort of automated performance test as part of
hadoopqa. A benchmark proving no degradation of a claimed improvement is
required.
Better was adding some set of general micro benchmarks... and then do them
on your machine before and after and paste findings.
Compat testing
Now that 0.96.0 is out the door, we should be careful about changes that
are compat-breaking. Any ideas on how to best avoid/detect such changes?
Also maybe have a discussion about how wire compat/protobufs changes how
we do things, how to best utilize it, pitfalls to avoid, what it can’t do?
Formalize compat matrix? Go over the matrix we had 1 year ago. Do we want
to support all combinations, or only between two major releases with some
rolling upgrade model? How to avoid current situation where client does not
scale -- found at last minute running on 80 node cluster!
Test framework do this? But would have to be a non-apache context because
no resources there to do it. There are the jdiff and compare. We have to
do rolling upgrade tests. Deprecations in .protos too. Remove after a
release. Just go version numbers. Write up the matrix. How to avoid issues
like the recent client not scaling. Need perf tests. Need
microbenchmarks. Modular so can pull out and simulate rpc. Tests like
that. Standup simple server done in proto
MTTRDistributed log replay
Relax semantics; it is ok to allow out-of-order edits? Or fix?
Distributed log replay going to be done in 0.98 using tags.
Region online for writes
Lets just do this.
Client
- Short term perf/scalability regression (0.96) time frame
- Long term (0.98/1.x/2.x) time frame.
- Asynchbase on 0.96 - almost there.
Should we remove support for setting timestamp. Lars says set it in
HTableDescription. Sequence number and ts. Sequenceid Perhaps add new
coordinate. Out-of-order deletes. Ts w/ the value. Make it configurable
and do optimization.
*Aditya brought up new c-client effort.*
0.98
- tags.
- reverse scan
- issue with cells? (intel guys at mtg?)
Where is the branch?
1.0
is 0.98 == 1.0?
- when to drop hadoop1 support? next year?
98.x becomes 1.0 Security not as a CP but in codepath. Do check in the
code. Make security first class. But security was always good for dev'ing
CP. Make it a required CP. Integrated w/ simple testing. Enable
Authorization by default. Permissive mode needs to be added. Small perf
price.
We need better defaults -- especially for 1.0. Revisit the docs, the
refguide. An edit. Questions, the docbook.... Formatting is a pain. Docathon?
….but no one would show up. Drop hadoop1 post 1.0.
New release adoption
- What can we do for 0.96 adoption
- People are still using 0.90
Within versions, update... You are in the middle of a rolling upgrade. Can
we make it easier for folks to update? Advertising that upgrade to 0.96 is
successful. Rolling upgrade needs to be in the master, not out in a script.
Release passes IT tests for time period before release?
48 hours Real honest, regionserver, datanodes, zookeepers all being
killed. Committers releasing it for a time. Failing a 48 hour test is
enough to sink an RC. We have a bar and we should keep raising it.
Elliott will put up bar for test release.
SequenceidMVCC
Unify sequenceid and mvcc
Do we need locks at all?
MultiwalSpeculative Read
Raise awareness with hdfs brothers and sisters
Favored node finish up
Add consideration to the balancer
Need to fix balancer. Not AM job. Its in AM and in Balancer.
CompactionsStripe?
Get some empirical data to see if different types of compactions are better.
Master Redo
Jimmy says evolve instead of rewrite
Secondary indices from phoenix?
Re: Minutes from Developer Meetup at HWX October 24th
Posted by Stack <st...@duboce.net>.
On Mon, Nov 4, 2013 at 9:13 PM, Andrew Purtell <ap...@apache.org> wrote:
> When last we spoke on dev@ about 0.98 I suggested branching one month
> after
> the 0.96.0 release. There was no comment or objection. That would be late
> November then. No problem to branch sooner.
>
> Sorry none of us at Intel could be there in person. For 0.98, to add to the
> below list there are a couple of security related issues we're aiming for
> the release, based on tags, coprocessors, and HFile v3: cell ACLs,
> visibility labels, and transparent encryption.
>
>
Branch end of Nov. sounds about right for getting the above in.
Good on you Andrew,
St.Ack
On Mon, Nov 4, 2013 at 9:13 PM, Andrew Purtell <ap...@apache.org> wrote:
> When last we spoke on dev@ about 0.98 I suggested branching one month
> after
> the 0.96.0 release. There was no comment or objection. That would be late
> November then. No problem to branch sooner.
>
> Sorry none of us at Intel could be there in person. For 0.98, to add to the
> below list there are a couple of security related issues we're aiming for
> the release, based on tags, coprocessors, and HFile v3: cell ACLs,
> visibility labels, and transparent encryption.
>
>
>
> On Saturday, November 2, 2013, Stack wrote:
>
> > Below are rough minutes from the developer meetup last Thursday week down
> > at HortonWorks: http://www.meetup.com/hackathon/events/144366512/
> >
> > The agenda was fast moving and the notes I kept were sparse (pardon me).
> > Hopefully the below at least conveys some flavor of what transpired.
> >
> >
> > Below is proposed agenda with discussion filled in in between in italics.
> >
> > Git/Gerrit<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.bi2n1jnf8wr9
> > >
> >
> > Lieutenants<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ts3zsiollp5s
> > >
> >
> > MTTR<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ngm4d0ee18u2
> > >
> >
> > Distributed log
> > replay<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.5nxjf2azyr63
> > >
> >
> > Region online for
> > writes<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.7h4u62q0yu1t
> > >
> >
> > Client<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.dxyzoaydy7up
> > >
> >
> > 0.98<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.jny6o6f3j3x8
> > >
> >
> > 1.0<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.4sjuoukolha1
> > >
> >
> > Release passes IT tests for time period before
> > release?<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ijcofohq0rk1
> > >
> >
> > Sequenceid<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.14eyjkb3glf1
> > >
> >
> > MVCC<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.hbv260gur4ui
> > >
> >
> > Multiwal<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.tab1edfjmpgq
> > >
> >
> > Speculative Read<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.3xr7mrnpmpby
> > >
> >
> > Favored node finish
> > up<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.2s9y1srcix0z
> > >
> >
> > Compactions<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.23pauxcj0ux8
> > >
> >
> > Stripe?<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.xsncn8yoed07
> > >
> >
> > Master Redo<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ngm4d0ee18u2
> > >
> >
> > Secondary indices from
> > phoenix<
> >
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.mk56oao2p3vj
> > >
> >
> >
> > Git/Gerrit
> >
> > Move to this? Or do Git without gerrit ?
> >
> > All were +1 on going to GIT. A gentleman Accumulo fellow present
> > volunteered to pass us his script for how they made the transition.
> > DISCUSSION already started on dev list.
> >
> > Lieutenants/Component Owners
> >
> > Review. How is this going? What can we do to improve? Not enough
> reviews
> > going on.
> >
> > The list of Component Owners needs an edit. It is stale. We need to
> > revive/renew/refresh this initiative.
> >
> > More Friction Committing
> >
> > Not enough reviews of stuff going in and this is a db after all? +1
> > (eclark) +1 from Dave
> >
> > Generally agreed. Pointed at Lieutenants/Component Owners role. Was
> > thought there should be some sort of automated performance test as part
> of
> > hadoopqa. A benchmark proving no degradation of a claimed improvement is
> > required.
> >
> > Better was adding some set of general micro benchmarks... and then do
> them
> > on your machine before and after and paste findings.
> >
> > Compat testing
> >
> > Now that 0.96.0 is out the door, we should be careful about changes that
> > are compat-breaking. Any ideas on how to best avoid/detect such changes?
> > Also maybe have a discussion about how wire compat/protobufs changes how
> > we do things, how to best utilize it, pitfalls to avoid, what it can’t
> do?
> >
> > Formalize compat matrix? Go over the matrix we had 1 year ago. Do we want
> > to support all combinations, or only between two major releases with some
> > rolling upgrade model? How to avoid current situation where client does
> not
> > scale -- found at last minute running on 80 node cluster!
> >
> > Test framework do this? But would have to be a non-apache context
> because
> > no resources there to do it. There are the jdiff and compare. We have to
> > do rolling upgrade tests. Deprecations in .protos too. Remove after a
> > release. Just go version numbers. Write up the matrix. How to avoid
> issues
> > like the recent client not scaling. Need perf tests. Need
> > microbenchmarks. Modular so can pull out and simulate rpc. Tests like
> > that. Standup simple server done in proto
> > MTTRDistributed log replay
> >
> > Relax semantics; it is ok to allow out-of-order edits? Or fix?
> >
> > Distributed log replay going to be done in 0.98 using tags.
> > Region online for writes
> >
> > Lets just do this.
> >
> > Client
> >
> > - Short term perf/scalability regression (0.96) time frame
> >
> > - Long term (0.98/1.x/2.x) time frame.
> >
> > - Asynchbase on 0.96 - almost there.
> >
> > Should we remove support for setting timestamp. Lars says set it in
> > HTableDescription. Sequence number and ts. Sequenceid Perhaps add new
> > coordinate. Out-of-order deletes. Ts w/ the value. Make it configurable
> > and do optimization.
> >
> > *Aditya brought up new c-client effort.*
> > 0.98
> >
> > - tags.
> >
> > - reverse scan
> >
> > - issue with cells? (intel guys at mtg?)
> >
> > Where is the branch?
> > 1.0
> >
> > is 0.98 == 1.0?
> >
> > - when to drop hadoop1 support? next year?
> >
> >
> > 98.x becomes 1.0 Security not as a CP but in codepath. Do check in the
> > code. Make security first class. But security was always good for dev'ing
> > CP. Make it a required CP. Integrated w/ simple testing. Enable
> > Authorization by default. Permissive mode needs to be added. Small perf
> > price.
> >
> >
> > We need better defaults -- especially for 1.0. Revisit the docs, the
> > refguide. An edit. Questions, the docbook.... Formatting is a pain.
> > Docathon?
> > ….but no one would show up. Drop hadoop1 post 1.0.
> >
> > New release adoption
> >
> > - What can we do for 0.96 adoption
> >
> > - People are still using 0.90
> >
> > Within versions, update... You are in the middle of a rolling upgrade.
> Can
> > we make it easier for folks to update? Advertising that upgrade to 0.96
> is
> > successful. Rolling upgrade needs to be in the master, not out in a
> script.
> > Release passes IT tests for time period before release?
> >
> > 48 hours Real honest, regionserver, datanodes, zookeepers all being
> > killed. Committers releasing it for a time. Failing a 48 hour test is
> > enough to sink an RC. We have a bar and we should keep raising it.
> >
> > Elliott will put up bar for test release.
> >
> > SequenceidMVCC
> >
> > Unify sequenceid and mvcc
> >
> > Do we need locks at all?
> > MultiwalSpeculative Read
> >
> > Raise awareness with hdfs brothers and sisters
> > Favored node finish up
> >
> > Add consideration to the balancer
> >
> > Need to fix balancer. Not AM job. Its in AM and in Balancer.
> >
> >
> > CompactionsStripe?
> >
> > Get some empirical data to see if different types of compactions are
> > better.
> > Master Redo
> >
> > Jimmy says evolve instead of rewrite
> > Secondary indices from phoenix?
> >
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: Minutes from Developer Meetup at HWX October 24th
Posted by Andrew Purtell <ap...@apache.org>.
When last we spoke on dev@ about 0.98 I suggested branching one month after
the 0.96.0 release. There was no comment or objection. That would be late
November then. No problem to branch sooner.
Sorry none of us at Intel could be there in person. For 0.98, to add to the
below list there are a couple of security related issues we're aiming for
the release, based on tags, coprocessors, and HFile v3: cell ACLs,
visibility labels, and transparent encryption.
On Saturday, November 2, 2013, Stack wrote:
> Below are rough minutes from the developer meetup last Thursday week down
> at HortonWorks: http://www.meetup.com/hackathon/events/144366512/
>
> The agenda was fast moving and the notes I kept were sparse (pardon me).
> Hopefully the below at least conveys some flavor of what transpired.
>
>
> Below is proposed agenda with discussion filled in in between in italics.
>
> Git/Gerrit<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.bi2n1jnf8wr9
> >
>
> Lieutenants<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ts3zsiollp5s
> >
>
> MTTR<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ngm4d0ee18u2
> >
>
> Distributed log
> replay<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.5nxjf2azyr63
> >
>
> Region online for
> writes<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.7h4u62q0yu1t
> >
>
> Client<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.dxyzoaydy7up
> >
>
> 0.98<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.jny6o6f3j3x8
> >
>
> 1.0<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.4sjuoukolha1
> >
>
> Release passes IT tests for time period before
> release?<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ijcofohq0rk1
> >
>
> Sequenceid<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.14eyjkb3glf1
> >
>
> MVCC<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.hbv260gur4ui
> >
>
> Multiwal<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.tab1edfjmpgq
> >
>
> Speculative Read<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.3xr7mrnpmpby
> >
>
> Favored node finish
> up<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.2s9y1srcix0z
> >
>
> Compactions<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.23pauxcj0ux8
> >
>
> Stripe?<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.xsncn8yoed07
> >
>
> Master Redo<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.ngm4d0ee18u2
> >
>
> Secondary indices from
> phoenix<
> https://docs.google.com/document/d/1pEFqKKxnJWLyOc_jsqmpWcBHl-kC98a969JJ9tVVZEE/edit#heading=h.mk56oao2p3vj
> >
>
>
> Git/Gerrit
>
> Move to this? Or do Git without gerrit ?
>
> All were +1 on going to GIT. A gentleman Accumulo fellow present
> volunteered to pass us his script for how they made the transition.
> DISCUSSION already started on dev list.
>
> Lieutenants/Component Owners
>
> Review. How is this going? What can we do to improve? Not enough reviews
> going on.
>
> The list of Component Owners needs an edit. It is stale. We need to
> revive/renew/refresh this initiative.
>
> More Friction Committing
>
> Not enough reviews of stuff going in and this is a db after all? +1
> (eclark) +1 from Dave
>
> Generally agreed. Pointed at Lieutenants/Component Owners role. Was
> thought there should be some sort of automated performance test as part of
> hadoopqa. A benchmark proving no degradation of a claimed improvement is
> required.
>
> Better was adding some set of general micro benchmarks... and then do them
> on your machine before and after and paste findings.
>
> Compat testing
>
> Now that 0.96.0 is out the door, we should be careful about changes that
> are compat-breaking. Any ideas on how to best avoid/detect such changes?
> Also maybe have a discussion about how wire compat/protobufs changes how
> we do things, how to best utilize it, pitfalls to avoid, what it can’t do?
>
> Formalize compat matrix? Go over the matrix we had 1 year ago. Do we want
> to support all combinations, or only between two major releases with some
> rolling upgrade model? How to avoid current situation where client does not
> scale -- found at last minute running on 80 node cluster!
>
> Test framework do this? But would have to be a non-apache context because
> no resources there to do it. There are the jdiff and compare. We have to
> do rolling upgrade tests. Deprecations in .protos too. Remove after a
> release. Just go version numbers. Write up the matrix. How to avoid issues
> like the recent client not scaling. Need perf tests. Need
> microbenchmarks. Modular so can pull out and simulate rpc. Tests like
> that. Standup simple server done in proto
> MTTRDistributed log replay
>
> Relax semantics; it is ok to allow out-of-order edits? Or fix?
>
> Distributed log replay going to be done in 0.98 using tags.
> Region online for writes
>
> Lets just do this.
>
> Client
>
> - Short term perf/scalability regression (0.96) time frame
>
> - Long term (0.98/1.x/2.x) time frame.
>
> - Asynchbase on 0.96 - almost there.
>
> Should we remove support for setting timestamp. Lars says set it in
> HTableDescription. Sequence number and ts. Sequenceid Perhaps add new
> coordinate. Out-of-order deletes. Ts w/ the value. Make it configurable
> and do optimization.
>
> *Aditya brought up new c-client effort.*
> 0.98
>
> - tags.
>
> - reverse scan
>
> - issue with cells? (intel guys at mtg?)
>
> Where is the branch?
> 1.0
>
> is 0.98 == 1.0?
>
> - when to drop hadoop1 support? next year?
>
>
> 98.x becomes 1.0 Security not as a CP but in codepath. Do check in the
> code. Make security first class. But security was always good for dev'ing
> CP. Make it a required CP. Integrated w/ simple testing. Enable
> Authorization by default. Permissive mode needs to be added. Small perf
> price.
>
>
> We need better defaults -- especially for 1.0. Revisit the docs, the
> refguide. An edit. Questions, the docbook.... Formatting is a pain.
> Docathon?
> ….but no one would show up. Drop hadoop1 post 1.0.
>
> New release adoption
>
> - What can we do for 0.96 adoption
>
> - People are still using 0.90
>
> Within versions, update... You are in the middle of a rolling upgrade. Can
> we make it easier for folks to update? Advertising that upgrade to 0.96 is
> successful. Rolling upgrade needs to be in the master, not out in a script.
> Release passes IT tests for time period before release?
>
> 48 hours Real honest, regionserver, datanodes, zookeepers all being
> killed. Committers releasing it for a time. Failing a 48 hour test is
> enough to sink an RC. We have a bar and we should keep raising it.
>
> Elliott will put up bar for test release.
>
> SequenceidMVCC
>
> Unify sequenceid and mvcc
>
> Do we need locks at all?
> MultiwalSpeculative Read
>
> Raise awareness with hdfs brothers and sisters
> Favored node finish up
>
> Add consideration to the balancer
>
> Need to fix balancer. Not AM job. Its in AM and in Balancer.
>
>
> CompactionsStripe?
>
> Get some empirical data to see if different types of compactions are
> better.
> Master Redo
>
> Jimmy says evolve instead of rewrite
> Secondary indices from phoenix?
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)