You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Devaraj Das <dd...@hortonworks.com> on 2013/02/20 02:27:11 UTC
Some notes from the HBase developer Pow-Wow
- Stabilizing 0.96 / CI
-- not running continuously
-- Roman: is it a good idea to run IT under Bigtop
- the HBase unit tests are good..
- What about HBase as a backend for hive - tests for those
- Do we think there is value with the integration with the rest
of the hadoop ecosystem
-- Jon: What's needed to make this work
-- Roman: Collective agreement that we want to solve this
-- Stack: We need to run the IT tests from Enis and Sergey running
continuously
-- Roman: If we all agree that this is something needs to be fixed
.. then yes we would talk about mechanics. Bigtop doesn't have
expertise in HBase and hence HBase folks would have to debug failures.
-- Roman: Bigtop is committed to tests. Less than a dozen tests for
hbase currently..
-- We have been running validation around RC time. Find all kinds of
issues - sometimes trivial (maybe a config issue),
-- Roman can offer CI for trunk but will work for only hadoop-2 line.
-- Roman does first line of triage.
-- No issues to do with other ecosystem artifacts. Bigtop ensures
the right artifacts are in place.
-- hadoop-2 is important but not particular about the version of
hadoop within 2. For example 2.0.2
-- Gary: Will be good to run security tests
-- Roman & Devaraj to talk on how this can be done/implemented
On 0.96 branching
- Lars:
-- We will have three branches to maintain.
- Stack: we need to stabilize quickly
- Enis: What about 0.95.
- Stack: Just do the snapshots thing. Every week, give a snapshot
- Enis: we've done a bunch of stuff in RPC.. If we have to break
something, we can if it is beta (0.96-beta).
- Agreement is there generally ... Debate on the name with snapshot
versus 95.0/96.0
-- 0.95 experimental .. 0.96 will be stable
-- If we go with 0.95, releasing will be easier as well..
-- 0.95 will not be in production..
-- 0.96 will be off 0.95 branch and not trunk based
- How do we go about committing issues.. (issue commit rate is low)
-- 0.94 is stable - 2 new commits and 2 new bugs a day
-- 0.96 has lots of issues not reviewed
-- Break up the patch into multiple smaller pieces to make review easier
-- Branching on a big feature was suggested -
--- Issues: committer needs to be there
Sergey: If the rate of change is high on common code (to the
branches) then merging will be tough
--- Jon: Refactor should be done in the main branch (since it
doesn't add any new funtionality)
--- Release often to reduce #backports overall and issues with that..
- Review process .. how to drive a review to closure. Effort goes
waste if the review process is not completed. The same reviewer should
continue to review the patch ..
- Hard to enforce any process
- Enis: there should be a summary of the patch and all that .. so that
the review process is helped.. Hard to understand the architecture of
the patch unless documented
- Jon: It should be easy to make a one-to-one correspondence between
the description and the patch
- The commit should have only the jira# as opposed to pages of description
- Component owners: is this working. Committers need to be forthcoming
with reviews
-- Maybe review the modules and add some more if needed.
-- Good that we have more contributions coming than we have
reviewers, but unless we keep up, we will plateau
-- Mail on dev@ list if review doesn't happen
- Dev co-ordination:
-- How best can we pull together
-- Priorities:
--- Getting 0.96 out is priority
--- Backports to 0.94 will happen .. until 0.96 is stable
--- 0.92 release ? Any committer who wants to make a release can do
so (maybe with some backports, etc.)
--- Backporting can be tough if there are bugs and the bugfix has
to be applied to all branches
- HBASE-2600 - this requires a change in the client and the server.
They have to be changed in lock-step. Its hard to do this .. Jon
doesn't want to have the fix for 0.96. So 0.98 might be another
singular release. Maybe do a rewrite of the meta after taking a lock
on the meta, do a shim layer to handle the backward compat. between
0.96 and 0.98
- What do people want to get into 0.94
-- The biggest thing - Snapshots, mostly new code, about a 3rd of the
stuff in 0.94 already
-- Compactions improvments - no backport
- How devs can better co-ordinate
-- Snapshots co-ordination working well
-- One page design is useful (makes it readable and all)
-- How about handling the stripe compaction - where an idea leads to
a bunch of others
-- Again write-up should be done
- Should we change the description to match the comments
-- Two ideas suggested:
--- We probably should have the description updated with the
"Date: new description" if the issue at hand is updated
--- Should we have a summary after a bunch of comments - yes
- The face-to-face meetings are useful. We should semd out the minutes
of the discussion to the dev list. We probably should have more
focused huddles. Discuss but don't decide (decide on the jira)!
- Jon: Would people be amenable to merge sooner rather than later on
snapshots? Tested and being beaten up.
- Stack: Yes
- What else goes in in 0.96:
-- RPC refactor
-- ROOT removal
-- Compaction stuff
-- Package name mailing list thread - there is now a jira on that.
We shouldn't break clients. Package name changes is not worth the
trouble.
- A bunch of discussions on the RPC with KeyValue/Cells
- What do we do about usability
-- It'll be nice if we don't need to change configs..
-- Maybe expose more metrics and then allow for online config
changes since automatic config is difficult and needs to be battle
tested and all.
- Benchmarking of the release:
-- We should measure the overhead of PB stuff
Re: Some notes from the HBase developer Pow-Wow
Posted by Stack <st...@duboce.net>.
On Tue, Feb 19, 2013 at 5:27 PM, Devaraj Das <dd...@hortonworks.com> wrote:
> - Stabilizing 0.96 / CI
>
Thanks Deveraj for doing Secretary duty.
St.Ack
Re: Some notes from the HBase developer Pow-Wow
Posted by Andrew Purtell <ap...@apache.org>.
Sorry I couldn't make it up for the powwow.
Thanks Devaraj for the excellent notes.
On this part:
-- Branching on a big feature was suggested -
--- Issues: committer needs to be there
Sergey: If the rate of change is high on common code (to
the branches)
then merging will be tough
--- Jon: Refactor should be done in the main branch (since it doesn't
add any new funtionality)
I like how snapshots was branched and worked on collaboratively up in
GitHub.
It's pretty painless to rebase with git but painful to keep a branch up to
date with subversion then merge back (at least for me).
I have something in the works that has to be out of tree for a while
because it depends on new APIs for Hadoop Common. Was thinking it should go
up into GH in the meantime.
Makes sense for big features to go on a branch and then be subject to the 3
+1 rule for merge. More eyes on the change. Also makes sense for refactors
to be done in place. They may cause pain for work on a branch, but it's
less painful to merge in as it happens on the main branch than sort out
collisions later if a refactor and feature want to go in at about the same
time.
And on this part:
- A bunch of discussions on the RPC with KeyValue/Cells
.... / Tags :-)
On Tue, Feb 19, 2013 at 5:27 PM, Devaraj Das <dd...@hortonworks.com> wrote:
> - Stabilizing 0.96 / CI
> -- not running continuously
> -- Roman: is it a good idea to run IT under Bigtop
> - the HBase unit tests are good..
> - What about HBase as a backend for hive - tests for those
> - Do we think there is value with the integration with the rest
> of the hadoop ecosystem
> -- Jon: What's needed to make this work
> -- Roman: Collective agreement that we want to solve this
> -- Stack: We need to run the IT tests from Enis and Sergey running
> continuously
> -- Roman: If we all agree that this is something needs to be fixed
> .. then yes we would talk about mechanics. Bigtop doesn't have
> expertise in HBase and hence HBase folks would have to debug failures.
> -- Roman: Bigtop is committed to tests. Less than a dozen tests for
> hbase currently..
> -- We have been running validation around RC time. Find all kinds of
> issues - sometimes trivial (maybe a config issue),
> -- Roman can offer CI for trunk but will work for only hadoop-2 line.
> -- Roman does first line of triage.
> -- No issues to do with other ecosystem artifacts. Bigtop ensures
> the right artifacts are in place.
> -- hadoop-2 is important but not particular about the version of
> hadoop within 2. For example 2.0.2
> -- Gary: Will be good to run security tests
> -- Roman & Devaraj to talk on how this can be done/implemented
>
> On 0.96 branching
> - Lars:
> -- We will have three branches to maintain.
> - Stack: we need to stabilize quickly
> - Enis: What about 0.95.
> - Stack: Just do the snapshots thing. Every week, give a snapshot
> - Enis: we've done a bunch of stuff in RPC.. If we have to break
> something, we can if it is beta (0.96-beta).
> - Agreement is there generally ... Debate on the name with snapshot
> versus 95.0/96.0
> -- 0.95 experimental .. 0.96 will be stable
> -- If we go with 0.95, releasing will be easier as well..
> -- 0.95 will not be in production..
> -- 0.96 will be off 0.95 branch and not trunk based
>
> - How do we go about committing issues.. (issue commit rate is low)
> -- 0.94 is stable - 2 new commits and 2 new bugs a day
> -- 0.96 has lots of issues not reviewed
> -- Break up the patch into multiple smaller pieces to make review easier
> -- Branching on a big feature was suggested -
> --- Issues: committer needs to be there
> Sergey: If the rate of change is high on common code (to the
> branches) then merging will be tough
> --- Jon: Refactor should be done in the main branch (since it
> doesn't add any new funtionality)
> --- Release often to reduce #backports overall and issues with that..
>
> - Review process .. how to drive a review to closure. Effort goes
> waste if the review process is not completed. The same reviewer should
> continue to review the patch ..
> - Hard to enforce any process
> - Enis: there should be a summary of the patch and all that .. so that
> the review process is helped.. Hard to understand the architecture of
> the patch unless documented
> - Jon: It should be easy to make a one-to-one correspondence between
> the description and the patch
> - The commit should have only the jira# as opposed to pages of description
>
> - Component owners: is this working. Committers need to be forthcoming
> with reviews
> -- Maybe review the modules and add some more if needed.
>
> -- Good that we have more contributions coming than we have
> reviewers, but unless we keep up, we will plateau
> -- Mail on dev@ list if review doesn't happen
>
> - Dev co-ordination:
> -- How best can we pull together
> -- Priorities:
> --- Getting 0.96 out is priority
> --- Backports to 0.94 will happen .. until 0.96 is stable
> --- 0.92 release ? Any committer who wants to make a release can do
> so (maybe with some backports, etc.)
> --- Backporting can be tough if there are bugs and the bugfix has
> to be applied to all branches
>
> - HBASE-2600 - this requires a change in the client and the server.
> They have to be changed in lock-step. Its hard to do this .. Jon
> doesn't want to have the fix for 0.96. So 0.98 might be another
> singular release. Maybe do a rewrite of the meta after taking a lock
> on the meta, do a shim layer to handle the backward compat. between
> 0.96 and 0.98
>
> - What do people want to get into 0.94
> -- The biggest thing - Snapshots, mostly new code, about a 3rd of the
> stuff in 0.94 already
> -- Compactions improvments - no backport
>
> - How devs can better co-ordinate
> -- Snapshots co-ordination working well
> -- One page design is useful (makes it readable and all)
> -- How about handling the stripe compaction - where an idea leads to
> a bunch of others
> -- Again write-up should be done
>
> - Should we change the description to match the comments
> -- Two ideas suggested:
> --- We probably should have the description updated with the
> "Date: new description" if the issue at hand is updated
> --- Should we have a summary after a bunch of comments - yes
>
> - The face-to-face meetings are useful. We should semd out the minutes
> of the discussion to the dev list. We probably should have more
> focused huddles. Discuss but don't decide (decide on the jira)!
>
> - Jon: Would people be amenable to merge sooner rather than later on
> snapshots? Tested and being beaten up.
> - Stack: Yes
>
> - What else goes in in 0.96:
> -- RPC refactor
> -- ROOT removal
> -- Compaction stuff
> -- Package name mailing list thread - there is now a jira on that.
> We shouldn't break clients. Package name changes is not worth the
> trouble.
>
> - A bunch of discussions on the RPC with KeyValue/Cells
>
> - What do we do about usability
> -- It'll be nice if we don't need to change configs..
> -- Maybe expose more metrics and then allow for online config
> changes since automatic config is difficult and needs to be battle
> tested and all.
>
> - Benchmarking of the release:
> -- We should measure the overhead of PB stuff
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: 答复: Some notes from the HBase developer Pow-Wow
Posted by ramkrishna vasudevan <ra...@gmail.com>.
I checked this out. I am able to hear the voice of the recording.
Regards
Ram
On Thu, Feb 21, 2013 at 7:26 AM, 谢良 <xi...@xiaomi.com> wrote:
> Yes, i am able to hear the recording as well :)
>
> Regards,
> Liang
> ________________________________________
> 发件人: Dave Wang [dsw@cloudera.com]
> 发送时间: 2013年2月21日 0:28
> 收件人: dev@hbase.apache.org
> 主题: Re: Some notes from the HBase developer Pow-Wow
>
> Jean-Marc,
>
> I just tried it and am able to hear the recording. Hmmm, not sure what to
> do here. Any one else able to access the recording?
>
> - Dave
>
>
> On Wed, Feb 20, 2013 at 7:51 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
> > Hi Dave,
> >
> > I' tried to access the link.I 'm able to load it, but there is no
> > sound. Sound is working if I play any other video, but back to webex,
> > still not sound.
> >
> > Is it working for you?
> >
> > JM
> >
> > 2013/2/19, Dave Wang <ds...@cloudera.com>:
> > > Thanks for the summary Deveraj.
> > >
> > > The webex recording for the meeting is at:
> > >
> > >
> >
> https://cloudera.webex.com/cloudera/ldr.php?AT=pb&SP=MC&rID=120418137&rKey=d058765b798c47c5
> > >
> > > Please let me know if you cannot access it.
> > >
> > > Regards,
> > >
> > > - Dave
> > >
> > >
> > > On Tue, Feb 19, 2013 at 5:30 PM, Jean-Marc Spaggiari <
> > > jean-marc@spaggiari.org> wrote:
> > >
> > >> Awesome!
> > >>
> > >> Thanks a for those notes Devaraj!
> > >>
> > >> Very useful for the unfortunates who did not got the chance to join
> the
> > >> meeting ...
> > >> Le 19 févr. 2013 20:27, "Devaraj Das" <dd...@hortonworks.com> a écrit
> :
> > >>
> > >> > - Stabilizing 0.96 / CI
> > >> > -- not running continuously
> > >> > -- Roman: is it a good idea to run IT under Bigtop
> > >> > - the HBase unit tests are good..
> > >> > - What about HBase as a backend for hive - tests for those
> > >> > - Do we think there is value with the integration with the
> rest
> > >> > of the hadoop ecosystem
> > >> > -- Jon: What's needed to make this work
> > >> > -- Roman: Collective agreement that we want to solve this
> > >> > -- Stack: We need to run the IT tests from Enis and Sergey running
> > >> > continuously
> > >> > -- Roman: If we all agree that this is something needs to be fixed
> > >> > .. then yes we would talk about mechanics. Bigtop doesn't have
> > >> > expertise in HBase and hence HBase folks would have to debug
> failures.
> > >> > -- Roman: Bigtop is committed to tests. Less than a dozen tests
> for
> > >> > hbase currently..
> > >> > -- We have been running validation around RC time. Find all kinds
> of
> > >> > issues - sometimes trivial (maybe a config issue),
> > >> > -- Roman can offer CI for trunk but will work for only hadoop-2
> > line.
> > >> > -- Roman does first line of triage.
> > >> > -- No issues to do with other ecosystem artifacts. Bigtop ensures
> > >> > the right artifacts are in place.
> > >> > -- hadoop-2 is important but not particular about the version of
> > >> > hadoop within 2. For example 2.0.2
> > >> > -- Gary: Will be good to run security tests
> > >> > -- Roman & Devaraj to talk on how this can be done/implemented
> > >> >
> > >> > On 0.96 branching
> > >> > - Lars:
> > >> > -- We will have three branches to maintain.
> > >> > - Stack: we need to stabilize quickly
> > >> > - Enis: What about 0.95.
> > >> > - Stack: Just do the snapshots thing. Every week, give a snapshot
> > >> > - Enis: we've done a bunch of stuff in RPC.. If we have to break
> > >> > something, we can if it is beta (0.96-beta).
> > >> > - Agreement is there generally ... Debate on the name with snapshot
> > >> > versus 95.0/96.0
> > >> > -- 0.95 experimental .. 0.96 will be stable
> > >> > -- If we go with 0.95, releasing will be easier as well..
> > >> > -- 0.95 will not be in production..
> > >> > -- 0.96 will be off 0.95 branch and not trunk based
> > >> >
> > >> > - How do we go about committing issues.. (issue commit rate is low)
> > >> > -- 0.94 is stable - 2 new commits and 2 new bugs a day
> > >> > -- 0.96 has lots of issues not reviewed
> > >> > -- Break up the patch into multiple smaller pieces to make review
> > >> > easier
> > >> > -- Branching on a big feature was suggested -
> > >> > --- Issues: committer needs to be there
> > >> > Sergey: If the rate of change is high on common code (to
> the
> > >> > branches) then merging will be tough
> > >> > --- Jon: Refactor should be done in the main branch (since it
> > >> > doesn't add any new funtionality)
> > >> > --- Release often to reduce #backports overall and issues with
> > >> > that..
> > >> >
> > >> > - Review process .. how to drive a review to closure. Effort goes
> > >> > waste if the review process is not completed. The same reviewer
> should
> > >> > continue to review the patch ..
> > >> > - Hard to enforce any process
> > >> > - Enis: there should be a summary of the patch and all that .. so
> that
> > >> > the review process is helped.. Hard to understand the architecture
> of
> > >> > the patch unless documented
> > >> > - Jon: It should be easy to make a one-to-one correspondence between
> > >> > the description and the patch
> > >> > - The commit should have only the jira# as opposed to pages of
> > >> description
> > >> >
> > >> > - Component owners: is this working. Committers need to be
> forthcoming
> > >> > with reviews
> > >> > -- Maybe review the modules and add some more if needed.
> > >> >
> > >> > -- Good that we have more contributions coming than we have
> > >> > reviewers, but unless we keep up, we will plateau
> > >> > -- Mail on dev@ list if review doesn't happen
> > >> >
> > >> > - Dev co-ordination:
> > >> > -- How best can we pull together
> > >> > -- Priorities:
> > >> > --- Getting 0.96 out is priority
> > >> > --- Backports to 0.94 will happen .. until 0.96 is stable
> > >> > --- 0.92 release ? Any committer who wants to make a release can
> do
> > >> > so (maybe with some backports, etc.)
> > >> > --- Backporting can be tough if there are bugs and the bugfix has
> > >> > to be applied to all branches
> > >> >
> > >> > - HBASE-2600 - this requires a change in the client and the server.
> > >> > They have to be changed in lock-step. Its hard to do this .. Jon
> > >> > doesn't want to have the fix for 0.96. So 0.98 might be another
> > >> > singular release. Maybe do a rewrite of the meta after taking a lock
> > >> > on the meta, do a shim layer to handle the backward compat. between
> > >> > 0.96 and 0.98
> > >> >
> > >> > - What do people want to get into 0.94
> > >> > -- The biggest thing - Snapshots, mostly new code, about a 3rd of
> the
> > >> > stuff in 0.94 already
> > >> > -- Compactions improvments - no backport
> > >> >
> > >> > - How devs can better co-ordinate
> > >> > -- Snapshots co-ordination working well
> > >> > -- One page design is useful (makes it readable and all)
> > >> > -- How about handling the stripe compaction - where an idea leads
> to
> > >> > a bunch of others
> > >> > -- Again write-up should be done
> > >> >
> > >> > - Should we change the description to match the comments
> > >> > -- Two ideas suggested:
> > >> > --- We probably should have the description updated with the
> > >> > "Date: new description" if the issue at hand is updated
> > >> > --- Should we have a summary after a bunch of comments - yes
> > >> >
> > >> > - The face-to-face meetings are useful. We should semd out the
> minutes
> > >> > of the discussion to the dev list. We probably should have more
> > >> > focused huddles. Discuss but don't decide (decide on the jira)!
> > >> >
> > >> > - Jon: Would people be amenable to merge sooner rather than later on
> > >> > snapshots? Tested and being beaten up.
> > >> > - Stack: Yes
> > >> >
> > >> > - What else goes in in 0.96:
> > >> > -- RPC refactor
> > >> > -- ROOT removal
> > >> > -- Compaction stuff
> > >> > -- Package name mailing list thread - there is now a jira on that.
> > >> > We shouldn't break clients. Package name changes is not worth the
> > >> > trouble.
> > >> >
> > >> > - A bunch of discussions on the RPC with KeyValue/Cells
> > >> >
> > >> > - What do we do about usability
> > >> > -- It'll be nice if we don't need to change configs..
> > >> > -- Maybe expose more metrics and then allow for online config
> > >> > changes since automatic config is difficult and needs to be battle
> > >> > tested and all.
> > >> >
> > >> > - Benchmarking of the release:
> > >> > -- We should measure the overhead of PB stuff
> > >> >
> > >>
> > >
> >
>
答复: Some notes from the HBase developer Pow-Wow
Posted by 谢良 <xi...@xiaomi.com>.
Yes, i am able to hear the recording as well :)
Regards,
Liang
________________________________________
发件人: Dave Wang [dsw@cloudera.com]
发送时间: 2013年2月21日 0:28
收件人: dev@hbase.apache.org
主题: Re: Some notes from the HBase developer Pow-Wow
Jean-Marc,
I just tried it and am able to hear the recording. Hmmm, not sure what to
do here. Any one else able to access the recording?
- Dave
On Wed, Feb 20, 2013 at 7:51 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi Dave,
>
> I' tried to access the link.I 'm able to load it, but there is no
> sound. Sound is working if I play any other video, but back to webex,
> still not sound.
>
> Is it working for you?
>
> JM
>
> 2013/2/19, Dave Wang <ds...@cloudera.com>:
> > Thanks for the summary Deveraj.
> >
> > The webex recording for the meeting is at:
> >
> >
> https://cloudera.webex.com/cloudera/ldr.php?AT=pb&SP=MC&rID=120418137&rKey=d058765b798c47c5
> >
> > Please let me know if you cannot access it.
> >
> > Regards,
> >
> > - Dave
> >
> >
> > On Tue, Feb 19, 2013 at 5:30 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> Awesome!
> >>
> >> Thanks a for those notes Devaraj!
> >>
> >> Very useful for the unfortunates who did not got the chance to join the
> >> meeting ...
> >> Le 19 févr. 2013 20:27, "Devaraj Das" <dd...@hortonworks.com> a écrit :
> >>
> >> > - Stabilizing 0.96 / CI
> >> > -- not running continuously
> >> > -- Roman: is it a good idea to run IT under Bigtop
> >> > - the HBase unit tests are good..
> >> > - What about HBase as a backend for hive - tests for those
> >> > - Do we think there is value with the integration with the rest
> >> > of the hadoop ecosystem
> >> > -- Jon: What's needed to make this work
> >> > -- Roman: Collective agreement that we want to solve this
> >> > -- Stack: We need to run the IT tests from Enis and Sergey running
> >> > continuously
> >> > -- Roman: If we all agree that this is something needs to be fixed
> >> > .. then yes we would talk about mechanics. Bigtop doesn't have
> >> > expertise in HBase and hence HBase folks would have to debug failures.
> >> > -- Roman: Bigtop is committed to tests. Less than a dozen tests for
> >> > hbase currently..
> >> > -- We have been running validation around RC time. Find all kinds of
> >> > issues - sometimes trivial (maybe a config issue),
> >> > -- Roman can offer CI for trunk but will work for only hadoop-2
> line.
> >> > -- Roman does first line of triage.
> >> > -- No issues to do with other ecosystem artifacts. Bigtop ensures
> >> > the right artifacts are in place.
> >> > -- hadoop-2 is important but not particular about the version of
> >> > hadoop within 2. For example 2.0.2
> >> > -- Gary: Will be good to run security tests
> >> > -- Roman & Devaraj to talk on how this can be done/implemented
> >> >
> >> > On 0.96 branching
> >> > - Lars:
> >> > -- We will have three branches to maintain.
> >> > - Stack: we need to stabilize quickly
> >> > - Enis: What about 0.95.
> >> > - Stack: Just do the snapshots thing. Every week, give a snapshot
> >> > - Enis: we've done a bunch of stuff in RPC.. If we have to break
> >> > something, we can if it is beta (0.96-beta).
> >> > - Agreement is there generally ... Debate on the name with snapshot
> >> > versus 95.0/96.0
> >> > -- 0.95 experimental .. 0.96 will be stable
> >> > -- If we go with 0.95, releasing will be easier as well..
> >> > -- 0.95 will not be in production..
> >> > -- 0.96 will be off 0.95 branch and not trunk based
> >> >
> >> > - How do we go about committing issues.. (issue commit rate is low)
> >> > -- 0.94 is stable - 2 new commits and 2 new bugs a day
> >> > -- 0.96 has lots of issues not reviewed
> >> > -- Break up the patch into multiple smaller pieces to make review
> >> > easier
> >> > -- Branching on a big feature was suggested -
> >> > --- Issues: committer needs to be there
> >> > Sergey: If the rate of change is high on common code (to the
> >> > branches) then merging will be tough
> >> > --- Jon: Refactor should be done in the main branch (since it
> >> > doesn't add any new funtionality)
> >> > --- Release often to reduce #backports overall and issues with
> >> > that..
> >> >
> >> > - Review process .. how to drive a review to closure. Effort goes
> >> > waste if the review process is not completed. The same reviewer should
> >> > continue to review the patch ..
> >> > - Hard to enforce any process
> >> > - Enis: there should be a summary of the patch and all that .. so that
> >> > the review process is helped.. Hard to understand the architecture of
> >> > the patch unless documented
> >> > - Jon: It should be easy to make a one-to-one correspondence between
> >> > the description and the patch
> >> > - The commit should have only the jira# as opposed to pages of
> >> description
> >> >
> >> > - Component owners: is this working. Committers need to be forthcoming
> >> > with reviews
> >> > -- Maybe review the modules and add some more if needed.
> >> >
> >> > -- Good that we have more contributions coming than we have
> >> > reviewers, but unless we keep up, we will plateau
> >> > -- Mail on dev@ list if review doesn't happen
> >> >
> >> > - Dev co-ordination:
> >> > -- How best can we pull together
> >> > -- Priorities:
> >> > --- Getting 0.96 out is priority
> >> > --- Backports to 0.94 will happen .. until 0.96 is stable
> >> > --- 0.92 release ? Any committer who wants to make a release can do
> >> > so (maybe with some backports, etc.)
> >> > --- Backporting can be tough if there are bugs and the bugfix has
> >> > to be applied to all branches
> >> >
> >> > - HBASE-2600 - this requires a change in the client and the server.
> >> > They have to be changed in lock-step. Its hard to do this .. Jon
> >> > doesn't want to have the fix for 0.96. So 0.98 might be another
> >> > singular release. Maybe do a rewrite of the meta after taking a lock
> >> > on the meta, do a shim layer to handle the backward compat. between
> >> > 0.96 and 0.98
> >> >
> >> > - What do people want to get into 0.94
> >> > -- The biggest thing - Snapshots, mostly new code, about a 3rd of the
> >> > stuff in 0.94 already
> >> > -- Compactions improvments - no backport
> >> >
> >> > - How devs can better co-ordinate
> >> > -- Snapshots co-ordination working well
> >> > -- One page design is useful (makes it readable and all)
> >> > -- How about handling the stripe compaction - where an idea leads to
> >> > a bunch of others
> >> > -- Again write-up should be done
> >> >
> >> > - Should we change the description to match the comments
> >> > -- Two ideas suggested:
> >> > --- We probably should have the description updated with the
> >> > "Date: new description" if the issue at hand is updated
> >> > --- Should we have a summary after a bunch of comments - yes
> >> >
> >> > - The face-to-face meetings are useful. We should semd out the minutes
> >> > of the discussion to the dev list. We probably should have more
> >> > focused huddles. Discuss but don't decide (decide on the jira)!
> >> >
> >> > - Jon: Would people be amenable to merge sooner rather than later on
> >> > snapshots? Tested and being beaten up.
> >> > - Stack: Yes
> >> >
> >> > - What else goes in in 0.96:
> >> > -- RPC refactor
> >> > -- ROOT removal
> >> > -- Compaction stuff
> >> > -- Package name mailing list thread - there is now a jira on that.
> >> > We shouldn't break clients. Package name changes is not worth the
> >> > trouble.
> >> >
> >> > - A bunch of discussions on the RPC with KeyValue/Cells
> >> >
> >> > - What do we do about usability
> >> > -- It'll be nice if we don't need to change configs..
> >> > -- Maybe expose more metrics and then allow for online config
> >> > changes since automatic config is difficult and needs to be battle
> >> > tested and all.
> >> >
> >> > - Benchmarking of the release:
> >> > -- We should measure the overhead of PB stuff
> >> >
> >>
> >
>
Re: Some notes from the HBase developer Pow-Wow
Posted by Dave Wang <ds...@cloudera.com>.
Jean-Marc,
I just tried it and am able to hear the recording. Hmmm, not sure what to
do here. Any one else able to access the recording?
- Dave
On Wed, Feb 20, 2013 at 7:51 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi Dave,
>
> I' tried to access the link.I 'm able to load it, but there is no
> sound. Sound is working if I play any other video, but back to webex,
> still not sound.
>
> Is it working for you?
>
> JM
>
> 2013/2/19, Dave Wang <ds...@cloudera.com>:
> > Thanks for the summary Deveraj.
> >
> > The webex recording for the meeting is at:
> >
> >
> https://cloudera.webex.com/cloudera/ldr.php?AT=pb&SP=MC&rID=120418137&rKey=d058765b798c47c5
> >
> > Please let me know if you cannot access it.
> >
> > Regards,
> >
> > - Dave
> >
> >
> > On Tue, Feb 19, 2013 at 5:30 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> Awesome!
> >>
> >> Thanks a for those notes Devaraj!
> >>
> >> Very useful for the unfortunates who did not got the chance to join the
> >> meeting ...
> >> Le 19 févr. 2013 20:27, "Devaraj Das" <dd...@hortonworks.com> a écrit :
> >>
> >> > - Stabilizing 0.96 / CI
> >> > -- not running continuously
> >> > -- Roman: is it a good idea to run IT under Bigtop
> >> > - the HBase unit tests are good..
> >> > - What about HBase as a backend for hive - tests for those
> >> > - Do we think there is value with the integration with the rest
> >> > of the hadoop ecosystem
> >> > -- Jon: What's needed to make this work
> >> > -- Roman: Collective agreement that we want to solve this
> >> > -- Stack: We need to run the IT tests from Enis and Sergey running
> >> > continuously
> >> > -- Roman: If we all agree that this is something needs to be fixed
> >> > .. then yes we would talk about mechanics. Bigtop doesn't have
> >> > expertise in HBase and hence HBase folks would have to debug failures.
> >> > -- Roman: Bigtop is committed to tests. Less than a dozen tests for
> >> > hbase currently..
> >> > -- We have been running validation around RC time. Find all kinds of
> >> > issues - sometimes trivial (maybe a config issue),
> >> > -- Roman can offer CI for trunk but will work for only hadoop-2
> line.
> >> > -- Roman does first line of triage.
> >> > -- No issues to do with other ecosystem artifacts. Bigtop ensures
> >> > the right artifacts are in place.
> >> > -- hadoop-2 is important but not particular about the version of
> >> > hadoop within 2. For example 2.0.2
> >> > -- Gary: Will be good to run security tests
> >> > -- Roman & Devaraj to talk on how this can be done/implemented
> >> >
> >> > On 0.96 branching
> >> > - Lars:
> >> > -- We will have three branches to maintain.
> >> > - Stack: we need to stabilize quickly
> >> > - Enis: What about 0.95.
> >> > - Stack: Just do the snapshots thing. Every week, give a snapshot
> >> > - Enis: we've done a bunch of stuff in RPC.. If we have to break
> >> > something, we can if it is beta (0.96-beta).
> >> > - Agreement is there generally ... Debate on the name with snapshot
> >> > versus 95.0/96.0
> >> > -- 0.95 experimental .. 0.96 will be stable
> >> > -- If we go with 0.95, releasing will be easier as well..
> >> > -- 0.95 will not be in production..
> >> > -- 0.96 will be off 0.95 branch and not trunk based
> >> >
> >> > - How do we go about committing issues.. (issue commit rate is low)
> >> > -- 0.94 is stable - 2 new commits and 2 new bugs a day
> >> > -- 0.96 has lots of issues not reviewed
> >> > -- Break up the patch into multiple smaller pieces to make review
> >> > easier
> >> > -- Branching on a big feature was suggested -
> >> > --- Issues: committer needs to be there
> >> > Sergey: If the rate of change is high on common code (to the
> >> > branches) then merging will be tough
> >> > --- Jon: Refactor should be done in the main branch (since it
> >> > doesn't add any new funtionality)
> >> > --- Release often to reduce #backports overall and issues with
> >> > that..
> >> >
> >> > - Review process .. how to drive a review to closure. Effort goes
> >> > waste if the review process is not completed. The same reviewer should
> >> > continue to review the patch ..
> >> > - Hard to enforce any process
> >> > - Enis: there should be a summary of the patch and all that .. so that
> >> > the review process is helped.. Hard to understand the architecture of
> >> > the patch unless documented
> >> > - Jon: It should be easy to make a one-to-one correspondence between
> >> > the description and the patch
> >> > - The commit should have only the jira# as opposed to pages of
> >> description
> >> >
> >> > - Component owners: is this working. Committers need to be forthcoming
> >> > with reviews
> >> > -- Maybe review the modules and add some more if needed.
> >> >
> >> > -- Good that we have more contributions coming than we have
> >> > reviewers, but unless we keep up, we will plateau
> >> > -- Mail on dev@ list if review doesn't happen
> >> >
> >> > - Dev co-ordination:
> >> > -- How best can we pull together
> >> > -- Priorities:
> >> > --- Getting 0.96 out is priority
> >> > --- Backports to 0.94 will happen .. until 0.96 is stable
> >> > --- 0.92 release ? Any committer who wants to make a release can do
> >> > so (maybe with some backports, etc.)
> >> > --- Backporting can be tough if there are bugs and the bugfix has
> >> > to be applied to all branches
> >> >
> >> > - HBASE-2600 - this requires a change in the client and the server.
> >> > They have to be changed in lock-step. Its hard to do this .. Jon
> >> > doesn't want to have the fix for 0.96. So 0.98 might be another
> >> > singular release. Maybe do a rewrite of the meta after taking a lock
> >> > on the meta, do a shim layer to handle the backward compat. between
> >> > 0.96 and 0.98
> >> >
> >> > - What do people want to get into 0.94
> >> > -- The biggest thing - Snapshots, mostly new code, about a 3rd of the
> >> > stuff in 0.94 already
> >> > -- Compactions improvments - no backport
> >> >
> >> > - How devs can better co-ordinate
> >> > -- Snapshots co-ordination working well
> >> > -- One page design is useful (makes it readable and all)
> >> > -- How about handling the stripe compaction - where an idea leads to
> >> > a bunch of others
> >> > -- Again write-up should be done
> >> >
> >> > - Should we change the description to match the comments
> >> > -- Two ideas suggested:
> >> > --- We probably should have the description updated with the
> >> > "Date: new description" if the issue at hand is updated
> >> > --- Should we have a summary after a bunch of comments - yes
> >> >
> >> > - The face-to-face meetings are useful. We should semd out the minutes
> >> > of the discussion to the dev list. We probably should have more
> >> > focused huddles. Discuss but don't decide (decide on the jira)!
> >> >
> >> > - Jon: Would people be amenable to merge sooner rather than later on
> >> > snapshots? Tested and being beaten up.
> >> > - Stack: Yes
> >> >
> >> > - What else goes in in 0.96:
> >> > -- RPC refactor
> >> > -- ROOT removal
> >> > -- Compaction stuff
> >> > -- Package name mailing list thread - there is now a jira on that.
> >> > We shouldn't break clients. Package name changes is not worth the
> >> > trouble.
> >> >
> >> > - A bunch of discussions on the RPC with KeyValue/Cells
> >> >
> >> > - What do we do about usability
> >> > -- It'll be nice if we don't need to change configs..
> >> > -- Maybe expose more metrics and then allow for online config
> >> > changes since automatic config is difficult and needs to be battle
> >> > tested and all.
> >> >
> >> > - Benchmarking of the release:
> >> > -- We should measure the overhead of PB stuff
> >> >
> >>
> >
>
Re: Some notes from the HBase developer Pow-Wow
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Dave,
I' tried to access the link.I 'm able to load it, but there is no
sound. Sound is working if I play any other video, but back to webex,
still not sound.
Is it working for you?
JM
2013/2/19, Dave Wang <ds...@cloudera.com>:
> Thanks for the summary Deveraj.
>
> The webex recording for the meeting is at:
>
> https://cloudera.webex.com/cloudera/ldr.php?AT=pb&SP=MC&rID=120418137&rKey=d058765b798c47c5
>
> Please let me know if you cannot access it.
>
> Regards,
>
> - Dave
>
>
> On Tue, Feb 19, 2013 at 5:30 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> Awesome!
>>
>> Thanks a for those notes Devaraj!
>>
>> Very useful for the unfortunates who did not got the chance to join the
>> meeting ...
>> Le 19 févr. 2013 20:27, "Devaraj Das" <dd...@hortonworks.com> a écrit :
>>
>> > - Stabilizing 0.96 / CI
>> > -- not running continuously
>> > -- Roman: is it a good idea to run IT under Bigtop
>> > - the HBase unit tests are good..
>> > - What about HBase as a backend for hive - tests for those
>> > - Do we think there is value with the integration with the rest
>> > of the hadoop ecosystem
>> > -- Jon: What's needed to make this work
>> > -- Roman: Collective agreement that we want to solve this
>> > -- Stack: We need to run the IT tests from Enis and Sergey running
>> > continuously
>> > -- Roman: If we all agree that this is something needs to be fixed
>> > .. then yes we would talk about mechanics. Bigtop doesn't have
>> > expertise in HBase and hence HBase folks would have to debug failures.
>> > -- Roman: Bigtop is committed to tests. Less than a dozen tests for
>> > hbase currently..
>> > -- We have been running validation around RC time. Find all kinds of
>> > issues - sometimes trivial (maybe a config issue),
>> > -- Roman can offer CI for trunk but will work for only hadoop-2 line.
>> > -- Roman does first line of triage.
>> > -- No issues to do with other ecosystem artifacts. Bigtop ensures
>> > the right artifacts are in place.
>> > -- hadoop-2 is important but not particular about the version of
>> > hadoop within 2. For example 2.0.2
>> > -- Gary: Will be good to run security tests
>> > -- Roman & Devaraj to talk on how this can be done/implemented
>> >
>> > On 0.96 branching
>> > - Lars:
>> > -- We will have three branches to maintain.
>> > - Stack: we need to stabilize quickly
>> > - Enis: What about 0.95.
>> > - Stack: Just do the snapshots thing. Every week, give a snapshot
>> > - Enis: we've done a bunch of stuff in RPC.. If we have to break
>> > something, we can if it is beta (0.96-beta).
>> > - Agreement is there generally ... Debate on the name with snapshot
>> > versus 95.0/96.0
>> > -- 0.95 experimental .. 0.96 will be stable
>> > -- If we go with 0.95, releasing will be easier as well..
>> > -- 0.95 will not be in production..
>> > -- 0.96 will be off 0.95 branch and not trunk based
>> >
>> > - How do we go about committing issues.. (issue commit rate is low)
>> > -- 0.94 is stable - 2 new commits and 2 new bugs a day
>> > -- 0.96 has lots of issues not reviewed
>> > -- Break up the patch into multiple smaller pieces to make review
>> > easier
>> > -- Branching on a big feature was suggested -
>> > --- Issues: committer needs to be there
>> > Sergey: If the rate of change is high on common code (to the
>> > branches) then merging will be tough
>> > --- Jon: Refactor should be done in the main branch (since it
>> > doesn't add any new funtionality)
>> > --- Release often to reduce #backports overall and issues with
>> > that..
>> >
>> > - Review process .. how to drive a review to closure. Effort goes
>> > waste if the review process is not completed. The same reviewer should
>> > continue to review the patch ..
>> > - Hard to enforce any process
>> > - Enis: there should be a summary of the patch and all that .. so that
>> > the review process is helped.. Hard to understand the architecture of
>> > the patch unless documented
>> > - Jon: It should be easy to make a one-to-one correspondence between
>> > the description and the patch
>> > - The commit should have only the jira# as opposed to pages of
>> description
>> >
>> > - Component owners: is this working. Committers need to be forthcoming
>> > with reviews
>> > -- Maybe review the modules and add some more if needed.
>> >
>> > -- Good that we have more contributions coming than we have
>> > reviewers, but unless we keep up, we will plateau
>> > -- Mail on dev@ list if review doesn't happen
>> >
>> > - Dev co-ordination:
>> > -- How best can we pull together
>> > -- Priorities:
>> > --- Getting 0.96 out is priority
>> > --- Backports to 0.94 will happen .. until 0.96 is stable
>> > --- 0.92 release ? Any committer who wants to make a release can do
>> > so (maybe with some backports, etc.)
>> > --- Backporting can be tough if there are bugs and the bugfix has
>> > to be applied to all branches
>> >
>> > - HBASE-2600 - this requires a change in the client and the server.
>> > They have to be changed in lock-step. Its hard to do this .. Jon
>> > doesn't want to have the fix for 0.96. So 0.98 might be another
>> > singular release. Maybe do a rewrite of the meta after taking a lock
>> > on the meta, do a shim layer to handle the backward compat. between
>> > 0.96 and 0.98
>> >
>> > - What do people want to get into 0.94
>> > -- The biggest thing - Snapshots, mostly new code, about a 3rd of the
>> > stuff in 0.94 already
>> > -- Compactions improvments - no backport
>> >
>> > - How devs can better co-ordinate
>> > -- Snapshots co-ordination working well
>> > -- One page design is useful (makes it readable and all)
>> > -- How about handling the stripe compaction - where an idea leads to
>> > a bunch of others
>> > -- Again write-up should be done
>> >
>> > - Should we change the description to match the comments
>> > -- Two ideas suggested:
>> > --- We probably should have the description updated with the
>> > "Date: new description" if the issue at hand is updated
>> > --- Should we have a summary after a bunch of comments - yes
>> >
>> > - The face-to-face meetings are useful. We should semd out the minutes
>> > of the discussion to the dev list. We probably should have more
>> > focused huddles. Discuss but don't decide (decide on the jira)!
>> >
>> > - Jon: Would people be amenable to merge sooner rather than later on
>> > snapshots? Tested and being beaten up.
>> > - Stack: Yes
>> >
>> > - What else goes in in 0.96:
>> > -- RPC refactor
>> > -- ROOT removal
>> > -- Compaction stuff
>> > -- Package name mailing list thread - there is now a jira on that.
>> > We shouldn't break clients. Package name changes is not worth the
>> > trouble.
>> >
>> > - A bunch of discussions on the RPC with KeyValue/Cells
>> >
>> > - What do we do about usability
>> > -- It'll be nice if we don't need to change configs..
>> > -- Maybe expose more metrics and then allow for online config
>> > changes since automatic config is difficult and needs to be battle
>> > tested and all.
>> >
>> > - Benchmarking of the release:
>> > -- We should measure the overhead of PB stuff
>> >
>>
>
Re: Some notes from the HBase developer Pow-Wow
Posted by Dave Wang <ds...@cloudera.com>.
Thanks for the summary Deveraj.
The webex recording for the meeting is at:
https://cloudera.webex.com/cloudera/ldr.php?AT=pb&SP=MC&rID=120418137&rKey=d058765b798c47c5
Please let me know if you cannot access it.
Regards,
- Dave
On Tue, Feb 19, 2013 at 5:30 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Awesome!
>
> Thanks a for those notes Devaraj!
>
> Very useful for the unfortunates who did not got the chance to join the
> meeting ...
> Le 19 févr. 2013 20:27, "Devaraj Das" <dd...@hortonworks.com> a écrit :
>
> > - Stabilizing 0.96 / CI
> > -- not running continuously
> > -- Roman: is it a good idea to run IT under Bigtop
> > - the HBase unit tests are good..
> > - What about HBase as a backend for hive - tests for those
> > - Do we think there is value with the integration with the rest
> > of the hadoop ecosystem
> > -- Jon: What's needed to make this work
> > -- Roman: Collective agreement that we want to solve this
> > -- Stack: We need to run the IT tests from Enis and Sergey running
> > continuously
> > -- Roman: If we all agree that this is something needs to be fixed
> > .. then yes we would talk about mechanics. Bigtop doesn't have
> > expertise in HBase and hence HBase folks would have to debug failures.
> > -- Roman: Bigtop is committed to tests. Less than a dozen tests for
> > hbase currently..
> > -- We have been running validation around RC time. Find all kinds of
> > issues - sometimes trivial (maybe a config issue),
> > -- Roman can offer CI for trunk but will work for only hadoop-2 line.
> > -- Roman does first line of triage.
> > -- No issues to do with other ecosystem artifacts. Bigtop ensures
> > the right artifacts are in place.
> > -- hadoop-2 is important but not particular about the version of
> > hadoop within 2. For example 2.0.2
> > -- Gary: Will be good to run security tests
> > -- Roman & Devaraj to talk on how this can be done/implemented
> >
> > On 0.96 branching
> > - Lars:
> > -- We will have three branches to maintain.
> > - Stack: we need to stabilize quickly
> > - Enis: What about 0.95.
> > - Stack: Just do the snapshots thing. Every week, give a snapshot
> > - Enis: we've done a bunch of stuff in RPC.. If we have to break
> > something, we can if it is beta (0.96-beta).
> > - Agreement is there generally ... Debate on the name with snapshot
> > versus 95.0/96.0
> > -- 0.95 experimental .. 0.96 will be stable
> > -- If we go with 0.95, releasing will be easier as well..
> > -- 0.95 will not be in production..
> > -- 0.96 will be off 0.95 branch and not trunk based
> >
> > - How do we go about committing issues.. (issue commit rate is low)
> > -- 0.94 is stable - 2 new commits and 2 new bugs a day
> > -- 0.96 has lots of issues not reviewed
> > -- Break up the patch into multiple smaller pieces to make review easier
> > -- Branching on a big feature was suggested -
> > --- Issues: committer needs to be there
> > Sergey: If the rate of change is high on common code (to the
> > branches) then merging will be tough
> > --- Jon: Refactor should be done in the main branch (since it
> > doesn't add any new funtionality)
> > --- Release often to reduce #backports overall and issues with that..
> >
> > - Review process .. how to drive a review to closure. Effort goes
> > waste if the review process is not completed. The same reviewer should
> > continue to review the patch ..
> > - Hard to enforce any process
> > - Enis: there should be a summary of the patch and all that .. so that
> > the review process is helped.. Hard to understand the architecture of
> > the patch unless documented
> > - Jon: It should be easy to make a one-to-one correspondence between
> > the description and the patch
> > - The commit should have only the jira# as opposed to pages of
> description
> >
> > - Component owners: is this working. Committers need to be forthcoming
> > with reviews
> > -- Maybe review the modules and add some more if needed.
> >
> > -- Good that we have more contributions coming than we have
> > reviewers, but unless we keep up, we will plateau
> > -- Mail on dev@ list if review doesn't happen
> >
> > - Dev co-ordination:
> > -- How best can we pull together
> > -- Priorities:
> > --- Getting 0.96 out is priority
> > --- Backports to 0.94 will happen .. until 0.96 is stable
> > --- 0.92 release ? Any committer who wants to make a release can do
> > so (maybe with some backports, etc.)
> > --- Backporting can be tough if there are bugs and the bugfix has
> > to be applied to all branches
> >
> > - HBASE-2600 - this requires a change in the client and the server.
> > They have to be changed in lock-step. Its hard to do this .. Jon
> > doesn't want to have the fix for 0.96. So 0.98 might be another
> > singular release. Maybe do a rewrite of the meta after taking a lock
> > on the meta, do a shim layer to handle the backward compat. between
> > 0.96 and 0.98
> >
> > - What do people want to get into 0.94
> > -- The biggest thing - Snapshots, mostly new code, about a 3rd of the
> > stuff in 0.94 already
> > -- Compactions improvments - no backport
> >
> > - How devs can better co-ordinate
> > -- Snapshots co-ordination working well
> > -- One page design is useful (makes it readable and all)
> > -- How about handling the stripe compaction - where an idea leads to
> > a bunch of others
> > -- Again write-up should be done
> >
> > - Should we change the description to match the comments
> > -- Two ideas suggested:
> > --- We probably should have the description updated with the
> > "Date: new description" if the issue at hand is updated
> > --- Should we have a summary after a bunch of comments - yes
> >
> > - The face-to-face meetings are useful. We should semd out the minutes
> > of the discussion to the dev list. We probably should have more
> > focused huddles. Discuss but don't decide (decide on the jira)!
> >
> > - Jon: Would people be amenable to merge sooner rather than later on
> > snapshots? Tested and being beaten up.
> > - Stack: Yes
> >
> > - What else goes in in 0.96:
> > -- RPC refactor
> > -- ROOT removal
> > -- Compaction stuff
> > -- Package name mailing list thread - there is now a jira on that.
> > We shouldn't break clients. Package name changes is not worth the
> > trouble.
> >
> > - A bunch of discussions on the RPC with KeyValue/Cells
> >
> > - What do we do about usability
> > -- It'll be nice if we don't need to change configs..
> > -- Maybe expose more metrics and then allow for online config
> > changes since automatic config is difficult and needs to be battle
> > tested and all.
> >
> > - Benchmarking of the release:
> > -- We should measure the overhead of PB stuff
> >
>
Re: Some notes from the HBase developer Pow-Wow
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Awesome!
Thanks a for those notes Devaraj!
Very useful for the unfortunates who did not got the chance to join the
meeting ...
Le 19 févr. 2013 20:27, "Devaraj Das" <dd...@hortonworks.com> a écrit :
> - Stabilizing 0.96 / CI
> -- not running continuously
> -- Roman: is it a good idea to run IT under Bigtop
> - the HBase unit tests are good..
> - What about HBase as a backend for hive - tests for those
> - Do we think there is value with the integration with the rest
> of the hadoop ecosystem
> -- Jon: What's needed to make this work
> -- Roman: Collective agreement that we want to solve this
> -- Stack: We need to run the IT tests from Enis and Sergey running
> continuously
> -- Roman: If we all agree that this is something needs to be fixed
> .. then yes we would talk about mechanics. Bigtop doesn't have
> expertise in HBase and hence HBase folks would have to debug failures.
> -- Roman: Bigtop is committed to tests. Less than a dozen tests for
> hbase currently..
> -- We have been running validation around RC time. Find all kinds of
> issues - sometimes trivial (maybe a config issue),
> -- Roman can offer CI for trunk but will work for only hadoop-2 line.
> -- Roman does first line of triage.
> -- No issues to do with other ecosystem artifacts. Bigtop ensures
> the right artifacts are in place.
> -- hadoop-2 is important but not particular about the version of
> hadoop within 2. For example 2.0.2
> -- Gary: Will be good to run security tests
> -- Roman & Devaraj to talk on how this can be done/implemented
>
> On 0.96 branching
> - Lars:
> -- We will have three branches to maintain.
> - Stack: we need to stabilize quickly
> - Enis: What about 0.95.
> - Stack: Just do the snapshots thing. Every week, give a snapshot
> - Enis: we've done a bunch of stuff in RPC.. If we have to break
> something, we can if it is beta (0.96-beta).
> - Agreement is there generally ... Debate on the name with snapshot
> versus 95.0/96.0
> -- 0.95 experimental .. 0.96 will be stable
> -- If we go with 0.95, releasing will be easier as well..
> -- 0.95 will not be in production..
> -- 0.96 will be off 0.95 branch and not trunk based
>
> - How do we go about committing issues.. (issue commit rate is low)
> -- 0.94 is stable - 2 new commits and 2 new bugs a day
> -- 0.96 has lots of issues not reviewed
> -- Break up the patch into multiple smaller pieces to make review easier
> -- Branching on a big feature was suggested -
> --- Issues: committer needs to be there
> Sergey: If the rate of change is high on common code (to the
> branches) then merging will be tough
> --- Jon: Refactor should be done in the main branch (since it
> doesn't add any new funtionality)
> --- Release often to reduce #backports overall and issues with that..
>
> - Review process .. how to drive a review to closure. Effort goes
> waste if the review process is not completed. The same reviewer should
> continue to review the patch ..
> - Hard to enforce any process
> - Enis: there should be a summary of the patch and all that .. so that
> the review process is helped.. Hard to understand the architecture of
> the patch unless documented
> - Jon: It should be easy to make a one-to-one correspondence between
> the description and the patch
> - The commit should have only the jira# as opposed to pages of description
>
> - Component owners: is this working. Committers need to be forthcoming
> with reviews
> -- Maybe review the modules and add some more if needed.
>
> -- Good that we have more contributions coming than we have
> reviewers, but unless we keep up, we will plateau
> -- Mail on dev@ list if review doesn't happen
>
> - Dev co-ordination:
> -- How best can we pull together
> -- Priorities:
> --- Getting 0.96 out is priority
> --- Backports to 0.94 will happen .. until 0.96 is stable
> --- 0.92 release ? Any committer who wants to make a release can do
> so (maybe with some backports, etc.)
> --- Backporting can be tough if there are bugs and the bugfix has
> to be applied to all branches
>
> - HBASE-2600 - this requires a change in the client and the server.
> They have to be changed in lock-step. Its hard to do this .. Jon
> doesn't want to have the fix for 0.96. So 0.98 might be another
> singular release. Maybe do a rewrite of the meta after taking a lock
> on the meta, do a shim layer to handle the backward compat. between
> 0.96 and 0.98
>
> - What do people want to get into 0.94
> -- The biggest thing - Snapshots, mostly new code, about a 3rd of the
> stuff in 0.94 already
> -- Compactions improvments - no backport
>
> - How devs can better co-ordinate
> -- Snapshots co-ordination working well
> -- One page design is useful (makes it readable and all)
> -- How about handling the stripe compaction - where an idea leads to
> a bunch of others
> -- Again write-up should be done
>
> - Should we change the description to match the comments
> -- Two ideas suggested:
> --- We probably should have the description updated with the
> "Date: new description" if the issue at hand is updated
> --- Should we have a summary after a bunch of comments - yes
>
> - The face-to-face meetings are useful. We should semd out the minutes
> of the discussion to the dev list. We probably should have more
> focused huddles. Discuss but don't decide (decide on the jira)!
>
> - Jon: Would people be amenable to merge sooner rather than later on
> snapshots? Tested and being beaten up.
> - Stack: Yes
>
> - What else goes in in 0.96:
> -- RPC refactor
> -- ROOT removal
> -- Compaction stuff
> -- Package name mailing list thread - there is now a jira on that.
> We shouldn't break clients. Package name changes is not worth the
> trouble.
>
> - A bunch of discussions on the RPC with KeyValue/Cells
>
> - What do we do about usability
> -- It'll be nice if we don't need to change configs..
> -- Maybe expose more metrics and then allow for online config
> changes since automatic config is difficult and needs to be battle
> tested and all.
>
> - Benchmarking of the release:
> -- We should measure the overhead of PB stuff
>