You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Andrew Purtell <ap...@apache.org> on 2010/04/17 19:28:49 UTC
Hackathon agenda
The Hackathon is basically agenda-less, but I'd like to propose a general topic of discussion we should cover while we are all in the room together:
- For HBASE-1964 (HBASE-2183, HBASE-2461, and related): injecting and/or mocking exceptions thrown up from DFSClient. I think we want a toolkit for that. Could be incorporated into the unit testing framework. Should be possible to swap out a jar or something and make it active running on a real cluster with real load. Should be possible to inject random exceptions with adjustable probability. So what does HDFS have already? What do we need? If we're adding something, does it make sense to put it into HBase or contribute to HDFS? I think the latter.
Let's gather a list of other topics, if any, that hackathon participants want to see covered so we can make sure it will happen.
- Andy
Re: Hackathon agenda
Posted by Stack <st...@duboce.net>.
On Sat, Apr 17, 2010 at 11:54 AM, Jonathan Gray <jg...@facebook.com> wrote:
> Agreed that it's good to try to be agenda-less, but in the past we've always taken the first couple hours to do a group discussion around some of the key topics. Given there's a bunch of fairly major changes/testing going on these days, I think there is a good bit of stuff that would benefit from group discussion. After that, we can break up into smaller groups or individually to start hacking away. Or for those not interested in the topics, you can just hack from the start.
>
The above sounds good, discussion of a few near future concerns. I
think it important we not let it go on too long.
All of the below look good. I've added a few comments. (Anyone else
have suggestions on what should be discussed?)
> More potential topics of discussion I had in mind:
>
> - Compaction, split, and flush policies/heuristics (HBASE-2453, HBASE-2462, HBASE-2457, HBASE-2375, HBASE-1892, etc...)
> - Define our desired behaviors related to versioning, deletes, and removal of deletes in minor/major compactions. (HBASE-2453, HBASE-2457, HBASE-2243, etc)
The new HBase ACID spec should get at least a passing airing (Its
'done'. All releases post 0.20.4 are required to adhere to it).
> - Brainstorm on doing better distributed scenario testing (HBASE-2414)
We also badly need to work on breaking down current daemons so their
functions can be made more standalone and thus testable; e.g. load
balancer in master, compacting code in regionservers, etc.
> - Brainstorm on performance improvement ideas (top HDFS issues, better use of HFile seeking, blooms, block pre-fetch, etc...) Would be cool to have a wiki page w/ a list of these things.
A wiki page of items to be discussed?
> - Brainstorm on new functionality / updated road map. What priorities do the various sponsoring companies have, what are nice to haves but not on anyones schedule yet, etc. Again, this can seed a new (or updated) wiki page and/or update the currently outdated road map wiki page.
I moved aside the current 0.21 roadmap moving it to a page of its own
rather than have it as a section on the roadmap page
(http://wiki.apache.org/hadoop/HBase/Original021Roadmap#preview), and
added comments on the state of roadmap items listed therein. A
roadmap discussion could start with what is listed in the old roadmap,
I'd suggest. I'd suggest that roadmap discussion should not go into
deep depth because it has a tendency to consume mostly because it
turns into a blue skying session and besides, 0.21 is fairly imminent
and there isn't that much we could get into this release anyways.
> - HBase PR. We could use a new web site (maven and otherwise), a centralized blog, and also a refresh/cleanup of documentation. There's also agreement on shipping w/ a few different configurations, which should be part of a new set of getting started / new user docs. Would like to get everyones thoughts and also come up with a schedule.
On monday our petition to become an apache top level project goes
before the apache board. If it passes, rolling out a site revamp
might be timed to match our move to TLP.
> - Ideas for future HUGs
>
> For anyone that will not be able to attend the hackathon we will post a wrap-up afterwards with notes about all the discussions we had. Whatever comes out of the hackathon should be posted into the proper jiras or mailing list for full community discussion.
>
> Also, if anyone was not able to sign up for the HUG or Hackathon (both are full now) and is a regular contributor, please contact me directly.
>
> Very awesome. Gonna be a great day of HBase!
>
Agreed.
St.Ack
> JG
>
> ________________________________________
> From: Andrew Purtell [apurtell@apache.org]
> Sent: Saturday, April 17, 2010 10:28 AM
> To: hbase-dev@hadoop.apache.org
> Cc: hbase-user@hadoop.apache.org
> Subject: Hackathon agenda
>
> The Hackathon is basically agenda-less, but I'd like to propose a general topic of discussion we should cover while we are all in the room together:
>
> - For HBASE-1964 (HBASE-2183, HBASE-2461, and related): injecting and/or mocking exceptions thrown up from DFSClient. I think we want a toolkit for that. Could be incorporated into the unit testing framework. Should be possible to swap out a jar or something and make it active running on a real cluster with real load. Should be possible to inject random exceptions with adjustable probability. So what does HDFS have already? What do we need? If we're adding something, does it make sense to put it into HBase or contribute to HDFS? I think the latter.
>
> Let's gather a list of other topics, if any, that hackathon participants want to see covered so we can make sure it will happen.
>
> - Andy
>
>
>
>
>
RE: Hackathon agenda
Posted by Jonathan Gray <jg...@facebook.com>.
Agreed that it's good to try to be agenda-less, but in the past we've always taken the first couple hours to do a group discussion around some of the key topics. Given there's a bunch of fairly major changes/testing going on these days, I think there is a good bit of stuff that would benefit from group discussion. After that, we can break up into smaller groups or individually to start hacking away. Or for those not interested in the topics, you can just hack from the start.
More potential topics of discussion I had in mind:
- Compaction, split, and flush policies/heuristics (HBASE-2453, HBASE-2462, HBASE-2457, HBASE-2375, HBASE-1892, etc...)
- Define our desired behaviors related to versioning, deletes, and removal of deletes in minor/major compactions. (HBASE-2453, HBASE-2457, HBASE-2243, etc)
- Brainstorm on doing better distributed scenario testing (HBASE-2414)
- Brainstorm on performance improvement ideas (top HDFS issues, better use of HFile seeking, blooms, block pre-fetch, etc...) Would be cool to have a wiki page w/ a list of these things.
- Brainstorm on new functionality / updated road map. What priorities do the various sponsoring companies have, what are nice to haves but not on anyones schedule yet, etc. Again, this can seed a new (or updated) wiki page and/or update the currently outdated road map wiki page.
- HBase PR. We could use a new web site (maven and otherwise), a centralized blog, and also a refresh/cleanup of documentation. There's also agreement on shipping w/ a few different configurations, which should be part of a new set of getting started / new user docs. Would like to get everyones thoughts and also come up with a schedule.
- Ideas for future HUGs
For anyone that will not be able to attend the hackathon we will post a wrap-up afterwards with notes about all the discussions we had. Whatever comes out of the hackathon should be posted into the proper jiras or mailing list for full community discussion.
Also, if anyone was not able to sign up for the HUG or Hackathon (both are full now) and is a regular contributor, please contact me directly.
Very awesome. Gonna be a great day of HBase!
JG
________________________________________
From: Andrew Purtell [apurtell@apache.org]
Sent: Saturday, April 17, 2010 10:28 AM
To: hbase-dev@hadoop.apache.org
Cc: hbase-user@hadoop.apache.org
Subject: Hackathon agenda
The Hackathon is basically agenda-less, but I'd like to propose a general topic of discussion we should cover while we are all in the room together:
- For HBASE-1964 (HBASE-2183, HBASE-2461, and related): injecting and/or mocking exceptions thrown up from DFSClient. I think we want a toolkit for that. Could be incorporated into the unit testing framework. Should be possible to swap out a jar or something and make it active running on a real cluster with real load. Should be possible to inject random exceptions with adjustable probability. So what does HDFS have already? What do we need? If we're adding something, does it make sense to put it into HBase or contribute to HDFS? I think the latter.
Let's gather a list of other topics, if any, that hackathon participants want to see covered so we can make sure it will happen.
- Andy
RE: Hackathon agenda
Posted by Jonathan Gray <jg...@facebook.com>.
Agreed that it's good to try to be agenda-less, but in the past we've always taken the first couple hours to do a group discussion around some of the key topics. Given there's a bunch of fairly major changes/testing going on these days, I think there is a good bit of stuff that would benefit from group discussion. After that, we can break up into smaller groups or individually to start hacking away. Or for those not interested in the topics, you can just hack from the start.
More potential topics of discussion I had in mind:
- Compaction, split, and flush policies/heuristics (HBASE-2453, HBASE-2462, HBASE-2457, HBASE-2375, HBASE-1892, etc...)
- Define our desired behaviors related to versioning, deletes, and removal of deletes in minor/major compactions. (HBASE-2453, HBASE-2457, HBASE-2243, etc)
- Brainstorm on doing better distributed scenario testing (HBASE-2414)
- Brainstorm on performance improvement ideas (top HDFS issues, better use of HFile seeking, blooms, block pre-fetch, etc...) Would be cool to have a wiki page w/ a list of these things.
- Brainstorm on new functionality / updated road map. What priorities do the various sponsoring companies have, what are nice to haves but not on anyones schedule yet, etc. Again, this can seed a new (or updated) wiki page and/or update the currently outdated road map wiki page.
- HBase PR. We could use a new web site (maven and otherwise), a centralized blog, and also a refresh/cleanup of documentation. There's also agreement on shipping w/ a few different configurations, which should be part of a new set of getting started / new user docs. Would like to get everyones thoughts and also come up with a schedule.
- Ideas for future HUGs
For anyone that will not be able to attend the hackathon we will post a wrap-up afterwards with notes about all the discussions we had. Whatever comes out of the hackathon should be posted into the proper jiras or mailing list for full community discussion.
Also, if anyone was not able to sign up for the HUG or Hackathon (both are full now) and is a regular contributor, please contact me directly.
Very awesome. Gonna be a great day of HBase!
JG
________________________________________
From: Andrew Purtell [apurtell@apache.org]
Sent: Saturday, April 17, 2010 10:28 AM
To: hbase-dev@hadoop.apache.org
Cc: hbase-user@hadoop.apache.org
Subject: Hackathon agenda
The Hackathon is basically agenda-less, but I'd like to propose a general topic of discussion we should cover while we are all in the room together:
- For HBASE-1964 (HBASE-2183, HBASE-2461, and related): injecting and/or mocking exceptions thrown up from DFSClient. I think we want a toolkit for that. Could be incorporated into the unit testing framework. Should be possible to swap out a jar or something and make it active running on a real cluster with real load. Should be possible to inject random exceptions with adjustable probability. So what does HDFS have already? What do we need? If we're adding something, does it make sense to put it into HBase or contribute to HDFS? I think the latter.
Let's gather a list of other topics, if any, that hackathon participants want to see covered so we can make sure it will happen.
- Andy