You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Jesse Yates <je...@gmail.com> on 2012/05/09 03:31:38 UTC
modularizing trunk
Hi devs,
I'm just finishing up work on
HBASE-4336<https://issues.apache.org/jira/browse/HBASE-4336>-
modularization of hbase - and am planning to work with Stack to commit
the patch to trunk. If there are no objections, the planned refactor is on *May
23, 2012 (hbase hackathon).*
*What does this mean for you?*
If at the commit, you don't have any patches you will just pull the latest
version, import the projects into eclipse and move on with your life. It
may or may not be slightly more complex to import, depending on how you are
currently importing the project, but details on specific procedures will
come with the commit.
If you do have open patches, its a simple process to convert them over. You
will export your diff into a patch file (lets assume here its the standard
--no-prefix format). After pulling down the new version, you will just cd
into hbase-core and apply your patch. Unless you are modifying the
reference guide, hbase/bin scripts, hbase/conf files or the pom (please
avoid until after the switch as we will be changing everything), then the
patch _should_ apply cleanly. If you are modifying those files, you will
need to split up the patch file and apply the first half from hbase and the
second from hbase-core.
*What is actually happening in this patch?*
The initial plan for HBASE-4336 was to break out each of the different
pieces of hbase into their own module. However, in practice doing the
one-shot refactor is a huge pain and not really feasible with the current
project velocity. Instead, 4336 sets up the framework to enable much easier
modularization. If we want to pull out client stuff, then we merely need to
dis-entangle the code, add a few lines to the hbase-assembly project, and
you are off to the races (more or less - see
HBASE-4336<https://issues.apache.org/jira/browse/HBASE-4336>for
caveats around tests).
After the patch goes in, we will have three modules: hbase, hbase-core,
hbase-assemble.
- hbase is the parent and defines the general properties used by all the
children, including managing dependency versions, plugin versions and
general properties, and managing the docbook and site.
- hbase-core is where all the code will go. It will still export a
regular jar and a test jar and support the security profile (though
activation will now be -Dsecurity, rather than -Psecurity)
- hbase-assemble will package everything together into the final
tarball/rpm/deb
In the future, packages like mapreduce, client, sever, wal, etc have been
discussed. The means easier testing, development and deployment.
*How can you help?
*1) Get as much of your code into the trunk before the refactor
2) Take a look at https://github.com/jyates/hbase/tree/refactor-modules and
comment on the recent patches there (this is pretty close to what the final
refactor will look like)
3) Start thinking about refactoring other modules out of hbase-core (or
even forking on github and doing it!)
Thanks,
Jesse
-------------------
Jesse Yates
@jesse_yates
jyates.github.com
Re: modularizing trunk
Posted by Stack <st...@duboce.net>.
On Thu, May 24, 2012 at 12:29 PM, Jesse Yates <je...@gmail.com> wrote:
> All,
>
> Its taking a little longer than hoped. Thanks for being patient with
> commits. Hopefully done shortly - will let you know when we have made the
> switch.
>
FYI, we are still at it. Hopefully we'll be done today.
(I looooovvvveeeee Maven)
St.Ack
Re: modularizing trunk
Posted by Andrew Purtell <ap...@apache.org>.
Sorry, I committed a doc update to trunk, 0.94, and 0.92 at noon.
On Thu, May 24, 2012 at 12:29 PM, Jesse Yates <je...@gmail.com> wrote:
> All,
>
> Its taking a little longer than hoped. Thanks for being patient with
> commits. Hopefully done shortly - will let you know when we have made the
> switch.
>
> -------------------
> Jesse Yates
> @jesse_yates
> jyates.github.com
>
>
> On Wed, May 23, 2012 at 11:27 PM, Jesse Yates <je...@gmail.com>wrote:
>
>> Hopefully, all the committers are tuckered out from hbasecon and the
>> hackathon this week (great events - thanks to everyone who organized and
>> made it out) as I'd like to ask to for a virtual lock on the repo until *Noon
>> PST Tomorrow (5/24)* so Stack and I can get the modularization committed.
>> That means, please no commits until then noon.
>>
>> After we do the commit, any outstanding patches will need to be rebased
>> onto the new trunk (see earlier emails in this thread for more details on
>> how this works).
>>
>> Thanks for your help in this effort.
>>
>> thanks,
>> Jesse
>>
>> ps. look forward to our third module soon after the switch - hbase-common!
>>
>> -------------------
>> Jesse Yates
>> @jesse_yates
>> jyates.github.com
>>
>>
>> On Wed, May 9, 2012 at 3:22 PM, Jesse Yates <je...@gmail.com>wrote:
>>
>>> Created
>>> * HBASE-5976 to discuss the name of the initial package
>>> * HBASE-5977 to discuss future naming/usage of packages
>>>
>>> Go nuts.
>>>
>>>
>>> -------------------
>>> Jesse Yates
>>> @jesse_yates
>>> jyates.github.com
>>>
>>>
>>> On Wed, May 9, 2012 at 2:18 PM, Elliott Clark <ec...@stumbleupon.com>wrote:
>>>
>>>> Starting with hbase-core seems to make sense. If we started with
>>>> hbase-server there would be a while until all of the client and other
>>>> packages were split up where org.apache.hadoop.hbase.client was in the
>>>> hadoop-server jar.
>>>>
>>>> On Wed, May 9, 2012 at 1:34 PM, Matt Corgan <mc...@hotpads.com> wrote:
>>>>
>>>> > Sorry about that - my intention was not to decide all future module
>>>> names,
>>>> > just to avoid having both core and common and to illustrate how we were
>>>> > heading that direction.
>>>> >
>>>> >
>>>> > On Wed, May 9, 2012 at 1:27 PM, Jesse Yates <je...@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > I was worried this discussion around naming might happen. Should I
>>>> open a
>>>> > > sub-jira on 4336 for how to name the modules, what future modules
>>>> should
>>>> > > be, etc.?
>>>> > >
>>>> > > -Jesse
>>>> > > -------------------
>>>> > > Jesse Yates
>>>> > > @jesse_yates
>>>> > > jyates.github.com
>>>> > >
>>>> > >
>>>> > > On Wed, May 9, 2012 at 12:06 PM, Andrew Purtell <apurtell@apache.org
>>>> >
>>>> > > wrote:
>>>> > >
>>>> > > > On Wed, May 9, 2012 at 12:01 PM, Stack <st...@duboce.net> wrote:
>>>> > > > > On Wed, May 9, 2012 at 11:18 AM, Matt Corgan <
>>>> mcorgan@hotpads.com>
>>>> > > > wrote:
>>>> > > > >> I'd suggest calling this first module hbase-server since the
>>>> > majority
>>>> > > of
>>>> > > > >> the classes are related to the master and regionservers. Then
>>>> we
>>>> > can
>>>> > > > pull
>>>> > > > >> out the fundamental classes (KeyValue, Bytes, etc) into a small
>>>> > module
>>>> > > > >> called hbase-core. After that, we can create an hbase-client
>>>> module
>>>> > > > that
>>>> > > > >> only depends on hbase-core (so few or no dependencies). My main
>>>> > point
>>>> > > > >> being that we'll want to reserve the name hbase-core for the
>>>> actual
>>>> > > core
>>>> > > > >> classes and not throw everything in there.
>>>> > > > >>
>>>> > > > >
>>>> > > > > We don't want hbase-common and then hbase-core for sure.
>>>> > > > >
>>>> > > > > The first module rightly should be called hbase-bucket since its
>>>> just
>>>> > > > > a holding module while we do our sort-through. Can we figure a
>>>> name
>>>> > > > > that better conveys the module as so? hbase-hbase?
>>>> hbase-bucket?
>>>> > > > > hbase-99%? hbase-pick-u-part? hbase-residuum?
>>>> > > >
>>>> > > > hbase-server is as reasonable as any, right?
>>>> > > >
>>>> > > > Best regards,
>>>> > > >
>>>> > > > - Andy
>>>> > > >
>>>> > > > Problems worthy of attack prove their worth by hitting back. - Piet
>>>> > > > Hein (via Tom White)
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
Re: modularizing trunk
Posted by Jesse Yates <je...@gmail.com>.
All,
Its taking a little longer than hoped. Thanks for being patient with
commits. Hopefully done shortly - will let you know when we have made the
switch.
-------------------
Jesse Yates
@jesse_yates
jyates.github.com
On Wed, May 23, 2012 at 11:27 PM, Jesse Yates <je...@gmail.com>wrote:
> Hopefully, all the committers are tuckered out from hbasecon and the
> hackathon this week (great events - thanks to everyone who organized and
> made it out) as I'd like to ask to for a virtual lock on the repo until *Noon
> PST Tomorrow (5/24)* so Stack and I can get the modularization committed.
> That means, please no commits until then noon.
>
> After we do the commit, any outstanding patches will need to be rebased
> onto the new trunk (see earlier emails in this thread for more details on
> how this works).
>
> Thanks for your help in this effort.
>
> thanks,
> Jesse
>
> ps. look forward to our third module soon after the switch - hbase-common!
>
> -------------------
> Jesse Yates
> @jesse_yates
> jyates.github.com
>
>
> On Wed, May 9, 2012 at 3:22 PM, Jesse Yates <je...@gmail.com>wrote:
>
>> Created
>> * HBASE-5976 to discuss the name of the initial package
>> * HBASE-5977 to discuss future naming/usage of packages
>>
>> Go nuts.
>>
>>
>> -------------------
>> Jesse Yates
>> @jesse_yates
>> jyates.github.com
>>
>>
>> On Wed, May 9, 2012 at 2:18 PM, Elliott Clark <ec...@stumbleupon.com>wrote:
>>
>>> Starting with hbase-core seems to make sense. If we started with
>>> hbase-server there would be a while until all of the client and other
>>> packages were split up where org.apache.hadoop.hbase.client was in the
>>> hadoop-server jar.
>>>
>>> On Wed, May 9, 2012 at 1:34 PM, Matt Corgan <mc...@hotpads.com> wrote:
>>>
>>> > Sorry about that - my intention was not to decide all future module
>>> names,
>>> > just to avoid having both core and common and to illustrate how we were
>>> > heading that direction.
>>> >
>>> >
>>> > On Wed, May 9, 2012 at 1:27 PM, Jesse Yates <je...@gmail.com>
>>> > wrote:
>>> >
>>> > > I was worried this discussion around naming might happen. Should I
>>> open a
>>> > > sub-jira on 4336 for how to name the modules, what future modules
>>> should
>>> > > be, etc.?
>>> > >
>>> > > -Jesse
>>> > > -------------------
>>> > > Jesse Yates
>>> > > @jesse_yates
>>> > > jyates.github.com
>>> > >
>>> > >
>>> > > On Wed, May 9, 2012 at 12:06 PM, Andrew Purtell <apurtell@apache.org
>>> >
>>> > > wrote:
>>> > >
>>> > > > On Wed, May 9, 2012 at 12:01 PM, Stack <st...@duboce.net> wrote:
>>> > > > > On Wed, May 9, 2012 at 11:18 AM, Matt Corgan <
>>> mcorgan@hotpads.com>
>>> > > > wrote:
>>> > > > >> I'd suggest calling this first module hbase-server since the
>>> > majority
>>> > > of
>>> > > > >> the classes are related to the master and regionservers. Then
>>> we
>>> > can
>>> > > > pull
>>> > > > >> out the fundamental classes (KeyValue, Bytes, etc) into a small
>>> > module
>>> > > > >> called hbase-core. After that, we can create an hbase-client
>>> module
>>> > > > that
>>> > > > >> only depends on hbase-core (so few or no dependencies). My main
>>> > point
>>> > > > >> being that we'll want to reserve the name hbase-core for the
>>> actual
>>> > > core
>>> > > > >> classes and not throw everything in there.
>>> > > > >>
>>> > > > >
>>> > > > > We don't want hbase-common and then hbase-core for sure.
>>> > > > >
>>> > > > > The first module rightly should be called hbase-bucket since its
>>> just
>>> > > > > a holding module while we do our sort-through. Can we figure a
>>> name
>>> > > > > that better conveys the module as so? hbase-hbase?
>>> hbase-bucket?
>>> > > > > hbase-99%? hbase-pick-u-part? hbase-residuum?
>>> > > >
>>> > > > hbase-server is as reasonable as any, right?
>>> > > >
>>> > > > Best regards,
>>> > > >
>>> > > > - Andy
>>> > > >
>>> > > > Problems worthy of attack prove their worth by hitting back. - Piet
>>> > > > Hein (via Tom White)
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>
Re: modularizing trunk
Posted by Jesse Yates <je...@gmail.com>.
Hopefully, all the committers are tuckered out from hbasecon and the
hackathon this week (great events - thanks to everyone who organized and
made it out) as I'd like to ask to for a virtual lock on the repo until *Noon
PST Tomorrow (5/24)* so Stack and I can get the modularization committed.
That means, please no commits until then noon.
After we do the commit, any outstanding patches will need to be rebased
onto the new trunk (see earlier emails in this thread for more details on
how this works).
Thanks for your help in this effort.
thanks,
Jesse
ps. look forward to our third module soon after the switch - hbase-common!
-------------------
Jesse Yates
@jesse_yates
jyates.github.com
On Wed, May 9, 2012 at 3:22 PM, Jesse Yates <je...@gmail.com> wrote:
> Created
> * HBASE-5976 to discuss the name of the initial package
> * HBASE-5977 to discuss future naming/usage of packages
>
> Go nuts.
>
>
> -------------------
> Jesse Yates
> @jesse_yates
> jyates.github.com
>
>
> On Wed, May 9, 2012 at 2:18 PM, Elliott Clark <ec...@stumbleupon.com>wrote:
>
>> Starting with hbase-core seems to make sense. If we started with
>> hbase-server there would be a while until all of the client and other
>> packages were split up where org.apache.hadoop.hbase.client was in the
>> hadoop-server jar.
>>
>> On Wed, May 9, 2012 at 1:34 PM, Matt Corgan <mc...@hotpads.com> wrote:
>>
>> > Sorry about that - my intention was not to decide all future module
>> names,
>> > just to avoid having both core and common and to illustrate how we were
>> > heading that direction.
>> >
>> >
>> > On Wed, May 9, 2012 at 1:27 PM, Jesse Yates <je...@gmail.com>
>> > wrote:
>> >
>> > > I was worried this discussion around naming might happen. Should I
>> open a
>> > > sub-jira on 4336 for how to name the modules, what future modules
>> should
>> > > be, etc.?
>> > >
>> > > -Jesse
>> > > -------------------
>> > > Jesse Yates
>> > > @jesse_yates
>> > > jyates.github.com
>> > >
>> > >
>> > > On Wed, May 9, 2012 at 12:06 PM, Andrew Purtell <ap...@apache.org>
>> > > wrote:
>> > >
>> > > > On Wed, May 9, 2012 at 12:01 PM, Stack <st...@duboce.net> wrote:
>> > > > > On Wed, May 9, 2012 at 11:18 AM, Matt Corgan <mcorgan@hotpads.com
>> >
>> > > > wrote:
>> > > > >> I'd suggest calling this first module hbase-server since the
>> > majority
>> > > of
>> > > > >> the classes are related to the master and regionservers. Then we
>> > can
>> > > > pull
>> > > > >> out the fundamental classes (KeyValue, Bytes, etc) into a small
>> > module
>> > > > >> called hbase-core. After that, we can create an hbase-client
>> module
>> > > > that
>> > > > >> only depends on hbase-core (so few or no dependencies). My main
>> > point
>> > > > >> being that we'll want to reserve the name hbase-core for the
>> actual
>> > > core
>> > > > >> classes and not throw everything in there.
>> > > > >>
>> > > > >
>> > > > > We don't want hbase-common and then hbase-core for sure.
>> > > > >
>> > > > > The first module rightly should be called hbase-bucket since its
>> just
>> > > > > a holding module while we do our sort-through. Can we figure a
>> name
>> > > > > that better conveys the module as so? hbase-hbase? hbase-bucket?
>> > > > > hbase-99%? hbase-pick-u-part? hbase-residuum?
>> > > >
>> > > > hbase-server is as reasonable as any, right?
>> > > >
>> > > > Best regards,
>> > > >
>> > > > - Andy
>> > > >
>> > > > Problems worthy of attack prove their worth by hitting back. - Piet
>> > > > Hein (via Tom White)
>> > > >
>> > >
>> >
>>
>
>
Re: modularizing trunk
Posted by Jesse Yates <je...@gmail.com>.
Created
* HBASE-5976 to discuss the name of the initial package
* HBASE-5977 to discuss future naming/usage of packages
Go nuts.
-------------------
Jesse Yates
@jesse_yates
jyates.github.com
On Wed, May 9, 2012 at 2:18 PM, Elliott Clark <ec...@stumbleupon.com>wrote:
> Starting with hbase-core seems to make sense. If we started with
> hbase-server there would be a while until all of the client and other
> packages were split up where org.apache.hadoop.hbase.client was in the
> hadoop-server jar.
>
> On Wed, May 9, 2012 at 1:34 PM, Matt Corgan <mc...@hotpads.com> wrote:
>
> > Sorry about that - my intention was not to decide all future module
> names,
> > just to avoid having both core and common and to illustrate how we were
> > heading that direction.
> >
> >
> > On Wed, May 9, 2012 at 1:27 PM, Jesse Yates <je...@gmail.com>
> > wrote:
> >
> > > I was worried this discussion around naming might happen. Should I
> open a
> > > sub-jira on 4336 for how to name the modules, what future modules
> should
> > > be, etc.?
> > >
> > > -Jesse
> > > -------------------
> > > Jesse Yates
> > > @jesse_yates
> > > jyates.github.com
> > >
> > >
> > > On Wed, May 9, 2012 at 12:06 PM, Andrew Purtell <ap...@apache.org>
> > > wrote:
> > >
> > > > On Wed, May 9, 2012 at 12:01 PM, Stack <st...@duboce.net> wrote:
> > > > > On Wed, May 9, 2012 at 11:18 AM, Matt Corgan <mc...@hotpads.com>
> > > > wrote:
> > > > >> I'd suggest calling this first module hbase-server since the
> > majority
> > > of
> > > > >> the classes are related to the master and regionservers. Then we
> > can
> > > > pull
> > > > >> out the fundamental classes (KeyValue, Bytes, etc) into a small
> > module
> > > > >> called hbase-core. After that, we can create an hbase-client
> module
> > > > that
> > > > >> only depends on hbase-core (so few or no dependencies). My main
> > point
> > > > >> being that we'll want to reserve the name hbase-core for the
> actual
> > > core
> > > > >> classes and not throw everything in there.
> > > > >>
> > > > >
> > > > > We don't want hbase-common and then hbase-core for sure.
> > > > >
> > > > > The first module rightly should be called hbase-bucket since its
> just
> > > > > a holding module while we do our sort-through. Can we figure a
> name
> > > > > that better conveys the module as so? hbase-hbase? hbase-bucket?
> > > > > hbase-99%? hbase-pick-u-part? hbase-residuum?
> > > >
> > > > hbase-server is as reasonable as any, right?
> > > >
> > > > Best regards,
> > > >
> > > > - Andy
> > > >
> > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > > > Hein (via Tom White)
> > > >
> > >
> >
>
Re: modularizing trunk
Posted by Elliott Clark <ec...@stumbleupon.com>.
Starting with hbase-core seems to make sense. If we started with
hbase-server there would be a while until all of the client and other
packages were split up where org.apache.hadoop.hbase.client was in the
hadoop-server jar.
On Wed, May 9, 2012 at 1:34 PM, Matt Corgan <mc...@hotpads.com> wrote:
> Sorry about that - my intention was not to decide all future module names,
> just to avoid having both core and common and to illustrate how we were
> heading that direction.
>
>
> On Wed, May 9, 2012 at 1:27 PM, Jesse Yates <je...@gmail.com>
> wrote:
>
> > I was worried this discussion around naming might happen. Should I open a
> > sub-jira on 4336 for how to name the modules, what future modules should
> > be, etc.?
> >
> > -Jesse
> > -------------------
> > Jesse Yates
> > @jesse_yates
> > jyates.github.com
> >
> >
> > On Wed, May 9, 2012 at 12:06 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > On Wed, May 9, 2012 at 12:01 PM, Stack <st...@duboce.net> wrote:
> > > > On Wed, May 9, 2012 at 11:18 AM, Matt Corgan <mc...@hotpads.com>
> > > wrote:
> > > >> I'd suggest calling this first module hbase-server since the
> majority
> > of
> > > >> the classes are related to the master and regionservers. Then we
> can
> > > pull
> > > >> out the fundamental classes (KeyValue, Bytes, etc) into a small
> module
> > > >> called hbase-core. After that, we can create an hbase-client module
> > > that
> > > >> only depends on hbase-core (so few or no dependencies). My main
> point
> > > >> being that we'll want to reserve the name hbase-core for the actual
> > core
> > > >> classes and not throw everything in there.
> > > >>
> > > >
> > > > We don't want hbase-common and then hbase-core for sure.
> > > >
> > > > The first module rightly should be called hbase-bucket since its just
> > > > a holding module while we do our sort-through. Can we figure a name
> > > > that better conveys the module as so? hbase-hbase? hbase-bucket?
> > > > hbase-99%? hbase-pick-u-part? hbase-residuum?
> > >
> > > hbase-server is as reasonable as any, right?
> > >
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> > > Hein (via Tom White)
> > >
> >
>
Re: modularizing trunk
Posted by Matt Corgan <mc...@hotpads.com>.
Sorry about that - my intention was not to decide all future module names,
just to avoid having both core and common and to illustrate how we were
heading that direction.
On Wed, May 9, 2012 at 1:27 PM, Jesse Yates <je...@gmail.com> wrote:
> I was worried this discussion around naming might happen. Should I open a
> sub-jira on 4336 for how to name the modules, what future modules should
> be, etc.?
>
> -Jesse
> -------------------
> Jesse Yates
> @jesse_yates
> jyates.github.com
>
>
> On Wed, May 9, 2012 at 12:06 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > On Wed, May 9, 2012 at 12:01 PM, Stack <st...@duboce.net> wrote:
> > > On Wed, May 9, 2012 at 11:18 AM, Matt Corgan <mc...@hotpads.com>
> > wrote:
> > >> I'd suggest calling this first module hbase-server since the majority
> of
> > >> the classes are related to the master and regionservers. Then we can
> > pull
> > >> out the fundamental classes (KeyValue, Bytes, etc) into a small module
> > >> called hbase-core. After that, we can create an hbase-client module
> > that
> > >> only depends on hbase-core (so few or no dependencies). My main point
> > >> being that we'll want to reserve the name hbase-core for the actual
> core
> > >> classes and not throw everything in there.
> > >>
> > >
> > > We don't want hbase-common and then hbase-core for sure.
> > >
> > > The first module rightly should be called hbase-bucket since its just
> > > a holding module while we do our sort-through. Can we figure a name
> > > that better conveys the module as so? hbase-hbase? hbase-bucket?
> > > hbase-99%? hbase-pick-u-part? hbase-residuum?
> >
> > hbase-server is as reasonable as any, right?
> >
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein (via Tom White)
> >
>
Re: modularizing trunk
Posted by Jesse Yates <je...@gmail.com>.
I was worried this discussion around naming might happen. Should I open a
sub-jira on 4336 for how to name the modules, what future modules should
be, etc.?
-Jesse
-------------------
Jesse Yates
@jesse_yates
jyates.github.com
On Wed, May 9, 2012 at 12:06 PM, Andrew Purtell <ap...@apache.org> wrote:
> On Wed, May 9, 2012 at 12:01 PM, Stack <st...@duboce.net> wrote:
> > On Wed, May 9, 2012 at 11:18 AM, Matt Corgan <mc...@hotpads.com>
> wrote:
> >> I'd suggest calling this first module hbase-server since the majority of
> >> the classes are related to the master and regionservers. Then we can
> pull
> >> out the fundamental classes (KeyValue, Bytes, etc) into a small module
> >> called hbase-core. After that, we can create an hbase-client module
> that
> >> only depends on hbase-core (so few or no dependencies). My main point
> >> being that we'll want to reserve the name hbase-core for the actual core
> >> classes and not throw everything in there.
> >>
> >
> > We don't want hbase-common and then hbase-core for sure.
> >
> > The first module rightly should be called hbase-bucket since its just
> > a holding module while we do our sort-through. Can we figure a name
> > that better conveys the module as so? hbase-hbase? hbase-bucket?
> > hbase-99%? hbase-pick-u-part? hbase-residuum?
>
> hbase-server is as reasonable as any, right?
>
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein (via Tom White)
>
Re: modularizing trunk
Posted by Andrew Purtell <ap...@apache.org>.
On Wed, May 9, 2012 at 12:01 PM, Stack <st...@duboce.net> wrote:
> On Wed, May 9, 2012 at 11:18 AM, Matt Corgan <mc...@hotpads.com> wrote:
>> I'd suggest calling this first module hbase-server since the majority of
>> the classes are related to the master and regionservers. Then we can pull
>> out the fundamental classes (KeyValue, Bytes, etc) into a small module
>> called hbase-core. After that, we can create an hbase-client module that
>> only depends on hbase-core (so few or no dependencies). My main point
>> being that we'll want to reserve the name hbase-core for the actual core
>> classes and not throw everything in there.
>>
>
> We don't want hbase-common and then hbase-core for sure.
>
> The first module rightly should be called hbase-bucket since its just
> a holding module while we do our sort-through. Can we figure a name
> that better conveys the module as so? hbase-hbase? hbase-bucket?
> hbase-99%? hbase-pick-u-part? hbase-residuum?
hbase-server is as reasonable as any, right?
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
Re: modularizing trunk
Posted by Stack <st...@duboce.net>.
On Wed, May 9, 2012 at 11:18 AM, Matt Corgan <mc...@hotpads.com> wrote:
> I'd suggest calling this first module hbase-server since the majority of
> the classes are related to the master and regionservers. Then we can pull
> out the fundamental classes (KeyValue, Bytes, etc) into a small module
> called hbase-core. After that, we can create an hbase-client module that
> only depends on hbase-core (so few or no dependencies). My main point
> being that we'll want to reserve the name hbase-core for the actual core
> classes and not throw everything in there.
>
We don't want hbase-common and then hbase-core for sure.
The first module rightly should be called hbase-bucket since its just
a holding module while we do our sort-through. Can we figure a name
that better conveys the module as so? hbase-hbase? hbase-bucket?
hbase-99%? hbase-pick-u-part? hbase-residuum?
I don't think we can go the other way and figure a module that is more
'core' than core: e.g. hbase-nucleus and hbase-core wouldn't work.
St.Ack
Re: modularizing trunk
Posted by Matt Corgan <mc...@hotpads.com>.
I talked briefly with Jesse about this before, but thought it was worth
bringing up one more time. I'm worried that the name hbase-core is a bad
choice for this first module that contains all the code.
As soon as the modules are created, I'm planning to submit
HBASE-4676<https://issues.apache.org/jira/browse/HBASE-4676>as a
separate module (hbase-prefix-trie) that has a dependency on just a
handful of existing classes (KeyValue, the DataBlockEncoder interface,
Bytes, and maybe a couple others). My idea was to pull those critical
classes into a small central module that contains the fundamental classes
of HBase and has no (or few) external dependencies. If the initial module
is called hbase-core, then the new shared module would have to be called
hbase-common. I'm worried that having an hbase-common and an hbase-core
will cause needless confusion for years to come.
I'd suggest calling this first module hbase-server since the majority of
the classes are related to the master and regionservers. Then we can pull
out the fundamental classes (KeyValue, Bytes, etc) into a small module
called hbase-core. After that, we can create an hbase-client module that
only depends on hbase-core (so few or no dependencies). My main point
being that we'll want to reserve the name hbase-core for the actual core
classes and not throw everything in there.
Another option is to treat the current code base as hbase-core (the current
plan), and pull 95% of the classes out to another module called
hbase-server. But, it seems like it would be easier to extract the smaller
number of core classes out to a new module. That is only a theory
though... maybe others who've tried to refactor in the past know better.
Anyway, just thought I'd throw that out there since the module naming
discussion in the jira got really confusing. Last thing I want to do is
sidetrack Jesse's progress...
Matt
On Wed, May 9, 2012 at 10:43 AM, Stack <st...@duboce.net> wrote:
> Sounds good Jesse.
>
> @Ted
>
> > w.r.t. timing of integration of HBASE-4336, we should plan in advance.
>
> Yes. Thats what Jesse has been doing (See the issue for a recounting
> of prep that has gotten us to here).
>
> > Currently there're several big patches pending checkin, e.g. HBASE-5732
> > (Remove the SecureRPCEngine and merge the security-related logic in the
> > core engine). Such patches would need rebasing if they get integrated
> after
> > HBASE-4336.
>
> Read Jesse's prescription above. HBASE-5732 will be in before May
> 22nd so that one in particular is a non-issue.
>
> > I feel HBASE-4336 should go in before (or after) 23rd so that people
> > working on other JIRAs have an easier time adjusting to the new
> structure.
> >
>
> Disagree. On 23rd there will be a bunch of contributors all in the
> one room. We'll be able to address issues and educate in person.
>
> St.Ack
>
Re: modularizing trunk
Posted by Stack <st...@duboce.net>.
Sounds good Jesse.
@Ted
> w.r.t. timing of integration of HBASE-4336, we should plan in advance.
Yes. Thats what Jesse has been doing (See the issue for a recounting
of prep that has gotten us to here).
> Currently there're several big patches pending checkin, e.g. HBASE-5732
> (Remove the SecureRPCEngine and merge the security-related logic in the
> core engine). Such patches would need rebasing if they get integrated after
> HBASE-4336.
Read Jesse's prescription above. HBASE-5732 will be in before May
22nd so that one in particular is a non-issue.
> I feel HBASE-4336 should go in before (or after) 23rd so that people
> working on other JIRAs have an easier time adjusting to the new structure.
>
Disagree. On 23rd there will be a bunch of contributors all in the
one room. We'll be able to address issues and educate in person.
St.Ack
Re: modularizing trunk
Posted by Ted Yu <yu...@gmail.com>.
Thanks Jesse for the nice writeup.
w.r.t. timing of integration of HBASE-4336, we should plan in advance.
Currently there're several big patches pending checkin, e.g. HBASE-5732
(Remove the SecureRPCEngine and merge the security-related logic in the
core engine). Such patches would need rebasing if they get integrated after
HBASE-4336.
On May 23, 2012, there would be quite a few patches becoming ready for
checkin.
I feel HBASE-4336 should go in before (or after) 23rd so that people
working on other JIRAs have an easier time adjusting to the new structure.
Cheers
On Tue, May 8, 2012 at 6:31 PM, Jesse Yates <je...@gmail.com> wrote:
> Hi devs,
>
> I'm just finishing up work on
> HBASE-4336<https://issues.apache.org/jira/browse/HBASE-4336>-
> modularization of hbase - and am planning to work with Stack to commit
> the patch to trunk. If there are no objections, the planned refactor is on
> *May
> 23, 2012 (hbase hackathon).*
>
> *What does this mean for you?*
>
> If at the commit, you don't have any patches you will just pull the latest
> version, import the projects into eclipse and move on with your life. It
> may or may not be slightly more complex to import, depending on how you are
> currently importing the project, but details on specific procedures will
> come with the commit.
>
> If you do have open patches, its a simple process to convert them over. You
> will export your diff into a patch file (lets assume here its the standard
> --no-prefix format). After pulling down the new version, you will just cd
> into hbase-core and apply your patch. Unless you are modifying the
> reference guide, hbase/bin scripts, hbase/conf files or the pom (please
> avoid until after the switch as we will be changing everything), then the
> patch _should_ apply cleanly. If you are modifying those files, you will
> need to split up the patch file and apply the first half from hbase and the
> second from hbase-core.
>
> *What is actually happening in this patch?*
>
> The initial plan for HBASE-4336 was to break out each of the different
> pieces of hbase into their own module. However, in practice doing the
> one-shot refactor is a huge pain and not really feasible with the current
> project velocity. Instead, 4336 sets up the framework to enable much easier
> modularization. If we want to pull out client stuff, then we merely need to
> dis-entangle the code, add a few lines to the hbase-assembly project, and
> you are off to the races (more or less - see
> HBASE-4336<https://issues.apache.org/jira/browse/HBASE-4336>for
> caveats around tests).
>
> After the patch goes in, we will have three modules: hbase, hbase-core,
> hbase-assemble.
>
> - hbase is the parent and defines the general properties used by all the
> children, including managing dependency versions, plugin versions and
> general properties, and managing the docbook and site.
> - hbase-core is where all the code will go. It will still export a
> regular jar and a test jar and support the security profile (though
> activation will now be -Dsecurity, rather than -Psecurity)
> - hbase-assemble will package everything together into the final
> tarball/rpm/deb
>
> In the future, packages like mapreduce, client, sever, wal, etc have been
> discussed. The means easier testing, development and deployment.
>
> *How can you help?
>
> *1) Get as much of your code into the trunk before the refactor
> 2) Take a look at https://github.com/jyates/hbase/tree/refactor-modulesand
> comment on the recent patches there (this is pretty close to what the final
> refactor will look like)
> 3) Start thinking about refactoring other modules out of hbase-core (or
> even forking on github and doing it!)
>
> Thanks,
> Jesse
> -------------------
> Jesse Yates
> @jesse_yates
> jyates.github.com
>