Project module granularity

[Brent Atkinson <br...@gmail.com>]

Hi,

This is mainly a background question, but is prompted by observations I
have had while working in the code base. The general question is, can the
project be simpler to work on and the build speed improved by removing or
redrawing module boundaries?

An example of this is what has been undertaken in Archiva recently, where
redback has been pulled up into the project. This essentially makes redback
part of the project so there it can more easily track with the project.
This was something I didn't appreciate until we had to consider reworking
all of the Struts 2 actions, etc. to update it for security reasons. It
would be nice to just replace what redback was doing with spring security,
but moving incrementally is more difficult with redback as a separate
project.

Another example is whether the degree to which project modules make sense
as modules. In the extreme, we might consider regrouping the existing
classes into external API, internal API, core implementation, and maybe a
few more. The goal would be to reduce the number of independent modules to
as close to 1 as possible. I'm curious about this because it seems the
coarser granularity could simplify the project considerably and possibly
speed up the compile/test cycle. I tried looking through the archive for
similar conversations and couldn't find any.

Have these been considered? Thoughts?

Re: Project module granularity

[Brent Atkinson <br...@gmail.com>]

The information you just offered was exactly the type of thing I was hoping
to learn. I see how we would not want to have pollute the core code with
dependencies only relevant for notifier implementations (or something
similar). SCM dependencies seem similar, but they are more central to CI
functions.

I will draft something up and I would love to shoot holes through it. The
general idea is to split into two major modules, an api module that
contains only interfaces, exceptions and stable domain classes and a core
module that includes the core implementation of the CI engine. Things like
the notifiers, where they don't make sense to pull into the core, would
depend on and implement the section of the api.

If successful the other projects, like notifiers, would be possible to
convert them to something more like plugins. Using the dependency injection
container and dynamically loading the classes is already there. One thing
that would be different is the stability of the APIs used for these.

I suspect the model classes may present a challenge if these modules are
coupled to them directly because as we change the domain. I would assume
that we'd have to have a more stable extensible model for this to be
successful.

Brent



On Mon, May 19, 2014 at 1:27 AM, Brett Porter <br...@apache.org> wrote:

> Agree with what has been said here, and thanks for raising the topic.
>
> I don't think there's been an explicit discussion about it within
> Continuum, and I think your observations are correct - many modules have
> gone well over the deep end in being split up, while there are things like
> continuum-core that contain the kitchen sink (and then some :) There are
> other problems too - like a lot of code duplication out of core into the
> build agents, and some tests that are slow due to using network or SCM
> operations.
>
> The split is always a trade off. I think it's good to consolidate along
> the lines you described, though I'd still make special purpose code
> (particularly with external dependencies) separate - things like the
> notifier implementations, and release code.
>
> Did you want to draft a module hierarchy that we can talk through?
>
> - Brett
>
> On 19 May 2014, at 4:38 am, Brent Atkinson <br...@gmail.com>
> wrote:
>
> > Christian,
> >
> > Thank you for chiming in. I wanted to raise this for general feedback,
> and
> > I think we're somewhat safe from Parkinson's law of triviality. I don't
> > think we run too great a risk of bogging down in discussion about this,
> > especially with this list's volume at the moment.
> >
> > From my point of view, it helps to know that someone's experience agrees
> > with this in concept, even if it isn't based on direct experience with
> > Continuum or Archiva.  In my opinion the number of modules makes
> > development more cumbersome all around, from building to IDE integration,
> > to developing content.
> >
> > I also raised it hoping someone with more direct experience than I have
> > might be able to offer insight into what value this would have, and maybe
> > suggest how it might make sense to tackle. I'd be very interesting in
> > tackling something like this myself if it makes sense.
> >
> > Brent
> >
> >
> > On Sun, May 18, 2014 at 12:37 PM, Christian Gruber <cgruber@israfil.net
> >wrote:
> >
> >> As an effective outsider (haven't really dealt with continuum or archive
> >> in a long time) I can back up this suggestion from my own experience
> with
> >> maven modules.  I have found that artifact divisions on component lines
> can
> >> be useful, but overkill, and a public API, (if relevant) SPI for
> extenders,
> >> and internal implementation artifacts are a useful division that
> balances
> >> things out and makes builds faster, etc.
> >>
> >> It is also useful that artifacts split on major boundaries of
> >> cohesion/coupling where reuse is desired. That is, if something is
> >> realistically suitable for re-use in other projects, it's reasonable to
> >> have an artifact on it - but at that point it really becomes another
> >> project.  When to do that is really an art.  But I tend to try to
> reserve
> >> such extraction to really highly valuable libraries where the
> bang-for-buck
> >> ratio is high enough to justify the extra overhead (conceptual and
> >> computational in the build).
> >>
> >> /shrug - sorry to bike shed, but I think the idea has merit and no one
> was
> >> really responding to Brent.
> >>
> >> christian.
> >>
> >>
> >> On 17 May 2014, at 10:02, Brent Atkinson wrote:
> >>
> >> Hi,
> >>>
> >>> This is mainly a background question, but is prompted by observations I
> >>> have had while working in the code base. The general question is, can
> the
> >>> project be simpler to work on and the build speed improved by removing
> or
> >>> redrawing module boundaries?
> >>>
> >>> An example of this is what has been undertaken in Archiva recently,
> where
> >>> redback has been pulled up into the project. This essentially makes
> >>> redback
> >>> part of the project so there it can more easily track with the project.
> >>> This was something I didn't appreciate until we had to consider
> reworking
> >>> all of the Struts 2 actions, etc. to update it for security reasons. It
> >>> would be nice to just replace what redback was doing with spring
> security,
> >>> but moving incrementally is more difficult with redback as a separate
> >>> project.
> >>>
> >>> Another example is whether the degree to which project modules make
> sense
> >>> as modules. In the extreme, we might consider regrouping the existing
> >>> classes into external API, internal API, core implementation, and
> maybe a
> >>> few more. The goal would be to reduce the number of independent
> modules to
> >>> as close to 1 as possible. I'm curious about this because it seems the
> >>> coarser granularity could simplify the project considerably and
> possibly
> >>> speed up the compile/test cycle. I tried looking through the archive
> for
> >>> similar conversations and couldn't find any.
> >>>
> >>> Have these been considered? Thoughts?
> >>>
> >>
> >>
> >> Christian Gruber - christianedwardgruber@gmail.com
> >>
>
>

Re: Project module granularity

[Brett Porter <br...@apache.org>]

Agree with what has been said here, and thanks for raising the topic.

I don't think there's been an explicit discussion about it within Continuum, and I think your observations are correct - many modules have gone well over the deep end in being split up, while there are things like continuum-core that contain the kitchen sink (and then some :) There are other problems too - like a lot of code duplication out of core into the build agents, and some tests that are slow due to using network or SCM operations.

The split is always a trade off. I think it's good to consolidate along the lines you described, though I'd still make special purpose code (particularly with external dependencies) separate - things like the notifier implementations, and release code.

Did you want to draft a module hierarchy that we can talk through?

- Brett

On 19 May 2014, at 4:38 am, Brent Atkinson <br...@gmail.com> wrote:

> Christian,
> 
> Thank you for chiming in. I wanted to raise this for general feedback, and
> I think we're somewhat safe from Parkinson's law of triviality. I don't
> think we run too great a risk of bogging down in discussion about this,
> especially with this list's volume at the moment.
> 
> From my point of view, it helps to know that someone's experience agrees
> with this in concept, even if it isn't based on direct experience with
> Continuum or Archiva.  In my opinion the number of modules makes
> development more cumbersome all around, from building to IDE integration,
> to developing content.
> 
> I also raised it hoping someone with more direct experience than I have
> might be able to offer insight into what value this would have, and maybe
> suggest how it might make sense to tackle. I'd be very interesting in
> tackling something like this myself if it makes sense.
> 
> Brent
> 
> 
> On Sun, May 18, 2014 at 12:37 PM, Christian Gruber <cg...@israfil.net>wrote:
> 
>> As an effective outsider (haven't really dealt with continuum or archive
>> in a long time) I can back up this suggestion from my own experience with
>> maven modules.  I have found that artifact divisions on component lines can
>> be useful, but overkill, and a public API, (if relevant) SPI for extenders,
>> and internal implementation artifacts are a useful division that balances
>> things out and makes builds faster, etc.
>> 
>> It is also useful that artifacts split on major boundaries of
>> cohesion/coupling where reuse is desired. That is, if something is
>> realistically suitable for re-use in other projects, it's reasonable to
>> have an artifact on it - but at that point it really becomes another
>> project.  When to do that is really an art.  But I tend to try to reserve
>> such extraction to really highly valuable libraries where the bang-for-buck
>> ratio is high enough to justify the extra overhead (conceptual and
>> computational in the build).
>> 
>> /shrug - sorry to bike shed, but I think the idea has merit and no one was
>> really responding to Brent.
>> 
>> christian.
>> 
>> 
>> On 17 May 2014, at 10:02, Brent Atkinson wrote:
>> 
>> Hi,
>>> 
>>> This is mainly a background question, but is prompted by observations I
>>> have had while working in the code base. The general question is, can the
>>> project be simpler to work on and the build speed improved by removing or
>>> redrawing module boundaries?
>>> 
>>> An example of this is what has been undertaken in Archiva recently, where
>>> redback has been pulled up into the project. This essentially makes
>>> redback
>>> part of the project so there it can more easily track with the project.
>>> This was something I didn't appreciate until we had to consider reworking
>>> all of the Struts 2 actions, etc. to update it for security reasons. It
>>> would be nice to just replace what redback was doing with spring security,
>>> but moving incrementally is more difficult with redback as a separate
>>> project.
>>> 
>>> Another example is whether the degree to which project modules make sense
>>> as modules. In the extreme, we might consider regrouping the existing
>>> classes into external API, internal API, core implementation, and maybe a
>>> few more. The goal would be to reduce the number of independent modules to
>>> as close to 1 as possible. I'm curious about this because it seems the
>>> coarser granularity could simplify the project considerably and possibly
>>> speed up the compile/test cycle. I tried looking through the archive for
>>> similar conversations and couldn't find any.
>>> 
>>> Have these been considered? Thoughts?
>>> 
>> 
>> 
>> Christian Gruber - christianedwardgruber@gmail.com
>> 

Re: Project module granularity

[Brent Atkinson <br...@gmail.com>]

Christian,

Thank you for chiming in. I wanted to raise this for general feedback, and
I think we're somewhat safe from Parkinson's law of triviality. I don't
think we run too great a risk of bogging down in discussion about this,
especially with this list's volume at the moment.

>From my point of view, it helps to know that someone's experience agrees
with this in concept, even if it isn't based on direct experience with
Continuum or Archiva.  In my opinion the number of modules makes
development more cumbersome all around, from building to IDE integration,
to developing content.

I also raised it hoping someone with more direct experience than I have
might be able to offer insight into what value this would have, and maybe
suggest how it might make sense to tackle. I'd be very interesting in
tackling something like this myself if it makes sense.

Brent


On Sun, May 18, 2014 at 12:37 PM, Christian Gruber <cg...@israfil.net>wrote:

> As an effective outsider (haven't really dealt with continuum or archive
> in a long time) I can back up this suggestion from my own experience with
> maven modules.  I have found that artifact divisions on component lines can
> be useful, but overkill, and a public API, (if relevant) SPI for extenders,
> and internal implementation artifacts are a useful division that balances
> things out and makes builds faster, etc.
>
> It is also useful that artifacts split on major boundaries of
> cohesion/coupling where reuse is desired. That is, if something is
> realistically suitable for re-use in other projects, it's reasonable to
> have an artifact on it - but at that point it really becomes another
> project.  When to do that is really an art.  But I tend to try to reserve
> such extraction to really highly valuable libraries where the bang-for-buck
> ratio is high enough to justify the extra overhead (conceptual and
> computational in the build).
>
> /shrug - sorry to bike shed, but I think the idea has merit and no one was
> really responding to Brent.
>
> christian.
>
>
> On 17 May 2014, at 10:02, Brent Atkinson wrote:
>
>  Hi,
>>
>> This is mainly a background question, but is prompted by observations I
>> have had while working in the code base. The general question is, can the
>> project be simpler to work on and the build speed improved by removing or
>> redrawing module boundaries?
>>
>> An example of this is what has been undertaken in Archiva recently, where
>> redback has been pulled up into the project. This essentially makes
>> redback
>> part of the project so there it can more easily track with the project.
>> This was something I didn't appreciate until we had to consider reworking
>> all of the Struts 2 actions, etc. to update it for security reasons. It
>> would be nice to just replace what redback was doing with spring security,
>> but moving incrementally is more difficult with redback as a separate
>> project.
>>
>> Another example is whether the degree to which project modules make sense
>> as modules. In the extreme, we might consider regrouping the existing
>> classes into external API, internal API, core implementation, and maybe a
>> few more. The goal would be to reduce the number of independent modules to
>> as close to 1 as possible. I'm curious about this because it seems the
>> coarser granularity could simplify the project considerably and possibly
>> speed up the compile/test cycle. I tried looking through the archive for
>> similar conversations and couldn't find any.
>>
>> Have these been considered? Thoughts?
>>
>
>
> Christian Gruber - christianedwardgruber@gmail.com
>

Re: Project module granularity

[Christian Gruber <cg...@israfil.net>]

As an effective outsider (haven't really dealt with continuum or archive 
in a long time) I can back up this suggestion from my own experience 
with maven modules.  I have found that artifact divisions on component 
lines can be useful, but overkill, and a public API, (if relevant) SPI 
for extenders, and internal implementation artifacts are a useful 
division that balances things out and makes builds faster, etc.

It is also useful that artifacts split on major boundaries of 
cohesion/coupling where reuse is desired. That is, if something is 
realistically suitable for re-use in other projects, it's reasonable to 
have an artifact on it - but at that point it really becomes another 
project.  When to do that is really an art.  But I tend to try to 
reserve such extraction to really highly valuable libraries where the 
bang-for-buck ratio is high enough to justify the extra overhead 
(conceptual and computational in the build).

/shrug - sorry to bike shed, but I think the idea has merit and no one 
was really responding to Brent.

christian.

On 17 May 2014, at 10:02, Brent Atkinson wrote:

> Hi,
>
> This is mainly a background question, but is prompted by observations 
> I
> have had while working in the code base. The general question is, can 
> the
> project be simpler to work on and the build speed improved by removing 
> or
> redrawing module boundaries?
>
> An example of this is what has been undertaken in Archiva recently, 
> where
> redback has been pulled up into the project. This essentially makes 
> redback
> part of the project so there it can more easily track with the 
> project.
> This was something I didn't appreciate until we had to consider 
> reworking
> all of the Struts 2 actions, etc. to update it for security reasons. 
> It
> would be nice to just replace what redback was doing with spring 
> security,
> but moving incrementally is more difficult with redback as a separate
> project.
>
> Another example is whether the degree to which project modules make 
> sense
> as modules. In the extreme, we might consider regrouping the existing
> classes into external API, internal API, core implementation, and 
> maybe a
> few more. The goal would be to reduce the number of independent 
> modules to
> as close to 1 as possible. I'm curious about this because it seems the
> coarser granularity could simplify the project considerably and 
> possibly
> speed up the compile/test cycle. I tried looking through the archive 
> for
> similar conversations and couldn't find any.
>
> Have these been considered? Thoughts?


Christian Gruber - christianedwardgruber@gmail.com