Multi-module builds in Continuum

[Marica Tan <ct...@exist.com>]

Hi everyone,

Current Scenario:
When changes/updates occured in between builds of a multi-module project,
some modules failed to build because of those changes not being reflected.

Let say we have a scheduled build of a multi-module project with sub-project
A, B and C, where C is dependent on A.
B is currently building when changes was made on A and C. So when C gets to
build it fails because changes on A wasn't reflected.


Our group here in Exist had some discussions on how to approach this
problem, which somehow also addresses
http://jira.codehaus.org/browse/CONTINUUM-798.
So far we've came up with a list of things that we would like to propose.

1. This is the current scenario when building a multi-module project: for
every module, the steps performed are checkout/update then build. What we're
proposing is to checkout/update group atomically before builds, meaning
there will only be one checkout/update and this would be at the group level.
The build order is still like the current process (by dependency order), the
only difference is that the checkout/update will be performed only once at
the group level and at the start of the group build.

2. Notification will be moved at the end of the group build.

3. Reflect the changes in UI. Maybe a tree view.

4. Add features related to building

5. Aggregated build result/in progress

6. Need skip/don't build state that retains previous state for statistics.

7. Cancel build is still per project. When a build is cancelled, it will
trigger a skip.
   - can set a flag that the build was cancelled / skipped, but the state is
still the previous state.

8. Add group cancel

9. Add transient state

10. Decision to build is still per project

11. Track hierarchy in config


Comments, suggestions, violent reactions are welcome :)

Thanks,
Marica

Re: Multi-module builds in Continuum

[Brett Porter <br...@apache.org>]

On 16/05/2008, at 6:06 AM, Emmanuel Venisse wrote:

> I think it would be better to use only one working copy for a whole
> multi-module project. With this, we'll save lot of space on the disk  
> and for
> each multi-module project we'd can do a single build with the  
> recursive
> mode.

Agreed - though I think the "recursive" mode should remain optional.

>
>
> An other point is we won't have a build in the middle and we'll can  
> support
> correctly multi-module projects with flat structure.

I still like the module-by-module build - just having one checkout  
means that you only build which modules changes (or their dependencies  
did), rather than rebuilding the parent and everything every time.

> I don't know yet if we can implement this in 1.X or if it would be  
> better to
> wait 2.0 because it is a big change.

This is one of my goals of the SCM changes :) Since it's mostly  
internal, I think it can be done in 1.x.

Cheers,
Brett

--
Brett Porter
brett@apache.org
http://blogs.exist.com/bporter/

Re: Multi-module builds in Continuum

[Emmanuel Venisse <em...@gmail.com>]

I think it would be better to use only one working copy for a whole
multi-module project. With this, we'll save lot of space on the disk and for
each multi-module project we'd can do a single build with the recursive
mode.

An other point is we won't have a build in the middle and we'll can support
correctly multi-module projects with flat structure.

I don't know yet if we can implement this in 1.X or if it would be better to
wait 2.0 because it is a big change.

Emmanuel

On Wed, Apr 30, 2008 at 10:01 AM, Marica Tan <ct...@exist.com> wrote:

> On Fri, Apr 18, 2008 at 12:26 PM, Marica Tan <ct...@exist.com> wrote:
>
> > Hi everyone,
> >
> > Current Scenario:
> > When changes/updates occured in between builds of a multi-module project,
> > some modules failed to build because of those changes not being
> reflected.
> >
> > Let say we have a scheduled build of a multi-module project with
> > sub-project A, B and C, where C is dependent on A.
> > B is currently building when changes was made on A and C. So when C gets
> > to build it fails because changes on A wasn't reflected.
> >
> >
> > Our group here in Exist had some discussions on how to approach this
> > problem, which somehow also addresses
> > http://jira.codehaus.org/browse/CONTINUUM-798.
> > So far we've came up with a list of things that we would like to propose.
> >
> > 1. This is the current scenario when building a multi-module project: for
> > every module, the steps performed are checkout/update then build. What
> we're
> > proposing is to checkout/update group atomically before builds, meaning
> > there will only be one checkout/update and this would be at the group
> level.
> > The build order is still like the current process (by dependency order),
> the
> > only difference is that the checkout/update will be performed only once
> at
> > the group level and at the start of the group build.
> >
> > 2. Notification will be moved at the end of the group build.
> >
> > 3. Reflect the changes in UI. Maybe a tree view.
> >
> > 4. Add features related to building
>
>
> > 5. Aggregated build result/in progress
>
>
> This is actually under number 4.  There will only be one build result per
> group instead of having one per project.
>
> >
> > 6. Need skip/don't build state that retains previous state for
> statistics.
> >
> > 7. Cancel build is still per project. When a build is cancelled, it will
> > trigger a skip.
> >    - can set a flag that the build was cancelled / skipped, but the state
> > is still the previous state.
> >
> > 8. Add group cancel
> >
> > 9. Add transient state
> >
>
> I'll create another thread for this.
>
> >
> > 10. Decision to build is still per project
> >
>
> This is just a note.
>
> >
> > 11. Track hierarchy in config
> >
>
> A project should be aware of its parent and its children or sub projects to
> handle building of project group that has more than one pom with modules.
>
> A
> |__B
> |    |__C
> |    |__D
> |__E
>
> From the figure above, after D finishes building, I could return to A and
> build E.
>
>
> >
> > Comments, suggestions, violent reactions are welcome :)
> >
> > Thanks,
> > Marica
> >
>

Re: Multi-module builds in Continuum

[Marica Tan <ct...@exist.com>]

On Fri, Apr 18, 2008 at 12:26 PM, Marica Tan <ct...@exist.com> wrote:

> Hi everyone,
>
> Current Scenario:
> When changes/updates occured in between builds of a multi-module project,
> some modules failed to build because of those changes not being reflected.
>
> Let say we have a scheduled build of a multi-module project with
> sub-project A, B and C, where C is dependent on A.
> B is currently building when changes was made on A and C. So when C gets
> to build it fails because changes on A wasn't reflected.
>
>
> Our group here in Exist had some discussions on how to approach this
> problem, which somehow also addresses
> http://jira.codehaus.org/browse/CONTINUUM-798.
> So far we've came up with a list of things that we would like to propose.
>
> 1. This is the current scenario when building a multi-module project: for
> every module, the steps performed are checkout/update then build. What we're
> proposing is to checkout/update group atomically before builds, meaning
> there will only be one checkout/update and this would be at the group level.
> The build order is still like the current process (by dependency order), the
> only difference is that the checkout/update will be performed only once at
> the group level and at the start of the group build.
>
> 2. Notification will be moved at the end of the group build.
>
> 3. Reflect the changes in UI. Maybe a tree view.
>
> 4. Add features related to building


> 5. Aggregated build result/in progress


This is actually under number 4.  There will only be one build result per
group instead of having one per project.

>
> 6. Need skip/don't build state that retains previous state for statistics.
>
> 7. Cancel build is still per project. When a build is cancelled, it will
> trigger a skip.
>    - can set a flag that the build was cancelled / skipped, but the state
> is still the previous state.
>
> 8. Add group cancel
>
> 9. Add transient state
>

I'll create another thread for this.

>
> 10. Decision to build is still per project
>

This is just a note.

>
> 11. Track hierarchy in config
>

A project should be aware of its parent and its children or sub projects to
handle building of project group that has more than one pom with modules.

A
|__B
|    |__C
|    |__D
|__E

>From the figure above, after D finishes building, I could return to A and
build E.


>
> Comments, suggestions, violent reactions are welcome :)
>
> Thanks,
> Marica
>

Re: Multi-module builds in Continuum

[Wendy Smoak <ws...@gmail.com>]

On Fri, Apr 18, 2008 at 12:26 PM, Marica Tan <ct...@exist.com> wrote:

>  Current Scenario:
>  When changes/updates occured in between builds of a multi-module project,
>  some modules failed to build because of those changes not being reflected.
>
>  Let say we have a scheduled build of a multi-module project with sub-project
>  A, B and C, where C is dependent on A.
>  B is currently building when changes was made on A and C. So when C gets to
>  build it fails because changes on A wasn't reflected.

Okay...

>  1. This is the current scenario when building a multi-module project: for
>  every module, the steps performed are checkout/update then build. What we're
>  proposing is to checkout/update group atomically before builds, meaning
>  there will only be one checkout/update and this would be at the group level.
>  The build order is still like the current process (by dependency order), the
>  only difference is that the checkout/update will be performed only once at
>  the group level and at the start of the group build.
>
>  2. Notification will be moved at the end of the group build.
>
>  3. Reflect the changes in UI. Maybe a tree view.

I'm with you up to this point.  And 6-8 seem fairly straightforward.
But I'm not really sure what the rest of the things on the list mean.
Can you add more details?  If you think it's something that needs
discussion, separate threads might work better, or just go ahead and
open the JIRA issues.

>  4. Add features related to building
>
>  5. Aggregated build result/in progress
>
>  6. Need skip/don't build state that retains previous state for statistics.
>
>  7. Cancel build is still per project. When a build is cancelled, it will
>  trigger a skip.
>    - can set a flag that the build was cancelled / skipped, but the state is
>  still the previous state.
>
>  8. Add group cancel
>
>  9. Add transient state
>
>  10. Decision to build is still per project
>
>  11. Track hierarchy in config

Thanks,
-- 
Wendy