Cleaning up the build

[ant elder <an...@gmail.com>]

I'm wondering if its got to the stage where we need to do something about
our builds as they take too long and are always broken.

The continuum builds almost never work, nor does a local build for me
usually, its got to the stage i think people often don't even bother trying
one before committing which is just exasperating the problem. Even on the
rare occasions when it is building cleanly it takes so long to run i'm
guessing most of us often don't run a build before committing anyway, and
one of the reasons there's seldom a completed 1.x continuum build is that it
takes so long it often gets killed before it finishes.

Right now the 1.x build takes over 70 minutes for me (and continuum), i'd
like to have a go to get that down to under 30 minutes or better. Disabling
the schema validation makes a difference but not enough so i'd like to start
taking things out of the build. I know that sounds a bit drastic but we've
quite a lot of stuff that has never been included in a release, or hasn't
been touched in ages, some doesn't even have any tests so we're just burning
time compiling with no idea if its doing anything useful.

I don't have a list, it would be easier to spend time on and off moving
things out as they're discovered, so would anyone object if this happened
CTR and anything can get put back if theres an issue?

Longer term i think we should think about separating some parts out into
separate builds and releases. Does any want to help or have any other ideas
about how to speed things up and get a more stable build?

   ...ant

Re: Cleaning up the build

[ant elder <an...@apache.org>]

On Tue, Apr 21, 2009 at 10:52 AM, ant elder <an...@apache.org> wrote:

>
> I've started doing some of this and plan to continue slowly over the next
> days. Feel free to help.
>

FYI, I've done most of these changes now moving out code that wasn't
included in the main 1.x build, so if nothing else a fresh svn
checkout is quicker.

The next ones would be to look at moving out code thats in the build
but not included in the binary release, but i'm not sure how much of
that we should do with the 1.5 release about to be taken though so was
thinking about holding off till after that?

It would be good to sort out the build relying on plugins in the build
so the buidl works ok on a fresh checkout, we could release the
plugins separately like 2.x does, or have a -Psetup profile, or just
doc that you need to build a few things first before running a full
build.

   ...ant

Re: Cleaning up the build

[ant elder <an...@apache.org>]

On Sat, Apr 18, 2009 at 9:28 AM, ant elder <an...@apache.org> wrote:

>
>
> On Thu, Apr 16, 2009 at 10:16 PM, Luciano Resende <lu...@gmail.com>wrote:
>
>> On Thu, Apr 16, 2009 at 1:33 PM, ant elder <an...@apache.org> wrote:
>> > Ok so ignoring what and why that may have happened in the past as you
>> were
>> > in favour of this approach before could i assume you yourself will be ok
>> > with it again this time?
>> >
>>
>> I think this would be a good improvement, we just need to come up with
>> a clear strategy for removing the modules and avoid complaints as we
>> had last time.
>>
>> Suggestions for the process ?
>>
>> How to ensure no users are using these modules ?
>>
>>
>> BTW, I'll listen to others process suggestions before giving my
>> opinion, as my approach didn't work very well last time :)
>>
>
> I'm not sure how easy it would be to come up with a precise strategy as
> each case will be slightly different. We should expect to move a few things
> and be asked to move them back without getting heat up about it. I think it
> will be easier now, there's already stuff moved out of 2.x into contrib
> which sets a precedence, just having 2.x changes things as thats where most
> new stuff should be happening anyway. The culture and climate of the project
> is a little different now too which should help too.
>
> So i say JFDI. Anything thats not included in the full build and isn't
> being actively worked on move it out, anything that _is_ in the build but
> isn't usually included in the binary release could be a candidate too. This
> is not a rip-and-replace opportunity though, we shouldn't move something out
> just as we want to rewrite it in our own style, so if something is moved out
> and later on we want to look at similar function again we should try to
> resurrect the old code not start from scratch.
>
>    ...ant
>
>
>
I've started doing some of this and plan to continue slowly over the next
days. Feel free to help.

   ...ant

Re: Cleaning up the build

[ant elder <an...@apache.org>]

On Thu, Apr 16, 2009 at 10:16 PM, Luciano Resende <lu...@gmail.com>wrote:

> On Thu, Apr 16, 2009 at 1:33 PM, ant elder <an...@apache.org> wrote:
> > Ok so ignoring what and why that may have happened in the past as you
> were
> > in favour of this approach before could i assume you yourself will be ok
> > with it again this time?
> >
>
> I think this would be a good improvement, we just need to come up with
> a clear strategy for removing the modules and avoid complaints as we
> had last time.
>
> Suggestions for the process ?
>
> How to ensure no users are using these modules ?
>
>
> BTW, I'll listen to others process suggestions before giving my
> opinion, as my approach didn't work very well last time :)
>

I'm not sure how easy it would be to come up with a precise strategy as each
case will be slightly different. We should expect to move a few things and
be asked to move them back without getting heat up about it. I think it will
be easier now, there's already stuff moved out of 2.x into contrib which
sets a precedence, just having 2.x changes things as thats where most new
stuff should be happening anyway. The culture and climate of the project is
a little different now too which should help too.

So i say JFDI. Anything thats not included in the full build and isn't being
actively worked on move it out, anything that _is_ in the build but isn't
usually included in the binary release could be a candidate too. This is not
a rip-and-replace opportunity though, we shouldn't move something out just
as we want to rewrite it in our own style, so if something is moved out and
later on we want to look at similar function again we should try to
resurrect the old code not start from scratch.

   ...ant

Re: Cleaning up the build

[Luciano Resende <lu...@gmail.com>]

On Thu, Apr 16, 2009 at 1:33 PM, ant elder <an...@apache.org> wrote:
> Ok so ignoring what and why that may have happened in the past as you were
> in favour of this approach before could i assume you yourself will be ok
> with it again this time?
>

I think this would be a good improvement, we just need to come up with
a clear strategy for removing the modules and avoid complaints as we
had last time.

Suggestions for the process ?

How to ensure no users are using these modules ?


BTW, I'll listen to others process suggestions before giving my
opinion, as my approach didn't work very well last time :)


-- 
Luciano Resende
Apache Tuscany, Apache PhotArk
http://people.apache.org/~lresende
http://lresende.blogspot.com/

Re: Cleaning up the build

[ant elder <an...@apache.org>]

On Thu, Apr 16, 2009 at 9:08 PM, Luciano Resende <lu...@gmail.com>wrote:

> On Thu, Apr 16, 2009 at 11:52 AM, ant elder <an...@gmail.com> wrote:
> > I'm wondering if its got to the stage where we need to do something about
> > our builds as they take too long and are always broken.
> >
>
> This is very simmilar to what I have proposed in [1]


There were some fine suggestions there, and over the years there have been
various other suggestion from other people all with pros and cons. This is a
point in time thing, whether or not suggestions have worked or been followed
through in the past is less important that what we can do today. The reality
is that the build takes ages and rarely runs cleanly and that makes day to
day development unnecessarily difficult and it will be off putting for those
thinking about contributing


>
>
> [1] http://www.mail-archive.com/dev@tuscany.apache.org/msg02236.html
>
> > The continuum builds almost never work, nor does a local build for me
> > usually, its got to the stage i think people often don't even bother
> trying
> > one before committing which is just exasperating the problem. Even on the
> > rare occasions when it is building cleanly it takes so long to run i'm
> > guessing most of us often don't run a build before committing anyway, and
> > one of the reasons there's seldom a completed 1.x continuum build is that
> it
> > takes so long it often gets killed before it finishes.
> >
>
> It' s really a shame to hear that people just dump code to trunk
> without building, I think we should rethink this.
>
> > Right now the 1.x build takes over 70 minutes for me (and continuum), i'd
> > like to have a go to get that down to under 30 minutes or better.
> Disabling
> > the schema validation makes a difference but not enough so i'd like to
> start
> > taking things out of the build. I know that sounds a bit drastic but
> we've
> > quite a lot of stuff that has never been included in a release, or hasn't
> > been touched in ages, some doesn't even have any tests so we're just
> burning
> > time compiling with no idea if its doing anything useful.
> >
> > I don't have a list, it would be easier to spend time on and off moving
> > things out as they're discovered, so would anyone object if this happened
> > CTR and anything can get put back if theres an issue?
> >
>
> For some suggestions, see
> http://www.mail-archive.com/dev@tuscany.apache.org/msg02236.html
>
> Last time I tried this CTR approach, I was asked to revert couple
> changes, so I don't think a simple CTR will work.
> See http://markmail.org/message/fypdpbffir4ghwoz
>

Ok so ignoring what and why that may have happened in the past as you were
in favour of this approach before could i assume you yourself will be ok
with it again this time?


> > Longer term i think we should think about separating some parts out into
> > separate builds and releases. Does any want to help or have any other
> ideas
> > about how to speed things up and get a more stable build?
> >
>
> We should think carefully about this, consider what we have in 2.x
> today, where we are having to release various plugins everytime we
> need to do a release. Also, if it's the other way around, where we
> never need to touch the code, it might get forgotten. I gues I would
> like to see a concrete plan of what goes where, before I could make my
> mind.
>

I don't think the plugin releases in 2.x are a problem, they're very easy to
release and having them separate has helped stabilise the 2.x build, i think
its a big improvement this way. Anyway, lets not get this thread too bogged
down in discussion about separate releasables, i wasn't proposing any of
that for right now.

   ...ant

Re: Cleaning up the build

[Luciano Resende <lu...@gmail.com>]

On Thu, Apr 16, 2009 at 11:52 AM, ant elder <an...@gmail.com> wrote:
> I'm wondering if its got to the stage where we need to do something about
> our builds as they take too long and are always broken.
>

This is very simmilar to what I have proposed in [1]

[1] http://www.mail-archive.com/dev@tuscany.apache.org/msg02236.html

> The continuum builds almost never work, nor does a local build for me
> usually, its got to the stage i think people often don't even bother trying
> one before committing which is just exasperating the problem. Even on the
> rare occasions when it is building cleanly it takes so long to run i'm
> guessing most of us often don't run a build before committing anyway, and
> one of the reasons there's seldom a completed 1.x continuum build is that it
> takes so long it often gets killed before it finishes.
>

It' s really a shame to hear that people just dump code to trunk
without building, I think we should rethink this.

> Right now the 1.x build takes over 70 minutes for me (and continuum), i'd
> like to have a go to get that down to under 30 minutes or better. Disabling
> the schema validation makes a difference but not enough so i'd like to start
> taking things out of the build. I know that sounds a bit drastic but we've
> quite a lot of stuff that has never been included in a release, or hasn't
> been touched in ages, some doesn't even have any tests so we're just burning
> time compiling with no idea if its doing anything useful.
>
> I don't have a list, it would be easier to spend time on and off moving
> things out as they're discovered, so would anyone object if this happened
> CTR and anything can get put back if theres an issue?
>

For some suggestions, see
http://www.mail-archive.com/dev@tuscany.apache.org/msg02236.html

Last time I tried this CTR approach, I was asked to revert couple
changes, so I don't think a simple CTR will work.
See http://markmail.org/message/fypdpbffir4ghwoz

> Longer term i think we should think about separating some parts out into
> separate builds and releases. Does any want to help or have any other ideas
> about how to speed things up and get a more stable build?
>

We should think carefully about this, consider what we have in 2.x
today, where we are having to release various plugins everytime we
need to do a release. Also, if it's the other way around, where we
never need to touch the code, it might get forgotten. I gues I would
like to see a concrete plan of what goes where, before I could make my
mind.


-- 
Luciano Resende
Apache Tuscany, Apache PhotArk
http://people.apache.org/~lresende
http://lresende.blogspot.com/