You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tuscany.apache.org by Simon Laws <si...@googlemail.com> on 2009/11/17 16:20:54 UTC
[DISCUSS] 2.x build structure
I raise this topic with some trepidation as we've been here before but
here goes. We have been building (and releasing) all of the code in
the 2.x code base which has worked well to date. However two things
make me pause for thought:
- we are not running the otests as part of the main build and we are
already seeing regressions because of this (this will happen more
often as more otests are brought in). This is not good.
- my 2.x build (without otests) is already up to 45 minutes and there
are still more extensions to bring back in
Now, there may some tuning I can do on my machine to improve build
performance but I would like to experiment with building the core
modules, along with a core set of extensions and associated otests
independently of other extensions.
Of course there is nothing stopping me doing this experiment locally
(and I will) but there are wider implications for this that I would
like us to discuss in parallel. My primary concern being that we can
produce and test a well defined extension SPI (which I don't think we
are very good at) and also separate those modules which we think are
becoming stable from those that are not ready yet.
I have avoided using words like features, shades, distributions etc
here on purpose. These may all be part of the answer. I've also not
touched on the release structure as I would like to keep that
separate. In the first instance I'm interested in peoples comments on
the principle of a more structured build.
Regards
Simon
--
Apache Tuscany committer: tuscany.apache.org
Co-author of a book about Tuscany and SCA: tuscanyinaction.com
Re: [DISCUSS] 2.x build structure
Posted by ant elder <an...@gmail.com>.
On Tue, Nov 17, 2009 at 3:20 PM, Simon Laws <si...@googlemail.com> wrote:
> - my 2.x build (without otests) is already up to 45 minutes and there
> are still more extensions to bring back in
>
It takes similar or even longer on my old machine, and thats with the
local repo already populated.
Doing a fresh checkout and build with an empty or out-of-date local
repository can take hours. Thats too long and i think one of the
reasons we find it so hard to attracted new committers - if someone
sets aside a few hours in an evening or weekend to play with Tuscany
and they can't even get a build done in that time then its going to be
difficult for them to contribute much. And thats not just newbies,
I've heard recently from a few experienced but less active Tuscany
people that they're afraid to do an svn up as they expect to then need
to spend hours or even days getting their environment working again.
> Now, there may some tuning I can do on my machine to improve build
> performance but I would like to experiment with building the core
> modules, along with a core set of extensions and associated otests
> independently of other extensions.
>
Giorgio tried something similar not so long ago with a Maven profile
for just some core modules and based on that I've added a "base"
profile and a "jms" profile - you can try those in the Tuscany top
level folder doing mvn -Pbase or mvn -Pjms or mvn -Pbase,jms. I've
also tried an alternative with svn externals so you can checkout and
build just a subset of the modules needed to run the SCA-J CAA tests -
https://svn.apache.org/repos/asf/tuscany/sandbox/scajcaa/
> In the first instance I'm interested in peoples comments on
> the principle of a more structured build.
>
We've seen from 1.x and now 2.x that the current approach doesn't
scale so I'm very much in favour of a more structured build. Those
maven profiles I mentioned above have been interesting but they're not
that usable, I'd like to rearrange the trunk structure with the
core/base and extensions separated and with things like the the
samples and tests for each extension all under its own subtree for the
extension.
...ant
Re: [DISCUSS] 2.x build structure
Posted by ant elder <an...@gmail.com>.
On Mon, Jan 11, 2010 at 9:15 AM, Simon Laws <si...@googlemail.com> wrote:
> Concurrently there are a few other things that are going to impact
> this. For example, we need to bring the otests into the picture as
> these need to be kept running cleanly as changes are made and there
> are undoubtedly things that are still in 1.x that we don't yet have in
> 2.x that we need to introduce
>
Its not just otests and other 1.x stuff - the 2.x code base and build
time will inevitably continue to grow over time as we add new
functions, features, test, samples, demos, tutorials, gsoc projects
etc etc. We can try to tweak the existing tests to make individual
modules build a bit quicker but in the big scheme of things thats just
not going to make much of a long term significant difference (IMHO).
> So coming back to the post that started this thread I'm still
> interested in looking at some profiles etc. to see if we can ease the
> pain. I'll experiment when I have some time.
>
Me too. A while back i added some build profiles based on what Giorgio
had been suggesting, right now theres just 'base' and 'jms' profiles,
I used them for a little while but found they didn't really work that
well for day to day dev use, has anyone else tried them? I'll help
with tinkering with profiles to see if they can be made more usable,
for example we can probably at least do things with them now that can
be used with Hudson to make the nightly builds more robust, but I'm
increasingly coming to the view that the only way we'll get a
significantly better development experience is by restructuring svn so
things are grouped into more separate functional areas.
...ant
Re: [DISCUSS] 2.x build structure
Posted by Simon Laws <si...@googlemail.com>.
Yep, starting to look more reasonable for the full build. After the
recent changes, and some changes to the way my machine is set up, I'm
starting to get better build times, although not quite as quick as
listed, and I'm sure we can do better.
So I think looking for further technical optimizations as you suggest
is an excellent idea.
Concurrently there are a few other things that are going to impact
this. For example, we need to bring the otests into the picture as
these need to be kept running cleanly as changes are made and there
are undoubtedly things that are still in 1.x that we don't yet have in
2.x that we need to introduce
So coming back to the post that started this thread I'm still
interested in looking at some profiles etc. to see if we can ease the
pain. I'll experiment when I have some time.
Regards
Simon
Re: [DISCUSS] 2.x build structure
Posted by Luciano Resende <lu...@gmail.com>.
I have spent some time looking at the current times of our 2.x build,
and have created a wiki page with the collected data [1]. Looks like
the latest changes from Simon around the distribution gave us couple
minutes back on the build execution, which is always good.
As a first step, I'd suggest we evaluate the projects/tests that are
taking the longest to build investigate possible optimizations.
Thoughts ?
[1] http://cwiki.apache.org/confluence/display/TUSCANYWIKI/2.x+Tuscany+Build+Performance
--
Luciano Resende
http://people.apache.org/~lresende
http://lresende.blogspot.com/
Re: [DISCUSS] 2.x build structure
Posted by Simon Laws <si...@googlemail.com>.
hmmm, ok let me take a look.
Simon
Re: [DISCUSS] 2.x build structure
Posted by Luciano Resende <lu...@gmail.com>.
On Tue, Nov 24, 2009 at 4:46 AM, Simon Laws <si...@googlemail.com> wrote:
> At r883670 I've changed the build so that mvn will not now build
> distribution archives and hence itest/distribution won't try and unzip
> an archive.
>
> To generate the distribution archives use mvn -Pdistribution. This
> also makes itest/distribution unpack the archive so is very similar to
> the scheme that we are using as default now.
>
> Knocks a few minutes off the build for me.
>
> Regards
>
> Simon
>
It looks like there are still places on the build relying on the
distribution, this is what I get with a build after removing
~/m2/repository/org/apache/tuscany/sca
[INFO] Failed to resolve artifact.
Missing:
----------
1) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
Path to dependency:
1) org.apache.tuscany.sca:itest-distribution-binding-ws-calculator:jar:2.0-SNAPSHOT
2) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
----------
1 required artifact is missing.
for artifact:
org.apache.tuscany.sca:itest-distribution-binding-ws-calculator:jar:2.0-SNAPSHOT
from the specified remote repositories:
indiana (http://ftp.ussg.iu.edu/eclipse/modeling/emf/emf/maven2/),
apache.snapshots (http://people.apache.org/repo/m2-snapshot-repository),
java.net (http://download.java.net/maven/1),
intalio.org (http://www.intalio.org/public/maven2),
maven.central (http://repo2.maven.org/maven2),
tuscany.repo (http://svn.apache.org/repos/asf/tuscany/maven),
apache.ws.zone (http://ws.zones.apache.org/repository2),
osuosl.org (http://ftp.osuosl.org/pub/eclipse/tools/emf/maven2),
central (http://repo1.maven.org/maven2),
apache.incubator (http://people.apache.org/repo/m2-incubating-repository),
java.net2 (http://download.java.net/maven/2)
[INFO] ------------------------------------------------------------------------
[INFO] Error for project: Apache Tuscany SCA iTest Distribution
calculator-rmi (during install)
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.
Missing:
----------
1) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
Path to dependency:
1) org.apache.tuscany.sca:itest-distribution-calculator-rmi:jar:2.0-SNAPSHOT
2) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
----------
1 required artifact is missing.
for artifact:
org.apache.tuscany.sca:itest-distribution-calculator-rmi:jar:2.0-SNAPSHOT
from the specified remote repositories:
indiana (http://ftp.ussg.iu.edu/eclipse/modeling/emf/emf/maven2/),
apache.snapshots (http://people.apache.org/repo/m2-snapshot-repository),
java.net (http://download.java.net/maven/1),
intalio.org (http://www.intalio.org/public/maven2),
maven.central (http://repo2.maven.org/maven2),
tuscany.repo (http://svn.apache.org/repos/asf/tuscany/maven),
apache.ws.zone (http://ws.zones.apache.org/repository2),
osuosl.org (http://ftp.osuosl.org/pub/eclipse/tools/emf/maven2),
central (http://repo1.maven.org/maven2),
apache.incubator (http://people.apache.org/repo/m2-incubating-repository),
java.net2 (http://download.java.net/maven/2)
[INFO] ------------------------------------------------------------------------
[INFO] Error for project: Apache Tuscany SCA iTest Distribution
implementation-java-calculator (during install)
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.
Missing:
----------
1) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
Path to dependency:
1) org.apache.tuscany.sca:itest-distribution-implementation-java-calculator:jar:2.0-SNAPSHOT
2) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
----------
1 required artifact is missing.
for artifact:
org.apache.tuscany.sca:itest-distribution-implementation-java-calculator:jar:2.0-SNAPSHOT
from the specified remote repositories:
indiana (http://ftp.ussg.iu.edu/eclipse/modeling/emf/emf/maven2/),
apache.snapshots (http://people.apache.org/repo/m2-snapshot-repository),
java.net (http://download.java.net/maven/1),
intalio.org (http://www.intalio.org/public/maven2),
maven.central (http://repo2.maven.org/maven2),
tuscany.repo (http://svn.apache.org/repos/asf/tuscany/maven),
apache.ws.zone (http://ws.zones.apache.org/repository2),
osuosl.org (http://ftp.osuosl.org/pub/eclipse/tools/emf/maven2),
central (http://repo1.maven.org/maven2),
apache.incubator (http://people.apache.org/repo/m2-incubating-repository),
java.net2 (http://download.java.net/maven/2)
[INFO] ------------------------------------------------------------------------
[INFO] Error for project: Apache Tuscany SCA iTest Distribution Legal
Checks (during install)
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.
Missing:
----------
1) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
Path to dependency:
1) org.apache.tuscany.sca:itest-distribution-legal-checks:jar:2.0-SNAPSHOT
2) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
----------
1 required artifact is missing.
for artifact:
org.apache.tuscany.sca:itest-distribution-legal-checks:jar:2.0-SNAPSHOT
from the specified remote repositories:
indiana (http://ftp.ussg.iu.edu/eclipse/modeling/emf/emf/maven2/),
apache.snapshots (http://people.apache.org/repo/m2-snapshot-repository),
java.net (http://download.java.net/maven/1),
intalio.org (http://www.intalio.org/public/maven2),
maven.central (http://repo2.maven.org/maven2),
tuscany.repo (http://svn.apache.org/repos/asf/tuscany/maven),
apache.ws.zone (http://ws.zones.apache.org/repository2),
osuosl.org (http://ftp.osuosl.org/pub/eclipse/tools/emf/maven2),
central (http://repo1.maven.org/maven2),
apache.incubator (http://people.apache.org/repo/m2-incubating-repository),
java.net2 (http://download.java.net/maven/2)
[INFO] ------------------------------------------------------------------------
[INFO] Error for project: Apache Tuscany SCA iTest Distribution Webapp
helloworld (during install)
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.
Missing:
----------
1) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
Path to dependency:
1) org.apache.tuscany.sca:itest-distribution-webapp-helloworld:jar:2.0-SNAPSHOT
2) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
----------
1 required artifact is missing.
for artifact:
org.apache.tuscany.sca:itest-distribution-webapp-helloworld:jar:2.0-SNAPSHOT
from the specified remote repositories:
indiana (http://ftp.ussg.iu.edu/eclipse/modeling/emf/emf/maven2/),
apache.snapshots (http://people.apache.org/repo/m2-snapshot-repository),
java.net (http://download.java.net/maven/1),
intalio.org (http://www.intalio.org/public/maven2),
maven.central (http://repo2.maven.org/maven2),
tuscany.repo (http://svn.apache.org/repos/asf/tuscany/maven),
apache.ws.zone (http://ws.zones.apache.org/repository2),
osuosl.org (http://ftp.osuosl.org/pub/eclipse/tools/emf/maven2),
central (http://repo1.maven.org/maven2),
apache.incubator (http://people.apache.org/repo/m2-incubating-repository),
java.net2 (http://download.java.net/maven/2)
[INFO] ------------------------------------------------------------------------
[INFO] Error for project: Apache Tuscany SCA iTest Distribution Webapp
helloworld-bpel (during install)
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.
Missing:
----------
1) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
Path to dependency:
1) org.apache.tuscany.sca:itest-distribution-webapp-helloworld-bpel:jar:2.0-SNAPSHOT
2) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
----------
1 required artifact is missing.
for artifact:
org.apache.tuscany.sca:itest-distribution-webapp-helloworld-bpel:jar:2.0-SNAPSHOT
from the specified remote repositories:
indiana (http://ftp.ussg.iu.edu/eclipse/modeling/emf/emf/maven2/),
apache.snapshots (http://people.apache.org/repo/m2-snapshot-repository),
java.net (http://download.java.net/maven/1),
intalio.org (http://www.intalio.org/public/maven2),
maven.central (http://repo2.maven.org/maven2),
tuscany.repo (http://svn.apache.org/repos/asf/tuscany/maven),
apache.ws.zone (http://ws.zones.apache.org/repository2),
osuosl.org (http://ftp.osuosl.org/pub/eclipse/tools/emf/maven2),
central (http://repo1.maven.org/maven2),
apache.incubator (http://people.apache.org/repo/m2-incubating-repository),
java.net2 (http://download.java.net/maven/2)
[INFO] ------------------------------------------------------------------------
[INFO] Error for project: Apache Tuscany SCA iTest Distribution Webapp
helloworld-js-client (during install)
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.
Missing:
----------
1) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
Path to dependency:
1) org.apache.tuscany.sca:itest-distribution-webapp-helloworld-js-client:jar:2.0-SNAPSHOT
2) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
----------
1 required artifact is missing.
for artifact:
org.apache.tuscany.sca:itest-distribution-webapp-helloworld-js-client:jar:2.0-SNAPSHOT
from the specified remote repositories:
indiana (http://ftp.ussg.iu.edu/eclipse/modeling/emf/emf/maven2/),
apache.snapshots (http://people.apache.org/repo/m2-snapshot-repository),
java.net (http://download.java.net/maven/1),
intalio.org (http://www.intalio.org/public/maven2),
maven.central (http://repo2.maven.org/maven2),
tuscany.repo (http://svn.apache.org/repos/asf/tuscany/maven),
apache.ws.zone (http://ws.zones.apache.org/repository2),
osuosl.org (http://ftp.osuosl.org/pub/eclipse/tools/emf/maven2),
central (http://repo1.maven.org/maven2),
apache.incubator (http://people.apache.org/repo/m2-incubating-repository),
java.net2 (http://download.java.net/maven/2)
[INFO] ------------------------------------------------------------------------
[INFO] Error for project: Apache Tuscany SCA iTest Distribution Webapp
helloworld-jsp (during install)
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.
Missing:
----------
1) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
Path to dependency:
1) org.apache.tuscany.sca:itest-distribution-webapp-helloworld-jsp:jar:2.0-SNAPSHOT
2) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
----------
1 required artifact is missing.
for artifact:
org.apache.tuscany.sca:itest-distribution-webapp-helloworld-jsp:jar:2.0-SNAPSHOT
from the specified remote repositories:
indiana (http://ftp.ussg.iu.edu/eclipse/modeling/emf/emf/maven2/),
apache.snapshots (http://people.apache.org/repo/m2-snapshot-repository),
java.net (http://download.java.net/maven/1),
intalio.org (http://www.intalio.org/public/maven2),
maven.central (http://repo2.maven.org/maven2),
tuscany.repo (http://svn.apache.org/repos/asf/tuscany/maven),
apache.ws.zone (http://ws.zones.apache.org/repository2),
osuosl.org (http://ftp.osuosl.org/pub/eclipse/tools/emf/maven2),
central (http://repo1.maven.org/maven2),
apache.incubator (http://people.apache.org/repo/m2-incubating-repository),
java.net2 (http://download.java.net/maven/2)
[INFO] ------------------------------------------------------------------------
[INFO] Error for project: Apache Tuscany SCA iTest Distribution Webapp
helloworld-servlet (during install)
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.
Missing:
----------
1) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
Path to dependency:
1) org.apache.tuscany.sca:itest-distribution-webapp-helloworld-servlet:jar:2.0-SNAPSHOT
2) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
----------
1 required artifact is missing.
for artifact:
org.apache.tuscany.sca:itest-distribution-webapp-helloworld-servlet:jar:2.0-SNAPSHOT
from the specified remote repositories:
indiana (http://ftp.ussg.iu.edu/eclipse/modeling/emf/emf/maven2/),
apache.snapshots (http://people.apache.org/repo/m2-snapshot-repository),
java.net (http://download.java.net/maven/1),
intalio.org (http://www.intalio.org/public/maven2),
maven.central (http://repo2.maven.org/maven2),
tuscany.repo (http://svn.apache.org/repos/asf/tuscany/maven),
apache.ws.zone (http://ws.zones.apache.org/repository2),
osuosl.org (http://ftp.osuosl.org/pub/eclipse/tools/emf/maven2),
central (http://repo1.maven.org/maven2),
apache.incubator (http://people.apache.org/repo/m2-incubating-repository),
java.net2 (http://download.java.net/maven/2)
[INFO] ------------------------------------------------------------------------
[INFO] Error for project: Apache Tuscany SCA iTest Distribution Webapp
helloworld-stripes (during install)
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.
Missing:
----------
1) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
Path to dependency:
1) org.apache.tuscany.sca:itest-distribution-webapp-helloworld-stripes:jar:2.0-SNAPSHOT
2) org.apache.tuscany.sca:itest-bin-distro-unzip:pom:2.0-SNAPSHOT
----------
1 required artifact is missing.
for artifact:
org.apache.tuscany.sca:itest-distribution-webapp-helloworld-stripes:jar:2.0-SNAPSHOT
from the specified remote repositories:
indiana (http://ftp.ussg.iu.edu/eclipse/modeling/emf/emf/maven2/),
apache.snapshots (http://people.apache.org/repo/m2-snapshot-repository),
java.net (http://download.java.net/maven/1),
intalio.org (http://www.intalio.org/public/maven2),
maven.central (http://repo2.maven.org/maven2),
tuscany.repo (http://svn.apache.org/repos/asf/tuscany/maven),
apache.ws.zone (http://ws.zones.apache.org/repository2),
osuosl.org (http://ftp.osuosl.org/pub/eclipse/tools/emf/maven2),
central (http://repo1.maven.org/maven2),
apache.incubator (http://people.apache.org/repo/m2-incubating-repository),
java.net2 (http://download.java.net/maven/2)
--
Luciano Resende
http://people.apache.org/~lresende
http://lresende.blogspot.com/
Re: [DISCUSS] 2.x build structure
Posted by Simon Laws <si...@googlemail.com>.
Also re. the distribution build change. I haven't yet made any changes
(consolidation or otherwise) to the various files that the new
bin-default.xml and bind-distribution.xml files refer to. I think Ant
mentioned this. Doesn't mean I don't think it's a good idea. Just
haven't done anything yet.
Simon
Re: [DISCUSS] 2.x build structure
Posted by Simon Laws <si...@googlemail.com>.
At r883670 I've changed the build so that mvn will not now build
distribution archives and hence itest/distribution won't try and unzip
an archive.
To generate the distribution archives use mvn -Pdistribution. This
also makes itest/distribution unpack the archive so is very similar to
the scheme that we are using as default now.
Knocks a few minutes off the build for me.
Regards
Simon
Re: [DISCUSS] 2.x build structure
Posted by Simon Laws <si...@googlemail.com>.
> And I agree with all that too. The itest\distribution\bin-distro-unzip
> unzips the zip anyway so it would be simpler to just replace that
> itest with <format>dir</format>. Where it starts to go wrong though
> is if there is separate assembly plugin bin.xml files, one for just
> the dir and another for dir, zip, tar. A way around that would be
> changing the <componentDescriptors> element to have a single generic
> componentDescriptor that both bin.xmls could share.
>
+1, if we have to have different config we've sunk.
Simon
Re: [DISCUSS] 2.x build structure
Posted by ant elder <an...@apache.org>.
On Fri, Nov 20, 2009 at 10:10 AM, Simon Laws <si...@googlemail.com> wrote:
>>
>> A comment on that is that if there are different configurations for
>> releases and regular builds then theres a good chance they will get
>> out of sync and we dont notice the difference until a release is being
>> reviewed so we lose a lot of the point of testing the distribution.
>> Thats exactly what used to happen and its one of the reasons we're now
>> doing it the way it is. It does seem a waste to be building the src
>> distro and then not testing it though, so if there is a way of only
>> doing the binary one and using the same bin.xml file that would be
>> good.
>
> I think there is a difference between three different scenarios we
> have mentioned.
>
> 1/ no automatic testing of distribution and samples. - the very
> painful 1.x scenario
> 2/ automatic testing of the distribution and samples including
> building all the archive formats - the current 2.x scenario
> 3/ automatic testing of the distribution and samples including
> building building only the zip archive format that is actually tests -
> my proposal.
>
> I believe there is a huge value of 3 over 1
> I believe there is no value of 2 over 3 (other than, in 3 we
> demonstrate that the tools are able to build tar.gz, -src.tar.gz,
> -srz.zip. Big deal as we don't actually test those archives)
>
Yep I agree with you (if that wasnt clear from my previous reply). I
was just adding that it would be good to try to find a way that uses
the same assembly plugin bin.xml file for both.
>>
>> A question is should we bother building the distribution as part of
>> the build? The original reason for that was so all the samples could
>> be tested from the distribution, but samples being added now are not
>> getting associated distribution itests so maybe we shouldnt bother.
>>
>> ...ant
>>
>
> I think we should fix the fact that the samples aren't included. IMHO
> any time we loose during the build (assuming we can get it under
> control) would be less that the time we take fixing the build up for
> each release. Originally when we set off down this path we have the
> assembly plugin configured to just generate the distro in directory
> form without building any archives. I'd be happy with that. If we find
> a problem related to building the archive and then unpacking it again
> it is unlikely to need a Tuscany fix.
>
And I agree with all that too. The itest\distribution\bin-distro-unzip
unzips the zip anyway so it would be simpler to just replace that
itest with <format>dir</format>. Where it starts to go wrong though
is if there is separate assembly plugin bin.xml files, one for just
the dir and another for dir, zip, tar. A way around that would be
changing the <componentDescriptors> element to have a single generic
componentDescriptor that both bin.xmls could share.
...ant
Re: [DISCUSS] 2.x build structure
Posted by Simon Laws <si...@googlemail.com>.
>
> A comment on that is that if there are different configurations for
> releases and regular builds then theres a good chance they will get
> out of sync and we dont notice the difference until a release is being
> reviewed so we lose a lot of the point of testing the distribution.
> Thats exactly what used to happen and its one of the reasons we're now
> doing it the way it is. It does seem a waste to be building the src
> distro and then not testing it though, so if there is a way of only
> doing the binary one and using the same bin.xml file that would be
> good.
I think there is a difference between three different scenarios we
have mentioned.
1/ no automatic testing of distribution and samples. - the very
painful 1.x scenario
2/ automatic testing of the distribution and samples including
building all the archive formats - the current 2.x scenario
3/ automatic testing of the distribution and samples including
building building only the zip archive format that is actually tests -
my proposal.
I believe there is a huge value of 3 over 1
I believe there is no value of 2 over 3 (other than, in 3 we
demonstrate that the tools are able to build tar.gz, -src.tar.gz,
-srz.zip. Big deal as we don't actually test those archives)
>
> A question is should we bother building the distribution as part of
> the build? The original reason for that was so all the samples could
> be tested from the distribution, but samples being added now are not
> getting associated distribution itests so maybe we shouldnt bother.
>
> ...ant
>
I think we should fix the fact that the samples aren't included. IMHO
any time we loose during the build (assuming we can get it under
control) would be less that the time we take fixing the build up for
each release. Originally when we set off down this path we have the
assembly plugin configured to just generate the distro in directory
form without building any archives. I'd be happy with that. If we find
a problem related to building the archive and then unpacking it again
it is unlikely to need a Tuscany fix.
Simon
Re: [DISCUSS] 2.x build structure
Posted by ant elder <an...@gmail.com>.
On Fri, Nov 20, 2009 at 9:20 AM, Simon Laws <si...@googlemail.com> wrote:
> I've done a bit of tidying, defragging etc on my machine and have got
> it down to approx 30 mins. Sometimes a little less sometimes a little
> more. Better but still too long and no where near the kind of time
> Luciano is getting.
>
> One of the outstanding issues is...
>
> [INFO] Apache Tuscany SCA All-in-one Distribution ............ SUCCESS
> [9:20.406s]
>
> We're building 4 distribution archives when we actually only test the
> bin zip archive. We could do with some kind of profile to separate
> building that one from building all of then for a full release.
>
> I'll experiment.
>
> Simon
>
A comment on that is that if there are different configurations for
releases and regular builds then theres a good chance they will get
out of sync and we dont notice the difference until a release is being
reviewed so we lose a lot of the point of testing the distribution.
Thats exactly what used to happen and its one of the reasons we're now
doing it the way it is. It does seem a waste to be building the src
distro and then not testing it though, so if there is a way of only
doing the binary one and using the same bin.xml file that would be
good.
A question is should we bother building the distribution as part of
the build? The original reason for that was so all the samples could
be tested from the distribution, but samples being added now are not
getting associated distribution itests so maybe we shouldnt bother.
...ant
Re: [DISCUSS] 2.x build structure
Posted by Simon Laws <si...@googlemail.com>.
I've done a bit of tidying, defragging etc on my machine and have got
it down to approx 30 mins. Sometimes a little less sometimes a little
more. Better but still too long and no where near the kind of time
Luciano is getting.
One of the outstanding issues is...
[INFO] Apache Tuscany SCA All-in-one Distribution ............ SUCCESS
[9:20.406s]
We're building 4 distribution archives when we actually only test the
bin zip archive. We could do with some kind of profile to separate
building that one from building all of then for a full release.
I'll experiment.
Simon
Re: [DISCUSS] 2.x build structure
Posted by Mike Edwards <mi...@gmail.com>.
Luciano Resende wrote:
> On Tue, Nov 17, 2009 at 7:20 AM, Simon Laws <si...@googlemail.com> wrote:
>> I raise this topic with some trepidation as we've been here before but
>> here goes. We have been building (and releasing) all of the code in
>> the 2.x code base which has worked well to date. However two things
>> make me pause for thought:
>>
>> - we are not running the otests as part of the main build and we are
>> already seeing regressions because of this (this will happen more
>> often as more otests are brought in). This is not good.
>> - my 2.x build (without otests) is already up to 45 minutes and there
>> are still more extensions to bring back in
>>
>
> Here is some numbers that can help in case we want to optimize the
> current build as a first step.
> http://people.apache.org/~lresende/tuscany/Tuscany%20Build.pdf
>
> This is based on my build machine, which is taking 12 mins for a full 2.x build
>
> [INFO] Total time: 12 minutes 10 seconds
> [INFO] Finished at: Tue Nov 17 08:07:47 PST 2009
> [INFO] Final Memory: 102M/210M
> [INFO] ------------------------------------------------------------------------
>
>
Folks,
I can see that Luciano has access to the Los Alamos supercomputer that does the nuclear simulations
;-) Can we have a go please?
The BPEL integration tests in 6 milliseconds is astoundingly good - how do you get that Luciano? My
figures running the BPEL iTest on its own give this:
Apache Tuscany SCA BPEL Integration Tests ............. SUCCESS [1.687s]
Apache Tuscany SCA iTest HelloWorld BPEL .............. SUCCESS [18.328s]
This is on a relatively speedy Lenovo T61p with 3Gb RAM under Win XP. Even the iTest HelloWorld
BPEL is 3x faster on Luciano's system. I am using Sun JDK 1.6.0_13.
One thing I note about the build is that we build all the Samples and run all the iTests.
I think that we should not build and test Samples by default. Let them be built on demand only.
No problem with running tests, but I note that even on Luciano's system, the tests seem to dominate
the larger times in the list. Are we running too many tests - and are the tests poorly structured?
We should give some thought to this.
Yours, Mike.
Re: [DISCUSS] 2.x build structure
Posted by Luciano Resende <lu...@gmail.com>.
On Tue, Nov 17, 2009 at 11:45 AM, Simon Laws <si...@googlemail.com> wrote:
> Firstly, what sort of machine do you have Luciano (I gotta get me one
> of those!) Secondly, do you have any specific configuration or tuning
> that allows you to get the build done in 12 minutes?
>
> Simon
>
The 12 mins build was on a shared desktop machine I have from my
employer, and to do a fair comparison I tried the same thing in my
laptop (MacBook Pro 2.53 GHZ with 4GB RAM) and below are the results
of a full 2.x build (mvn -fae clean install)
INFO] ------------------------------------------------------------------------
[INFO] Total time: 14 minutes 3 seconds
[INFO] Finished at: Tue Nov 17 15:22:05 PST 2009
[INFO] Final Memory: 179M/335M
[INFO] ------------------------------------------------------------------------
Note that I have no specific tweaks or special tunings, except for
using -o to stay offline and speed up things.
--
Luciano Resende
http://people.apache.org/~lresende
http://lresende.blogspot.com/
Re: [DISCUSS] 2.x build structure
Posted by Simon Laws <si...@googlemail.com>.
Firstly, what sort of machine do you have Luciano (I gotta get me one
of those!) Secondly, do you have any specific configuration or tuning
that allows you to get the build done in 12 minutes?
Simon
Re: [DISCUSS] 2.x build structure
Posted by Luciano Resende <lu...@gmail.com>.
On Tue, Nov 17, 2009 at 7:20 AM, Simon Laws <si...@googlemail.com> wrote:
> I raise this topic with some trepidation as we've been here before but
> here goes. We have been building (and releasing) all of the code in
> the 2.x code base which has worked well to date. However two things
> make me pause for thought:
>
> - we are not running the otests as part of the main build and we are
> already seeing regressions because of this (this will happen more
> often as more otests are brought in). This is not good.
> - my 2.x build (without otests) is already up to 45 minutes and there
> are still more extensions to bring back in
>
Here is some numbers that can help in case we want to optimize the
current build as a first step.
http://people.apache.org/~lresende/tuscany/Tuscany%20Build.pdf
This is based on my build machine, which is taking 12 mins for a full 2.x build
[INFO] Total time: 12 minutes 10 seconds
[INFO] Finished at: Tue Nov 17 08:07:47 PST 2009
[INFO] Final Memory: 102M/210M
[INFO] ------------------------------------------------------------------------
--
Luciano Resende
http://people.apache.org/~lresende
http://lresende.blogspot.com/