You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by Juha Ranta <ju...@iki.fi> on 2008/02/29 20:24:25 UTC
Continuous integration and deciding whether to build
Hello,
As an introduction, I have an existing build system that I'm trying to
improve by using Ivy. I think Ivy is very cool, so big thanks for it!
I have a build server with projects checked out from CVS in one
directory like this:
projectA/
projectB/
projectC/
projectD/
....
Each project has in its root directory a build.xml and ivy.xml which
also come from CVS.
Each of these projects can build, tag, etc, itself and finally publish
its artifact to an Ivy repository independently without directly
referencing other projects. Now I'd like to implement a simple script
which polls CVS and builds automatically builds new artifacts from
projects which, well, need to be built. I can quite easily get a list
of these projects with the Ant task buildlist sorted so that, for
example, if A depends on B, B gets built and published before A.
During the buildlist processing, the script goes through each of the
projects to see if there are any changes in CVS and if there are, the
project gets built. This works fine except that there are also the
dynamic "latest.integration" revisions in the ivy.xmls and the script
doesn't know if some if the dynamic revisions have changed by just
polling CVS.
I think that my script would work like I wanted it to if it would
trigger a new build if the project has changed in CVS since the last
build *or* if the dynamic revisions have changed. Therefore I'm now
wondering what's the best way to check whether the dynamic revisions
have changed. My script can get to the "static" ivy.xml of the
previous build. It perhaps would work if the script executed
"resolve", "delivered" a new static ivy.xml and then somehow diffed
the static dependencies of the previous static ivy.xml to the new
static ivy.xml to decide whether a new build should be triggered.
Are the some easy ways to check if the dependencies of the latest
resolved ivy.xml differ from those of a previously resolved ivy.xml?
Or you think my approach is way off and that I should do it some other
way?
Re: Continuous integration and deciding whether to build
Posted by Xavier Hanin <xa...@gmail.com>.
Besides the chicken and egg problem and continuous integration server
integration which would in itself deserve a lot of discussion, since this is
something that is not addressed properly in any continuous integration
server AFAIK, for the question of knowing if your dependencies have changed
you may be able to use the ivy.deps.changed property that the resolve task
if dependencies have changed. But this require to use the same resolution
cache between each resolve process, which may be difficult on a continuous
integration server.
Otherwise I would go the way that has already been suggested, storing the
dependency report for comparison with the new one. If your dependency
metadata in the repository is using only fixed version, you can even limit
the first resolve (used only to know if something has changed) to the direct
dependencies (disable transitivity) to improve performance.
Xavier
On Sat, Mar 1, 2008 at 12:20 AM, Juha Ranta <ju...@iki.fi> wrote:
>
> I know pretty well what I'd like to do: check if there are any changes
> in CVS or in the resolved dependencies since the last build, and if
> there are, trigger a new build. However, there are kind of chicken and
> egg problems because, for instance, to see if there are changes in the
> dependencies, I need to do a resolve, and typically that resolve is
> done during the Ant build task. It seems that to get this to work the
> way I want, I'd need to insert the checking part inside the project's
> Ant build, which complicates the build script, or else do one resolve
> outside the Ant build and then another during the actual build.
>
> I could also do it so that when a project finally does its Ivy
> publish, it generates a buildlist from the leaf to the root and then
> evokes the build scripts of those projects that depend on it, but this
> leaf build never gets triggered if there is no other process polling
> CVS..
>
> I'll have to think about this more later, but not right now, since
> it's Friday! Thanks to all for all of the help.
>
> On 29.2.2008, at 23:56, Shawn Castrianni wrote:
>
> > Here is link to that discussion:
> >
> > http://www.mail-archive.com/ivy-user@ant.apache.org/msg00185.html
> >
> >
> > ---
> > Shawn Castrianni
> >
> >
> > -----Original Message-----
> > From: Juha Ranta [mailto:juha.ranta@iki.fi]
> > Sent: Friday, February 29, 2008 3:53 PM
> > To: ivy-user@ant.apache.org
> > Subject: Re: Continuous integration and deciding whether to build
> >
> >
> > On 29.2.2008, at 23:37, Shawn Castrianni wrote:
> >
> >> You probably could somehow use the ivy:repreport task to get xml
> >> version of reporting metadata about what versions are out there and
> >> stuff. Then you could use the oopsconsultancy xmltask to scan the
> >> xml report files to check if something has changed. Obviously,
> >> there are a lot of details I am leaving out, but I did post a
> >> previous message to this mailing list where I did something similar
> >> and showed my ANT source code for it. I used the report task and
> >> scanned the xml report file with the xmltask.
> >
> > Thanks! I'll look to it.
> >
> > ----------------------------------------------------------------------
> > This e-mail, including any attached files, may contain confidential
> > and privileged information for the sole use of the intended
> > recipient. Any review, use, distribution, or disclosure by others
> > is strictly prohibited. If you are not the intended recipient (or
> > authorized to receive information for the intended recipient),
> > please contact the sender by reply e-mail and delete all copies of
> > this message.
>
>
--
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://ant.apache.org/ivy/
http://www.xoocode.org/
Re: Continuous integration and deciding whether to build
Posted by Juha Ranta <ju...@iki.fi>.
I know pretty well what I'd like to do: check if there are any changes
in CVS or in the resolved dependencies since the last build, and if
there are, trigger a new build. However, there are kind of chicken and
egg problems because, for instance, to see if there are changes in the
dependencies, I need to do a resolve, and typically that resolve is
done during the Ant build task. It seems that to get this to work the
way I want, I'd need to insert the checking part inside the project's
Ant build, which complicates the build script, or else do one resolve
outside the Ant build and then another during the actual build.
I could also do it so that when a project finally does its Ivy
publish, it generates a buildlist from the leaf to the root and then
evokes the build scripts of those projects that depend on it, but this
leaf build never gets triggered if there is no other process polling
CVS..
I'll have to think about this more later, but not right now, since
it's Friday! Thanks to all for all of the help.
On 29.2.2008, at 23:56, Shawn Castrianni wrote:
> Here is link to that discussion:
>
> http://www.mail-archive.com/ivy-user@ant.apache.org/msg00185.html
>
>
> ---
> Shawn Castrianni
>
>
> -----Original Message-----
> From: Juha Ranta [mailto:juha.ranta@iki.fi]
> Sent: Friday, February 29, 2008 3:53 PM
> To: ivy-user@ant.apache.org
> Subject: Re: Continuous integration and deciding whether to build
>
>
> On 29.2.2008, at 23:37, Shawn Castrianni wrote:
>
>> You probably could somehow use the ivy:repreport task to get xml
>> version of reporting metadata about what versions are out there and
>> stuff. Then you could use the oopsconsultancy xmltask to scan the
>> xml report files to check if something has changed. Obviously,
>> there are a lot of details I am leaving out, but I did post a
>> previous message to this mailing list where I did something similar
>> and showed my ANT source code for it. I used the report task and
>> scanned the xml report file with the xmltask.
>
> Thanks! I'll look to it.
>
> ----------------------------------------------------------------------
> This e-mail, including any attached files, may contain confidential
> and privileged information for the sole use of the intended
> recipient. Any review, use, distribution, or disclosure by others
> is strictly prohibited. If you are not the intended recipient (or
> authorized to receive information for the intended recipient),
> please contact the sender by reply e-mail and delete all copies of
> this message.
RE: Continuous integration and deciding whether to build
Posted by Shawn Castrianni <Sh...@halliburton.com>.
Here is link to that discussion:
http://www.mail-archive.com/ivy-user@ant.apache.org/msg00185.html
---
Shawn Castrianni
-----Original Message-----
From: Juha Ranta [mailto:juha.ranta@iki.fi]
Sent: Friday, February 29, 2008 3:53 PM
To: ivy-user@ant.apache.org
Subject: Re: Continuous integration and deciding whether to build
On 29.2.2008, at 23:37, Shawn Castrianni wrote:
> You probably could somehow use the ivy:repreport task to get xml
> version of reporting metadata about what versions are out there and
> stuff. Then you could use the oopsconsultancy xmltask to scan the
> xml report files to check if something has changed. Obviously,
> there are a lot of details I am leaving out, but I did post a
> previous message to this mailing list where I did something similar
> and showed my ANT source code for it. I used the report task and
> scanned the xml report file with the xmltask.
Thanks! I'll look to it.
----------------------------------------------------------------------
This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.
Re: Continuous integration and deciding whether to build
Posted by Juha Ranta <ju...@iki.fi>.
On 29.2.2008, at 23:37, Shawn Castrianni wrote:
> You probably could somehow use the ivy:repreport task to get xml
> version of reporting metadata about what versions are out there and
> stuff. Then you could use the oopsconsultancy xmltask to scan the
> xml report files to check if something has changed. Obviously,
> there are a lot of details I am leaving out, but I did post a
> previous message to this mailing list where I did something similar
> and showed my ANT source code for it. I used the report task and
> scanned the xml report file with the xmltask.
Thanks! I'll look to it.
RE: Continuous integration and deciding whether to build
Posted by Shawn Castrianni <Sh...@halliburton.com>.
You probably could somehow use the ivy:repreport task to get xml version of reporting metadata about what versions are out there and stuff. Then you could use the oopsconsultancy xmltask to scan the xml report files to check if something has changed. Obviously, there are a lot of details I am leaving out, but I did post a previous message to this mailing list where I did something similar and showed my ANT source code for it. I used the report task and scanned the xml report file with the xmltask.
---
Shawn Castrianni
-----Original Message-----
From: Juha Ranta [mailto:juha.ranta@iki.fi]
Sent: Friday, February 29, 2008 3:25 PM
To: ivy-user@ant.apache.org
Subject: Re: Continuous integration and deciding whether to build
On 29.2.2008, at 23:02, Troy Bull wrote:
> On Fri, Feb 29, 2008 at 1:24 PM, Juha Ranta <ju...@iki.fi> wrote:
>> Hello,
>>
>> As an introduction, I have an existing build system that I'm trying
>> to
>> improve by using Ivy. I think Ivy is very cool, so big thanks for it!
>>
>> I have a build server with projects checked out from CVS in one
>> directory like this:
>>
>> projectA/
>> projectB/
>> projectC/
>> projectD/
>> ....
>
>
> Have you looked at CruiseControl? Thats what we use to do something
> very similar to what you are wondering..
>
> troy
I've looked at CruiseControl but not actually used it. I think that a
build server would be a better option than using some Ant Contrib
tasks to make Ant do things it wasn't really created for. However, it
seems to be a somewhat tedious process at my current workplace to
introduce new products such as CruiseControl. Secondly, I'd like to
first create a simple demonstration of how the build system could
work. Thirdly, even though I haven't looked that much in CruiseControl
or other build servers, I don't see how they could see if the resolved
"latest.integration" dependencies have changed since the previous
build, unless someone has written a plugin or something equivalent
especially for Ivy. I'm interested if there are such plugins for any
build servers.
----------------------------------------------------------------------
This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.
Re: Continuous integration and deciding whether to build
Posted by Juha Ranta <ju...@iki.fi>.
On 29.2.2008, at 23:02, Troy Bull wrote:
> On Fri, Feb 29, 2008 at 1:24 PM, Juha Ranta <ju...@iki.fi> wrote:
>> Hello,
>>
>> As an introduction, I have an existing build system that I'm trying
>> to
>> improve by using Ivy. I think Ivy is very cool, so big thanks for it!
>>
>> I have a build server with projects checked out from CVS in one
>> directory like this:
>>
>> projectA/
>> projectB/
>> projectC/
>> projectD/
>> ....
>
>
> Have you looked at CruiseControl? Thats what we use to do something
> very similar to what you are wondering..
>
> troy
I've looked at CruiseControl but not actually used it. I think that a
build server would be a better option than using some Ant Contrib
tasks to make Ant do things it wasn't really created for. However, it
seems to be a somewhat tedious process at my current workplace to
introduce new products such as CruiseControl. Secondly, I'd like to
first create a simple demonstration of how the build system could
work. Thirdly, even though I haven't looked that much in CruiseControl
or other build servers, I don't see how they could see if the resolved
"latest.integration" dependencies have changed since the previous
build, unless someone has written a plugin or something equivalent
especially for Ivy. I'm interested if there are such plugins for any
build servers.
Re: Continuous integration and deciding whether to build
Posted by Troy Bull <tr...@gmail.com>.
On Fri, Feb 29, 2008 at 1:24 PM, Juha Ranta <ju...@iki.fi> wrote:
> Hello,
>
> As an introduction, I have an existing build system that I'm trying to
> improve by using Ivy. I think Ivy is very cool, so big thanks for it!
>
> I have a build server with projects checked out from CVS in one
> directory like this:
>
> projectA/
> projectB/
> projectC/
> projectD/
> ....
Have you looked at CruiseControl? Thats what we use to do something
very similar to what you are wondering..
troy
Re: Continuous integration and deciding whether to build
Posted by Juha Ranta <ju...@iki.fi>.
On 29.2.2008, at 23:55, Nicolas Lalevée wrote:
> ah OK, I was thinking that the dynamic revision you talk about was a
> project one. So your issue is in fact with your repository.
I want to create "snapshots" or "baselines" of my project. I want to
create "snapshots" so that I can reproduce my build from the source,
branch that snapshot to a release branch, etc. If I tag a project with
an ivy.xml that has dynamic dependencies in it, I can't reproduce that
build with the tag alone. I've already figured out a way so that I can
reproduce my builds, but I haven't yet got the builds to figure out
when there's new dependencies to build against.
>
>
> First, maybe it is a bad idea to have lastest.integration on a
> repository, see Stephane's mail in the "ivy integration with
> automated build tools" thread in this mailing list.
> If you really want to deal with it, maybe you can try to work with
> the cache which holds that last dependency report. First you save
> the old report, then you trigger an ivy:resolve which will generate
> another report, and finally we could compare the both report to see
> if something changed. And it might be interesting to have a such
> feature in Ivy if you are willing to contribute :)
>
I'll try to see that thread, but I'm not putting ivy files that say
"latest.integration" in a repository. However, I do think that I'll
put the artifacts created by CI server to repository and even tag
them. This will create a lot of artifacts, tags and other mess, but
it's possible to clean these. I like this approach over creating Maven
style snapshot artifacs for various reasons.
>
> Nicolas
>
Re: Continuous integration and deciding whether to build
Posted by Nicolas Lalevée <ni...@anyware-tech.com>.
Le 29 févr. 08 à 22:12, Juha Ranta a écrit :
>
> Hi Nicolas,
>
> On 29.2.2008, at 22:40, Nicolas Lalevée wrote:
>>
>> I have no answer but a question :)
>> It seems strange to me of having a dynamic revision that does not
>> change between two builds. What is the rationale behind it ?
>>
>> Nicolas
>>
>
>
> Each project in CVS has its own ivy.xml in the root directory. As an
> example, let's consider that the mainline development of a projects
> takes place in the trunk of CVS and the ivy.xml in this projects
> says "latest.integration" as the revision for some of the
> dependencies. When the build server checks out the project from CVS,
> it also gets the ivy.xml with the "latest.integration"
> dependencies". However, when Ivy resolves the "latest.integration"
> dependencies, those resolved dependencies get an actual revision.
> For example, the build may resolve and retrieve revision 13 of
> dom4.jar and use that during the build. When the next build is
> triggered, it may be that there's revision 14 of dom4j.jar in the
> repository, so this new build will retrieve revision 14 of
> dom4j.jar. The "dynamic" ivy.xml of the project in CVS has not
> changed at all, but the generated "static" ivy.xml has different
> revisions in the dom4j.jar dependency.
>
> During my builds from the build server, my script actually checks in
> the generated "static" ivy.xml to CVS, tags it with the build tag,
> and then checks in the original "dynamic" ivy.xml. This way the
> builds are reproducible and this is the foundation from which I
> create release or bug fix branches, for instance. I know it's not
> necessary to make this kind of "snapshots" of every single
> continuous integration build like I'm doing, but making this kind of
> "snapshots" at milestones is very important if you, for instance,
> possible need to make make bug fixes on earlier versions of your
> project while you're already developing new stuff.
ah OK, I was thinking that the dynamic revision you talk about was a
project one. So your issue is in fact with your repository.
First, maybe it is a bad idea to have lastest.integration on a
repository, see Stephane's mail in the "ivy integration with automated
build tools" thread in this mailing list.
If you really want to deal with it, maybe you can try to work with the
cache which holds that last dependency report. First you save the old
report, then you trigger an ivy:resolve which will generate another
report, and finally we could compare the both report to see if
something changed. And it might be interesting to have a such feature
in Ivy if you are willing to contribute :)
Nicolas
Re: Continuous integration and deciding whether to build
Posted by Juha Ranta <ju...@iki.fi>.
Hi Nicolas,
On 29.2.2008, at 22:40, Nicolas Lalevée wrote:
>
> I have no answer but a question :)
> It seems strange to me of having a dynamic revision that does not
> change between two builds. What is the rationale behind it ?
>
> Nicolas
>
Each project in CVS has its own ivy.xml in the root directory. As an
example, let's consider that the mainline development of a projects
takes place in the trunk of CVS and the ivy.xml in this projects says
"latest.integration" as the revision for some of the dependencies.
When the build server checks out the project from CVS, it also gets
the ivy.xml with the "latest.integration" dependencies". However, when
Ivy resolves the "latest.integration" dependencies, those resolved
dependencies get an actual revision. For example, the build may
resolve and retrieve revision 13 of dom4.jar and use that during the
build. When the next build is triggered, it may be that there's
revision 14 of dom4j.jar in the repository, so this new build will
retrieve revision 14 of dom4j.jar. The "dynamic" ivy.xml of the
project in CVS has not changed at all, but the generated "static"
ivy.xml has different revisions in the dom4j.jar dependency.
During my builds from the build server, my script actually checks in
the generated "static" ivy.xml to CVS, tags it with the build tag, and
then checks in the original "dynamic" ivy.xml. This way the builds are
reproducible and this is the foundation from which I create release or
bug fix branches, for instance. I know it's not necessary to make this
kind of "snapshots" of every single continuous integration build like
I'm doing, but making this kind of "snapshots" at milestones is very
important if you, for instance, possible need to make make bug fixes
on earlier versions of your project while you're already developing
new stuff.
Re: Continuous integration and deciding whether to build
Posted by Nicolas Lalevée <ni...@anyware-tech.com>.
Le 29 févr. 08 à 20:24, Juha Ranta a écrit :
> Hello,
>
> As an introduction, I have an existing build system that I'm trying
> to improve by using Ivy. I think Ivy is very cool, so big thanks for
> it!
>
> I have a build server with projects checked out from CVS in one
> directory like this:
>
> projectA/
> projectB/
> projectC/
> projectD/
> ....
>
> Each project has in its root directory a build.xml and ivy.xml which
> also come from CVS.
>
> Each of these projects can build, tag, etc, itself and finally
> publish its artifact to an Ivy repository independently without
> directly referencing other projects. Now I'd like to implement a
> simple script which polls CVS and builds automatically builds new
> artifacts from projects which, well, need to be built. I can quite
> easily get a list of these projects with the Ant task buildlist
> sorted so that, for example, if A depends on B, B gets built and
> published before A. During the buildlist processing, the script goes
> through each of the projects to see if there are any changes in CVS
> and if there are, the project gets built. This works fine except
> that there are also the dynamic "latest.integration" revisions in
> the ivy.xmls and the script doesn't know if some if the dynamic
> revisions have changed by just polling CVS.
>
> I think that my script would work like I wanted it to if it would
> trigger a new build if the project has changed in CVS since the last
> build *or* if the dynamic revisions have changed. Therefore I'm now
> wondering what's the best way to check whether the dynamic revisions
> have changed. My script can get to the "static" ivy.xml of the
> previous build. It perhaps would work if the script executed
> "resolve", "delivered" a new static ivy.xml and then somehow diffed
> the static dependencies of the previous static ivy.xml to the new
> static ivy.xml to decide whether a new build should be triggered.
>
> Are the some easy ways to check if the dependencies of the latest
> resolved ivy.xml differ from those of a previously resolved ivy.xml?
> Or you think my approach is way off and that I should do it some
> other way?
I have no answer but a question :)
It seems strange to me of having a dynamic revision that does not
change between two builds. What is the rationale behind it ?
Nicolas