You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by "Roberto C. Sánchez" <ro...@connexer.com> on 2015/04/03 03:01:37 UTC
Difficult with concurrent builds
I have a project using Ivy which is built via Jenkins. I have two
branches for this project (a stable branch and a current development
branch, which is the master branch in Git).
In Jenkins I have one build configured for each branch (I don't want to
mix the build histories). I have a post-commit hook in my Git
repository which checks for the branch(es) of the commit and then uses
to wget to retrieve the URL that triggers the build that corresponds to
the branch of the commit.
The problem that I am experiencing has to do with making changes in
quick succession on the two branches. My Ant build.xml does the normal
build routine and at the end publishes the artifacts into my own local
Ivy repository. When the builds run concurrently (because, for example,
I make a commit one branch and then immediately cherry-pick the commit
to the other branch, the build that finshes last uses the ivy.xml
desriptor from the build that finishes first. This is very frustrating
as it breaks all sorts of things. I can go into Jenkins and force a
rebuild of the one that finished last and everything turns out
correctly, with the correct descriptor being published the second time.
I looked through the Ivy source code and found a reference to a delivery
list file that looks in ${java.io.tmpdir}/delivery.properties. I tried
explicitly setting the ivy.delivery.list.file using Ant's <tempfile/>
task to ensure that concurrent builds won't accidentally touch the same
file. I was able to confirm that instead of /tmp/delivery.properties
that the Ivy publish task was using the randomly generated file name Ant
put into the ivy.delivery.list.file property.
Even with that, though, I am still seeing the buggy behavior. That is,
the build that finishes last uses the same descriptor as the build that
finishes first (line for line identical except for the revision and
publication attributes of the info tag.
Any help on this would be appreciated.
Regards,
-Roberto
--
Roberto C. Sánchez
http://people.connexer.com/~roberto
http://www.connexer.com
RE: Difficult with concurrent builds
Posted by Loren Kratzke <LK...@blueorigin.com>.
Your comment about Ivy not always delivering the latest artifact caught my attention. That in combination with perceived cache issues makes me just shake my head.
Deleting cache directories is only treating the symptoms. When you think about it, what is the point of a cache if you delete it? Furthermore, you consider the cache poison to the build because it is. It's a bad place to be. I have been there my friend.
See my recommendations on how to take absolute control of your dependency resolution and retrieval strategies in Ivy. Use it, like it, then scream it from the mountain tops until it becomes documentation on the site. It's good stuff.
http://mail-archives.apache.org/mod_mbox/ant-ivy-user/201504.mbox/%3C2ffef9f4335143be92cd684f2199808c%40MAIL02.blueorigin.com%3E
L.K.
-----Original Message-----
From: Qazwart [mailto:qazwart@gmail.com]
Sent: Tuesday, April 07, 2015 8:48 AM
To: ivy-user@ant.apache.org
Subject: Re: Difficult with concurrent builds
Here's how I fixed the ivy cache issue with Jenkins:
In my ivysettings.xml:
<ivysettings>
<property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
<caches
defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
<settings defaultResolver="default"/>
<include file="${ivy.dir}/ivysettings-public.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
</ivysettings>
Jenkins sets the "EXECUTOR_NUMBER" environment variable depending on which executor is used. If the build is run locally, the default cache will be $HOME/.ivy2/cache-0.
I ran into this cache issue because we clean out the cache from our locally built jars with each build. This guarantees we're working with the latest version of those jars (something that Ivy didn't always do all the time). Deleting jars while an Ivy build is running doesn't make Jenkins very happy. It took us a while to realize why builds were failing.
--
David Weintraub
qazwart@gmail.com
perl -e 'print "Just another second rate Perl Hacker\n";'
> On Apr 7, 2015, at 9:58 AM, Roberto C. Sánchez <ro...@connexer.com> wrote:
>
> On Tue, Apr 07, 2015 at 09:53:08AM +0200, Geißler, Daniel wrote:
>> Hi Roberto,
>>
>> did you have a look into locking strategies, as far as i know ivy does no locking by default wich may lead to your problems:
>>
>> http://ant.apache.org/ivy/history/latest-milestone/settings/lock-stra
>> tegies.html
>>
> Hi Daniel,
>
> Thank you for responding. This appears to be a low traffic list :-)
>
> I did not realize that multiple builds sharing the cache could present
> a problem. In any event, each of my builds is configured to use its
> own Ivy cache directory based on the Jenkins workspace directory. I
> don't think that affects the part of the build with which I am
> currently experiencing a problem. Additionally, the documentation
> link you provided does not give an example of how I would use the
> locking strategy.
>
> I think that the problem has something to do with the deliver/publish
> logic, but I cannot seem to figure out precisely what or where. I
> have not observed any problem with any of my downloaded artifacts for
> the dependencies. I have only seen a problem with the
> delivered/published ivy.xml pertaining to the project being built at
> the same time as it is being built by another process (but on a different branch).
>
> Regards,
>
> -Roberto
> --
> Roberto C. Sánchez
> http://people.connexer.com/~roberto
> http://www.connexer.com
Re: Difficult with concurrent builds
Posted by Qazwart <qa...@gmail.com>.
Here’s how I fixed the ivy cache issue with Jenkins:
In my ivysettings.xml:
<ivysettings>
<property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
<caches
defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
<settings defaultResolver="default"/>
<include file="${ivy.dir}/ivysettings-public.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
</ivysettings>
Jenkins sets the “EXECUTOR_NUMBER” environment variable depending on which executor is used. If the build is run locally, the default cache will be $HOME/.ivy2/cache-0.
I ran into this cache issue because we clean out the cache from our locally built jars with each build. This guarantees we’re working with the latest version of those jars (something that Ivy didn’t always do all the time). Deleting jars while an Ivy build is running doesn’t make Jenkins very happy. It took us a while to realize why builds were failing.
--
David Weintraub
qazwart@gmail.com
perl -e 'print "Just another second rate Perl Hacker\n";'
> On Apr 7, 2015, at 9:58 AM, Roberto C. Sánchez <ro...@connexer.com> wrote:
>
> On Tue, Apr 07, 2015 at 09:53:08AM +0200, Geißler, Daniel wrote:
>> Hi Roberto,
>>
>> did you have a look into locking strategies, as far as i know ivy does no locking by default wich may lead to your problems:
>>
>> http://ant.apache.org/ivy/history/latest-milestone/settings/lock-strategies.html
>>
> Hi Daniel,
>
> Thank you for responding. This appears to be a low traffic list :-)
>
> I did not realize that multiple builds sharing the cache could present a
> problem. In any event, each of my builds is configured to use its own
> Ivy cache directory based on the Jenkins workspace directory. I don't
> think that affects the part of the build with which I am currently
> experiencing a problem. Additionally, the documentation link you
> provided does not give an example of how I would use the locking
> strategy.
>
> I think that the problem has something to do with the deliver/publish
> logic, but I cannot seem to figure out precisely what or where. I have
> not observed any problem with any of my downloaded artifacts for the
> dependencies. I have only seen a problem with the delivered/published
> ivy.xml pertaining to the project being built at the same time as it is
> being built by another process (but on a different branch).
>
> Regards,
>
> -Roberto
> --
> Roberto C. Sánchez
> http://people.connexer.com/~roberto
> http://www.connexer.com
Re: Difficult with concurrent builds
Posted by "Roberto C. Sánchez" <ro...@connexer.com>.
On Tue, Apr 07, 2015 at 09:53:08AM +0200, Geißler, Daniel wrote:
> Hi Roberto,
>
> did you have a look into locking strategies, as far as i know ivy does no locking by default wich may lead to your problems:
>
> http://ant.apache.org/ivy/history/latest-milestone/settings/lock-strategies.html
>
Hi Daniel,
Thank you for responding. This appears to be a low traffic list :-)
I did not realize that multiple builds sharing the cache could present a
problem. In any event, each of my builds is configured to use its own
Ivy cache directory based on the Jenkins workspace directory. I don't
think that affects the part of the build with which I am currently
experiencing a problem. Additionally, the documentation link you
provided does not give an example of how I would use the locking
strategy.
I think that the problem has something to do with the deliver/publish
logic, but I cannot seem to figure out precisely what or where. I have
not observed any problem with any of my downloaded artifacts for the
dependencies. I have only seen a problem with the delivered/published
ivy.xml pertaining to the project being built at the same time as it is
being built by another process (but on a different branch).
Regards,
-Roberto
--
Roberto C. Sánchez
http://people.connexer.com/~roberto
http://www.connexer.com
AW: Difficult with concurrent builds
Posted by Geißler,
Daniel <da...@salt-solutions.de>.
Hi Roberto,
did you have a look into locking strategies, as far as i know ivy does no locking by default wich may lead to your problems:
http://ant.apache.org/ivy/history/latest-milestone/settings/lock-strategies.html
kind regards
Daniel
_____________________________________
www.salt-solutions.de
Geschäftsführer: Dr. Bernhard Blüthner, Dieter Heyde, Markus Honold
Sitz: München, AG München, HRB 146081
-----Ursprüngliche Nachricht-----
Von: Roberto C. Sánchez [mailto:roberto@connexer.com]
Gesendet: Fr, 3. April 2015 03:02
An: ivy-user@ant.apache.org
Betreff: Difficult with concurrent builds
I have a project using Ivy which is built via Jenkins. I have two branches for this project (a stable branch and a current development branch, which is the master branch in Git).
In Jenkins I have one build configured for each branch (I don't want to mix the build histories). I have a post-commit hook in my Git repository which checks for the branch(es) of the commit and then uses to wget to retrieve the URL that triggers the build that corresponds to the branch of the commit.
The problem that I am experiencing has to do with making changes in quick succession on the two branches. My Ant build.xml does the normal build routine and at the end publishes the artifacts into my own local Ivy repository. When the builds run concurrently (because, for example, I make a commit one branch and then immediately cherry-pick the commit to the other branch, the build that finshes last uses the ivy.xml desriptor from the build that finishes first. This is very frustrating as it breaks all sorts of things. I can go into Jenkins and force a rebuild of the one that finished last and everything turns out correctly, with the correct descriptor being published the second time.
I looked through the Ivy source code and found a reference to a delivery list file that looks in ${java.io.tmpdir}/delivery.properties. I tried explicitly setting the ivy.delivery.list.file using Ant's <tempfile/> task to ensure that concurrent builds won't accidentally touch the same file. I was able to confirm that instead of /tmp/delivery.properties that the Ivy publish task was using the randomly generated file name Ant put into the ivy.delivery.list.file property.
Even with that, though, I am still seeing the buggy behavior. That is, the build that finishes last uses the same descriptor as the build that finishes first (line for line identical except for the revision and publication attributes of the info tag.
Any help on this would be appreciated.
Regards,
-Roberto
--
Roberto C. Sánchez
http://people.connexer.com/~roberto
http://www.connexer.com