You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by David Weintraub <qa...@gmail.com> on 2012/11/29 16:35:05 UTC
vs.
When is it better to use <ivy:settings> vs. <ivy:configure>?
I ask because I was having a problem with my Ivy setup. We use Subversion
and have multiple projects setup that use Ant. In order to centralize our
jar dependencies, I've decided to implement Ivy into our process.
I did this by creating an Ivy project (https://github.com/qazwart/ivy.dir)
in Subversion. Users merely set `svn:externals` to include this project,
make some minor changes in their build.xml, and add in a "ivy.xml" file,
and everything is set. I control the ivysettings.xml file, so I point Ivy
to our repository. The Ivy jar is part of the Ivy project, so users don't
have to install it. The idea was to make Ivy integration as painless as
possible.
We use Jenkins as our build system, and I decided it would be good if we
could clean the Ivy cache before each build. However, out Jenkins system
has six executors, so as many as six builds could happen at once. Cleaning
the cache in one build while another one runs could cause problems, so I
modified my ivysettings.xml file to use a different cache depending upon
the executor:
<ivysettings>
<property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
<caches
defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
<settings defaultResolver="default"/>
<include file="${ivy.dir}/ivysettings-public.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
<include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
<include
url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
</ivysettings>
The problem is that if I did an <ivy:cleancache>, the cache ID was always
set to "cache-0". When I switched from "<ivy:settings>" to
"<ivy:configure>", the problem went away. (In the standard build process,
the <ivy:cleancache> task is executed before the <ivy:resolve> task).
I'm told that "<ivy:settings>" can do multiple configurations, but it looks
like <ivy:configure> also can do multiple settings too. I also read that
"<ivy:configure>" was deprecated in some mailings, but it doesn't state
that in the on line documentation.
So what is the difference between <ivy:settings> and <ivy:configure>, and
when should I use one over the other?
--
David Weintraub
qazwart@gmail.com
Re: vs.
Posted by David Weintraub <qa...@gmail.com>.
> The essence of the "correct" solution is to control the location of the
> parent Ivy user directory, which by default is user.home/.ivy2/. Developers
> are OK to use that default. For the Jenkins build, you want to override
> that in the user properties that the Jenkins job passes to Ant. The
> property you want to set is ivy.default.ivy.user.dir, and I believe you
> want to set it to the Jenkins-specific WORKSPACE variable. From here, you
> can define the caches relative to ${ivy.default.ivy.user.dir}.
That actually makes a bit more sense. This way, I don't have to go through the rigamarole of checking the value of the ivy.cleancache property I go through when I do a clean, and I don't have to depend upon the developers putting the <ivy:cleancache> in their build.xml.
I'm going to see if I can set this property in my ivy.tasks.xml file, so I don't have to redefine all of my Jenkins jobs. I'll also have to see if this solves the <ivy:configuration> vs. the <ivy:setitngs> issue which caused this whole thing to come up in the first place.
Thank you for your answer.
On Dec 1, 2012, at 12:04 PM, Mitch Gitman <mg...@gmail.com> wrote:
> I'm a bit embarrassed by the answer I gave you because it was as recently
> as April that I was actively supporting a Jenkins cluster, and in that
> short time my Jenkins knowledge has slipped a bit. Since I'm going to have
> to do another Jenkins setup myself in about a week, I decided to refresh
> myself a bit about what I'd done before.
>
> I didn't get very far, but the most important bit of advice I can give you,
> even if it might not be what you want to hear, is this: don't be mapping
> Ivy caches to Jenkins executors.
>
> The essence of the "correct" solution is to control the location of the
> parent Ivy user directory, which by default is user.home/.ivy2/. Developers
> are OK to use that default. For the Jenkins build, you want to override
> that in the user properties that the Jenkins job passes to Ant. The
> property you want to set is ivy.default.ivy.user.dir, and I believe you
> want to set it to the Jenkins-specific WORKSPACE variable. From here, you
> can define the caches relative to ${ivy.default.ivy.user.dir}.
>
> This solves your problem: "This is the problem: Both developers and Jenkins
> use the same build process, and the same build files." The cache will be
> preserved for developers—and trusted for all modules except those that are
> publishing continually changing integration versions. The cache will be
> wiped out every time on the CI server because you've told the CI build
> (unlike the dev build) to run ivy:cleancache, but this is fine because for
> a CI build there's a greater premium on pristineness and reproducibility
> than speed. Because each Jenkins job has its own dedicated workspace and no
> two executors will be running the same job simultaneously, you're assured
> that there will be no concurrency issues between parallel builds.
>
> One clarification about terminology. The two types of cache are the
> resolution cache and the repository cache.
>
> If you're not satisfied about the Jenkins build wiping out the repository
> cache every time, there are steps you can take. But get the safe thing
> working first and optimize later. And even then, you should only be
> creating a cache and trusting it for those dependencies (like third-party
> libraries) that never change within the same version.
>
> On Thu, Nov 29, 2012 at 11:18 PM, David Weintraub <qa...@gmail.com> wrote:
>
>> On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mg...@gmail.com> wrote:
>>> As you describe, the immediate problem is that, with settings, the
>>> EXECUTOR_NUMBER environment variable is not being picked up. The
>> following
>>> line is really just establishing a default:
>>> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
>>
>> That's correct. The environment variable EXECUTOR_NUMBER is set by
>> Jenkins, If you're not running on Jenkins, then, EXECUTOR_NUMBER isn't set.
>> I use that line to set it if it's not already set.
>>
>>> For debugging's sake, I'd try leaving that out. The executor number
>> should
>>> be required.
>>>
>>> For debugging's sake, I'd also try outputting the value in the Ant build
>>> just prior to calling ivy:settings.
>>>
>>> Also, I'd make sure that ivy:settings is always getting called before
>>> ivy:cleancache.
>>
>> All three of these are true. In fact, when I first used my ivysettings.xml
>> file, I set the executor number manually in my environment. When I did the
>> build, the cache directory was being set correctly. At least when the
>> <ivy:resolve> took place, and the correct cache was being created. I also
>> made sure that the cache was set to $HOME/.ivy2/cache-o when there was no
>> EXECUTOR_NUMBER environment variable. To me, this was the default way the
>> user would run it. Everything looked great.
>>
>> Suddenly builds on Jenkins started to fail and I figured that happened
>> when parallel builds took place and were wiping out each other's cache.
>> Printing a few debug statements before and after the <ivy:settings> task
>> led me to realize that the cache wasn't being set. That's when I read
>> through the <ivy:settings> task documentation and discovered that
>> <ivy:configure> did more or less the same thing, but that the
>> <ivy:configure> task did the configuration immediately instead of "when
>> needed". Switching to <ivy:configure> fixed the issue.
>>
>>> Also, I'd make sure that ivy:settings is always getting called before
>>> ivy:cleancache.
>>
>> It is. My Ivy project is a subdirectory to the user's project. The
>> developer puts in the lines "<property name="ivy.dir"
>> value="${basedir}/ivy.dir"/> and <import file="${ivy.dir}/ivy.tasks.xml"/>
>> into their build. The "ivy.task.xml" file calls the "<ivy:settings>" task
>> for the user. It runs before any other task is called. The user calls
>> <ivy:cleancache> this way:
>>
>> <property name="ivy.cleancache" value="true"/>
>>
>> <target name="clean">
>> <if>
>> <istrue value="${ivy.cleancache}"/>
>> <then>
>> <ivy:cleancache/>
>> </then>
>> </if>
>> <delete>
>> <fileset dir="${target.dir}"/>
>> </fileset>
>> </target>
>>
>> The user can override the value of ${ivy.cleancache} via a
>> build.properties file, so the cache isn't cleaned every time the developer
>> runs the "clean" target. Thus, <ivy:settings/> is called first before
>> <ivy:cleancache/>.
>>
>>> It appears that your resolution cache is not being defined on a
>>> per-executor basis. If each executor build has its own target directory,
>>> that's OK. But then, why not do that for both your resolution cache and
>>> your repository cache? So you'd specify resolutionCacheDir and
>>> repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.
>>
>> The resolution cache needs to be defined so that parallel builds don't
>> interfere with each other. There can also be an issue of a developer using
>> multiple working directories of the same project, and this is also an issue
>> with Jenkins too. The resolution cache gets built pretty quickly, so if you
>> delete it and it has to be built from scratch, it takes maybe another
>> second to do. Thus, putting the resolution cache per checkout resolves the
>> issue of multiple builds sharing the same resolution cache since each will
>> have its own. When the developer does a clean, they delete the "target"
>> directory and the resolution cache. No real problem since it can be quickly
>> rebuilt.
>>
>> The Ivy cache is a different issue. If this cache is deleted, all the jars
>> must be redownloaded which takes time. Besides if project A downloads
>> version 1.3 of foo.jar, and project B also needs that jar, project B
>> doesn't have to download it. That's the whole purpose of the Ivy cache.
>> Thus, the resolution cache is placed per project while the Ivy jar cache is
>> shared between different projects.
>>
>>> I forget if Jenkins checks out the source into a different location for
>>> each executor build (it should), but if that's the case and you're
>> sending
>>> your output to a location relative to your source, then establishing a
>>> relative path to your caches should suffice, and there's no need to
>>> establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now
>> that I
>>> think of it, specifying Ivy caches for a CI build that are based on
>>> ivy.default.ivy.user.dir, considering that its default is
>> user.home/.ivy2/,
>>> is a bit of an antipattern.
>>
>> No, each job has an established working directory and it uses the same
>> working directory no matter which executor is running. This makes sense
>> since the first time a job might run, it runs on executor #1 and the second
>> time on executor #0. If there were separate working directories per
>> executor, you couldn't take advantage of a version control update which
>> only has to copy a few files over instead of a whole new working directory.
>>
>> The "cache" really doesn't exist on Jenkins because I wipe it out for each
>> build. That's why each executor needs a separate build. I could just as
>> easily put that under the "target" directory like I do the resolution
>> cache. Unfortunately, the same build.xml is also used by the developer.
>> Their machines are slower and on a slower network connection than the
>> servers. Jenkins can download all of the jars in seconds. On the
>> developer's machines, it can take minutes. Thus, the cache is very
>> important to the developers.
>>
>> This is the problem: Both developers and Jenkins use the same build
>> process, and the same build files. But, both use the Ivy cache very
>> differently. Developers want it to act like a cache: Saving them valuable
>> time by not having to redownload files over and over. Jenkins wants a clean
>> slate to make sure that there isn't a jar issue during its build process.
>> Satisfying these two completely different requirements while using the same
>> build process and files lead me to this design.
>>
>> On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mg...@gmail.com> wrote:
>>> Let me take a stab at addressing your immediate problem, even though I
>>> don't have a direct answer. But then I'm going to suggest a different
>>> approach that avoids that problem.
>>>
>>> Frankly, I've never used configure, only settings. The configure
>>> documentation explains the primary difference (basically settings is
>> lazy):
>>> http://ant.apache.org/ivy/history/latest-milestone/use/configure.html
>>>
>>> As you describe, the immediate problem is that, with settings, the
>>> EXECUTOR_NUMBER environment variable is not being picked up. The
>> following
>>> line is really just establishing a default:
>>> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
>>>
>>> For debugging's sake, I'd try leaving that out. The executor number
>> should
>>> be required.
>>>
>>> For debugging's sake, I'd also try outputting the value in the Ant build
>>> just prior to calling ivy:settings.
>>>
>>> Also, I'd make sure that ivy:settings is always getting called before
>>> ivy:cleancache.
>>>
>>> And there's something I find odd about your caches specification:
>>> <caches
>>>
>> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
>>> resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
>>>
>>> It appears that your resolution cache is not being defined on a
>>> per-executor basis. If each executor build has its own target directory,
>>> that's OK. But then, why not do that for both your resolution cache and
>>> your repository cache? So you'd specify resolutionCacheDir and
>>> repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.
>>>
>>> I forget if Jenkins checks out the source into a different location for
>>> each executor build (it should), but if that's the case and you're
>> sending
>>> your output to a location relative to your source, then establishing a
>>> relative path to your caches should suffice, and there's no need to
>>> establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now
>> that I
>>> think of it, specifying Ivy caches for a CI build that are based on
>>> ivy.default.ivy.user.dir, considering that its default is
>> user.home/.ivy2/,
>>> is a bit of an antipattern.
>>>
>>> On Thu, Nov 29, 2012 at 8:35 AM, David Weintraub <qa...@gmail.com>
>> wrote:
>>>
>>>> When is it better to use <ivy:settings> vs. <ivy:configure>?
>>>>
>>>> I ask because I was having a problem with my Ivy setup. We use
>> Subversion
>>>> and have multiple projects setup that use Ant. In order to centralize
>> our
>>>> jar dependencies, I've decided to implement Ivy into our process.
>>>>
>>>> I did this by creating an Ivy project (
>> https://github.com/qazwart/ivy.dir)
>>>> in Subversion. Users merely set `svn:externals` to include this project,
>>>> make some minor changes in their build.xml, and add in a "ivy.xml" file,
>>>> and everything is set. I control the ivysettings.xml file, so I point
>> Ivy
>>>> to our repository. The Ivy jar is part of the Ivy project, so users
>> don't
>>>> have to install it. The idea was to make Ivy integration as painless as
>>>> possible.
>>>>
>>>> We use Jenkins as our build system, and I decided it would be good if we
>>>> could clean the Ivy cache before each build. However, out Jenkins system
>>>> has six executors, so as many as six builds could happen at once.
>> Cleaning
>>>> the cache in one build while another one runs could cause problems, so I
>>>> modified my ivysettings.xml file to use a different cache depending upon
>>>> the executor:
>>>>
>>>> <ivysettings>
>>>> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
>>>> <caches
>>>>
>>>>
>> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
>>>> resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
>>>> <settings defaultResolver="default"/>
>>>> <include file="${ivy.dir}/ivysettings-public.xml"/>
>>>> <include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
>>>> <include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
>>>> <include
>> url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
>>>> <include
>>>> url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
>>>> </ivysettings>
>>>>
>>>> The problem is that if I did an <ivy:cleancache>, the cache ID was
>> always
>>>> set to "cache-0". When I switched from "<ivy:settings>" to
>>>> "<ivy:configure>", the problem went away. (In the standard build
>> process,
>>>> the <ivy:cleancache> task is executed before the <ivy:resolve> task).
>>>>
>>>> I'm told that "<ivy:settings>" can do multiple configurations, but it
>> looks
>>>> like <ivy:configure> also can do multiple settings too. I also read that
>>>> "<ivy:configure>" was deprecated in some mailings, but it doesn't state
>>>> that in the on line documentation.
>>>>
>>>> So what is the difference between <ivy:settings> and <ivy:configure>,
>> and
>>>> when should I use one over the other?
>>>>
>>>> --
>>>> David Weintraub
>>>> qazwart@gmail.com
>>>>
>>
>>
Re: vs.
Posted by Mitch Gitman <mg...@gmail.com>.
I'm a bit embarrassed by the answer I gave you because it was as recently
as April that I was actively supporting a Jenkins cluster, and in that
short time my Jenkins knowledge has slipped a bit. Since I'm going to have
to do another Jenkins setup myself in about a week, I decided to refresh
myself a bit about what I'd done before.
I didn't get very far, but the most important bit of advice I can give you,
even if it might not be what you want to hear, is this: don't be mapping
Ivy caches to Jenkins executors.
The essence of the "correct" solution is to control the location of the
parent Ivy user directory, which by default is user.home/.ivy2/. Developers
are OK to use that default. For the Jenkins build, you want to override
that in the user properties that the Jenkins job passes to Ant. The
property you want to set is ivy.default.ivy.user.dir, and I believe you
want to set it to the Jenkins-specific WORKSPACE variable. From here, you
can define the caches relative to ${ivy.default.ivy.user.dir}.
This solves your problem: "This is the problem: Both developers and Jenkins
use the same build process, and the same build files." The cache will be
preserved for developers—and trusted for all modules except those that are
publishing continually changing integration versions. The cache will be
wiped out every time on the CI server because you've told the CI build
(unlike the dev build) to run ivy:cleancache, but this is fine because for
a CI build there's a greater premium on pristineness and reproducibility
than speed. Because each Jenkins job has its own dedicated workspace and no
two executors will be running the same job simultaneously, you're assured
that there will be no concurrency issues between parallel builds.
One clarification about terminology. The two types of cache are the
resolution cache and the repository cache.
If you're not satisfied about the Jenkins build wiping out the repository
cache every time, there are steps you can take. But get the safe thing
working first and optimize later. And even then, you should only be
creating a cache and trusting it for those dependencies (like third-party
libraries) that never change within the same version.
On Thu, Nov 29, 2012 at 11:18 PM, David Weintraub <qa...@gmail.com> wrote:
> On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mg...@gmail.com> wrote:
> > As you describe, the immediate problem is that, with settings, the
> > EXECUTOR_NUMBER environment variable is not being picked up. The
> following
> > line is really just establishing a default:
> > <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
>
> That's correct. The environment variable EXECUTOR_NUMBER is set by
> Jenkins, If you're not running on Jenkins, then, EXECUTOR_NUMBER isn't set.
> I use that line to set it if it's not already set.
>
> > For debugging's sake, I'd try leaving that out. The executor number
> should
> > be required.
> >
> > For debugging's sake, I'd also try outputting the value in the Ant build
> > just prior to calling ivy:settings.
> >
> > Also, I'd make sure that ivy:settings is always getting called before
> > ivy:cleancache.
>
> All three of these are true. In fact, when I first used my ivysettings.xml
> file, I set the executor number manually in my environment. When I did the
> build, the cache directory was being set correctly. At least when the
> <ivy:resolve> took place, and the correct cache was being created. I also
> made sure that the cache was set to $HOME/.ivy2/cache-o when there was no
> EXECUTOR_NUMBER environment variable. To me, this was the default way the
> user would run it. Everything looked great.
>
> Suddenly builds on Jenkins started to fail and I figured that happened
> when parallel builds took place and were wiping out each other's cache.
> Printing a few debug statements before and after the <ivy:settings> task
> led me to realize that the cache wasn't being set. That's when I read
> through the <ivy:settings> task documentation and discovered that
> <ivy:configure> did more or less the same thing, but that the
> <ivy:configure> task did the configuration immediately instead of "when
> needed". Switching to <ivy:configure> fixed the issue.
>
> > Also, I'd make sure that ivy:settings is always getting called before
> > ivy:cleancache.
>
> It is. My Ivy project is a subdirectory to the user's project. The
> developer puts in the lines "<property name="ivy.dir"
> value="${basedir}/ivy.dir"/> and <import file="${ivy.dir}/ivy.tasks.xml"/>
> into their build. The "ivy.task.xml" file calls the "<ivy:settings>" task
> for the user. It runs before any other task is called. The user calls
> <ivy:cleancache> this way:
>
> <property name="ivy.cleancache" value="true"/>
>
> <target name="clean">
> <if>
> <istrue value="${ivy.cleancache}"/>
> <then>
> <ivy:cleancache/>
> </then>
> </if>
> <delete>
> <fileset dir="${target.dir}"/>
> </fileset>
> </target>
>
> The user can override the value of ${ivy.cleancache} via a
> build.properties file, so the cache isn't cleaned every time the developer
> runs the "clean" target. Thus, <ivy:settings/> is called first before
> <ivy:cleancache/>.
>
> > It appears that your resolution cache is not being defined on a
> > per-executor basis. If each executor build has its own target directory,
> > that's OK. But then, why not do that for both your resolution cache and
> > your repository cache? So you'd specify resolutionCacheDir and
> > repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.
>
> The resolution cache needs to be defined so that parallel builds don't
> interfere with each other. There can also be an issue of a developer using
> multiple working directories of the same project, and this is also an issue
> with Jenkins too. The resolution cache gets built pretty quickly, so if you
> delete it and it has to be built from scratch, it takes maybe another
> second to do. Thus, putting the resolution cache per checkout resolves the
> issue of multiple builds sharing the same resolution cache since each will
> have its own. When the developer does a clean, they delete the "target"
> directory and the resolution cache. No real problem since it can be quickly
> rebuilt.
>
> The Ivy cache is a different issue. If this cache is deleted, all the jars
> must be redownloaded which takes time. Besides if project A downloads
> version 1.3 of foo.jar, and project B also needs that jar, project B
> doesn't have to download it. That's the whole purpose of the Ivy cache.
> Thus, the resolution cache is placed per project while the Ivy jar cache is
> shared between different projects.
>
> > I forget if Jenkins checks out the source into a different location for
> > each executor build (it should), but if that's the case and you're
> sending
> > your output to a location relative to your source, then establishing a
> > relative path to your caches should suffice, and there's no need to
> > establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now
> that I
> > think of it, specifying Ivy caches for a CI build that are based on
> > ivy.default.ivy.user.dir, considering that its default is
> user.home/.ivy2/,
> > is a bit of an antipattern.
>
> No, each job has an established working directory and it uses the same
> working directory no matter which executor is running. This makes sense
> since the first time a job might run, it runs on executor #1 and the second
> time on executor #0. If there were separate working directories per
> executor, you couldn't take advantage of a version control update which
> only has to copy a few files over instead of a whole new working directory.
>
> The "cache" really doesn't exist on Jenkins because I wipe it out for each
> build. That's why each executor needs a separate build. I could just as
> easily put that under the "target" directory like I do the resolution
> cache. Unfortunately, the same build.xml is also used by the developer.
> Their machines are slower and on a slower network connection than the
> servers. Jenkins can download all of the jars in seconds. On the
> developer's machines, it can take minutes. Thus, the cache is very
> important to the developers.
>
> This is the problem: Both developers and Jenkins use the same build
> process, and the same build files. But, both use the Ivy cache very
> differently. Developers want it to act like a cache: Saving them valuable
> time by not having to redownload files over and over. Jenkins wants a clean
> slate to make sure that there isn't a jar issue during its build process.
> Satisfying these two completely different requirements while using the same
> build process and files lead me to this design.
>
> On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mg...@gmail.com> wrote:
> > Let me take a stab at addressing your immediate problem, even though I
> > don't have a direct answer. But then I'm going to suggest a different
> > approach that avoids that problem.
> >
> > Frankly, I've never used configure, only settings. The configure
> > documentation explains the primary difference (basically settings is
> lazy):
> > http://ant.apache.org/ivy/history/latest-milestone/use/configure.html
> >
> > As you describe, the immediate problem is that, with settings, the
> > EXECUTOR_NUMBER environment variable is not being picked up. The
> following
> > line is really just establishing a default:
> > <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
> >
> > For debugging's sake, I'd try leaving that out. The executor number
> should
> > be required.
> >
> > For debugging's sake, I'd also try outputting the value in the Ant build
> > just prior to calling ivy:settings.
> >
> > Also, I'd make sure that ivy:settings is always getting called before
> > ivy:cleancache.
> >
> > And there's something I find odd about your caches specification:
> > <caches
> >
> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
> > resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
> >
> > It appears that your resolution cache is not being defined on a
> > per-executor basis. If each executor build has its own target directory,
> > that's OK. But then, why not do that for both your resolution cache and
> > your repository cache? So you'd specify resolutionCacheDir and
> > repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.
> >
> > I forget if Jenkins checks out the source into a different location for
> > each executor build (it should), but if that's the case and you're
> sending
> > your output to a location relative to your source, then establishing a
> > relative path to your caches should suffice, and there's no need to
> > establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now
> that I
> > think of it, specifying Ivy caches for a CI build that are based on
> > ivy.default.ivy.user.dir, considering that its default is
> user.home/.ivy2/,
> > is a bit of an antipattern.
> >
> > On Thu, Nov 29, 2012 at 8:35 AM, David Weintraub <qa...@gmail.com>
> wrote:
> >
> >> When is it better to use <ivy:settings> vs. <ivy:configure>?
> >>
> >> I ask because I was having a problem with my Ivy setup. We use
> Subversion
> >> and have multiple projects setup that use Ant. In order to centralize
> our
> >> jar dependencies, I've decided to implement Ivy into our process.
> >>
> >> I did this by creating an Ivy project (
> https://github.com/qazwart/ivy.dir)
> >> in Subversion. Users merely set `svn:externals` to include this project,
> >> make some minor changes in their build.xml, and add in a "ivy.xml" file,
> >> and everything is set. I control the ivysettings.xml file, so I point
> Ivy
> >> to our repository. The Ivy jar is part of the Ivy project, so users
> don't
> >> have to install it. The idea was to make Ivy integration as painless as
> >> possible.
> >>
> >> We use Jenkins as our build system, and I decided it would be good if we
> >> could clean the Ivy cache before each build. However, out Jenkins system
> >> has six executors, so as many as six builds could happen at once.
> Cleaning
> >> the cache in one build while another one runs could cause problems, so I
> >> modified my ivysettings.xml file to use a different cache depending upon
> >> the executor:
> >>
> >> <ivysettings>
> >> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
> >> <caches
> >>
> >>
> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
> >> resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
> >> <settings defaultResolver="default"/>
> >> <include file="${ivy.dir}/ivysettings-public.xml"/>
> >> <include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
> >> <include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
> >> <include
> url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
> >> <include
> >> url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
> >> </ivysettings>
> >>
> >> The problem is that if I did an <ivy:cleancache>, the cache ID was
> always
> >> set to "cache-0". When I switched from "<ivy:settings>" to
> >> "<ivy:configure>", the problem went away. (In the standard build
> process,
> >> the <ivy:cleancache> task is executed before the <ivy:resolve> task).
> >>
> >> I'm told that "<ivy:settings>" can do multiple configurations, but it
> looks
> >> like <ivy:configure> also can do multiple settings too. I also read that
> >> "<ivy:configure>" was deprecated in some mailings, but it doesn't state
> >> that in the on line documentation.
> >>
> >> So what is the difference between <ivy:settings> and <ivy:configure>,
> and
> >> when should I use one over the other?
> >>
> >> --
> >> David Weintraub
> >> qazwart@gmail.com
> >>
>
>
Re: vs.
Posted by David Weintraub <qa...@gmail.com>.
On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mg...@gmail.com> wrote:
> As you describe, the immediate problem is that, with settings, the
> EXECUTOR_NUMBER environment variable is not being picked up. The following
> line is really just establishing a default:
> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
That's correct. The environment variable EXECUTOR_NUMBER is set by Jenkins, If you're not running on Jenkins, then, EXECUTOR_NUMBER isn't set. I use that line to set it if it's not already set.
> For debugging's sake, I'd try leaving that out. The executor number should
> be required.
>
> For debugging's sake, I'd also try outputting the value in the Ant build
> just prior to calling ivy:settings.
>
> Also, I'd make sure that ivy:settings is always getting called before
> ivy:cleancache.
All three of these are true. In fact, when I first used my ivysettings.xml file, I set the executor number manually in my environment. When I did the build, the cache directory was being set correctly. At least when the <ivy:resolve> took place, and the correct cache was being created. I also made sure that the cache was set to $HOME/.ivy2/cache-o when there was no EXECUTOR_NUMBER environment variable. To me, this was the default way the user would run it. Everything looked great.
Suddenly builds on Jenkins started to fail and I figured that happened when parallel builds took place and were wiping out each other's cache. Printing a few debug statements before and after the <ivy:settings> task led me to realize that the cache wasn't being set. That's when I read through the <ivy:settings> task documentation and discovered that <ivy:configure> did more or less the same thing, but that the <ivy:configure> task did the configuration immediately instead of "when needed". Switching to <ivy:configure> fixed the issue.
> Also, I'd make sure that ivy:settings is always getting called before
> ivy:cleancache.
It is. My Ivy project is a subdirectory to the user's project. The developer puts in the lines "<property name="ivy.dir" value="${basedir}/ivy.dir"/> and <import file="${ivy.dir}/ivy.tasks.xml"/> into their build. The "ivy.task.xml" file calls the "<ivy:settings>" task for the user. It runs before any other task is called. The user calls <ivy:cleancache> this way:
<property name="ivy.cleancache" value="true"/>
<target name="clean">
<if>
<istrue value="${ivy.cleancache}"/>
<then>
<ivy:cleancache/>
</then>
</if>
<delete>
<fileset dir="${target.dir}"/>
</fileset>
</target>
The user can override the value of ${ivy.cleancache} via a build.properties file, so the cache isn't cleaned every time the developer runs the "clean" target. Thus, <ivy:settings/> is called first before <ivy:cleancache/>.
> It appears that your resolution cache is not being defined on a
> per-executor basis. If each executor build has its own target directory,
> that's OK. But then, why not do that for both your resolution cache and
> your repository cache? So you'd specify resolutionCacheDir and
> repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.
The resolution cache needs to be defined so that parallel builds don't interfere with each other. There can also be an issue of a developer using multiple working directories of the same project, and this is also an issue with Jenkins too. The resolution cache gets built pretty quickly, so if you delete it and it has to be built from scratch, it takes maybe another second to do. Thus, putting the resolution cache per checkout resolves the issue of multiple builds sharing the same resolution cache since each will have its own. When the developer does a clean, they delete the "target" directory and the resolution cache. No real problem since it can be quickly rebuilt.
The Ivy cache is a different issue. If this cache is deleted, all the jars must be redownloaded which takes time. Besides if project A downloads version 1.3 of foo.jar, and project B also needs that jar, project B doesn't have to download it. That's the whole purpose of the Ivy cache. Thus, the resolution cache is placed per project while the Ivy jar cache is shared between different projects.
> I forget if Jenkins checks out the source into a different location for
> each executor build (it should), but if that's the case and you're sending
> your output to a location relative to your source, then establishing a
> relative path to your caches should suffice, and there's no need to
> establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now that I
> think of it, specifying Ivy caches for a CI build that are based on
> ivy.default.ivy.user.dir, considering that its default is user.home/.ivy2/,
> is a bit of an antipattern.
No, each job has an established working directory and it uses the same working directory no matter which executor is running. This makes sense since the first time a job might run, it runs on executor #1 and the second time on executor #0. If there were separate working directories per executor, you couldn't take advantage of a version control update which only has to copy a few files over instead of a whole new working directory.
The "cache" really doesn't exist on Jenkins because I wipe it out for each build. That's why each executor needs a separate build. I could just as easily put that under the "target" directory like I do the resolution cache. Unfortunately, the same build.xml is also used by the developer. Their machines are slower and on a slower network connection than the servers. Jenkins can download all of the jars in seconds. On the developer's machines, it can take minutes. Thus, the cache is very important to the developers.
This is the problem: Both developers and Jenkins use the same build process, and the same build files. But, both use the Ivy cache very differently. Developers want it to act like a cache: Saving them valuable time by not having to redownload files over and over. Jenkins wants a clean slate to make sure that there isn't a jar issue during its build process. Satisfying these two completely different requirements while using the same build process and files lead me to this design.
On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mg...@gmail.com> wrote:
> Let me take a stab at addressing your immediate problem, even though I
> don't have a direct answer. But then I'm going to suggest a different
> approach that avoids that problem.
>
> Frankly, I've never used configure, only settings. The configure
> documentation explains the primary difference (basically settings is lazy):
> http://ant.apache.org/ivy/history/latest-milestone/use/configure.html
>
> As you describe, the immediate problem is that, with settings, the
> EXECUTOR_NUMBER environment variable is not being picked up. The following
> line is really just establishing a default:
> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
>
> For debugging's sake, I'd try leaving that out. The executor number should
> be required.
>
> For debugging's sake, I'd also try outputting the value in the Ant build
> just prior to calling ivy:settings.
>
> Also, I'd make sure that ivy:settings is always getting called before
> ivy:cleancache.
>
> And there's something I find odd about your caches specification:
> <caches
> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
> resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
>
> It appears that your resolution cache is not being defined on a
> per-executor basis. If each executor build has its own target directory,
> that's OK. But then, why not do that for both your resolution cache and
> your repository cache? So you'd specify resolutionCacheDir and
> repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.
>
> I forget if Jenkins checks out the source into a different location for
> each executor build (it should), but if that's the case and you're sending
> your output to a location relative to your source, then establishing a
> relative path to your caches should suffice, and there's no need to
> establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now that I
> think of it, specifying Ivy caches for a CI build that are based on
> ivy.default.ivy.user.dir, considering that its default is user.home/.ivy2/,
> is a bit of an antipattern.
>
> On Thu, Nov 29, 2012 at 8:35 AM, David Weintraub <qa...@gmail.com> wrote:
>
>> When is it better to use <ivy:settings> vs. <ivy:configure>?
>>
>> I ask because I was having a problem with my Ivy setup. We use Subversion
>> and have multiple projects setup that use Ant. In order to centralize our
>> jar dependencies, I've decided to implement Ivy into our process.
>>
>> I did this by creating an Ivy project (https://github.com/qazwart/ivy.dir)
>> in Subversion. Users merely set `svn:externals` to include this project,
>> make some minor changes in their build.xml, and add in a "ivy.xml" file,
>> and everything is set. I control the ivysettings.xml file, so I point Ivy
>> to our repository. The Ivy jar is part of the Ivy project, so users don't
>> have to install it. The idea was to make Ivy integration as painless as
>> possible.
>>
>> We use Jenkins as our build system, and I decided it would be good if we
>> could clean the Ivy cache before each build. However, out Jenkins system
>> has six executors, so as many as six builds could happen at once. Cleaning
>> the cache in one build while another one runs could cause problems, so I
>> modified my ivysettings.xml file to use a different cache depending upon
>> the executor:
>>
>> <ivysettings>
>> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
>> <caches
>>
>> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
>> resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
>> <settings defaultResolver="default"/>
>> <include file="${ivy.dir}/ivysettings-public.xml"/>
>> <include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
>> <include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
>> <include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
>> <include
>> url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
>> </ivysettings>
>>
>> The problem is that if I did an <ivy:cleancache>, the cache ID was always
>> set to "cache-0". When I switched from "<ivy:settings>" to
>> "<ivy:configure>", the problem went away. (In the standard build process,
>> the <ivy:cleancache> task is executed before the <ivy:resolve> task).
>>
>> I'm told that "<ivy:settings>" can do multiple configurations, but it looks
>> like <ivy:configure> also can do multiple settings too. I also read that
>> "<ivy:configure>" was deprecated in some mailings, but it doesn't state
>> that in the on line documentation.
>>
>> So what is the difference between <ivy:settings> and <ivy:configure>, and
>> when should I use one over the other?
>>
>> --
>> David Weintraub
>> qazwart@gmail.com
>>
Re: vs.
Posted by Mitch Gitman <mg...@gmail.com>.
Let me take a stab at addressing your immediate problem, even though I
don't have a direct answer. But then I'm going to suggest a different
approach that avoids that problem.
Frankly, I've never used configure, only settings. The configure
documentation explains the primary difference (basically settings is lazy):
http://ant.apache.org/ivy/history/latest-milestone/use/configure.html
As you describe, the immediate problem is that, with settings, the
EXECUTOR_NUMBER environment variable is not being picked up. The following
line is really just establishing a default:
<property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
For debugging's sake, I'd try leaving that out. The executor number should
be required.
For debugging's sake, I'd also try outputting the value in the Ant build
just prior to calling ivy:settings.
Also, I'd make sure that ivy:settings is always getting called before
ivy:cleancache.
And there's something I find odd about your caches specification:
<caches
defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
It appears that your resolution cache is not being defined on a
per-executor basis. If each executor build has its own target directory,
that's OK. But then, why not do that for both your resolution cache and
your repository cache? So you'd specify resolutionCacheDir and
repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.
I forget if Jenkins checks out the source into a different location for
each executor build (it should), but if that's the case and you're sending
your output to a location relative to your source, then establishing a
relative path to your caches should suffice, and there's no need to
establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now that I
think of it, specifying Ivy caches for a CI build that are based on
ivy.default.ivy.user.dir, considering that its default is user.home/.ivy2/,
is a bit of an antipattern.
On Thu, Nov 29, 2012 at 8:35 AM, David Weintraub <qa...@gmail.com> wrote:
> When is it better to use <ivy:settings> vs. <ivy:configure>?
>
> I ask because I was having a problem with my Ivy setup. We use Subversion
> and have multiple projects setup that use Ant. In order to centralize our
> jar dependencies, I've decided to implement Ivy into our process.
>
> I did this by creating an Ivy project (https://github.com/qazwart/ivy.dir)
> in Subversion. Users merely set `svn:externals` to include this project,
> make some minor changes in their build.xml, and add in a "ivy.xml" file,
> and everything is set. I control the ivysettings.xml file, so I point Ivy
> to our repository. The Ivy jar is part of the Ivy project, so users don't
> have to install it. The idea was to make Ivy integration as painless as
> possible.
>
> We use Jenkins as our build system, and I decided it would be good if we
> could clean the Ivy cache before each build. However, out Jenkins system
> has six executors, so as many as six builds could happen at once. Cleaning
> the cache in one build while another one runs could cause problems, so I
> modified my ivysettings.xml file to use a different cache depending upon
> the executor:
>
> <ivysettings>
> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
> <caches
>
> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
> resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
> <settings defaultResolver="default"/>
> <include file="${ivy.dir}/ivysettings-public.xml"/>
> <include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
> <include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
> <include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
> <include
> url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
> </ivysettings>
>
> The problem is that if I did an <ivy:cleancache>, the cache ID was always
> set to "cache-0". When I switched from "<ivy:settings>" to
> "<ivy:configure>", the problem went away. (In the standard build process,
> the <ivy:cleancache> task is executed before the <ivy:resolve> task).
>
> I'm told that "<ivy:settings>" can do multiple configurations, but it looks
> like <ivy:configure> also can do multiple settings too. I also read that
> "<ivy:configure>" was deprecated in some mailings, but it doesn't state
> that in the on line documentation.
>
> So what is the difference between <ivy:settings> and <ivy:configure>, and
> when should I use one over the other?
>
> --
> David Weintraub
> qazwart@gmail.com
>