You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by Loren Kratzke <LK...@blueorigin.com> on 2015/03/20 23:12:02 UTC
cache busting and integration question
I am encountering challenges with the Ivy cache and also with setting up what I would consider to be a typical (if not classic) developer workflow. Here is the desired workflow:
I have a Nexus repository (releases and snapshots) plus a local file system repository used for local development. I have Project which depends upon Library. I wish to modify Library, publish locally, and then pull it into the local build of Project.
And for my Jenkins "continuous" integration server, if the copy of Library-1.0.0-SNAPSHOT in Nexus is newer than what is in the cache on the build machine, then I would like Ivy to grab the newer version (just like Maven does).
ISSUE #1
Assuming I start with an empty cache and empty internal repo on my local dev machine, I build Project and it pulls Library from Nexus. Perfect. Works great. But then if I publish Library locally and build Project again, I will get the cached version of Library that came from Nexus. No matter what I do, I will always get the cached copy. This happens even though my chain resolver specifies my internal repo before the Nexus repo.
If I blow away the cache, then I get my locally published build, but only if I blow away the cache. Otherwise I get stale stuff. Issue #2 below is all about stale stuff.
I debugged Ivy (for hours upon hours) and during the resolve, it checks my internal repo, finds the fresh artifact (yay!) and the next thing it does is checks its cache, finds a module descriptor that points at Nexus, and proceeds to pull in the wrong artifact into Project.
The behavior is either a bug, or implies that Ivy assumes that an artifact can only come from one place, and if it came from there once, it will come from there forever, and will never change. Again, Ivy finds the artifact in local repo, stops searching, then ends up delivering the cached artifact that came from Nexus sometime earlier.
ISSUE #2
I have been working on this problem for quite some time and I thought I had it fixed, but I don't. That is, my integration server needs the latest snapshot build of Library-1.0.0-SNAPSHOT. But what I get is the cached version from the Ivy cache. An SVN commit happens, triggers a build of Library, Library get published, triggers a build of Project, and a stale build of Library is delivered to Project from the cache (which breaks Project build).
SUMMARY
Ivy will resolve and cache deps, but once a dep is in the cache, that's what I get forever until I blow away the cache. I see this question asked quite often regarding obtaining latest integration versions, and I see that Ivy claims to support the local development workflow, but I have yet to see a working example of either of these. Surely some of you have solved these issues.
Is there any place anywhere in the world that has an example of development workflow configuration and/or integration configuration? Can anybody provide an example? So far I have not found anything, and advice on the net seems to be general suggestions like "try this, try that" and nobody really has the configuration solution. Is Ivy capable of these behaviors or do I need to hand roll a dependency management solution. Maven is too strict for these projects (C++ code). Ivy is good if I can just obtain these fundamental behaviors.
Note that I have not provided any configs in this message that demonstrate the issues I am having but definitely can upon request.
Thanks in advance,
L.K.
RE: cache busting and integration question
Posted by Loren Kratzke <LK...@blueorigin.com>.
Very interesting. After reading this, I configured one cache per repository and I do believe it fixed one of odd behaviors that I saw earlier.
The odd behavior was an artifact was selected as a winner from the local repository, but a subsequent last second check for that artifact in the cache resulted in a cached artifact from a remote repository being delivered. This no longer seems to be an issue but I am also testing a few other changes so can't be sure if this is fixed and by this change. But I think it is.
Now I can't imagine why I would ever want a shared cache for multiple repositories. I think I like one cache per repo much better.
Thanks for the input. It made a tangible positive difference in my testing.
I am currently testing a potential bug fix in the ChainResolver. It might not be a bug, in which case it would make a great enhancement. Either way, I like it very much.
The fix is to get the chain resolver to return the latest snapshot from any of all configured repositories. Some code looked odd in that the intent looked to be to return the most recent artifact given two artifacts, but no date comparison was ever performed. Rather, the method always returned the same Boolean regardless of the age of the artifacts. Performing an actual date comparison at that point unlocked a bunch of great behaviors.
It was a 4 line fix in ChainResolver. I will share asap.
L.K.
-----Original Message-----
From: Sven Zethelius [mailto:svenz@expedia.com]
Sent: Monday, March 23, 2015 4:07 PM
To: ivy-user@ant.apache.org
Subject: Re: cache busting and integration question
There are uses where a FileSystemRepository need to be cached for performance. On Windows, UNC path files will be resolved by the FileSystemRepository. On Linux, smb or NFS file mounts will benefit from local caching as well.
Have you tried configuring caching differently for each repository in your ivy.settings.xml. Here¹s a snippet from mine. We make use of the local cache as a repository when the build is given special instructions to be completely offline.
<caches default="remote-cache"
defaultCacheDir="${ivycache.base}"
ivyPattern="[organisation]/[module]/ivy-[revision].xml"
artifactPattern="[organisation]/[module]/[artifact]-[revision].[ext]">
<cache name="local-cache" useOrigin="true" />
<cache name="remote-cache" useOrigin="false"
defaultTTL="${ivycache.timeout}" />
<cache name="offline-cache" basedir="${ivycache.base}/../offline"
useOrigin="true" />
</caches>
Then in each resolver, I specify which of Œlocal-cache¹, Œremote-cache¹, Œoffline-cache¹ it should use using the cache attribute.
On 3/23/15, 3:09 PM, "Loren Kratzke" <LK...@blueorigin.com> wrote:
>Ivy works perfectly with a single repository or even a chain of
>repositories provided that a dependency does not try to come from two
>places.
>
>Ivy records where an artifact comes from but does not take into account
>that it may come from a different place next time. When Ivy finds a
>candidate artifact in a repository somewhere, it checks the .ivy cache
>for that artifact, finds an artifact in the cache (that happens to
>originate from somewhere else), doesn't check or realize that it is
>from somewhere else, checks the module descriptor against the origin
>and determines that the stale artifact is up to date with its stale
>origin, and then assumes that the stale artifact is the same as the
>found artifact. It does not compare the locations and invalidate the
>cached artifact. Rather, it delivers the stale artifact that originated
>from a different location than the found artifact.
>
>THAT is a bug. It has to be a bug. There is no way that this is not a
>bug. Got that out of my system. Whew!
>
>That said, using a local repo and "force" tends to overcome this
>behavior (in theory, and in my testing, and only sometimes). Results of
>whether you get a fresh artifact range from "never", to "the second
>time and after that", to "always but only ever from one place"
>depending upon the configuration.
>
>I think this is fixable or at least patchable though.
>
>For example, when I find an artifact in a file system repository, WHY
>would I want to check my cache for that artifact? The artifact is
>already right-freaking-there on my file system. It is "here". It can't
>be any closer without already being delivered to my project. Only
>trouble can come from querying the cache when you have the desired
>artifact in your hand already. And to then proceed to find and return
>an older artifact from the cache that came from a different repo is just not cool at all.
>
>**Should not check the cache for an artifact that was found in a file
>system repository.
>**Should not assume that cached artifact came from same repo as found
>artifact.
>**Should at least check the date to know that the cached artifact about
>to be returned is older than the found artifact.
>
>Perhaps using a different cache directory for each repository would
>help on that last point, but there is nothing that makes me think that
>Ivy would not search all of its caches and produce a similar outcome of
>providing a stale artifact fetched at an earlier time and from a
>different repository. I may be a bit jaded at this point. But I have
>many hours in on this so can vent a little.
>
>Part of the problem is that Ivy developers somewhere along the line
>separated the module descriptor from the module that it describes, and
>both of those from the repository which provided each, recording the
>origin location of the artifact in the cached module descriptor instead
>of using independent cache directories per repo and using metadata
>files, and then not leveraging that origin information to know that it
>has totally not found a cached version of a found artifact. That's 4
>strikes that all add up to a very broken and at times quite silly cache
>mechanism.
>
>I am banking on some of this being a bug and some of it being by
>design, but mostly bug. When I go back into Ivy source code, I am going
>to see if I can disable the cache checking when an artifact is found
>using a file system resolver. That would be a major step forward and
>would make much of this odd behavior go away.
>
>L.K.
>
>-----Original Message-----
>From: Marc De Boeck [mailto:mdeboec@gmail.com]
>Sent: Monday, March 23, 2015 1:29 PM
>To: ivy-user@ant.apache.org
>Subject: Re: cache busting and integration question
>
>It's an interesting problem to investigate, and I understand your
>frustration. But since we don't have exactly the same setup as you,
>it's not evident to tell exactly what the problem is and how you can solve it.
>We also have an environment with a local and a central repo, but we
>don't use the checkmodified attribute. If it would work, it is indeed
>a good idea to setup such a resolver chain with SNAPSHOT builds for the
>developers.
>In our case (without checkmodified=true) the detection of changes in
>both local and central repo works correctly for us.
>
>Now, I can only search with you in the docs hoping to find a missing
>setting (if it's not an actual bug as you suggested).
>
>I suppose that the publication date of the ivy.xml that you published
>in your local repo is more recent than the one in your cache ? Did you
>also try setting the "changing" attribute directly in the dependency
>itself (allthough I expect the behaviour will be similar).
>Could you try a setup where you only publish and resolve from the local
>repo (so not chained with your Nexus) ?
>
>You also wrote that latest.integration is a non-starter for you ? So
>could you describe how you specify your dependencies ? If you specify
>an exact revision in your dependency (like rev="1.0.0-SNAPSHOT), then
>maybe you could also try to set the attribute
>"alwaysCheckExactRevision" in the settings of your resolver.
>
>
>Regards,
>Marc
>
>
>2015-03-23 17:33 GMT+01:00 Loren Kratzke <LK...@blueorigin.com>:
>
>> I am not getting the results that you describe...
>
Re: cache busting and integration question
Posted by Sven Zethelius <sv...@expedia.com>.
There are uses where a FileSystemRepository need to be cached for
performance. On Windows, UNC path files will be resolved by the
FileSystemRepository. On Linux, smb or NFS file mounts will benefit from
local caching as well.
Have you tried configuring caching differently for each repository in your
ivy.settings.xml. Here¹s a snippet from mine. We make use of the local
cache as a repository when the build is given special instructions to be
completely offline.
<caches default="remote-cache"
defaultCacheDir="${ivycache.base}"
ivyPattern="[organisation]/[module]/ivy-[revision].xml"
artifactPattern="[organisation]/[module]/[artifact]-[revision].[ext]">
<cache name="local-cache" useOrigin="true" />
<cache name="remote-cache" useOrigin="false"
defaultTTL="${ivycache.timeout}" />
<cache name="offline-cache" basedir="${ivycache.base}/../offline"
useOrigin="true" />
</caches>
Then in each resolver, I specify which of Œlocal-cache¹, Œremote-cache¹,
Œoffline-cache¹ it should use using the cache attribute.
On 3/23/15, 3:09 PM, "Loren Kratzke" <LK...@blueorigin.com> wrote:
>Ivy works perfectly with a single repository or even a chain of
>repositories provided that a dependency does not try to come from two
>places.
>
>Ivy records where an artifact comes from but does not take into account
>that it may come from a different place next time. When Ivy finds a
>candidate artifact in a repository somewhere, it checks the .ivy cache
>for that artifact, finds an artifact in the cache (that happens to
>originate from somewhere else), doesn't check or realize that it is from
>somewhere else, checks the module descriptor against the origin and
>determines that the stale artifact is up to date with its stale origin,
>and then assumes that the stale artifact is the same as the found
>artifact. It does not compare the locations and invalidate the cached
>artifact. Rather, it delivers the stale artifact that originated from a
>different location than the found artifact.
>
>THAT is a bug. It has to be a bug. There is no way that this is not a
>bug. Got that out of my system. Whew!
>
>That said, using a local repo and "force" tends to overcome this behavior
>(in theory, and in my testing, and only sometimes). Results of whether
>you get a fresh artifact range from "never", to "the second time and
>after that", to "always but only ever from one place" depending upon the
>configuration.
>
>I think this is fixable or at least patchable though.
>
>For example, when I find an artifact in a file system repository, WHY
>would I want to check my cache for that artifact? The artifact is already
>right-freaking-there on my file system. It is "here". It can't be any
>closer without already being delivered to my project. Only trouble can
>come from querying the cache when you have the desired artifact in your
>hand already. And to then proceed to find and return an older artifact
>from the cache that came from a different repo is just not cool at all.
>
>**Should not check the cache for an artifact that was found in a file
>system repository.
>**Should not assume that cached artifact came from same repo as found
>artifact.
>**Should at least check the date to know that the cached artifact about
>to be returned is older than the found artifact.
>
>Perhaps using a different cache directory for each repository would help
>on that last point, but there is nothing that makes me think that Ivy
>would not search all of its caches and produce a similar outcome of
>providing a stale artifact fetched at an earlier time and from a
>different repository. I may be a bit jaded at this point. But I have many
>hours in on this so can vent a little.
>
>Part of the problem is that Ivy developers somewhere along the line
>separated the module descriptor from the module that it describes, and
>both of those from the repository which provided each, recording the
>origin location of the artifact in the cached module descriptor instead
>of using independent cache directories per repo and using metadata files,
>and then not leveraging that origin information to know that it has
>totally not found a cached version of a found artifact. That's 4 strikes
>that all add up to a very broken and at times quite silly cache
>mechanism.
>
>I am banking on some of this being a bug and some of it being by design,
>but mostly bug. When I go back into Ivy source code, I am going to see if
>I can disable the cache checking when an artifact is found using a file
>system resolver. That would be a major step forward and would make much
>of this odd behavior go away.
>
>L.K.
>
>-----Original Message-----
>From: Marc De Boeck [mailto:mdeboec@gmail.com]
>Sent: Monday, March 23, 2015 1:29 PM
>To: ivy-user@ant.apache.org
>Subject: Re: cache busting and integration question
>
>It's an interesting problem to investigate, and I understand your
>frustration. But since we don't have exactly the same setup as you, it's
>not evident to tell exactly what the problem is and how you can solve it.
>We also have an environment with a local and a central repo, but we don't
>use the checkmodified attribute. If it would work, it is indeed a good
>idea to setup such a resolver chain with SNAPSHOT builds for the
>developers.
>In our case (without checkmodified=true) the detection of changes in both
>local and central repo works correctly for us.
>
>Now, I can only search with you in the docs hoping to find a missing
>setting (if it's not an actual bug as you suggested).
>
>I suppose that the publication date of the ivy.xml that you published in
>your local repo is more recent than the one in your cache ? Did you also
>try setting the "changing" attribute directly in the dependency itself
>(allthough I expect the behaviour will be similar).
>Could you try a setup where you only publish and resolve from the local
>repo (so not chained with your Nexus) ?
>
>You also wrote that latest.integration is a non-starter for you ? So
>could you describe how you specify your dependencies ? If you specify an
>exact revision in your dependency (like rev="1.0.0-SNAPSHOT), then maybe
>you could also try to set the attribute "alwaysCheckExactRevision" in the
>settings of your resolver.
>
>
>Regards,
>Marc
>
>
>2015-03-23 17:33 GMT+01:00 Loren Kratzke <LK...@blueorigin.com>:
>
>> I am not getting the results that you describe...
>
RE: cache busting and integration question
Posted by Loren Kratzke <LK...@blueorigin.com>.
Ivy works perfectly with a single repository or even a chain of repositories provided that a dependency does not try to come from two places.
Ivy records where an artifact comes from but does not take into account that it may come from a different place next time. When Ivy finds a candidate artifact in a repository somewhere, it checks the .ivy cache for that artifact, finds an artifact in the cache (that happens to originate from somewhere else), doesn't check or realize that it is from somewhere else, checks the module descriptor against the origin and determines that the stale artifact is up to date with its stale origin, and then assumes that the stale artifact is the same as the found artifact. It does not compare the locations and invalidate the cached artifact. Rather, it delivers the stale artifact that originated from a different location than the found artifact.
THAT is a bug. It has to be a bug. There is no way that this is not a bug. Got that out of my system. Whew!
That said, using a local repo and "force" tends to overcome this behavior (in theory, and in my testing, and only sometimes). Results of whether you get a fresh artifact range from "never", to "the second time and after that", to "always but only ever from one place" depending upon the configuration.
I think this is fixable or at least patchable though.
For example, when I find an artifact in a file system repository, WHY would I want to check my cache for that artifact? The artifact is already right-freaking-there on my file system. It is "here". It can't be any closer without already being delivered to my project. Only trouble can come from querying the cache when you have the desired artifact in your hand already. And to then proceed to find and return an older artifact from the cache that came from a different repo is just not cool at all.
**Should not check the cache for an artifact that was found in a file system repository.
**Should not assume that cached artifact came from same repo as found artifact.
**Should at least check the date to know that the cached artifact about to be returned is older than the found artifact.
Perhaps using a different cache directory for each repository would help on that last point, but there is nothing that makes me think that Ivy would not search all of its caches and produce a similar outcome of providing a stale artifact fetched at an earlier time and from a different repository. I may be a bit jaded at this point. But I have many hours in on this so can vent a little.
Part of the problem is that Ivy developers somewhere along the line separated the module descriptor from the module that it describes, and both of those from the repository which provided each, recording the origin location of the artifact in the cached module descriptor instead of using independent cache directories per repo and using metadata files, and then not leveraging that origin information to know that it has totally not found a cached version of a found artifact. That's 4 strikes that all add up to a very broken and at times quite silly cache mechanism.
I am banking on some of this being a bug and some of it being by design, but mostly bug. When I go back into Ivy source code, I am going to see if I can disable the cache checking when an artifact is found using a file system resolver. That would be a major step forward and would make much of this odd behavior go away.
L.K.
-----Original Message-----
From: Marc De Boeck [mailto:mdeboec@gmail.com]
Sent: Monday, March 23, 2015 1:29 PM
To: ivy-user@ant.apache.org
Subject: Re: cache busting and integration question
It's an interesting problem to investigate, and I understand your frustration. But since we don't have exactly the same setup as you, it's not evident to tell exactly what the problem is and how you can solve it.
We also have an environment with a local and a central repo, but we don't use the checkmodified attribute. If it would work, it is indeed a good idea to setup such a resolver chain with SNAPSHOT builds for the developers.
In our case (without checkmodified=true) the detection of changes in both local and central repo works correctly for us.
Now, I can only search with you in the docs hoping to find a missing setting (if it's not an actual bug as you suggested).
I suppose that the publication date of the ivy.xml that you published in your local repo is more recent than the one in your cache ? Did you also try setting the "changing" attribute directly in the dependency itself (allthough I expect the behaviour will be similar).
Could you try a setup where you only publish and resolve from the local repo (so not chained with your Nexus) ?
You also wrote that latest.integration is a non-starter for you ? So could you describe how you specify your dependencies ? If you specify an exact revision in your dependency (like rev="1.0.0-SNAPSHOT), then maybe you could also try to set the attribute "alwaysCheckExactRevision" in the settings of your resolver.
Regards,
Marc
2015-03-23 17:33 GMT+01:00 Loren Kratzke <LK...@blueorigin.com>:
> I am not getting the results that you describe...
Re: cache busting and integration question
Posted by Josh Suereth <jo...@gmail.com>.
On Mar 23, 2015 4:29 PM, "Marc De Boeck" <md...@gmail.com> wrote:
>
> It's an interesting problem to investigate, and I understand your
> frustration. But since we don't have exactly the same setup as you, it's
> not evident to tell exactly what the problem is and how you can solve it.
> We also have an environment with a local and a central repo, but we don't
> use the checkmodified attribute. If it would work, it is indeed a good
> idea to setup such a resolver chain with SNAPSHOT builds for the
developers.
> In our case (without checkmodified=true) the detection of changes in both
> local and central repo works correctly for us.
>
> Now, I can only search with you in the docs hoping to find a missing
> setting (if it's not an actual bug as you suggested).
>
> I suppose that the publication date of the ivy.xml that you published in
> your local repo is more recent than the one in your cache ? Did you also
> try setting the "changing" attribute directly in the dependency itself
> (allthough I expect the behaviour will be similar).
> Could you try a setup where you only publish and resolve from the local
> repo (so not chained with your Nexus) ?
The issue is that publication date is not correctly detected for pom.xml
files, generally.
> You also wrote that latest.integration is a non-starter for you ? So could
> you describe how you specify your dependencies ? If you specify an exact
> revision in your dependency (like rev="1.0.0-SNAPSHOT), then maybe you
> could also try to set the attribute "alwaysCheckExactRevision" in the
> settings of your resolver.
>
>
> Regards,
> Marc
>
>
> 2015-03-23 17:33 GMT+01:00 Loren Kratzke <LK...@blueorigin.com>:
>
> > I am not getting the results that you describe. I am testing with a
simple
> > project that publishes a text file with the current date and time so it
is
> > very easy to see when I get a fresh or stale artifact. When I run using
the
> > configuration below, I get the following results when I publish my
> > dependency and then consume it in the downstream project:
> >
> > [pub nexus] [get nexus] good
> > [pub nexus] [get nexus] good
> > [pub local] [get cached nexus artifact] bad
> > [pub nexus] [get cached nexus artfiact] bad
> >
> > So as the first two tests indicate, I can get the latest snapshot from
> > Nexus. That will work well for the integration server if we leave the
local
> > repo out.
> >
> > But as the 3rd and 4th tests indicate, if I do a local publish then
> > everything goes to hell. I do not get my local artifact, I get the
cached
> > artifact. And if I publish to Nexus again, I still get the cached
artifact.
> > In fact I get the cached artifact forever after a local publish no
matter
> > what gets published or where.
> >
> > That can't be desired behavior. There has to be a bug here somewhere,
but
> > I would be most happy with a work around. This behavior was witnessed
when
> > I debugged. It would find a local artifact, check its cache to find an
ivy
> > file that points at Nexus, and ultimately return a stale Nexus artifact
> > from the cache.
> >
> > My setup is very simple - One Nexus repository and one local repository.
> > Nothing fancy here. I have tried every combination of "checkmodified",
> > "changingPattern", and "force" that the configuration can tolerate
without
> > it quacking a fur ball back at me but I cannot seem to get the desired
> > behavior. Please find the err in my ways and show me where I have
wandered
> > off the ranch.
> >
> > I thought I had this working - once - and I changed something and can't
> > seem to get it back. I thought I had it working as you described where
it
> > would pick up the snapshots, then it would pick up my local artifact,
and
> > it would always deliver the local artifact if present until the local
cache
> > was torched. That behavior is ok with me if I could get it back.
> >
> > Here is my settings.xml
> >
> > <ivysettings>
> > <caches defaultCacheDir="${user.home}/.ivy2/cache"/>
> > <settings defaultResolver="chain"/>
> > <credentials host="nexus.mycompany.com" realm="Sonatype Nexus
> > Repository Manager" username="name" passwd="pass"/>
> >
> > <property name="m2pattern"
> >
value="[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]"/>
> > <property name="internal.repo.dir" value="${user.home}/.ivy2/m2"/>
> > <property name="mycompany.repo.url" value="
> > http://nexus.mycompany.com:8081/nexus/content/repositories"/>
> >
> > <resolvers>
> > <filesystem name="internal" force="true" checkmodified="true"
> > changingPattern=".*">
> > <ivy
> > pattern="${internal.repo.dir}/[module]/ivy-[revision].xml" />
> > <artifact pattern="${internal.repo.dir}/${m2pattern}" />
> > </filesystem>
> >
> > <url name="nexus-snapshots-noplatform" m2compatible="true"
> > checkmodified="true" changingPattern=".*">
> > <artifact
> > pattern="${mycompany.repo.url}/snapshots/${m2pattern}" />
> > </url>
> >
> > <chain name="chain" checkmodified="true">
> > <resolver ref="internal"/>
> > <resolver ref="nexus-snapshots-noplatform"/>
> > </chain>
> > </resolvers>
> >
> > </ivysettings>
> >
> > L.K.
> >
> >
> > -----Original Message-----
> > From: Zac Jacobson [mailto:pie.fed@gmail.com]
> > Sent: Sunday, March 22, 2015 6:07 PM
> > To: ivy-user@ant.apache.org
> > Subject: Re: cache busting and integration question
> >
> > If you set your resolver’s checkmodified flag to true, it will verify
the
> > timestamp of the published artifact and pull down a more recent version
if
> > the repo is newer than the cache. I’ve found that if you have chained
> > resolvers, you need to do this at the top level and it affects all
repos,
> > not just your snapshot repository.
> >
> > Also, if you set the force attribute for your local resolver, it will
> > always take a version from there rather than going to other resolvers in
> > the chain. Once you’ve got changes published to your snapshot or release
> > repo, you’ll want to clean that artifact out of your local repo.
> >
> >
http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html
> > <
> >
http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html
> > >
> > > On Mar 20, 2015, at 15:12, Loren Kratzke <LK...@blueorigin.com>
> > wrote:
> > >
> > > I am encountering challenges with the Ivy cache and also with setting
up
> > what I would consider to be a typical (if not classic) developer
workflow.
> > Here is the desired workflow:
> > >
> > > I have a Nexus repository (releases and snapshots) plus a local file
> > system repository used for local development. I have Project which
depends
> > upon Library. I wish to modify Library, publish locally, and then pull
it
> > into the local build of Project.
> > >
> > > And for my Jenkins "continuous" integration server, if the copy of
> > Library-1.0.0-SNAPSHOT in Nexus is newer than what is in the cache on
the
> > build machine, then I would like Ivy to grab the newer version (just
like
> > Maven does).
> > >
> > > ISSUE #1
> > >
> > > Assuming I start with an empty cache and empty internal repo on my
local
> > dev machine, I build Project and it pulls Library from Nexus. Perfect.
> > Works great. But then if I publish Library locally and build Project
again,
> > I will get the cached version of Library that came from Nexus. No matter
> > what I do, I will always get the cached copy. This happens even though
my
> > chain resolver specifies my internal repo before the Nexus repo.
> > >
> > > If I blow away the cache, then I get my locally published build, but
> > only if I blow away the cache. Otherwise I get stale stuff. Issue #2
below
> > is all about stale stuff.
> > >
> > > I debugged Ivy (for hours upon hours) and during the resolve, it
checks
> > my internal repo, finds the fresh artifact (yay!) and the next thing it
> > does is checks its cache, finds a module descriptor that points at
Nexus,
> > and proceeds to pull in the wrong artifact into Project.
> > >
> > > The behavior is either a bug, or implies that Ivy assumes that an
> > artifact can only come from one place, and if it came from there once,
it
> > will come from there forever, and will never change. Again, Ivy finds
the
> > artifact in local repo, stops searching, then ends up delivering the
cached
> > artifact that came from Nexus sometime earlier.
> > >
> > > ISSUE #2
> > >
> > > I have been working on this problem for quite some time and I thought
I
> > had it fixed, but I don't. That is, my integration server needs the
latest
> > snapshot build of Library-1.0.0-SNAPSHOT. But what I get is the cached
> > version from the Ivy cache. An SVN commit happens, triggers a build of
> > Library, Library get published, triggers a build of Project, and a stale
> > build of Library is delivered to Project from the cache (which breaks
> > Project build).
> > >
> > > SUMMARY
> > >
> > > Ivy will resolve and cache deps, but once a dep is in the cache,
that's
> > what I get forever until I blow away the cache. I see this question
asked
> > quite often regarding obtaining latest integration versions, and I see
that
> > Ivy claims to support the local development workflow, but I have yet to
see
> > a working example of either of these. Surely some of you have solved
these
> > issues.
> > >
> > > Is there any place anywhere in the world that has an example of
> > development workflow configuration and/or integration configuration? Can
> > anybody provide an example? So far I have not found anything, and
advice on
> > the net seems to be general suggestions like "try this, try that" and
> > nobody really has the configuration solution. Is Ivy capable of these
> > behaviors or do I need to hand roll a dependency management solution.
Maven
> > is too strict for these projects (C++ code). Ivy is good if I can just
> > obtain these fundamental behaviors.
> > >
> > > Note that I have not provided any configs in this message that
> > demonstrate the issues I am having but definitely can upon request.
> > >
> > > Thanks in advance,
> > >
> > > L.K.
> > >
> >
> >
Re: cache busting and integration question
Posted by Marc De Boeck <md...@gmail.com>.
It's an interesting problem to investigate, and I understand your
frustration. But since we don't have exactly the same setup as you, it's
not evident to tell exactly what the problem is and how you can solve it.
We also have an environment with a local and a central repo, but we don't
use the checkmodified attribute. If it would work, it is indeed a good
idea to setup such a resolver chain with SNAPSHOT builds for the developers.
In our case (without checkmodified=true) the detection of changes in both
local and central repo works correctly for us.
Now, I can only search with you in the docs hoping to find a missing
setting (if it's not an actual bug as you suggested).
I suppose that the publication date of the ivy.xml that you published in
your local repo is more recent than the one in your cache ? Did you also
try setting the "changing" attribute directly in the dependency itself
(allthough I expect the behaviour will be similar).
Could you try a setup where you only publish and resolve from the local
repo (so not chained with your Nexus) ?
You also wrote that latest.integration is a non-starter for you ? So could
you describe how you specify your dependencies ? If you specify an exact
revision in your dependency (like rev="1.0.0-SNAPSHOT), then maybe you
could also try to set the attribute "alwaysCheckExactRevision" in the
settings of your resolver.
Regards,
Marc
2015-03-23 17:33 GMT+01:00 Loren Kratzke <LK...@blueorigin.com>:
> I am not getting the results that you describe. I am testing with a simple
> project that publishes a text file with the current date and time so it is
> very easy to see when I get a fresh or stale artifact. When I run using the
> configuration below, I get the following results when I publish my
> dependency and then consume it in the downstream project:
>
> [pub nexus] [get nexus] good
> [pub nexus] [get nexus] good
> [pub local] [get cached nexus artifact] bad
> [pub nexus] [get cached nexus artfiact] bad
>
> So as the first two tests indicate, I can get the latest snapshot from
> Nexus. That will work well for the integration server if we leave the local
> repo out.
>
> But as the 3rd and 4th tests indicate, if I do a local publish then
> everything goes to hell. I do not get my local artifact, I get the cached
> artifact. And if I publish to Nexus again, I still get the cached artifact.
> In fact I get the cached artifact forever after a local publish no matter
> what gets published or where.
>
> That can't be desired behavior. There has to be a bug here somewhere, but
> I would be most happy with a work around. This behavior was witnessed when
> I debugged. It would find a local artifact, check its cache to find an ivy
> file that points at Nexus, and ultimately return a stale Nexus artifact
> from the cache.
>
> My setup is very simple - One Nexus repository and one local repository.
> Nothing fancy here. I have tried every combination of "checkmodified",
> "changingPattern", and "force" that the configuration can tolerate without
> it quacking a fur ball back at me but I cannot seem to get the desired
> behavior. Please find the err in my ways and show me where I have wandered
> off the ranch.
>
> I thought I had this working - once - and I changed something and can't
> seem to get it back. I thought I had it working as you described where it
> would pick up the snapshots, then it would pick up my local artifact, and
> it would always deliver the local artifact if present until the local cache
> was torched. That behavior is ok with me if I could get it back.
>
> Here is my settings.xml
>
> <ivysettings>
> <caches defaultCacheDir="${user.home}/.ivy2/cache"/>
> <settings defaultResolver="chain"/>
> <credentials host="nexus.mycompany.com" realm="Sonatype Nexus
> Repository Manager" username="name" passwd="pass"/>
>
> <property name="m2pattern"
> value="[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]"/>
> <property name="internal.repo.dir" value="${user.home}/.ivy2/m2"/>
> <property name="mycompany.repo.url" value="
> http://nexus.mycompany.com:8081/nexus/content/repositories"/>
>
> <resolvers>
> <filesystem name="internal" force="true" checkmodified="true"
> changingPattern=".*">
> <ivy
> pattern="${internal.repo.dir}/[module]/ivy-[revision].xml" />
> <artifact pattern="${internal.repo.dir}/${m2pattern}" />
> </filesystem>
>
> <url name="nexus-snapshots-noplatform" m2compatible="true"
> checkmodified="true" changingPattern=".*">
> <artifact
> pattern="${mycompany.repo.url}/snapshots/${m2pattern}" />
> </url>
>
> <chain name="chain" checkmodified="true">
> <resolver ref="internal"/>
> <resolver ref="nexus-snapshots-noplatform"/>
> </chain>
> </resolvers>
>
> </ivysettings>
>
> L.K.
>
>
> -----Original Message-----
> From: Zac Jacobson [mailto:pie.fed@gmail.com]
> Sent: Sunday, March 22, 2015 6:07 PM
> To: ivy-user@ant.apache.org
> Subject: Re: cache busting and integration question
>
> If you set your resolver’s checkmodified flag to true, it will verify the
> timestamp of the published artifact and pull down a more recent version if
> the repo is newer than the cache. I’ve found that if you have chained
> resolvers, you need to do this at the top level and it affects all repos,
> not just your snapshot repository.
>
> Also, if you set the force attribute for your local resolver, it will
> always take a version from there rather than going to other resolvers in
> the chain. Once you’ve got changes published to your snapshot or release
> repo, you’ll want to clean that artifact out of your local repo.
>
> http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html
> <
> http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html
> >
> > On Mar 20, 2015, at 15:12, Loren Kratzke <LK...@blueorigin.com>
> wrote:
> >
> > I am encountering challenges with the Ivy cache and also with setting up
> what I would consider to be a typical (if not classic) developer workflow.
> Here is the desired workflow:
> >
> > I have a Nexus repository (releases and snapshots) plus a local file
> system repository used for local development. I have Project which depends
> upon Library. I wish to modify Library, publish locally, and then pull it
> into the local build of Project.
> >
> > And for my Jenkins "continuous" integration server, if the copy of
> Library-1.0.0-SNAPSHOT in Nexus is newer than what is in the cache on the
> build machine, then I would like Ivy to grab the newer version (just like
> Maven does).
> >
> > ISSUE #1
> >
> > Assuming I start with an empty cache and empty internal repo on my local
> dev machine, I build Project and it pulls Library from Nexus. Perfect.
> Works great. But then if I publish Library locally and build Project again,
> I will get the cached version of Library that came from Nexus. No matter
> what I do, I will always get the cached copy. This happens even though my
> chain resolver specifies my internal repo before the Nexus repo.
> >
> > If I blow away the cache, then I get my locally published build, but
> only if I blow away the cache. Otherwise I get stale stuff. Issue #2 below
> is all about stale stuff.
> >
> > I debugged Ivy (for hours upon hours) and during the resolve, it checks
> my internal repo, finds the fresh artifact (yay!) and the next thing it
> does is checks its cache, finds a module descriptor that points at Nexus,
> and proceeds to pull in the wrong artifact into Project.
> >
> > The behavior is either a bug, or implies that Ivy assumes that an
> artifact can only come from one place, and if it came from there once, it
> will come from there forever, and will never change. Again, Ivy finds the
> artifact in local repo, stops searching, then ends up delivering the cached
> artifact that came from Nexus sometime earlier.
> >
> > ISSUE #2
> >
> > I have been working on this problem for quite some time and I thought I
> had it fixed, but I don't. That is, my integration server needs the latest
> snapshot build of Library-1.0.0-SNAPSHOT. But what I get is the cached
> version from the Ivy cache. An SVN commit happens, triggers a build of
> Library, Library get published, triggers a build of Project, and a stale
> build of Library is delivered to Project from the cache (which breaks
> Project build).
> >
> > SUMMARY
> >
> > Ivy will resolve and cache deps, but once a dep is in the cache, that's
> what I get forever until I blow away the cache. I see this question asked
> quite often regarding obtaining latest integration versions, and I see that
> Ivy claims to support the local development workflow, but I have yet to see
> a working example of either of these. Surely some of you have solved these
> issues.
> >
> > Is there any place anywhere in the world that has an example of
> development workflow configuration and/or integration configuration? Can
> anybody provide an example? So far I have not found anything, and advice on
> the net seems to be general suggestions like "try this, try that" and
> nobody really has the configuration solution. Is Ivy capable of these
> behaviors or do I need to hand roll a dependency management solution. Maven
> is too strict for these projects (C++ code). Ivy is good if I can just
> obtain these fundamental behaviors.
> >
> > Note that I have not provided any configs in this message that
> demonstrate the issues I am having but definitely can upon request.
> >
> > Thanks in advance,
> >
> > L.K.
> >
>
>
RE: cache busting and integration question
Posted by Loren Kratzke <LK...@blueorigin.com>.
That is excellent insight. Thank you for that. The things you found encourage me to fix the issues that I am seeing (or patch them).
We are publishing and consuming C++ artifacts using Ivy/Nexus. We also have a Maven side of the house but they use a different Nexus repo dedicated to pure Java artifacts. So I have not seen any Ivy/Maven cross pollination issues.
Based upon your comments however I will inspect the metadata files and check into the timestamps. Hopefully the parts that we are using on the Ivy side are ok.
I agree about the cache implementation being tied to the resolvers. I considered writing a new resolver but the resolution and download of the artifacts works just fine. It is higher up where the problems begin, like the chain resolver and cache as you mentioned.
The one thing that is working for us right now is torching the ivy cache before every build. That does not imply very nice things about the ivy cache functionality. It is more of a liability than as asset to us.
I will try to fix this. Ivy is a good technology. It just needs some tough love.
L.K.
-----Original Message-----
From: Josh Suereth [mailto:joshua.suereth@gmail.com]
Sent: Monday, March 23, 2015 2:16 PM
To: ivy-user@ant.apache.org
Subject: RE: cache busting and integration question
We experienced this bug in sbt as well.
If I recall correctly, it turns out that the implementation of most resolvers is rather tied to the implementation of the cache. One artifact of this is that the chain resolver wasn't ever looking for -SNAPSHOTs when we wanted it to.
We actually did the following, recently ->
1. A new chain resolver implementation which forces a date check against
the remote repository. It turns out this exposed issues in publication
date not being a universal thing. E.g. pom.xml files weren't reporting legitimate publication dates, leading to any attempt to pull "latest" for a snapshot being wrong.
2. We created a new implementation of DependencyResolver which actually
uses aether. This had two side effects:
2a. We avoid parent pom issues which still sneak in when using -SNAPSHOT parents or weird profiles in POM's (why hadoop, why!!!)
2b. We always correctly update maven-metadata.xml and maven-metadata-local.xml. this means we'll have a somewhat legitimate publication date to do loose comparisons on (not exact but good enough).
Both are still in experimental stage, but both now pass our previously failing test cases around -SNAPSHOT updates, when used together.
What does this mean for ivy?
1. Ivy should probably support maven-metadata(-local).xml directly in its maven code.
2. Ivy chain resolver / ivy cache may need tweaks for dynamic revisions to make sure, even if the version number is the same, that remote repositories (or just metadata.xml) is checked.
This adventure lead me to believe that the ivy cache api itself may need a
3.0 overhaul, and the maven code may need to be further separated from "core" usage. While the loss of reuse would be sad, I think the reality is maven just doesn't work the way ivy was designed.
On Mar 23, 2015 12:33 PM, "Loren Kratzke" <LK...@blueorigin.com> wrote:
> I am not getting the results that you describe. I am testing with a
> simple project that publishes a text file with the current date and
> time so it is very easy to see when I get a fresh or stale artifact.
> When I run using the configuration below, I get the following results
> when I publish my dependency and then consume it in the downstream project:
>
> [pub nexus] [get nexus] good
> [pub nexus] [get nexus] good
> [pub local] [get cached nexus artifact] bad [pub nexus] [get cached
> nexus artfiact] bad
>
> So as the first two tests indicate, I can get the latest snapshot from
> Nexus. That will work well for the integration server if we leave the
> local repo out.
>
> But as the 3rd and 4th tests indicate, if I do a local publish then
> everything goes to hell. I do not get my local artifact, I get the
> cached artifact. And if I publish to Nexus again, I still get the cached artifact.
> In fact I get the cached artifact forever after a local publish no
> matter what gets published or where.
>
> That can't be desired behavior. There has to be a bug here somewhere,
> but I would be most happy with a work around. This behavior was
> witnessed when I debugged. It would find a local artifact, check its
> cache to find an ivy file that points at Nexus, and ultimately return
> a stale Nexus artifact from the cache.
>
> My setup is very simple - One Nexus repository and one local repository.
> Nothing fancy here. I have tried every combination of "checkmodified",
> "changingPattern", and "force" that the configuration can tolerate
> without it quacking a fur ball back at me but I cannot seem to get the
> desired behavior. Please find the err in my ways and show me where I
> have wandered off the ranch.
>
> I thought I had this working - once - and I changed something and
> can't seem to get it back. I thought I had it working as you described
> where it would pick up the snapshots, then it would pick up my local
> artifact, and it would always deliver the local artifact if present
> until the local cache was torched. That behavior is ok with me if I could get it back.
>
> Here is my settings.xml
>
> <ivysettings>
> <caches defaultCacheDir="${user.home}/.ivy2/cache"/>
> <settings defaultResolver="chain"/>
> <credentials host="nexus.mycompany.com" realm="Sonatype Nexus
> Repository Manager" username="name" passwd="pass"/>
>
> <property name="m2pattern"
> value="[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]"/>
> <property name="internal.repo.dir" value="${user.home}/.ivy2/m2"/>
> <property name="mycompany.repo.url" value="
> http://nexus.mycompany.com:8081/nexus/content/repositories"/>
>
> <resolvers>
> <filesystem name="internal" force="true" checkmodified="true"
> changingPattern=".*">
> <ivy
> pattern="${internal.repo.dir}/[module]/ivy-[revision].xml" />
> <artifact pattern="${internal.repo.dir}/${m2pattern}" />
> </filesystem>
>
> <url name="nexus-snapshots-noplatform" m2compatible="true"
> checkmodified="true" changingPattern=".*">
> <artifact
> pattern="${mycompany.repo.url}/snapshots/${m2pattern}" />
> </url>
>
> <chain name="chain" checkmodified="true">
> <resolver ref="internal"/>
> <resolver ref="nexus-snapshots-noplatform"/>
> </chain>
> </resolvers>
>
> </ivysettings>
>
> L.K.
RE: cache busting and integration question
Posted by Josh Suereth <jo...@gmail.com>.
We experienced this bug in sbt as well.
If I recall correctly, it turns out that the implementation of most
resolvers is rather tied to the implementation of the cache. One artifact
of this is that the chain resolver wasn't ever looking for -SNAPSHOTs when
we wanted it to.
We actually did the following, recently ->
1. A new chain resolver implementation which forces a date check against
the remote repository. It turns out this exposed issues in publication
date not being a universal thing. E.g. pom.xml files weren't reporting
legitimate publication dates, leading to any attempt to pull "latest" for a
snapshot being wrong.
2. We created a new implementation of DependencyResolver which actually
uses aether. This had two side effects:
2a. We avoid parent pom issues which still sneak in when using -SNAPSHOT
parents or weird profiles in POM's (why hadoop, why!!!)
2b. We always correctly update maven-metadata.xml and
maven-metadata-local.xml. this means we'll have a somewhat legitimate
publication date to do loose comparisons on (not exact but good enough).
Both are still in experimental stage, but both now pass our previously
failing test cases around -SNAPSHOT updates, when used together.
What does this mean for ivy?
1. Ivy should probably support maven-metadata(-local).xml directly in its
maven code.
2. Ivy chain resolver / ivy cache may need tweaks for dynamic revisions to
make sure, even if the version number is the same, that remote repositories
(or just metadata.xml) is checked.
This adventure lead me to believe that the ivy cache api itself may need a
3.0 overhaul, and the maven code may need to be further separated from
"core" usage. While the loss of reuse would be sad, I think the reality is
maven just doesn't work the way ivy was designed.
On Mar 23, 2015 12:33 PM, "Loren Kratzke" <LK...@blueorigin.com> wrote:
> I am not getting the results that you describe. I am testing with a simple
> project that publishes a text file with the current date and time so it is
> very easy to see when I get a fresh or stale artifact. When I run using the
> configuration below, I get the following results when I publish my
> dependency and then consume it in the downstream project:
>
> [pub nexus] [get nexus] good
> [pub nexus] [get nexus] good
> [pub local] [get cached nexus artifact] bad
> [pub nexus] [get cached nexus artfiact] bad
>
> So as the first two tests indicate, I can get the latest snapshot from
> Nexus. That will work well for the integration server if we leave the local
> repo out.
>
> But as the 3rd and 4th tests indicate, if I do a local publish then
> everything goes to hell. I do not get my local artifact, I get the cached
> artifact. And if I publish to Nexus again, I still get the cached artifact.
> In fact I get the cached artifact forever after a local publish no matter
> what gets published or where.
>
> That can't be desired behavior. There has to be a bug here somewhere, but
> I would be most happy with a work around. This behavior was witnessed when
> I debugged. It would find a local artifact, check its cache to find an ivy
> file that points at Nexus, and ultimately return a stale Nexus artifact
> from the cache.
>
> My setup is very simple - One Nexus repository and one local repository.
> Nothing fancy here. I have tried every combination of "checkmodified",
> "changingPattern", and "force" that the configuration can tolerate without
> it quacking a fur ball back at me but I cannot seem to get the desired
> behavior. Please find the err in my ways and show me where I have wandered
> off the ranch.
>
> I thought I had this working - once - and I changed something and can't
> seem to get it back. I thought I had it working as you described where it
> would pick up the snapshots, then it would pick up my local artifact, and
> it would always deliver the local artifact if present until the local cache
> was torched. That behavior is ok with me if I could get it back.
>
> Here is my settings.xml
>
> <ivysettings>
> <caches defaultCacheDir="${user.home}/.ivy2/cache"/>
> <settings defaultResolver="chain"/>
> <credentials host="nexus.mycompany.com" realm="Sonatype Nexus
> Repository Manager" username="name" passwd="pass"/>
>
> <property name="m2pattern"
> value="[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]"/>
> <property name="internal.repo.dir" value="${user.home}/.ivy2/m2"/>
> <property name="mycompany.repo.url" value="
> http://nexus.mycompany.com:8081/nexus/content/repositories"/>
>
> <resolvers>
> <filesystem name="internal" force="true" checkmodified="true"
> changingPattern=".*">
> <ivy
> pattern="${internal.repo.dir}/[module]/ivy-[revision].xml" />
> <artifact pattern="${internal.repo.dir}/${m2pattern}" />
> </filesystem>
>
> <url name="nexus-snapshots-noplatform" m2compatible="true"
> checkmodified="true" changingPattern=".*">
> <artifact
> pattern="${mycompany.repo.url}/snapshots/${m2pattern}" />
> </url>
>
> <chain name="chain" checkmodified="true">
> <resolver ref="internal"/>
> <resolver ref="nexus-snapshots-noplatform"/>
> </chain>
> </resolvers>
>
> </ivysettings>
>
> L.K.
>
>
> -----Original Message-----
> From: Zac Jacobson [mailto:pie.fed@gmail.com]
> Sent: Sunday, March 22, 2015 6:07 PM
> To: ivy-user@ant.apache.org
> Subject: Re: cache busting and integration question
>
> If you set your resolver’s checkmodified flag to true, it will verify the
> timestamp of the published artifact and pull down a more recent version if
> the repo is newer than the cache. I’ve found that if you have chained
> resolvers, you need to do this at the top level and it affects all repos,
> not just your snapshot repository.
>
> Also, if you set the force attribute for your local resolver, it will
> always take a version from there rather than going to other resolvers in
> the chain. Once you’ve got changes published to your snapshot or release
> repo, you’ll want to clean that artifact out of your local repo.
>
> http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html
> <
> http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html
> >
> > On Mar 20, 2015, at 15:12, Loren Kratzke <LK...@blueorigin.com>
> wrote:
> >
> > I am encountering challenges with the Ivy cache and also with setting up
> what I would consider to be a typical (if not classic) developer workflow.
> Here is the desired workflow:
> >
> > I have a Nexus repository (releases and snapshots) plus a local file
> system repository used for local development. I have Project which depends
> upon Library. I wish to modify Library, publish locally, and then pull it
> into the local build of Project.
> >
> > And for my Jenkins "continuous" integration server, if the copy of
> Library-1.0.0-SNAPSHOT in Nexus is newer than what is in the cache on the
> build machine, then I would like Ivy to grab the newer version (just like
> Maven does).
> >
> > ISSUE #1
> >
> > Assuming I start with an empty cache and empty internal repo on my local
> dev machine, I build Project and it pulls Library from Nexus. Perfect.
> Works great. But then if I publish Library locally and build Project again,
> I will get the cached version of Library that came from Nexus. No matter
> what I do, I will always get the cached copy. This happens even though my
> chain resolver specifies my internal repo before the Nexus repo.
> >
> > If I blow away the cache, then I get my locally published build, but
> only if I blow away the cache. Otherwise I get stale stuff. Issue #2 below
> is all about stale stuff.
> >
> > I debugged Ivy (for hours upon hours) and during the resolve, it checks
> my internal repo, finds the fresh artifact (yay!) and the next thing it
> does is checks its cache, finds a module descriptor that points at Nexus,
> and proceeds to pull in the wrong artifact into Project.
> >
> > The behavior is either a bug, or implies that Ivy assumes that an
> artifact can only come from one place, and if it came from there once, it
> will come from there forever, and will never change. Again, Ivy finds the
> artifact in local repo, stops searching, then ends up delivering the cached
> artifact that came from Nexus sometime earlier.
> >
> > ISSUE #2
> >
> > I have been working on this problem for quite some time and I thought I
> had it fixed, but I don't. That is, my integration server needs the latest
> snapshot build of Library-1.0.0-SNAPSHOT. But what I get is the cached
> version from the Ivy cache. An SVN commit happens, triggers a build of
> Library, Library get published, triggers a build of Project, and a stale
> build of Library is delivered to Project from the cache (which breaks
> Project build).
> >
> > SUMMARY
> >
> > Ivy will resolve and cache deps, but once a dep is in the cache, that's
> what I get forever until I blow away the cache. I see this question asked
> quite often regarding obtaining latest integration versions, and I see that
> Ivy claims to support the local development workflow, but I have yet to see
> a working example of either of these. Surely some of you have solved these
> issues.
> >
> > Is there any place anywhere in the world that has an example of
> development workflow configuration and/or integration configuration? Can
> anybody provide an example? So far I have not found anything, and advice on
> the net seems to be general suggestions like "try this, try that" and
> nobody really has the configuration solution. Is Ivy capable of these
> behaviors or do I need to hand roll a dependency management solution. Maven
> is too strict for these projects (C++ code). Ivy is good if I can just
> obtain these fundamental behaviors.
> >
> > Note that I have not provided any configs in this message that
> demonstrate the issues I am having but definitely can upon request.
> >
> > Thanks in advance,
> >
> > L.K.
> >
>
>
RE: cache busting and integration question
Posted by Loren Kratzke <LK...@blueorigin.com>.
I am not getting the results that you describe. I am testing with a simple project that publishes a text file with the current date and time so it is very easy to see when I get a fresh or stale artifact. When I run using the configuration below, I get the following results when I publish my dependency and then consume it in the downstream project:
[pub nexus] [get nexus] good
[pub nexus] [get nexus] good
[pub local] [get cached nexus artifact] bad
[pub nexus] [get cached nexus artfiact] bad
So as the first two tests indicate, I can get the latest snapshot from Nexus. That will work well for the integration server if we leave the local repo out.
But as the 3rd and 4th tests indicate, if I do a local publish then everything goes to hell. I do not get my local artifact, I get the cached artifact. And if I publish to Nexus again, I still get the cached artifact. In fact I get the cached artifact forever after a local publish no matter what gets published or where.
That can't be desired behavior. There has to be a bug here somewhere, but I would be most happy with a work around. This behavior was witnessed when I debugged. It would find a local artifact, check its cache to find an ivy file that points at Nexus, and ultimately return a stale Nexus artifact from the cache.
My setup is very simple - One Nexus repository and one local repository. Nothing fancy here. I have tried every combination of "checkmodified", "changingPattern", and "force" that the configuration can tolerate without it quacking a fur ball back at me but I cannot seem to get the desired behavior. Please find the err in my ways and show me where I have wandered off the ranch.
I thought I had this working - once - and I changed something and can't seem to get it back. I thought I had it working as you described where it would pick up the snapshots, then it would pick up my local artifact, and it would always deliver the local artifact if present until the local cache was torched. That behavior is ok with me if I could get it back.
Here is my settings.xml
<ivysettings>
<caches defaultCacheDir="${user.home}/.ivy2/cache"/>
<settings defaultResolver="chain"/>
<credentials host="nexus.mycompany.com" realm="Sonatype Nexus Repository Manager" username="name" passwd="pass"/>
<property name="m2pattern" value="[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]"/>
<property name="internal.repo.dir" value="${user.home}/.ivy2/m2"/>
<property name="mycompany.repo.url" value="http://nexus.mycompany.com:8081/nexus/content/repositories"/>
<resolvers>
<filesystem name="internal" force="true" checkmodified="true" changingPattern=".*">
<ivy pattern="${internal.repo.dir}/[module]/ivy-[revision].xml" />
<artifact pattern="${internal.repo.dir}/${m2pattern}" />
</filesystem>
<url name="nexus-snapshots-noplatform" m2compatible="true" checkmodified="true" changingPattern=".*">
<artifact pattern="${mycompany.repo.url}/snapshots/${m2pattern}" />
</url>
<chain name="chain" checkmodified="true">
<resolver ref="internal"/>
<resolver ref="nexus-snapshots-noplatform"/>
</chain>
</resolvers>
</ivysettings>
L.K.
-----Original Message-----
From: Zac Jacobson [mailto:pie.fed@gmail.com]
Sent: Sunday, March 22, 2015 6:07 PM
To: ivy-user@ant.apache.org
Subject: Re: cache busting and integration question
If you set your resolver’s checkmodified flag to true, it will verify the timestamp of the published artifact and pull down a more recent version if the repo is newer than the cache. I’ve found that if you have chained resolvers, you need to do this at the top level and it affects all repos, not just your snapshot repository.
Also, if you set the force attribute for your local resolver, it will always take a version from there rather than going to other resolvers in the chain. Once you’ve got changes published to your snapshot or release repo, you’ll want to clean that artifact out of your local repo.
http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html <http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html>
> On Mar 20, 2015, at 15:12, Loren Kratzke <LK...@blueorigin.com> wrote:
>
> I am encountering challenges with the Ivy cache and also with setting up what I would consider to be a typical (if not classic) developer workflow. Here is the desired workflow:
>
> I have a Nexus repository (releases and snapshots) plus a local file system repository used for local development. I have Project which depends upon Library. I wish to modify Library, publish locally, and then pull it into the local build of Project.
>
> And for my Jenkins "continuous" integration server, if the copy of Library-1.0.0-SNAPSHOT in Nexus is newer than what is in the cache on the build machine, then I would like Ivy to grab the newer version (just like Maven does).
>
> ISSUE #1
>
> Assuming I start with an empty cache and empty internal repo on my local dev machine, I build Project and it pulls Library from Nexus. Perfect. Works great. But then if I publish Library locally and build Project again, I will get the cached version of Library that came from Nexus. No matter what I do, I will always get the cached copy. This happens even though my chain resolver specifies my internal repo before the Nexus repo.
>
> If I blow away the cache, then I get my locally published build, but only if I blow away the cache. Otherwise I get stale stuff. Issue #2 below is all about stale stuff.
>
> I debugged Ivy (for hours upon hours) and during the resolve, it checks my internal repo, finds the fresh artifact (yay!) and the next thing it does is checks its cache, finds a module descriptor that points at Nexus, and proceeds to pull in the wrong artifact into Project.
>
> The behavior is either a bug, or implies that Ivy assumes that an artifact can only come from one place, and if it came from there once, it will come from there forever, and will never change. Again, Ivy finds the artifact in local repo, stops searching, then ends up delivering the cached artifact that came from Nexus sometime earlier.
>
> ISSUE #2
>
> I have been working on this problem for quite some time and I thought I had it fixed, but I don't. That is, my integration server needs the latest snapshot build of Library-1.0.0-SNAPSHOT. But what I get is the cached version from the Ivy cache. An SVN commit happens, triggers a build of Library, Library get published, triggers a build of Project, and a stale build of Library is delivered to Project from the cache (which breaks Project build).
>
> SUMMARY
>
> Ivy will resolve and cache deps, but once a dep is in the cache, that's what I get forever until I blow away the cache. I see this question asked quite often regarding obtaining latest integration versions, and I see that Ivy claims to support the local development workflow, but I have yet to see a working example of either of these. Surely some of you have solved these issues.
>
> Is there any place anywhere in the world that has an example of development workflow configuration and/or integration configuration? Can anybody provide an example? So far I have not found anything, and advice on the net seems to be general suggestions like "try this, try that" and nobody really has the configuration solution. Is Ivy capable of these behaviors or do I need to hand roll a dependency management solution. Maven is too strict for these projects (C++ code). Ivy is good if I can just obtain these fundamental behaviors.
>
> Note that I have not provided any configs in this message that demonstrate the issues I am having but definitely can upon request.
>
> Thanks in advance,
>
> L.K.
>
RE: cache busting and integration question
Posted by Loren Kratzke <LK...@blueorigin.com>.
Here are the projects that I am using to test and debug this situation. Each project uses identical build.xml and settings.xml. They differ only by their ivy.xml files. Here is the project layout:
ivy_stale_app
- build.xml
- ivy.xml
- ivysettings.xml
ivy_stale_dep
- build.xml
- ivy.xml
- ivysettings.xml
If you create the projects above using the files below, and edit such that you can publish/retrieve from a Nexus server somewhere, then you can reproduce this issue.
The dep project creates a file named date_time.txt with the date and time, creates an artifact named mydep-1.0.0-SNAPSHOT from that file, and publishes it. The app project consumes that dependency (and generates a similar but otherwise useless artifact that can be ignored here). Monitor the date/time value in ivy_stale_app/lib/mydep-1.0.0-SNAPSHOT in the app project to know if/when you pick up a fresh artifact.
Here is the messed up behavior with this setup:
1. If you publish the dep to Nexus (over and over) you will get the fresh artifact in the app every time. (GOOD)
2. If you then publish to the internal repo, you will get the cached Nexus artifact the first time. (BAD)
3. If you publish to the internal repo a second time, you will get the internal repo artifact. (GOOD after BAD)
4. If you publish to the internal repo over and over, you will get fresh artifacts in the app every time. (GOOD)
5. If you publish to Nexus any time after publishing to internal, you get the internal artifact. (regardless of the "force" attribute value)
The "force" setting on the internal repo had no effect one way or the other in my testing with the settings below. All other setting changes have only random and degrading effect from the above description (such as never getting anything fresh from anywhere ever).
If I could fix #2 above I could live with Ivy. But I can't tell my customers to be sure to build twice if developing locally. I just can't do that.
So given all of that info, and the repo steps, and the files, please anybody show me what I am doing wrong, or agree that this is odd and undesirable behavior that should be fixed, or tell me that Ivy just can't do what I want it to do.
*** BEGIN build.xml ***
<project name="ivy_stale_app" xmlns:ivy="antlib:org.apache.ivy.ant">
<description>Cache test - consumer</description>
<taskdef resource="net/sf/antcontrib/antlib.xml"/>
<taskdef resource="org/apache/ivy/ant/antlib.xml"/>
<ivy:settings file="ivysettings.xml"/>
<tstamp>
<format property="date_time" pattern="yyyy-MM-dd_HH:mm:ss"/>
</tstamp>
<property name="build.version" value="1.0.0-SNAPSHOT"/>
<target name="clean" description="Clean the project">
<delete dir="build"/>
<delete dir="dist"/>
<delete dir="lib"/>
</target>
<target name="retrieve" description="Fetch dependencies">
<ivy:resolve/>
<ivy:retrieve pattern="lib/[artifact]-[revision](-[classifier]).[ext]"/>
</target>
<target name="build" depends="clean,retrieve" description="build the project">
<mkdir dir="build"/>
<echo message="${date_time}" file="build/date_time.txt"/>
</target>
<target name="dist" depends="build" description="create dist that can be published">
<mkdir dir="dist"/>
<copy file="build/date_time.txt" tofile="dist/${ivy.module}-${build.version}.txt"/>
</target>
<target name="preparePublish" depends="dist" description="Prepare for publish">
<ivy:deliver deliverpattern="dist/ivy-${build.version}.xml"
pubrevision="${build.version}"
status="integration"/>
<ivy:makepom ivyfile="dist/ivy-${build.version}.xml"
pomfile="dist/${ivy.module}-${build.version}.pom"/>
</target>
<target name="publish-local" depends="preparePublish" description="Publish artifacts locally">
<ivy:publish artifactspattern="dist/[artifact]-${build.version}(-[classifier]).[ext]"
resolver="internal"
pubrevision="${build.version}"
status="integration"
overwrite="true"/>
</target>
<target name="publish-nexus" depends="preparePublish" description="Publish artifacts to Nexus">
<ivy:publish artifactspattern="dist/[artifact]-${build.version}(-[classifier]).[ext]"
resolver="nexus-snapshots-noplatform"
pubrevision="${build.version}"
status="integration"
overwrite="true"/>
</target>
</project>
*** END build.xml ***
*** BEGIN ivysettings.xml ***
<ivysettings>
<caches defaultCacheDir="${user.home}/.ivy2/cache"/>
<settings defaultResolver="chain"/>
<credentials host="nexus.mycompany.com" realm="Sonatype Nexus Repository Manager" username="name" passwd="pass"/>
<property name="m2pattern" value="[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]"/>
<property name="internal.repo.dir" value="${user.home}/.ivy2/m2"/>
<property name="mycompany.repo.url" value="http://nexus.mycompany.com:8081/nexus/content/repositories"/>
<!--<property name="ivy.resolver.default.check.modified" value="true"/>-->
<resolvers>
<filesystem name="internal" force="true" checkmodified="true" changingPattern=".*">
<ivy pattern="${internal.repo.dir}/[module]/ivy-[revision].xml" />
<artifact pattern="${internal.repo.dir}/${m2pattern}" />
</filesystem>
<url name="nexus-snapshots-noplatform" m2compatible="true" checkmodified="true" changingPattern=".*">
<artifact pattern="${mycompany.repo.url}/snapshots/${m2pattern}" />
</url>
<chain name="chain" checkmodified="true" changingPattern=".*">
<resolver ref="internal"/>
<resolver ref="nexus-snapshots-noplatform"/>
</chain>
</resolvers>
</ivysettings>
*** END ivysettings.xml ***
*** BEGIN ivy.xml for dependency project ***
<?xml version="1.0" encoding="utf-8"?>
<ivy-module version="2.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd"
xmlns:m="http://ant.apache.org/ivy/maven">
<info organisation="example.scratch" module="mydep"/>
<publications>
<artifact name="mydep" type="pom"/>
<artifact name="mydep" type="txt"/>
</publications>
</ivy-module>
*** END ivy.xml for dependency project ***
*** BEGIN ivy.xml for consumer of dependency ***
<?xml version="1.0" encoding="utf-8"?>
<ivy-module version="2.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd"
xmlns:m="http://ant.apache.org/ivy/maven">
<info organisation="example.scratch" module="myapp"/>
<publications>
<artifact name="myapp" type="pom"/>
<artifact name="myapp" type="txt"/>
</publications>
<dependencies>
<dependency org="example.scratch" name="mydep" rev="1.0.0-SNAPSHOT">
<artifact name="mydep" type="txt"/>
</dependency>
</dependencies>
</ivy-module>
*** END ivy.xml for consumer of dependency ***
L.K.
-----Original Message-----
From: Zac Jacobson [mailto:pie.fed@gmail.com]
Sent: Sunday, March 22, 2015 6:07 PM
To: ivy-user@ant.apache.org
Subject: Re: cache busting and integration question
If you set your resolver’s checkmodified flag to true, it will verify the timestamp of the published artifact and pull down a more recent version if the repo is newer than the cache. I’ve found that if you have chained resolvers, you need to do this at the top level and it affects all repos, not just your snapshot repository.
Also, if you set the force attribute for your local resolver, it will always take a version from there rather than going to other resolvers in the chain. Once you’ve got changes published to your snapshot or release repo, you’ll want to clean that artifact out of your local repo.
Re: cache busting and integration question
Posted by Zac Jacobson <pi...@gmail.com>.
If you set your resolver’s checkmodified flag to true, it will verify the timestamp of the published artifact and pull down a more recent version if the repo is newer than the cache. I’ve found that if you have chained resolvers, you need to do this at the top level and it affects all repos, not just your snapshot repository.
Also, if you set the force attribute for your local resolver, it will always take a version from there rather than going to other resolvers in the chain. Once you’ve got changes published to your snapshot or release repo, you’ll want to clean that artifact out of your local repo.
http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html <http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html>
> On Mar 20, 2015, at 15:12, Loren Kratzke <LK...@blueorigin.com> wrote:
>
> I am encountering challenges with the Ivy cache and also with setting up what I would consider to be a typical (if not classic) developer workflow. Here is the desired workflow:
>
> I have a Nexus repository (releases and snapshots) plus a local file system repository used for local development. I have Project which depends upon Library. I wish to modify Library, publish locally, and then pull it into the local build of Project.
>
> And for my Jenkins "continuous" integration server, if the copy of Library-1.0.0-SNAPSHOT in Nexus is newer than what is in the cache on the build machine, then I would like Ivy to grab the newer version (just like Maven does).
>
> ISSUE #1
>
> Assuming I start with an empty cache and empty internal repo on my local dev machine, I build Project and it pulls Library from Nexus. Perfect. Works great. But then if I publish Library locally and build Project again, I will get the cached version of Library that came from Nexus. No matter what I do, I will always get the cached copy. This happens even though my chain resolver specifies my internal repo before the Nexus repo.
>
> If I blow away the cache, then I get my locally published build, but only if I blow away the cache. Otherwise I get stale stuff. Issue #2 below is all about stale stuff.
>
> I debugged Ivy (for hours upon hours) and during the resolve, it checks my internal repo, finds the fresh artifact (yay!) and the next thing it does is checks its cache, finds a module descriptor that points at Nexus, and proceeds to pull in the wrong artifact into Project.
>
> The behavior is either a bug, or implies that Ivy assumes that an artifact can only come from one place, and if it came from there once, it will come from there forever, and will never change. Again, Ivy finds the artifact in local repo, stops searching, then ends up delivering the cached artifact that came from Nexus sometime earlier.
>
> ISSUE #2
>
> I have been working on this problem for quite some time and I thought I had it fixed, but I don't. That is, my integration server needs the latest snapshot build of Library-1.0.0-SNAPSHOT. But what I get is the cached version from the Ivy cache. An SVN commit happens, triggers a build of Library, Library get published, triggers a build of Project, and a stale build of Library is delivered to Project from the cache (which breaks Project build).
>
> SUMMARY
>
> Ivy will resolve and cache deps, but once a dep is in the cache, that's what I get forever until I blow away the cache. I see this question asked quite often regarding obtaining latest integration versions, and I see that Ivy claims to support the local development workflow, but I have yet to see a working example of either of these. Surely some of you have solved these issues.
>
> Is there any place anywhere in the world that has an example of development workflow configuration and/or integration configuration? Can anybody provide an example? So far I have not found anything, and advice on the net seems to be general suggestions like "try this, try that" and nobody really has the configuration solution. Is Ivy capable of these behaviors or do I need to hand roll a dependency management solution. Maven is too strict for these projects (C++ code). Ivy is good if I can just obtain these fundamental behaviors.
>
> Note that I have not provided any configs in this message that demonstrate the issues I am having but definitely can upon request.
>
> Thanks in advance,
>
> L.K.
>
RE: cache busting and integration question
Posted by Loren Kratzke <LK...@blueorigin.com>.
I actually do implement a "dual publish". I publish a unique snapshot that includes an svn rev number in the version number, and then a generic snapshot that does not. I publish the same artifact twice. For example, myartifact-1.0.0_5353-SNAPSHOT and myartifact-1.0.0-SNAPSHOT. If you want the latest, depend upon the latter (sans the cache issues that I am having).
The latter is always the "latest" build and not a "specific" build. That is by design. I want to be able to perform an integration build against the most recent commits (every commit if possible) and break early if there is a compilation problem. Jenkins will report the latest commit that broke the build. I don't need a unique version number to figure that part out.
Then for functional testing, we use the uniquely versioned artifacts for obvious reasons. You can't test a moving target.
What I wish to avoid is having Ivy evaluate version numbers and determine which is the latest version for integration purposes. Developers often like to get their freak on when it comes to version numbering and eventually Ivy is not going to like it. The only way to determine the "latest" version of something with certainty is in the time domain. Latest means "most recent in time" not alphabetically last, or numerically larger, or it matches my funky regex, or anything else. It is nice that Ivy can attempt to determine the latest version, and perhaps it does so very well, but eventually actual lives will be in the balance and blaming a developer for creating a breaking version number will not be an option for me. I have no choice but to use clock time to determine what the "latest" version of something is. So latest.integration is a non-starter for me.
Please do not take that as a lecture. I am just explaining my constraints in painful detail.
I have been over the pages that you referenced many times. And again, this response is like so many other Ivy responses. It points to a page, which points at configuration documentation, which fails to provide a complete example of the desired behavior. Within the docs, it tells me what you said, that I should really use unique version numbers. And above I have explained in great detail that I do, and why I cannot allow Ivy to interpret those numbers to determine the latest version. I can only use actual time. Nothing less.
In my other response, I displayed a configuration that correctly obtains the latest snapshot from Nexus every time until a local publish to the internal repo happens. Then Ivy delivers only the cached version from the external repo forever (not even the local publish). It will no longer pick up new snapshots from the Nexus repo, and will not deliver the locally published artifact.
When debugging Ivy, Ivy would find the artifact in my internal repository and proceed to deliver the stale artifact from the cache thinking that it is a cached version of the artifact from the internal repository, but it is actually a cached version of the Nexus artifact. I am fairly certain that Ivy is broken. This is definitely a bug.
Either that, or Ivy assumes that an artifact can only come from one place, and once it comes from that place, it can come from no other place, ever. I don't believe that was the intent of the designers.
One last possibility is that I have conjured up a very unfortunate combination of configuration attributes. I hope that this is the case.
I appreciate the pointers to the docs. I am sorry but they fall short otherwise I would have had this working by now. I need a working config, not more Ivy docs. I have been reading those for a year now. Yes, it is possible that I am missing something. I hope that I am. But all indications are that Ivy is broken.
Failing a working config, I will move the discussion to the developers list. If Ivy would not substitute the cached version of the incorrect artifact for the found artifact at the last moment, everything would work properly.
L.K.
-----Original Message-----
From: Marc De Boeck [mailto:mdeboec@gmail.com]
Sent: Sunday, March 22, 2015 2:30 PM
To: ivy-user@ant.apache.org
Subject: Re: cache busting and integration question
Please have a look at the paragraphs "Dealing with integration versions" in
http://ant.apache.org/ivy/history/latest-milestone/bestpractices.html:
The "typical" developer workflow using SNAPSHOT builds of a release, has become popular because it is the standard in Maven. A disadvantage of using SNAPSHOT-builds, is however that you can never be sure which build you have finally gotten after a dependency has been resolved.
A more controlled way of dealing with your dependencies, is to create a new version for every build that is made, and to resolve for your local builds using the revision 'latest.integration' for "home-made" artifacts.
Ivy supports both methods of resolving dependencies. The URL http://ant.apache.org/ivy/history/latest-milestone/bestpractices.html
describes how you can implement the "SNAPSHOT" approach using Ivy.
Regards,
Marc
2015-03-20 23:12 GMT+01:00 Loren Kratzke <LK...@blueorigin.com>:
> I am encountering challenges with the Ivy cache and also with setting
> up what I would consider to be a typical (if not classic) developer workflow.
> Here is the desired workflow:
>
> I have a Nexus repository (releases and snapshots) plus a local file
> system repository used for local development. I have Project which
> depends upon Library. I wish to modify Library, publish locally, and
> then pull it into the local build of Project.
>
> And for my Jenkins "continuous" integration server, if the copy of
> Library-1.0.0-SNAPSHOT in Nexus is newer than what is in the cache on
> the build machine, then I would like Ivy to grab the newer version
> (just like Maven does).
>
> ISSUE #1
>
> Assuming I start with an empty cache and empty internal repo on my
> local dev machine, I build Project and it pulls Library from Nexus. Perfect.
> Works great. But then if I publish Library locally and build Project
> again, I will get the cached version of Library that came from Nexus.
> No matter what I do, I will always get the cached copy. This happens
> even though my chain resolver specifies my internal repo before the Nexus repo.
>
> If I blow away the cache, then I get my locally published build, but
> only if I blow away the cache. Otherwise I get stale stuff. Issue #2
> below is all about stale stuff.
>
> I debugged Ivy (for hours upon hours) and during the resolve, it
> checks my internal repo, finds the fresh artifact (yay!) and the next
> thing it does is checks its cache, finds a module descriptor that
> points at Nexus, and proceeds to pull in the wrong artifact into Project.
>
> The behavior is either a bug, or implies that Ivy assumes that an
> artifact can only come from one place, and if it came from there once,
> it will come from there forever, and will never change. Again, Ivy
> finds the artifact in local repo, stops searching, then ends up
> delivering the cached artifact that came from Nexus sometime earlier.
>
> ISSUE #2
>
> I have been working on this problem for quite some time and I thought
> I had it fixed, but I don't. That is, my integration server needs the
> latest snapshot build of Library-1.0.0-SNAPSHOT. But what I get is the
> cached version from the Ivy cache. An SVN commit happens, triggers a
> build of Library, Library get published, triggers a build of Project,
> and a stale build of Library is delivered to Project from the cache
> (which breaks Project build).
>
> SUMMARY
>
> Ivy will resolve and cache deps, but once a dep is in the cache,
> that's what I get forever until I blow away the cache. I see this
> question asked quite often regarding obtaining latest integration
> versions, and I see that Ivy claims to support the local development
> workflow, but I have yet to see a working example of either of these.
> Surely some of you have solved these issues.
>
> Is there any place anywhere in the world that has an example of
> development workflow configuration and/or integration configuration?
> Can anybody provide an example? So far I have not found anything, and
> advice on the net seems to be general suggestions like "try this, try
> that" and nobody really has the configuration solution. Is Ivy capable
> of these behaviors or do I need to hand roll a dependency management
> solution. Maven is too strict for these projects (C++ code). Ivy is
> good if I can just obtain these fundamental behaviors.
>
> Note that I have not provided any configs in this message that
> demonstrate the issues I am having but definitely can upon request.
>
> Thanks in advance,
>
> L.K.
>
>
Re: cache busting and integration question
Posted by Marc De Boeck <md...@gmail.com>.
Please have a look at the paragraphs "Dealing with integration versions" in
http://ant.apache.org/ivy/history/latest-milestone/bestpractices.html:
The "typical" developer workflow using SNAPSHOT builds of a release, has
become popular because it is the standard in Maven. A disadvantage of using
SNAPSHOT-builds, is however that you can never be sure which build you have
finally gotten after a dependency has been resolved.
A more controlled way of dealing with your dependencies, is to create a new
version for every build that is made, and to resolve for your local builds
using the revision 'latest.integration' for "home-made" artifacts.
Ivy supports both methods of resolving dependencies. The URL
http://ant.apache.org/ivy/history/latest-milestone/bestpractices.html
describes how you can implement the "SNAPSHOT" approach using Ivy.
Regards,
Marc
2015-03-20 23:12 GMT+01:00 Loren Kratzke <LK...@blueorigin.com>:
> I am encountering challenges with the Ivy cache and also with setting up
> what I would consider to be a typical (if not classic) developer workflow.
> Here is the desired workflow:
>
> I have a Nexus repository (releases and snapshots) plus a local file
> system repository used for local development. I have Project which depends
> upon Library. I wish to modify Library, publish locally, and then pull it
> into the local build of Project.
>
> And for my Jenkins "continuous" integration server, if the copy of
> Library-1.0.0-SNAPSHOT in Nexus is newer than what is in the cache on the
> build machine, then I would like Ivy to grab the newer version (just like
> Maven does).
>
> ISSUE #1
>
> Assuming I start with an empty cache and empty internal repo on my local
> dev machine, I build Project and it pulls Library from Nexus. Perfect.
> Works great. But then if I publish Library locally and build Project again,
> I will get the cached version of Library that came from Nexus. No matter
> what I do, I will always get the cached copy. This happens even though my
> chain resolver specifies my internal repo before the Nexus repo.
>
> If I blow away the cache, then I get my locally published build, but only
> if I blow away the cache. Otherwise I get stale stuff. Issue #2 below is
> all about stale stuff.
>
> I debugged Ivy (for hours upon hours) and during the resolve, it checks my
> internal repo, finds the fresh artifact (yay!) and the next thing it does
> is checks its cache, finds a module descriptor that points at Nexus, and
> proceeds to pull in the wrong artifact into Project.
>
> The behavior is either a bug, or implies that Ivy assumes that an artifact
> can only come from one place, and if it came from there once, it will come
> from there forever, and will never change. Again, Ivy finds the artifact in
> local repo, stops searching, then ends up delivering the cached artifact
> that came from Nexus sometime earlier.
>
> ISSUE #2
>
> I have been working on this problem for quite some time and I thought I
> had it fixed, but I don't. That is, my integration server needs the latest
> snapshot build of Library-1.0.0-SNAPSHOT. But what I get is the cached
> version from the Ivy cache. An SVN commit happens, triggers a build of
> Library, Library get published, triggers a build of Project, and a stale
> build of Library is delivered to Project from the cache (which breaks
> Project build).
>
> SUMMARY
>
> Ivy will resolve and cache deps, but once a dep is in the cache, that's
> what I get forever until I blow away the cache. I see this question asked
> quite often regarding obtaining latest integration versions, and I see that
> Ivy claims to support the local development workflow, but I have yet to see
> a working example of either of these. Surely some of you have solved these
> issues.
>
> Is there any place anywhere in the world that has an example of
> development workflow configuration and/or integration configuration? Can
> anybody provide an example? So far I have not found anything, and advice on
> the net seems to be general suggestions like "try this, try that" and
> nobody really has the configuration solution. Is Ivy capable of these
> behaviors or do I need to hand roll a dependency management solution. Maven
> is too strict for these projects (C++ code). Ivy is good if I can just
> obtain these fundamental behaviors.
>
> Note that I have not provided any configs in this message that demonstrate
> the issues I am having but definitely can upon request.
>
> Thanks in advance,
>
> L.K.
>
>