You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by Cody Casterline <co...@gmail.com> on 2011/05/25 01:11:42 UTC
Building a Repo -- Excludes Question
I'm looking into switching to Ivy for managing our dependencies and I was
happy to find this page:
http://ant.apache.org/ivy/history/latest-milestone/tutorial/build-repository.html
which describes how to create a local repository using ivy:install so that
builds don't have to hit the network. (Yay!)
I grabbed
https://svn.apache.org/repos/asf/ant/ivy/core/trunk/src/example/build-a-ivy-repositoryand
started tweaking it to test my understanding, but I'm running into a
couple issues.
First, my dependencies:
<dependencies>
<dependency org="commons-io" name="commons-io" rev="1.4"/>
<dependency org="org.apache.cxf"
name="cxf-rt-frontend-jaxws" rev="2.4.0">
<!--
<exclude org="javax.activation"/>
-->
</dependency>
</dependencies>
I've also switched ivysettings-basic.xml to use the local repository
('my-repository') by default. (It doesn't seem that ivy:resolve or
ivy:retrieve accept a resolver name as an option? is ivy:install the only
place where you can specify resolvers by name?)
<ivysettings>
<settings defaultCache="${ivy.cache.dir}/no-namespace"
defaultResolver="my-repository"
defaultConflictManager="all" />
<resolvers>
<filesystem name="my-repository">
<ivy
pattern="${dest.repo.dir}/no-namespace/[organisation]/[module]/ivys/ivy-[revision].xml"/>
<artifact
pattern="${dest.repo.dir}/no-namespace/[organisation]/[module]/[type]s/[artifact]-[revision].[ext]"/>
</filesystem>
<ibiblio name="libraries" m2compatible="true" />
</resolvers>
</ivysettings>
My thinking is that I can define my dependencies, then try a build. if the
build can't resolve a dependency from my local repo, I can run ivy:install
to install that dep. locally and try again. This way I can iteratively
build up my local repo.
when I run ivy:install on org.apache.cxf#cxf-rt-frontend-jaxws;2.4.0, it
installs lots of dependencies but fails with:
[ivy:install] :: problems summary ::
[ivy:install] :::: WARNINGS
[ivy:install] [NOT FOUND ]
javax.activation#activation;1.0.2!activation.jar (128ms)
[ivy:install] ==== libraries: tried
[ivy:install]
http://repo1.maven.org/maven2/javax/activation/activation/1.0.2/activation-1.0.2.jar
[ivy:install] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:install] :: FAILED DOWNLOADS ::
[ivy:install] :: ^ see resolution messages for details ^ ::
[ivy:install] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:install] :: javax.activation#activation;1.0.2!activation.jar
[ivy:install] ::::::::::::::::::::::::::::::::::::::::::::::
... despite the fact that when I look at the ivy files in my repo, and the
pom files on maven.org, they claim to exclude javax.activation.
If I uncomment the extra exclude rule above, ivy:resolve will work. Why
does my exclude work where the ones in pom and ivy files didn't?
Also, while I've got everyone's attention, is there an easier way to just
tell Ivy to install all dependencies for a project into a repository? If
I've got them listed in ivy.xml, entering them manually into ivy:install
commands seems redundant. :)
Thanks!
Re: Building a Repo -- Excludes Question
Posted by Geoff Clitheroe <g....@gmail.com>.
My take;
if you are ever going to share that repo with other developers or have
your CI system publish to it on successful builds (we do) then use a
repo manager from the start. If you want it to be local for now then
is 127.0.0.1:80 local enough?
Also, you asked if it is possible to install direct from your ivy.xml
definition. I don't _think_ so (anyone else?) but a repo manager
being a proxy will take care of this.
FWIW these discussions and questions are pretty common. It begs a
project that is really just a repackage of other projects. Something
along this lines of an Ivy repo quick start.
Package archiva (already has Jetty), an Ivy jar, some ant scripts to
bootstrap Ivy and do the settings, build base files for Netbeans and
whatever Eclipse and Ivyde uses. Maybe add Ivy round up. Getting Ivy
running with a local repo (which could then become remote) should then
be as simple starting Jetty. I don't have time but if anyone does
then take what ever is useful from here:
http://codegeo.org/confluence/display/codegeo/Common+Build+Environment
http://codegeo.org/repos/codegeo/build/trunk/
Spend your time on planes drinking and sleeping. If you don't look
forwards to long haul as a well earned break you aren't working hard
enough in the office.
Cheers,
Geoff
On Wed, May 25, 2011 at 1:51 PM, Scott Palmer <sw...@gmail.com> wrote:
>
> On 2011-05-24, at 9:07 PM, Cody Casterline wrote:
>
>> I want to make an on-disk repo and
>> check it in with my source so that it's always available. Even if the
>> network goes down or I'm on a plane.
>
> Slightly off-topic, but I thought I would comment and someone can tell me if I'm missing something…
>
> Checking in the repo defeats it's purpose. One of the reasons to use a dependency manager is to keep stuff that isn't source code out of your source control system. A good rule of thumb is that if it doesn't create a readable "diff" it probably doesn't belong in your SCM. (Icons, images, sound effects, small media files being an allowable exception.)
>
> Ask your self what you are getting by checking-in your Ivy repo? Versioning? No, the ivy repo handles versioning of the resources internally. You are basically just making a redundant copy… but your SCM isn't a backup tool.
>
> But I'm totally with you in terms of having a local repository as a more visible and controllable "local cache"… (unless there is a better way?)
>
> Regards,
>
> Scott
Re: Building a Repo -- Excludes Question
Posted by Scott Palmer <sw...@gmail.com>.
On 2011-05-25, at 11:28 AM, Cody Casterline wrote:
>> One of the reasons to use a dependency manager is to keep stuff that isn't
>> source code out of your source control system. A good rule of thumb is that
>> if it doesn't create a readable "diff" it probably doesn't belong in your
>> SCM. (Icons, images, sound effects, small media files being an allowable
>> exception.)
>>
>
> I used to think that too. Then I started working at my current company,
> where they check all dependencies into the SCM. I was horrified at first,
> but I've actually since realized that it's a good practice for two very
> important reasons:
>
> * Build stability -- Your build never fails because some dependency isn't
> present. Everything required to build a our code is checked in with our
> code. You wouldn't not check in some of your code and require fetching it
> before compiling. Why do that with your JARs?
Fetching from your Ivy/Maven repo should be as reliable as fetching from your SCM. And you use the best tool suited for each job.
>
> * Build reproducibility -- When you are trying to track down where a bug was
> introduced, or need to do another build of an old version, the libraries
> used to do that build are present. You don't have to worry about whether
> the public maven repo still has that old dependency in it.
Right.. dependng on a public repo for builds I think is ridiculous. The public repos should only be used to populate your corporate repo or local repo.
> Or worry, in the
> case of using a repo manager, that your sysadmin has gone in and "cleaned
> up" dependencies no longer used.
That shouldn't happen either. You could just say that they cleaned up old revisions of the source that were no longer used. It's the same problem.
>
>
>> Ask your self what you are getting by checking-in your Ivy repo?
>> Versioning? No, the ivy repo handles versioning of the resources
>> internally. You are basically just making a redundant copy… but your SCM
>> isn't a backup tool.
>>
>
> Yes! Versioning! Including versioning of the dependencies used to build
> the project. You say "the ivy repo handles versioning of the resources
> internally" -- but does it? If I check out the project on another computer
> and ivy:resolve, am I guaranteed to get the exact same things I got on a
> build I did years ago?
Unless you changed you replaced stuff you already had in your internal repo, of course you should get the exact same things. If you go and swap out commons-net-1.4.1.jar with another different file that has the same name you will of course be screwing yourself. But is that a real risk?
> Even if you say "yes", please pardon me if I don't
> believe you and rely on my own history (via my SCM) to verify that. :)
I do say "yes :-) .. but if you don't trust the tool to do it's job then why use it? It's almost like fetching the dependences from the public repos on the internet, checking them in with your code and not using a dependency manager locally at all.
Not exactly the same, since if you use the same lib in the next project it is nice to have it locally in a system that knows the dependencies still.
>
> I wish I could find a link to someone making the argument better than I am.
> (I would've sworn that my coworkers pointed me to one when I first made
> your argument here.) But it really just boils down to reducing external
> dependencies and moving parts from your project to make for a more stable,
> reproducible build environment.
>
> - Cody
>
> P.S.: I realize I may have gone a bit off-topic for ivy-user. But I heard
> the same arguments when I was asking questions to the Maven community, and
> Geoff brought up the same issues, so it seems somewhat relevant? Well, now
> that I think about it -- I guess it's very relevant in that it's something I
> want to do with Ivy. :)
I think this is a good discussion… as my company is currently moving from the way you want to do it (checking in dependencies) to using a separate dependency manager (ultimately a Maven repo with Maven/Ivy/Gradle on the client side). So it's good to hear other opinions while we try to find the solution that is right for us.
I was pushing for Ant+Ivy because I find Maven too strict - anything slightly out of the ordinary and they tell you to go write a plugin (e.g. it sucks for JNI stuff which we have a lot of). Though it looks like the Maven side may win for 100% Java projects.. the problem is we do a lot of native code as well and none of these tools seem to have solved the dependency management problem for native code. I thought Ivy might be the most flexible.. but am now thinking of using Gradle on the client side for the native code or anything that doesn't fit the project structure that Maven decided was the "one true way"
Now I'm really getting off-topic though.. sorry… I'll shut up :-)
Cheers,
Scott
Re: Building a Repo -- Excludes Question
Posted by Archie Cobbs <ar...@dellroad.org>.
Some thoughts, not sure all are relevant... (not sure exactly what the goal
is)
You could create (and commit to SVN) your own private copy of the Ivy
RoundUp repository. Since it doesn't contain artifacts, you won't be
checking in large files. But, you could pre-load all of the archive files
into ~/.ivy2/packager/cache (or wherever you want your packager resolver
cache to be), so that your builds would work without any network access.
Because SHA1 checksums are used, you can be assured the archive files won't
change without breaking the build.
Or, if a specific version of Ivy Roundup is suitable for you without
modification, then just SVN checkout that revision instead of creating your
own repository.
-Archie
--
Archie L. Cobbs
Re: Building a Repo -- Excludes Question
Posted by Cody Casterline <co...@gmail.com>.
>
> Checking in the repo defeats it's purpose.
That depends on your purpose for using Ivy (or Maven or whatever). :) I
want it to:
1) Easily fetch a library and its dependencies the first time I need use
it.
2) Handle transitive dependencies for those libraries so that I can just
declare a dependency on BobsWidget, and not the dozens of libraries *it*
depends on. (In this case, Apache CXF is what finally prompted me to look
into Maven and Ivy.)
3) And handle those transitive dependencies between our own sub-projects.
> One of the reasons to use a dependency manager is to keep stuff that isn't
> source code out of your source control system. A good rule of thumb is that
> if it doesn't create a readable "diff" it probably doesn't belong in your
> SCM. (Icons, images, sound effects, small media files being an allowable
> exception.)
>
I used to think that too. Then I started working at my current company,
where they check all dependencies into the SCM. I was horrified at first,
but I've actually since realized that it's a good practice for two very
important reasons:
* Build stability -- Your build never fails because some dependency isn't
present. Everything required to build a our code is checked in with our
code. You wouldn't not check in some of your code and require fetching it
before compiling. Why do that with your JARs?
* Build reproducibility -- When you are trying to track down where a bug was
introduced, or need to do another build of an old version, the libraries
used to do that build are present. You don't have to worry about whether
the public maven repo still has that old dependency in it. Or worry, in the
case of using a repo manager, that your sysadmin has gone in and "cleaned
up" dependencies no longer used.
> Ask your self what you are getting by checking-in your Ivy repo?
> Versioning? No, the ivy repo handles versioning of the resources
> internally. You are basically just making a redundant copy… but your SCM
> isn't a backup tool.
>
Yes! Versioning! Including versioning of the dependencies used to build
the project. You say "the ivy repo handles versioning of the resources
internally" -- but does it? If I check out the project on another computer
and ivy:resolve, am I guaranteed to get the exact same things I got on a
build I did years ago? Even if you say "yes", please pardon me if I don't
believe you and rely on my own history (via my SCM) to verify that. :)
I wish I could find a link to someone making the argument better than I am.
(I would've sworn that my coworkers pointed me to one when I first made
your argument here.) But it really just boils down to reducing external
dependencies and moving parts from your project to make for a more stable,
reproducible build environment.
- Cody
P.S.: I realize I may have gone a bit off-topic for ivy-user. But I heard
the same arguments when I was asking questions to the Maven community, and
Geoff brought up the same issues, so it seems somewhat relevant? Well, now
that I think about it -- I guess it's very relevant in that it's something I
want to do with Ivy. :)
Re: Building a Repo -- Excludes Question
Posted by Scott Palmer <sw...@gmail.com>.
On 2011-05-24, at 9:07 PM, Cody Casterline wrote:
> I want to make an on-disk repo and
> check it in with my source so that it's always available. Even if the
> network goes down or I'm on a plane.
Slightly off-topic, but I thought I would comment and someone can tell me if I'm missing something…
Checking in the repo defeats it's purpose. One of the reasons to use a dependency manager is to keep stuff that isn't source code out of your source control system. A good rule of thumb is that if it doesn't create a readable "diff" it probably doesn't belong in your SCM. (Icons, images, sound effects, small media files being an allowable exception.)
Ask your self what you are getting by checking-in your Ivy repo? Versioning? No, the ivy repo handles versioning of the resources internally. You are basically just making a redundant copy… but your SCM isn't a backup tool.
But I'm totally with you in terms of having a local repository as a more visible and controllable "local cache"… (unless there is a better way?)
Regards,
Scott
Re: Building a Repo -- Excludes Question
Posted by Cody Casterline <co...@gmail.com>.
On Tue, May 24, 2011 at 7:04 PM, Geoff Clitheroe <g....@gmail.com>wrote:
> I don't see a direct dependency on javax.activation for
> cxf-rt-frontend-jaxws so something else is calling for it.
>
>
> http://search.maven.org/#artifactdetails%7Corg.apache.cxf%7Ccxf-rt-frontend-jaxws%7C2.4.0%7Cjar
I figured as much, but I was hoping someone a bit more experienced might be
able to pinpoint the issue and clarify why it's happening.
I did a bit more searching (thanks, ivy:report!) and I think I figured it
out:
org.apache.cxf#cxf-rt-frontend-jaxws
depends on: org.springframework#spring-webmvc
depends on: org.springframework#spring-web
depends on: javax.xml.soap#saaj-api
depends on: javax.activation#activation
While some other dependencies exclude javax.activation (like
org.apache.cxf#cxf-api), because nothing in the above chain does, without an
exclusion, my project does too. (Do please correct me if I'm wrong here.)
:)
Hmm, though, spring-web only depends on saaj-api as scope "provided", so it
shouldn't have to download it, should it?
see:
http://search.maven.org/remotecontent?filepath=org/springframework/spring-web/3.0.5.RELEASE/spring-web-3.0.5.RELEASE.pom
You will occasionally hit these problems.
Doh! Ok, then my next question is what's the best way to handle these when
they come up?
> You may want to check out
> http://code.google.com/p/ivyroundup/
Hmm. I'm not quite sure what this offers me over just using an <ibiblio>
resolver.
> And to answer the final question. Yes! Run a repo manager as a proxy.
>
> http://maven.apache.org/repository-management.html
Yikes! The reason I liked the solution at
http://ant.apache.org/ivy/history/latest-milestone/tutorial/build-repository.html
was
because it *doesn't* require using a repo manager. I really don't want to
have to hit the network to do builds. I want to make an on-disk repo and
check it in with my source so that it's always available. Even if the
network goes down or I'm on a plane.
> This also gives you some options to do manual fixes for problems like
> javax.activation
I can do that with an on-disk repo too, though, can't I? Though I'm a bit
vague on the best way to fix this case.
Re: Building a Repo -- Excludes Question
Posted by Geoff Clitheroe <g....@gmail.com>.
I don't see a direct dependency on javax.activation for
cxf-rt-frontend-jaxws so something else is calling for it.
http://search.maven.org/#artifactdetails%7Corg.apache.cxf%7Ccxf-rt-frontend-jaxws%7C2.4.0%7Cjar
You will occasionally hit these problems. You may want to check out
http://code.google.com/p/ivyroundup/
And to answer the final question. Yes! Run a repo manager as a proxy.
http://maven.apache.org/repository-management.html
I'm most familiar with Archiva for this. It has chains of remote
repos and they are proxied and cached locally. Your Ivy settings just
then refer to the local Archiva repo
e.g.,
http://codegeo.org/repos/codegeo/build/trunk/ivysettings.xml
which is referring to all the repos we care about and making the
artifacts available in one place including the ones we publish:
http://codegeo.org/archiva/browse
This also gives you some options to do manual fixes for problems like
javax.activation
Cheers,
Geoff
On Wed, May 25, 2011 at 11:11 AM, Cody Casterline
<co...@gmail.com> wrote:
> I'm looking into switching to Ivy for managing our dependencies and I was
> happy to find this page:
> http://ant.apache.org/ivy/history/latest-milestone/tutorial/build-repository.html
> which describes how to create a local repository using ivy:install so that
> builds don't have to hit the network. (Yay!)
>
> I grabbed
> https://svn.apache.org/repos/asf/ant/ivy/core/trunk/src/example/build-a-ivy-repositoryand
> started tweaking it to test my understanding, but I'm running into a
> couple issues.
>
> First, my dependencies:
>
> <dependencies>
> <dependency org="commons-io" name="commons-io" rev="1.4"/>
> <dependency org="org.apache.cxf"
> name="cxf-rt-frontend-jaxws" rev="2.4.0">
> <!--
> <exclude org="javax.activation"/>
> -->
> </dependency>
> </dependencies>
>
>
> I've also switched ivysettings-basic.xml to use the local repository
> ('my-repository') by default. (It doesn't seem that ivy:resolve or
> ivy:retrieve accept a resolver name as an option? is ivy:install the only
> place where you can specify resolvers by name?)
>
> <ivysettings>
> <settings defaultCache="${ivy.cache.dir}/no-namespace"
> defaultResolver="my-repository"
> defaultConflictManager="all" />
> <resolvers>
> <filesystem name="my-repository">
> <ivy
> pattern="${dest.repo.dir}/no-namespace/[organisation]/[module]/ivys/ivy-[revision].xml"/>
> <artifact
> pattern="${dest.repo.dir}/no-namespace/[organisation]/[module]/[type]s/[artifact]-[revision].[ext]"/>
> </filesystem>
> <ibiblio name="libraries" m2compatible="true" />
> </resolvers>
> </ivysettings>
>
> My thinking is that I can define my dependencies, then try a build. if the
> build can't resolve a dependency from my local repo, I can run ivy:install
> to install that dep. locally and try again. This way I can iteratively
> build up my local repo.
>
> when I run ivy:install on org.apache.cxf#cxf-rt-frontend-jaxws;2.4.0, it
> installs lots of dependencies but fails with:
>
> [ivy:install] :: problems summary ::
> [ivy:install] :::: WARNINGS
> [ivy:install] [NOT FOUND ]
> javax.activation#activation;1.0.2!activation.jar (128ms)
> [ivy:install] ==== libraries: tried
> [ivy:install]
> http://repo1.maven.org/maven2/javax/activation/activation/1.0.2/activation-1.0.2.jar
> [ivy:install] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:install] :: FAILED DOWNLOADS ::
> [ivy:install] :: ^ see resolution messages for details ^ ::
> [ivy:install] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:install] :: javax.activation#activation;1.0.2!activation.jar
> [ivy:install] ::::::::::::::::::::::::::::::::::::::::::::::
>
> ... despite the fact that when I look at the ivy files in my repo, and the
> pom files on maven.org, they claim to exclude javax.activation.
>
> If I uncomment the extra exclude rule above, ivy:resolve will work. Why
> does my exclude work where the ones in pom and ivy files didn't?
>
> Also, while I've got everyone's attention, is there an easier way to just
> tell Ivy to install all dependencies for a project into a repository? If
> I've got them listed in ivy.xml, entering them manually into ivy:install
> commands seems redundant. :)
>
> Thanks!
>