Ivy multi project setup with mixed binary and source dependencies

[Jochen <jo...@gmail.com>]

Hi,

we hava a non-trivial (but required) project setup that has mixed 
dependencies, either sources checked out from th SCM repository or 
binaries downloaded from a Maven repository. The most important thing 
is that we have to handle archives, e.g. resources or C headers that 
are contained in a tar archive for binary distribution, and must be 
unpacked when building.

What we need to achieve is the following:

1. Each source dependency should be checked out into its own directory, 
resulting e.g. in the following directory structure: 
Workspace/Project1, Workspace/Project2, …

2. The binary dependencies can be stored anywhere, e.g. in the ivy cache

3. If a project is missing, it should be checkout out from the SCM 
repository, if the user has access to it. This step is optional, but we 
would like to automate it

4. If Project1 has a dependency on an archive from Project2, a symlink 
should be created, e.g. "lib/Project2/archive" in Project1. The target 
of this symlink must be the directory in Project2 that contains the 
unpacked build results for the archive. For example, Project2 hast a 
target directory "Project2/build/Headers", so link 
"Project1/lib/Project2/Headers" should point to 
"Project2/build/Headers". I tried this using a file system resolve, but 
could not get it to work without publishing "Project2" to the Ivy 
cache. However, we absolutely need the direct link into Project2's 
source.

5. If Project2 cannot be obtained from the SCM repository, it should be 
downloaded from a Maven repository and unpacked someplace. The link 
"Project1/lib/Project2/Headers" should point to the location where the 
artifact was unpacked.

Ivy brings along a lot of the things we need, especially the 
configuration management.

It would be great if we could use it, so the question is: Is the setup 
described above possible using Ivy, and if yes, how?

Thanks very much for your help,
Jochen

Re: Ivy multi project setup with mixed binary and source dependencies

[Mitch Gitman <mg...@gmail.com>]

On Project1, you could make use of the ivy:resolve Ant task to get
Project1's dependencies. Then there's an ivy:cachefileset Ant task which
could be used to get all the files belonging to the dependencies. You'd
probably want to set transitive="false" to only get the direct
dependencies. From here you could cycle through the libraries, expand them
to the desired source location, and create the symlink.

However, what I've just described only works for those dependencies that
reside in your Maven repository. It doesn't account for projects that
reside in the SCM repository, which is something you say you also want to
automatically handle.

In this respect, though, it sounds like you're trying to conflate or munge
two distinct things: binary artifacts and the source they originate from.
There is a Subversion Ivy resolver, but that is still meant to be used for
deliverable artifacts (be they binary or text) stored in an SCM repo, not
the true source in the SCM repo.

I'd say a better way to go about thinking this through is to let users
check out everything they might want from source control (this shouldn't be
so hard, since you control the source), and then only do the ivy:resolve
and extract on those projects that you don't control.

Even this, however, kinda smells funny because Project2 shouldn't care
about the source of Project1. It should only care about Project1 as it has
been published to the Maven repo. In that case, it makes sense to expand
the dependency Project2/build/Headers to a place totally distinct from
anything checked out from source, so that the two might never mix. Never
can the source for one project peek into the source for another; it's only
aware of extracted source.

Hope this makes sense. I don't think I'll have time to answer any follow-up
questions.

On Tue, Jan 10, 2012 at 3:44 AM, Jochen <jo...@gmail.com> wrote:

> Hi,
>
> we hava a non-trivial (but required) project setup that has mixed
> dependencies, either sources checked out from th SCM repository or binaries
> downloaded from a Maven repository. The most important thing is that we
> have to handle archives, e.g. resources or C headers that are contained in
> a tar archive for binary distribution, and must be unpacked when building.
>
> What we need to achieve is the following:
>
> 1. Each source dependency should be checked out into its own directory,
> resulting e.g. in the following directory structure: Workspace/Project1,
> Workspace/Project2, …
>
> 2. The binary dependencies can be stored anywhere, e.g. in the ivy cache
>
> 3. If a project is missing, it should be checkout out from the SCM
> repository, if the user has access to it. This step is optional, but we
> would like to automate it
>
> 4. If Project1 has a dependency on an archive from Project2, a symlink
> should be created, e.g. "lib/Project2/archive" in Project1. The target of
> this symlink must be the directory in Project2 that contains the unpacked
> build results for the archive. For example, Project2 hast a target
> directory "Project2/build/Headers", so link "Project1/lib/Project2/**Headers"
> should point to "Project2/build/Headers". I tried this using a file system
> resolve, but could not get it to work without publishing "Project2" to the
> Ivy cache. However, we absolutely need the direct link into Project2's
> source.
>
> 5. If Project2 cannot be obtained from the SCM repository, it should be
> downloaded from a Maven repository and unpacked someplace. The link
> "Project1/lib/Project2/**Headers" should point to the location where the
> artifact was unpacked.
>
> Ivy brings along a lot of the things we need, especially the configuration
> management.
>
> It would be great if we could use it, so the question is: Is the setup
> described above possible using Ivy, and if yes, how?
>
> Thanks very much for your help,
> Jochen
>
>
>