making an SCM managed copy of your external artifacts

[Steve Loughran <st...@apache.org>]

I want to set up a project which has all the things we depend on in SCM, 
plan is to have two repositories

repo/internal  - in house built artifacts, SCM managed release 
propagation, etc.

repo/external - everything that normally comes out of ibiblio, jboss, 
reslet m2 repositories, which ivy pulls down, and the odd Sun JAR that 
isn't normally online

Question is, how best to build that external repository, especially, how 
best to automate building that repository.


1. Manual M2 layout. From the dependency data, grab the JARs and POM 
files, lay them out maven style.

2. Copy ivy cache. clean your local cache, do a release, copy 
~/.ivy/cache to repo/external, delete things you don't want there and 
check in the rest.

Has anyone else done this? What was their tactic?

Re: making an SCM managed copy of your external artifacts

[Steve Loughran <st...@apache.org>]

On 10/08/10 18:31, Archie Cobbs wrote:

> As a side note, I do the same thing with RPM's. We have an internal RPM
> repository plus the public openSUSE ones. Internal RPM builds publish to the
> internal RPM repository. openSUSE's zypper(1) command functions like ivy but
> in the RPM world. RPM dependencies function like ivy dependencies. RPM spec
> files are like ivy.xml files. These two parallel universes are very
> analogous.
> 

I create RPMS too, here are some of the things I have to do it against
artifacts from outside and published locally with ivy

Ivy file to pull in the published stuff of other projects
http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/release/ivy.xml?revision=8363&view=markup

a .spec file which at build time is copied with property expansion, so
every artifact gets its right versions.

http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/release/metadata/rpm/smartfrog.spec?revision=8363&view=markup

The rpm build file is
http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/release/build.xml?revision=8363&view=markup


There's some cuteness here as we test against a linux VM (centos and
java6) by scp-ing up the artifacts, then doing various RPM status checks
(which would be easier if the commands returned error codes), and by
issuing remote commands on the server to verify things got stuck on the
path and actually work. If I was really devious I'd actually snapshot
the VM on every release, so as to test rpm upgrades, but they scare me
so I don't do that, instead I say "we prefer clean installs".

-steve

Re: making an SCM managed copy of your external artifacts

[Archie Cobbs <ar...@dellroad.org>]

Here's what I do...

1. Set up a normal internal ivy repository. On the main/build server, a
"publish" ant task publishes a built project to the internal repo (version
number automatically includes SVN revision).

2. On the main server, SVN checkout Ivy RoundUp and publish this tree on the
internal web server. This is just an optimization to make access to Ivy
RoundUp files faster.

3. Individual projects use svn:externals to incorporate standard build
macros, etc. This stuff includes an ivysettings.xml file that configures the
above two repositories for ivy.

As an Ivy RoundUp maintainer I'm of course biased but this seems like a nice
and clean way to do things. There is a repo for internal stuff and there is
a repo for public stuff. Done. We don't use maven so there is no maven
weirdness, everything is pure ivy.

The only "glitch" is when something we need is not in Ivy RoundUp, but then
the task of adding it is doubly worthwhile because it serves both our own
purposes and other users of the project (i.e. positive open source karma).

As a side note, I do the same thing with RPM's. We have an internal RPM
repository plus the public openSUSE ones. Internal RPM builds publish to the
internal RPM repository. openSUSE's zypper(1) command functions like ivy but
in the RPM world. RPM dependencies function like ivy dependencies. RPM spec
files are like ivy.xml files. These two parallel universes are very
analogous.

-Archie

On Mon, Aug 9, 2010 at 6:24 AM, Steve Loughran <st...@apache.org> wrote:

>
> I want to set up a project which has all the things we depend on in SCM,
> plan is to have two repositories
>
> repo/internal  - in house built artifacts, SCM managed release propagation,
> etc.
>
> repo/external - everything that normally comes out of ibiblio, jboss,
> reslet m2 repositories, which ivy pulls down, and the odd Sun JAR that isn't
> normally online
>
> Question is, how best to build that external repository, especially, how
> best to automate building that repository.
>
>
> 1. Manual M2 layout. From the dependency data, grab the JARs and POM files,
> lay them out maven style.
>
> 2. Copy ivy cache. clean your local cache, do a release, copy ~/.ivy/cache
> to repo/external, delete things you don't want there and check in the rest.
>
> Has anyone else done this? What was their tactic?
>



-- 
Archie L. Cobbs

Re: making an SCM managed copy of your external artifacts

[Steve Loughran <st...@apache.org>]

On 11/08/10 01:56, Geoff Clitheroe wrote:
> Hi,
> 
> I've used 'both approaches'
> 
> 1. Use Ivy to build and maintain the repo - I use a master Ivy file with all
> the deps we use specified and then a resolve and rsync to the enterprise
> repo (which is Apache httpd).  This also holds our shared ivy settings.
> This way I have more control over what goes into the enterprise repo.
> 
> 2. We recently used Archiva on a different repo.  It was in many ways harder
> to get running (how we wanted) but does have the advantage of acting more
> like a persistent proxy for central etc.  This also means developers are in
> control of what gets into this enterprise repo.
> 
> Either way work fine I think the only real difference for us is who controls
> what gets into the repo.  If you're pushing a lot of jars then the search
> features of some of the archive managers may be nice.
> 

Thanks. I don't need the search stuff, I just want something which
builds and works offline, with complex build processes like "user brings
up windows 2003 vm image, VPNs to work and then does an SVN update into
the host filesystem", that being how I have to code on the road from a
linux laptop.

I'll probably try option 1 -ivy and sync, but look at the other options
too. I don't want to get into the game of managing a live service for
other people as it just becomes a support call problem, but updating
into SVN seems viable. I may also just code up some ant task to grab a
bit of the maven artifacts and metadata for an artifact and force you to
do that for everything you list in a file. If you don't do transient
stuff, that's easy, it just forces me to create a file listing every
artifact/version we ship. But I should be doing that anyway.

Re: making an SCM managed copy of your external artifacts

[Geoff Clitheroe <g....@gmail.com>]

Hi,

I've used 'both approaches'

1. Use Ivy to build and maintain the repo - I use a master Ivy file with all
the deps we use specified and then a resolve and rsync to the enterprise
repo (which is Apache httpd).  This also holds our shared ivy settings.
This way I have more control over what goes into the enterprise repo.

2. We recently used Archiva on a different repo.  It was in many ways harder
to get running (how we wanted) but does have the advantage of acting more
like a persistent proxy for central etc.  This also means developers are in
control of what gets into this enterprise repo.

Either way work fine I think the only real difference for us is who controls
what gets into the repo.  If you're pushing a lot of jars then the search
features of some of the archive managers may be nice.

Cheers,
Geoff


On Mon, Aug 9, 2010 at 11:24 PM, Steve Loughran <st...@apache.org> wrote:

>
> I want to set up a project which has all the things we depend on in SCM,
> plan is to have two repositories
>
> repo/internal  - in house built artifacts, SCM managed release propagation,
> etc.
>
> repo/external - everything that normally comes out of ibiblio, jboss,
> reslet m2 repositories, which ivy pulls down, and the odd Sun JAR that isn't
> normally online
>
> Question is, how best to build that external repository, especially, how
> best to automate building that repository.
>
>
> 1. Manual M2 layout. From the dependency data, grab the JARs and POM files,
> lay them out maven style.
>
> 2. Copy ivy cache. clean your local cache, do a release, copy ~/.ivy/cache
> to repo/external, delete things you don't want there and check in the rest.
>
> Has anyone else done this? What was their tactic?
>