You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by Troy Self <ts...@bbn.com> on 2009/10/05 14:04:56 UTC
Publishing 3rd party libraries to local repo
Hello,
I am new to Ivy and am trying to setup a repository for my software team. I have
gone through all of the tutorials on the web site and have some questions.
Is there a tried and true approach for quickly creating repository entries for
3rd party libraries? I have used the ivy:install command to copy items that are
already in Ibiblio or ivyrep, but what is the approach when neither of those
public repositories have the library you want, or they don't have the latest
version? Some examples are the slf4j-api and slf4j-log4j libraries. My project
depends on the latest versions, which are not published publicly, so I'd like to
add them to my own repo. Do you setup a project for each library and then
ivy:publish them? Or do you just manually create ivy.xml files and carefully
establish the correct directory/file structure in your repo?
Is there a way to query for the configurations of a repository entry? In setting
up my own dependencies, I need to map my own confs to the dependency's confs.
Right now that requires browsing to the repository to find the ivy.xml files for
each dependency and reading them. Is there a way to query for details about a
particular artifact, including its confs?
Is there a best practices for defining the various confs for a project? There
seems to be some inconsistency between what 'runtime' and 'default' means across
different dependencies.
Thanks very much for any help,
-- tBs
Troy Self
Re: Publishing 3rd party libraries to local repo
Posted by Kirby Files <kf...@masergy.com>.
Troy Self wrote on 10/05/2009 08:04 AM:
> Is there a tried and true approach for quickly creating repository entries for
> 3rd party libraries? I have used the ivy:install command to copy items that are
> already in Ibiblio or ivyrep, but what is the approach when neither of those
> public repositories have the library you want, or they don't have the latest
> version?
The ivy:install task can copy a module from one repository to another
(optionally creating the ivy.xml in the process). I use this both to
copy modules from iBiblio (creating an ivy.xml from the POM), and to
copy from a filesystem resolver, like our old "big-pile-of-jars-libdir"
(ivy.xml created from the specified org, module, and rev).
Here's what I wrote in response to a previous poster with this question.
I use this script:
--------------------------------------
#!/bin/csh
if ($1 == "") then
echo "Usage: installjar <org> <module> <revision> <path-to-file>"
else
/usr/local/ant/bin/ant -Dorg="$1" -Dmodule="$2" -Drev="$3"
-Dinstall.path="$4" $5 $6 $7 $8 $9
endif
--------------------------------------
With the following build target:
--------------------------------------
<target name="installjar"
depends="init-ivy"
description="--> install jar to masrep">
<ivy:install settingsRef="masergy.settings" organisation="${org}"
module="${module}" revision="${rev}" type="jar"
from="${localjar.resolver}" to="${to.resolver}" />
</target>
--------------------------------------
Where my build.properties defines these variables:
localjar.resolver=localjars
to.resolver=masrep_sftp
And my ivysettings.xml includes:
<resolvers>
<filesystem name="localjars">
<artifact pattern="${install.path}/[artifact]-[revision].[ext]" />
<artifact pattern="${install.path}/[artifact].[ext]" />
</filesystem>
</resolvers>
This script allows me to specify a jar on the local filesystem by its
name and the directory in which it resides (install.path), add the
org, and rev metadata, and publish it to one of my resolvers (I use
SFTP with a keyfile to push files up to our webserver; you may use a
filesystem resolver or webdav as a destination). This works well
enough as long as your jars are named like this: module-x.y.z.jar
I also have a bigger hack of a perl script which attempts to read a
directory of jars and guess the org, module, and rev (validating against
maven2), or prompt the user when it's unclear. See below.
---
Kirby Files
Software Architect
Masergy Communications
kfiles@masergy.com
------------------------ >8 CUT HERE 8< ----------------------
#!/usr/bin/perl
# parse_jars.pl
# Looks for jars in the incoming directory, attemps to parse the
# needed ivy info from the filename, and if successful, publishes
# the jar to the Masergy ivrep repository.
# If using linux with package mgmt, dependencies are:
# perl-libwww-perl
# perl-XML-Xpath
# perl-Archive-Zip
use LWP::UserAgent;
use XML::XPath;
use XML::XPath::XMLParser;
use Archive::Zip;
use strict qw(vars);
use vars qw($indir $ant $rep $pre $post);
$indir = "lib";
$ant = "/usr/local/ant/bin/ant";
$rep = "newjars";
$pre="<ivy:install settingsRef=\"masergy.settings\" organisation=\"";
$post=" from=\"\${from.resolver}\" to=\"\${to.resolver}\" />";
open (DIR, "ls -1 $indir|") || die "could not open $indir\n";
while (<DIR>) {
my ($org, $module, $jar, $ver);
next unless /jar/;
chomp;
my $file = $indir . "/" . $_;
$org=$_;
$module=$_;
$jar=$_;
$ver=$_;
$module=~s/^(.*)\.jar/$1/;
$module=~s/([^.]*)-\d.*/$1/;
$ver=~s/^[^.]*-([\d.]+.*)\.jar/$1/;
$ver="unknown" unless $ver =~ /\d/;
## Figure out the org
## Ask the user, if there's ambiguity
$org = &maven_search_org( $module );
if (-1 == $org) {
$org = &guess_org($module);
print "Org guessed from module name:\n $org\n";
$org = &classes_org($file);
print "Org guessed from jar package naming:\n $org\n"
unless (-1 == $org);
do {
$org = &get_user_org();
} while ($org =~ /^$/);
}
if ($ver eq "unknown" || ($module !~ /^\w+$/) || $org !~ /\w+/) {
print "Skipping jar $jar:\n";
print " $org / $module / $ver\n";
next;
} else {
&publish_jar($jar, $org, $module, $ver);
}
#&write_ivy_conf($jar, $org, $module, $ver);
}
## Use ivy::install to publish the jar via sftp to ivyrep
sub publish_jar
{
my ($jar, $org, $module, $ver) = @_;
print "Uploading $org / $module / $jar ... ";
my $resp = `$ant -Dorg=$org -Dmodule=$module -Drev=$ver
-Dinstall.path=./$indir 2>&1`;
if ($resp =~ /failed/i) {
print STDERR "Error uploading $jar:\n", $resp;
print "\n";
} else {
print "Done.\n";
}
}
## Find a directory in the jar to propose as the org
## searches for any dirs with two path elements
sub classes_org
{
my ($file) = @_;
my $somezip = Archive::Zip->new();
unless ( $somezip->read( $file ) == AZ_OK ) {
die "Error reading jarfile $file";
}
my @members = $somezip->members();
my @dirnames = ();
foreach (@members) {
if ($_->isDirectory()) {
my $fname = $_->fileName();
if ($fname =~ /^\w+\/\w+\/$/) {
$fname =~ s/^(\w+)\/(\w+)\/$/$1.$2/;
return $fname;
}
}
}
return -1;
}
## Ask the user for the best org name
sub get_user_org
{
print "Enter preferred org:> ";
my $line = <STDIN>;
chomp($line);
return $line;
}
## Make a guess about the org name from the module name
sub guess_org
{
my ($module) = @_;
my $org = $module;
$org=~s/spring.*/springframework/;
$org=~s/hiber.*/hibernate/;
$org=~s/standard.*/taglibs/;
$org=~s/jstl.*/taglibs/;
$org=~s/persistence.*/sun/;
$org=~s/jta.*/sun/;
$org=~s/servlet-api.*/tomcat/;
$org=~s/ojdbc.*/oracle/;
$org=~s/cglib.*/cglib/;
$org=~s/javassist.*/jboss/;
$org=~s/([^.]*)-\d.*jar/$1/;
$org=~s/(.*)\.jar/$1/;
return $org;
}
## Search the maven repo for a module matching this jar
## If found, return the module's organization
## if none, or more than one, are found, ask the user
sub maven_search_org
{
my ($module) = @_;
my $ua = LWP::UserAgent->new;
$ua->agent('Maven Repo Search');
my $service=
'http://maven.ozacc.com/search?format=xml&type=jar&keyword=';
my $url=URI->new($service . $module);
my $req=HTTP::Request->new;
$req->method('GET');
$req->uri($url);
my $resp = $ua->request($req)->content;
my $org = &parse_resp( $module, $resp );
return $org;
}
## Parse the XML response from the Maven search service
sub parse_resp
{
my ($module, $data) = @_;
my $xp = XML::XPath->new(xml => $data);
my $nodeset = $xp->find('//groupId'); # find all paragraphs
# Get rid of duplicates
my %uniq=();
foreach my $node ($nodeset->get_nodelist) {
$uniq{ $xp->getNodeText( $node ) } = 1;
}
my @keys = (sort keys %uniq);
if ($#keys == 0) {
return ($keys[0]);
} elsif ($#keys == -1) {
print "Found no module matching $module in Maven repo\n";
return -1;
} else {
print "Found multiple possible orgs for $module in Maven repo:\n";
foreach (@keys) {
print " $_\n";
}
return -1;
}
}
## (obsolete) Write ant task and ivy.xml conf lines to create repo
sub write_ivy_conf
{
my ($jar, $org, $module, $ver) = @_;
my $line = $pre . $org . "\" module=\"" . $module . "\" revision=\""
. $ver . "\"". $post . "\n";
system("mkdir -p $rep/$org/$module/jars");
link "$indir/$jar", "$rep/$org/$module/jars/$jar";
print $line;
print "<dependency org=\"" . $org . "\" name=\"" . $module . "\"
rev=\"" . $ver . "\"/>\n";
}
RE: Publishing 3rd party libraries to local repo
Posted by Troy Self <ts...@bbn.com>.
I got things working. I'm going to answer my own question for anyone else that happens upon this. My problems were with my ivy.xml file and my ivy:publish ant task. Corrections in-line:
> Here is an example where I am trying to publish the slf4j-log4j12 library:
>
> -------------------
> ivy.xml
> -------------------
> <ivy-module version="2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-
> instance"
>
> xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
> <info
> organisation="slf4j"
> module="slf4j-log4j12"
> status="release">
Remove the 'status' attribute from the ivy.xml file. This gets added by ivy:deliver.
> </info>
> <configurations>
> <conf name="default"
> description="Provides this module and its runtime deps."
> visibility="public" />
> </configurations>
> <publications>
> <artifact name="slf4j-log4j" />
The publications element is unnecessary if you are publishing a single jar w/ the same name as the module. Otherwise, you can specify specific publications as follows:
<publications>
<artifact /> <!-- shortcut for name same as module, type=jar, ext=jar -->
<artifact type="dll" ext="dll" conf="*" />
</publications>
What I did there was define 2 artifacts. One is a jar file and the other is a dll. They are both assumed to be named the same as the module. Otherwise you can specify a name attribute for each artifact.
> </publications>
> <dependencies>
> <dependency conf="*->default" org="slf4j" name="slf4j-api" rev="1.5.8"
> />
> <dependency conf="*->default" org="log4j" name="log4j"
> rev="latest.version" />
> </dependencies>
> </ivy-module>
>
> -----------------
> build.xml: publish
> -----------------
> <target name="publish" description="publish this module to ${to.resolver}">
> <ivy:resolve />
> <ivy:publish organisation="slf4j" module="slf4j-log4j12"
> resolver="${to.resolver}" revision="1.5.8">
> <artifacts pattern="jars/[artifact].[ext]" />
> </ivy:publish>
> </target>
Change the revision attribute to pubrevision. Then change the artifact pattern to "[type]/[artifact].[ext]". I then changed my directories so that the jar file is in the 'jar' directory and the dll is in the 'dll' directory.
With this setup, Ivy will create a 'published' version of your ivy.xml that replaces dynamic version numbers with real version numbers. It is going to drop it in the first artifact directory it uses. You can also specify the attribute 'srcivypattern' to put it somewhere explicitly (e.g. "ivy/ivy.xml"). That keeps it separate where it is easy to delete as part of your 'ant clean' task.
Thanks for the earlier help, since it led me in the right direction to figuring this out. If I get a convenient script written that makes it easy to quickly publish 3rd party libraries, I'll pass it along.
-- tBs
RE: Publishing 3rd party libraries to local repo
Posted by Troy Self <ts...@bbn.com>.
Thank you for the quick response. Follow-ups in-line:
> The most common thing I've seen people do to create third-party Ivy modules
> from scratch is to effectively blur the line between a source Ivy module
> and
> a published Ivy module. So instead of running the ivy:publish Ant task to
> publish the Ivy module to the repository, they would manually add it. This
> means going into the ivy.xml file and manually adding the content that
> would
> normally distinguish a published ivy.xml from a source one. This includes
> the publication and status attributes in the /ivy-module/info element.
This might explain my problem. I was adding those elements to the ivy.xml file and then using the ivy:publish ant task. Is ivy:publish supposed to automatically add publication and status for me? How does it know what artifacts from my build need to be sent?
Here is an example where I am trying to publish the slf4j-log4j library:
-------------------
ivy.xml
-------------------
<ivy-module version="2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
<info
organisation="slf4j"
module="slf4j-log4j12"
status="release">
</info>
<configurations>
<conf name="default"
description="Provides this module and its runtime deps."
visibility="public" />
</configurations>
<publications>
<artifact name="slf4j-log4j" />
</publications>
<dependencies>
<dependency conf="*->default" org="slf4j" name="slf4j-api" rev="1.5.8" />
<dependency conf="*->default" org="log4j" name="log4j" rev="latest.version" />
</dependencies>
</ivy-module>
-----------------
build.xml: publish
-----------------
<target name="publish" description="publish this module to ${to.resolver}">
<ivy:resolve />
<ivy:publish organisation="slf4j" module="slf4j-log4j12" resolver="${to.resolver}" revision="1.5.8">
<artifacts pattern="jars/[artifact].[ext]" />
</ivy:publish>
</target>
Here's what I get as a result:
Buildfile: build.xml
publish:
[ivy:resolve] :: Ivy 2.1.0-rc2 - 20090704004254 :: http://ant.apache.org/ivy/ ::
[ivy:resolve] :: loading settings :: file = C:\workspaces\code-workspace\IvyAdmin\ivysettings.xml
[ivy:resolve] :: resolving dependencies :: slf4j#slf4j-log4j12;working@ARLTSELF
[ivy:resolve] confs: [default]
[ivy:resolve] found slf4j#slf4j-api;1.5.8 in local
[ivy:resolve] found log4j#log4j;1.2.13 in default
[ivy:resolve] [1.2.13] log4j#log4j;latest.version
[ivy:resolve] :: resolution report :: resolve 3212ms :: artifacts dl 13ms
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 2 | 1 | 0 | 0 || 2 | 0 |
---------------------------------------------------------------------
:: delivering :: slf4j#slf4j-log4j12;1.5.8 :: 1.5.8 :: integration :: Mon Oct 05 14:11:04 EDT 2009
BUILD FAILED
C:\workspaces\code-workspace\IvyAdmin\build.xml:48: slf4j#slf4j-log4j12;1.5.8: j
ava.lang.IllegalStateException: ivy file not found in cache for slf4j#slf4j-log4
j12;1.5.8: please resolve dependencies before delivering (C:\Users\tself\.ivy2\c
ache\resolved-slf4j-slf4j-log4j12-1.5.8.xml)
It appears that it wants to resolve slf4j-log4j as a dependency before it will publish it.
Thanks for any help.
-- tBs
Troy Self
Re: Publishing 3rd party libraries to local repo
Posted by Mitch Gitman <mg...@gmail.com>.
Answers inline.
On Mon, Oct 5, 2009 at 5:04 AM, Troy Self <ts...@bbn.com> wrote:
> Hello,
>
> I am new to Ivy and am trying to setup a repository for my software team. I
> have
> gone through all of the tutorials on the web site and have some questions.
>
> Is there a tried and true approach for quickly creating repository entries
> for
> 3rd party libraries? I have used the ivy:install command to copy items that
> are
> already in Ibiblio or ivyrep, but what is the approach when neither of
> those
> public repositories have the library you want, or they don't have the
> latest
> version? Some examples are the slf4j-api and slf4j-log4j libraries. My
> project
> depends on the latest versions, which are not published publicly, so I'd
> like to
> add them to my own repo. Do you setup a project for each library and then
> ivy:publish them? Or do you just manually create ivy.xml files and
> carefully
> establish the correct directory/file structure in your repo?
>
The most common thing I've seen people do to create third-party Ivy modules
from scratch is to effectively blur the line between a source Ivy module and
a published Ivy module. So instead of running the ivy:publish Ant task to
publish the Ivy module to the repository, they would manually add it. This
means going into the ivy.xml file and manually adding the content that would
normally distinguish a published ivy.xml from a source one. This includes
the publication and status attributes in the /ivy-module/info element. One
thing you're lacking when you do this manual "pseudo-publish" is the
creation of the .md5 and .sha1 one-way hash files for the ivy.xml and the
artifacts.
A more correct--but still simple approach--is to create a build script that
does an ad hoc Ivy publish for an Ivy module whose name and location you
pass in. The easier to use you can make this build script, the better off
you'll be.
These are the two approaches I can think of that would be accessible to
someone new to Ivy.
>
> Is there a way to query for the configurations of a repository entry? In
> setting
> up my own dependencies, I need to map my own confs to the dependency's
> confs.
> Right now that requires browsing to the repository to find the ivy.xml
> files for
> each dependency and reading them. Is there a way to query for details about
> a
> particular artifact, including its confs?
>
I think most people don't find it too onerous to go straight to the ivy.xml
itself for confs and artifacts. Beyond that, you can use the ivy:report or
ivy:repreport Ant tasks to produce an HTML representation of Ivy modules.
The HTML report is arguably more readable.
>
> Is there a best practices for defining the various confs for a project?
> There
> seems to be some inconsistency between what 'runtime' and 'default' means
> across
> different dependencies.
>
Keep in mind that Ivy confs or conf names are arbitrary. You define your
own. So my suggestion is to establish your own naming conventions and then
just stick with them. Names like runtime or default are pretty much
interchangeable. The idea, though, is not to become too tied to your
standard palette of confs so that you can create new ones as needed to
accurately describe the published artifacts or special dependency needs of a
particular module. Beyond this, I would recommend making rigorous use of the
visibility attribute and trying to clearly delineate which confs are public
and which are private.
>
> Thanks very much for any help,
>
> -- tBs
>
> Troy Self
>
>
>