You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by jo...@barclayscapital.com on 2011/05/02 21:00:04 UTC
How to create a filesystem repository
Hello to all,
Can someone tell me how I can create a local filesystem repository with Ivy? I've been through the documentation and it is not clear how I can get started.
My jar files are in the format
Libdirectory/module/jar1.jar
Libdirectory/module/jar2.jar
Libdirectory/module/jar3.jar
Thanks in advance,
--Jose
_______________________________________________
This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group.
_______________________________________________
Re: How to create a filesystem repository
Posted by Kirby Files <kf...@masergy.com>.
jose.nunez-zuleta@barclayscapital.com wrote on 05/02/2011 04:55 PM:
> Hello Kirby,
>
> Bootstrapping to an existing repository is the best way? By bootstrapping a monstrous repository will be created with a lot of stuff that is not really needed in the end. How difficult is to create an 'empty repository' and fill it with the right contents? (your Perl script is definitely useful, same as the ant task to publish to the new repository).
>
> I'm trying to follow this short tutorial:
>
> http://www.drillio.com/en/software-build/apache-ivy/create-local-repository-guide/
I think we're saying the same thing. I didn't mean to suggest that you
copy the *entire* contents of the SpringSource, Jboss, or Maven repos
into yours, just that you use ivy:install to copy your public
dependencies from one of those repos (use the main Maven repo as a
last resort, as its metadata is truly inconsistent). So everything in
your repo should be necessary -- a dependency (direct or transitive)
of your project.
Ivy will create its own ivy.xml from poms, though you may desire to
edit it, if the current metadata is a mess.
Thanks,
---
Kirby Files
Software Architect
Masergy Communications
kfiles@masergy.com
RE: How to create a filesystem repository
Posted by jo...@barclayscapital.com.
Hello Kirby,
Bootstrapping to an existing repository is the best way? By bootstrapping a monstrous repository will be created with a lot of stuff that is not really needed in the end. How difficult is to create an 'empty repository' and fill it with the right contents? (your Perl script is definitely useful, same as the ant task to publish to the new repository).
I'm trying to follow this short tutorial:
http://www.drillio.com/en/software-build/apache-ivy/create-local-repository-guide/
Not exactly what I need but I like the fact that you are in control on what is inside the repository. I'll let you know how far I get with this :-)
Anybody else has other suggestions?
Thanks in advance!
--Jose
-----Original Message-----
From: Kirby Files [mailto:kfiles@masergy.com]
Sent: 02 May 2011 16:42
To: ivy-user@ant.apache.org
Cc: Nunez Zuleta, Jose V: IT (NYK)
Subject: Re: How to create a filesystem repository
jose.nunez-zuleta@barclayscapital.com wrote on 05/02/2011 03:00 PM:
> Hello to all,
>
> Can someone tell me how I can create a local filesystem repository with Ivy? I've been through the documentation and it is not clear how I can get started.
>
> My jar files are in the format
>
> Libdirectory/module/jar1.jar
> Libdirectory/module/jar2.jar
Yes, Jose, this is one of the more frustrating deficiencies in the Ivy
toolset: it's difficult to migrate from the most common situation (a
directory full of jars with no metadata) to an ivy.xml file with
dependencies and a repository containing those jars with metadata.
I have some suggestions:
1) Start with some decent metadata and existing sources of dependency
management: use the IvyRoundup packager repository or the public Maven
repos (with the ibiblio resolver) to bootstrap your filesystem
repository with the <ivy:install> task (it can copy from a source repo
to a dest repo, if necessary converting the maven poms to ivy.xml in
the process).
2) Create an ivysettings.xml with all of the repos, public and
private, that you'll need to access.
3) Create an ivy.xml and a ivy:publish target for each of your
projects, and publish those to your repository.
3) Use scripts to install jars not available publicly, nor produced
internally, from a directory to your repository.
I use this script:
--------------------------------------
#!/bin/csh
if ($1 == "") then
echo "Usage: installjar <org> <module> <revision> <path-to-file>"
else
/usr/local/ant/bin/ant -Dorg="$1" -Dmodule="$2" -Drev="$3"
-Dinstall.path="$4" $5 $6 $7 $8 $9
endif
--------------------------------------
With the following build target:
--------------------------------------
<target name="installjar"
depends="init-ivy"
description="--> install jar to masrep">
<ivy:install settingsRef="masergy.settings" organisation="${org}"
module="${module}" revision="${rev}" type="jar"
from="${localjar.resolver}" to="${to.resolver}" />
</target>
--------------------------------------
Where my build.properties defines these variables:
localjar.resolver=localjars
to.resolver=masrep_sftp
And my ivysettings.xml includes:
<resolvers>
<filesystem name="localjars">
<artifact pattern="${install.path}/[artifact]-[revision].[ext]" />
<artifact pattern="${install.path}/[artifact].[ext]" />
</filesystem>
</resolvers>
This script allows me to specify a jar on the local filesystem by its
name and the directory in which it resides (install.path), add the
org, and rev metadata, and publish it to one of my resolvers (I use
SFTP with a keyfile to push files up to our webserver; you may use a
filesystem resolver or webdav as a destination).
I also have a bigger hack of a perl script which attempts to read a
directory of jars and guess the org, modul, and rev (validating
against maven2), or prompt the user when it's unclear. See below.
Thanks,
---
Kirby Files
Software Architect
Masergy Communications
kfiles@masergy.com
----------------------------------------------
#!/usr/bin/perl
# Looks for jars in the incoming directory, attemps to parse the
# needed ivy info from the filename, and if successful, publishes
# the jar to the Masergy ivrep repository.
# If using linux with package mgmt, dependencies are:
# perl-libwww-perl
# perl-XML-Xpath
# perl-Archive-Zip
use LWP::UserAgent;
use XML::XPath;
use XML::XPath::XMLParser;
use Archive::Zip;
use Cwd;
use strict qw(vars);
use vars qw($indir $ant $rep $pre $post);
$indir = shift(@ARGV);
if ($indir !~ m#^/#) {
my $cwd = getcwd();
$indir =~ s/^/$cwd\//;
print "CWD: $indir\n";
}
$ant = "/usr/local/ant/bin/ant";
$rep = "newjars";
$pre="<ivy:install settingsRef=\"masergy.settings\" organisation=\"";
$post=" from=\"\${from.resolver}\" to=\"\${to.resolver}\" />";
open (DIR, "ls -1 $indir|") || die "could not open $indir\n";
while (<DIR>) {
my ($org, $module, $jar, $ver);
next unless /jar/;
chomp;
my $file = $indir . "/" . $_;
$org=$_;
$module=$_;
$jar=$_;
$ver=$_;
$module=~s/^(.*)\.jar/$1/;
$module=~s/([^.]*)-\d.*/$1/;
$ver=~s/^[^.]*-([\d.]+.*)\.jar/$1/;
$ver="unknown" unless $ver =~ /\d/;
## Figure out the org
## Ask the user, if there's ambiguity
$org = &maven_search_org( $module );
if (-1 == $org) {
$org = &guess_org($module);
print "Org guessed from module name:\n $org\n";
$org = &classes_org($file);
print "Org guessed from jar package naming:\n $org\n"
unless (-1 == $org);
do {
$org = &get_user_org();
} while ($org =~ /^$/);
}
if ($ver eq "unknown" || ($module !~ /^\w+$/) || $org !~ /\w+/) {
print "Skipping jar $jar:\n";
print " $org / $module / $ver\n";
next;
} else {
&publish_jar($jar, $org, $module, $ver);
}
#&write_ivy_conf($jar, $org, $module, $ver);
}
## Use ivy::install to publish the jar via sftp to ivyrep
sub publish_jar
{
my ($jar, $org, $module, $ver) = @_;
print "Uploading $org / $module / $jar ... ";
my $resp = `$ant -Dorg=$org -Dmodule=$module -Drev=$ver
-Dinstall.path=$indir 2>&1`;
if ($resp =~ /failed/i) {
print STDERR "Error uploading $jar:\n", $resp;
print "\n";
} else {
print "Done.\n";
}
}
## Find a directory in the jar to propose as the org
## searches for any dirs with two path elements
sub classes_org
{
my ($file) = @_;
my $somezip = Archive::Zip->new();
unless ( $somezip->read( $file ) == AZ_OK ) {
die "Error reading jarfile $file";
}
my @members = $somezip->members();
my @dirnames = ();
foreach (@members) {
if ($_->isDirectory()) {
my $fname = $_->fileName();
if ($fname =~ /^\w+\/\w+\/$/) {
$fname =~ s/^(\w+)\/(\w+)\/$/$1.$2/;
return $fname;
}
}
}
return -1;
}
## Ask the user for the best org name
sub get_user_org
{
print "Enter preffered org:> ";
my $line = <STDIN>;
chomp($line);
return $line;
}
## Make a guess about the org name from the module name
sub guess_org
{
my ($module) = @_;
my $org = $module;
$org=~s/spring.*/springframework/;
$org=~s/hiber.*/hibernate/;
$org=~s/standard.*/taglibs/;
$org=~s/jstl.*/taglibs/;
$org=~s/persistence.*/sun/;
$org=~s/jta.*/sun/;
$org=~s/servlet-api.*/tomcat/;
$org=~s/ojdbc.*/oracle/;
$org=~s/cglib.*/cglib/;
$org=~s/javassist.*/jboss/;
$org=~s/([^.]*)-\d.*jar/$1/;
$org=~s/(.*)\.jar/$1/;
return $org;
}
## Search the maven repo for a module matching this jar
## If found, return the module's organization
## if none, or more than one, are found, ask the user
sub maven_search_org
{
my ($module) = @_;
my $ua = LWP::UserAgent->new;
$ua->agent('Maven Repo Search');
my $service=
'http://maven.ozacc.com/search?format=xml&type=jar&keyword=';
my $url=URI->new($service . $module);
my $req=HTTP::Request->new;
$req->method('GET');
$req->uri($url);
my $resp = $ua->request($req)->content;
my $org = &parse_resp( $module, $resp );
return $org;
}
## Parse the XML response from the Maven search service
sub parse_resp
{
my ($module, $data) = @_;
my $xp = XML::XPath->new(xml => $data);
my $nodeset = $xp->find('//groupId'); # find all paragraphs
# Get rid of duplicates
my %uniq=();
foreach my $node ($nodeset->get_nodelist) {
$uniq{ $xp->getNodeText( $node ) } = 1;
}
my @keys = (sort keys %uniq);
if ($#keys == 0) {
return ($keys[0]);
} elsif ($#keys == -1) {
print "Found no module matching $module in Maven repo\n";
return -1;
} else {
print "Found multiple possible orgs for $module in Maven repo:\n";
foreach (@keys) {
print " $_\n";
}
return -1;
}
}
## (obsolete) Write ant task and ivy.xml conf lines to create repo
sub write_ivy_conf
{
my ($jar, $org, $module, $ver) = @_;
my $line = $pre . $org . "\" module=\"" . $module . "\"
revision=\"" . $ver . "\"". $post . "\n";
system("mkdir -p $rep/$org/$module/jars");
link "$indir/$jar", "$rep/$org/$module/jars/$jar";
print $line;
print "<dependency org=\"" . $org . "\" name=\"" . $module . "\"
rev=\"" . $ver . "\"/>\n";
}
_______________________________________________
This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group.
_______________________________________________
Re: How to create a filesystem repository
Posted by Kirby Files <kf...@masergy.com>.
jose.nunez-zuleta@barclayscapital.com wrote on 05/02/2011 03:00 PM:
> Hello to all,
>
> Can someone tell me how I can create a local filesystem repository with Ivy? I've been through the documentation and it is not clear how I can get started.
>
> My jar files are in the format
>
> Libdirectory/module/jar1.jar
> Libdirectory/module/jar2.jar
Yes, Jose, this is one of the more frustrating deficiencies in the Ivy
toolset: it's difficult to migrate from the most common situation (a
directory full of jars with no metadata) to an ivy.xml file with
dependencies and a repository containing those jars with metadata.
I have some suggestions:
1) Start with some decent metadata and existing sources of dependency
management: use the IvyRoundup packager repository or the public Maven
repos (with the ibiblio resolver) to bootstrap your filesystem
repository with the <ivy:install> task (it can copy from a source repo
to a dest repo, if necessary converting the maven poms to ivy.xml in
the process).
2) Create an ivysettings.xml with all of the repos, public and
private, that you'll need to access.
3) Create an ivy.xml and a ivy:publish target for each of your
projects, and publish those to your repository.
3) Use scripts to install jars not available publicly, nor produced
internally, from a directory to your repository.
I use this script:
--------------------------------------
#!/bin/csh
if ($1 == "") then
echo "Usage: installjar <org> <module> <revision> <path-to-file>"
else
/usr/local/ant/bin/ant -Dorg="$1" -Dmodule="$2" -Drev="$3"
-Dinstall.path="$4" $5 $6 $7 $8 $9
endif
--------------------------------------
With the following build target:
--------------------------------------
<target name="installjar"
depends="init-ivy"
description="--> install jar to masrep">
<ivy:install settingsRef="masergy.settings" organisation="${org}"
module="${module}" revision="${rev}" type="jar"
from="${localjar.resolver}" to="${to.resolver}" />
</target>
--------------------------------------
Where my build.properties defines these variables:
localjar.resolver=localjars
to.resolver=masrep_sftp
And my ivysettings.xml includes:
<resolvers>
<filesystem name="localjars">
<artifact pattern="${install.path}/[artifact]-[revision].[ext]" />
<artifact pattern="${install.path}/[artifact].[ext]" />
</filesystem>
</resolvers>
This script allows me to specify a jar on the local filesystem by its
name and the directory in which it resides (install.path), add the
org, and rev metadata, and publish it to one of my resolvers (I use
SFTP with a keyfile to push files up to our webserver; you may use a
filesystem resolver or webdav as a destination).
I also have a bigger hack of a perl script which attempts to read a
directory of jars and guess the org, modul, and rev (validating
against maven2), or prompt the user when it's unclear. See below.
Thanks,
---
Kirby Files
Software Architect
Masergy Communications
kfiles@masergy.com
----------------------------------------------
#!/usr/bin/perl
# Looks for jars in the incoming directory, attemps to parse the
# needed ivy info from the filename, and if successful, publishes
# the jar to the Masergy ivrep repository.
# If using linux with package mgmt, dependencies are:
# perl-libwww-perl
# perl-XML-Xpath
# perl-Archive-Zip
use LWP::UserAgent;
use XML::XPath;
use XML::XPath::XMLParser;
use Archive::Zip;
use Cwd;
use strict qw(vars);
use vars qw($indir $ant $rep $pre $post);
$indir = shift(@ARGV);
if ($indir !~ m#^/#) {
my $cwd = getcwd();
$indir =~ s/^/$cwd\//;
print "CWD: $indir\n";
}
$ant = "/usr/local/ant/bin/ant";
$rep = "newjars";
$pre="<ivy:install settingsRef=\"masergy.settings\" organisation=\"";
$post=" from=\"\${from.resolver}\" to=\"\${to.resolver}\" />";
open (DIR, "ls -1 $indir|") || die "could not open $indir\n";
while (<DIR>) {
my ($org, $module, $jar, $ver);
next unless /jar/;
chomp;
my $file = $indir . "/" . $_;
$org=$_;
$module=$_;
$jar=$_;
$ver=$_;
$module=~s/^(.*)\.jar/$1/;
$module=~s/([^.]*)-\d.*/$1/;
$ver=~s/^[^.]*-([\d.]+.*)\.jar/$1/;
$ver="unknown" unless $ver =~ /\d/;
## Figure out the org
## Ask the user, if there's ambiguity
$org = &maven_search_org( $module );
if (-1 == $org) {
$org = &guess_org($module);
print "Org guessed from module name:\n $org\n";
$org = &classes_org($file);
print "Org guessed from jar package naming:\n $org\n"
unless (-1 == $org);
do {
$org = &get_user_org();
} while ($org =~ /^$/);
}
if ($ver eq "unknown" || ($module !~ /^\w+$/) || $org !~ /\w+/) {
print "Skipping jar $jar:\n";
print " $org / $module / $ver\n";
next;
} else {
&publish_jar($jar, $org, $module, $ver);
}
#&write_ivy_conf($jar, $org, $module, $ver);
}
## Use ivy::install to publish the jar via sftp to ivyrep
sub publish_jar
{
my ($jar, $org, $module, $ver) = @_;
print "Uploading $org / $module / $jar ... ";
my $resp = `$ant -Dorg=$org -Dmodule=$module -Drev=$ver
-Dinstall.path=$indir 2>&1`;
if ($resp =~ /failed/i) {
print STDERR "Error uploading $jar:\n", $resp;
print "\n";
} else {
print "Done.\n";
}
}
## Find a directory in the jar to propose as the org
## searches for any dirs with two path elements
sub classes_org
{
my ($file) = @_;
my $somezip = Archive::Zip->new();
unless ( $somezip->read( $file ) == AZ_OK ) {
die "Error reading jarfile $file";
}
my @members = $somezip->members();
my @dirnames = ();
foreach (@members) {
if ($_->isDirectory()) {
my $fname = $_->fileName();
if ($fname =~ /^\w+\/\w+\/$/) {
$fname =~ s/^(\w+)\/(\w+)\/$/$1.$2/;
return $fname;
}
}
}
return -1;
}
## Ask the user for the best org name
sub get_user_org
{
print "Enter preffered org:> ";
my $line = <STDIN>;
chomp($line);
return $line;
}
## Make a guess about the org name from the module name
sub guess_org
{
my ($module) = @_;
my $org = $module;
$org=~s/spring.*/springframework/;
$org=~s/hiber.*/hibernate/;
$org=~s/standard.*/taglibs/;
$org=~s/jstl.*/taglibs/;
$org=~s/persistence.*/sun/;
$org=~s/jta.*/sun/;
$org=~s/servlet-api.*/tomcat/;
$org=~s/ojdbc.*/oracle/;
$org=~s/cglib.*/cglib/;
$org=~s/javassist.*/jboss/;
$org=~s/([^.]*)-\d.*jar/$1/;
$org=~s/(.*)\.jar/$1/;
return $org;
}
## Search the maven repo for a module matching this jar
## If found, return the module's organization
## if none, or more than one, are found, ask the user
sub maven_search_org
{
my ($module) = @_;
my $ua = LWP::UserAgent->new;
$ua->agent('Maven Repo Search');
my $service=
'http://maven.ozacc.com/search?format=xml&type=jar&keyword=';
my $url=URI->new($service . $module);
my $req=HTTP::Request->new;
$req->method('GET');
$req->uri($url);
my $resp = $ua->request($req)->content;
my $org = &parse_resp( $module, $resp );
return $org;
}
## Parse the XML response from the Maven search service
sub parse_resp
{
my ($module, $data) = @_;
my $xp = XML::XPath->new(xml => $data);
my $nodeset = $xp->find('//groupId'); # find all paragraphs
# Get rid of duplicates
my %uniq=();
foreach my $node ($nodeset->get_nodelist) {
$uniq{ $xp->getNodeText( $node ) } = 1;
}
my @keys = (sort keys %uniq);
if ($#keys == 0) {
return ($keys[0]);
} elsif ($#keys == -1) {
print "Found no module matching $module in Maven repo\n";
return -1;
} else {
print "Found multiple possible orgs for $module in Maven repo:\n";
foreach (@keys) {
print " $_\n";
}
return -1;
}
}
## (obsolete) Write ant task and ivy.xml conf lines to create repo
sub write_ivy_conf
{
my ($jar, $org, $module, $ver) = @_;
my $line = $pre . $org . "\" module=\"" . $module . "\"
revision=\"" . $ver . "\"". $post . "\n";
system("mkdir -p $rep/$org/$module/jars");
link "$indir/$jar", "$rep/$org/$module/jars/$jar";
print $line;
print "<dependency org=\"" . $org . "\" name=\"" . $module . "\"
rev=\"" . $ver . "\"/>\n";
}