You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ivy-user@ant.apache.org by Francisco Peredo <fr...@tabasco.gob.mx> on 2009/03/24 19:56:44 UTC
private repository MANIFEST.MF guessing the right repository
structure
Hi!
Where I work we have a heavily restricted and extremely slow Internet
access, so letting Ivy download the dependencies for us from the web is just
not an option.
We download stuff at our houses, and then copy them in usb portable storage
and bring them to work.
The problem is that we did not use anything like Ivy, so everything was just
copied in to the WEB-INF\lib folders....
Now we want to use Ivy to manage library dependencies better...
So, we want to upload our WEB-INF\lib .jar files (for hibernate, seam,
spring, commons, etc) in to a shared repository to a server inside our
intranet...
I guess that to do that we need to use the task ivy:publish... Am I right?
But since we have hundreds of files, we would like to be able to do it in
single shot, something like
<ivy:publis-as-repository source-dir="WEB-INF\lib\">
And have Ivy read each of the MANIFEST.MF files inside each of the .jar
files and "guess" the right repository structure by reading stuff like
(from commons-beansutils.jar)
Inside its MANIFEST.MF we can read:
Implementation-Vendor: Apache Software Foundation
Implementation-Version: 1.6
and generating a structure like
shared-repository\apache\commons-beanutils\1.6\commons-beansutils.jar
Is it possible to automate this with Ivy? or do I have to write my own
ivy:publis-as-repository ant task?
Thanks
Regards,
Francisco Jose Peredo
--
View this message in context: http://www.nabble.com/private-repository-MANIFEST.MF-guessing-the-right-repository-structure-tp22687302p22687302.html
Sent from the ivy-user mailing list archive at Nabble.com.
Re: private repository MANIFEST.MF guessing the right repository
structure
Posted by Francisco Peredo <fr...@tabasco.gob.mx>.
Hi!
Thanks a lot! Interesting approach.... sadly I do not have perl in my
computer right now... I'll
download it at my house and bring it with me tomorrow...
I really think this should be a part of Ivy, it would really help to make
the transition from "everything copied in to WEB-INF/lib" in to the
centralized .jar repository approach. Maybe I'll create a JIRA issue...
Thanks
Regards,
Francisco Jose Peredo
Kirby Files wrote:
>
> Francisco,
>
> Francisco Peredo wrote on 03/24/2009 02:56 PM:
>> We download stuff at our houses, and then copy them in usb portable
>> storage
>> and bring them to work.
>>
> [...]
>> So, we want to upload our WEB-INF\lib .jar files (for hibernate, seam,
>> spring, commons, etc) in to a shared repository to a server inside our
>> intranet...
>>
> [...]
>> I guess that to do that we need to use the task ivy:publish... Am I
>> right?
>> But since we have hundreds of files, we would like to be able to do it in
>> single shot, something like
>>
>> <ivy:publis-as-repository source-dir="WEB-INF\lib\">
>
> Well, ivy:install makes more sense to me, as you don't have an ivy.xml
> file for each module.
>
> ivy:install is good for moving things from one repo to another. The
> source repo could be the ibiblio default repo, the IvyRoundup packager
> resolver, or a local filesystem.
>
> You could take another approach, and just use your home bandwidth to
> install all of your required jars (if you knew their org/module/rev)
> to a local filesystem resolver on a USB stick, using ibiblio as the
> source repo.
>
>>
>> Is it possible to automate this with Ivy? or do I have to write my own
>> ivy:publis-as-repository ant task?
>
> Well, it's probably not elegant, but when faced with the same problem,
> I wrote the following perl script, parse_jars.pl, which will use ant
> with a default target like this:
>
> <target name="installjar"
> depends="init-ivy"
> description="--> install jar to shared repo">
> <ivy:install
> settingsRef="masergy.settings"
> organisation="${org}"
> module="${module}"
> revision="${rev}"
> type="jar"
> from="${localjar.resolver}"
> to="${to.resolver}" />
> </target>
>
> My ivysettings.xml includes:
>
> <filesystem name="localjars">
> <artifact pattern="${install.path}/[artifact]-[revision].[ext]" />
> <artifact pattern="${install.path}/[artifact].[ext]" />
> </filesystem>
>
> The script will attempt to divine the correct org/module using 4
> techniques: (1) the ozacc Maven search API; (2) pattern-matching on
> the module name; (3) the top-level of the jar packages; (4) if all
> else fails. just ask the user.
>
> Note the script requires all jars to follow the naming convention
> module-ver.jar. Too few jars have relevant and consistent metadata
> within their MANIFEST to do otherwise.
>
> Thanks,
> ---
> Kirby Files
> Software Architect
> Masergy Communications
> kfiles@masergy.com
>
> 8< ------------------------ parse_jars.pl ---------------------- >8
> #!/usr/bin/perl
> # Looks for jars in the incoming directory, attemps to parse the
> # needed ivy info from the filename, and if successful, publishes
> # the jar to the Masergy ivrep repository.
>
> # If using linux with package mgmt, dependencies are:
> # perl-libwww-perl
> # perl-XML-Xpath
> # perl-Archive-Zip
> use LWP::UserAgent;
> use XML::XPath;
> use XML::XPath::XMLParser;
> use Archive::Zip;
> use Cwd;
> use strict qw(vars);
>
> use vars qw($indir $ant $rep $pre $post);
>
> $indir = shift(@ARGV);
> if ($indir !~ m#^/#) {
> my $cwd = getcwd();
> $indir =~ s/^/$cwd\//;
> print "CWD: $indir\n";
> }
> $ant = "/usr/local/ant/bin/ant";
> $rep = "newjars";
> $pre="<ivy:install settingsRef=\"masergy.settings\" organisation=\"";
> $post=" from=\"\${from.resolver}\" to=\"\${to.resolver}\" />";
>
> open (DIR, "ls -1 $indir|") || die "could not open $indir\n";
> while (<DIR>) {
> my ($org, $module, $jar, $ver);
> next unless /jar/;
> chomp;
> my $file = $indir . "/" . $_;
> $org=$_;
> $module=$_;
> $jar=$_;
> $ver=$_;
> $module=~s/^(.*)\.jar/$1/;
> $module=~s/([^.]*)-\d.*/$1/;
> $ver=~s/^[^.]*-([\d.]+.*)\.jar/$1/;
> $ver="unknown" unless $ver =~ /\d/;
> ## Figure out the org
> ## Ask the user, if there's ambiguity
> $org = &maven_search_org( $module );
> if (-1 == $org) {
> $org = &guess_org($module);
> print "Org guessed from module name:\n $org\n";
> $org = &classes_org($file);
> print "Org guessed from jar package naming:\n $org\n"
> unless (-1 == $org);
> do {
> $org = &get_user_org();
> } while ($org =~ /^$/);
> }
> if ($ver eq "unknown" || ($module !~ /^\w+$/) || $org !~ /\w+/) {
> print "Skipping jar $jar:\n";
> print " $org / $module / $ver\n";
> next;
> } else {
> &publish_jar($jar, $org, $module, $ver);
> }
> #&write_ivy_conf($jar, $org, $module, $ver);
> }
>
> ## Use ivy::install to publish the jar via sftp to ivyrep
> sub publish_jar
> {
> my ($jar, $org, $module, $ver) = @_;
> print "Uploading $org / $module / $jar ... ";
> my $resp = `$ant -Dorg=$org -Dmodule=$module -Drev=$ver
> -Dinstall.path=$indir 2>&1`;
> if ($resp =~ /failed/i) {
> print STDERR "Error uploading $jar:\n", $resp;
> print "\n";
> } else {
> print "Done.\n";
> }
> }
>
> ## Find a directory in the jar to propose as the org
> ## searches for any dirs with two path elements
> sub classes_org
> {
> my ($file) = @_;
> my $somezip = Archive::Zip->new();
> unless ( $somezip->read( $file ) == AZ_OK ) {
> die "Error reading jarfile $file";
> }
> my @members = $somezip->members();
> my @dirnames = ();
> foreach (@members) {
> if ($_->isDirectory()) {
> my $fname = $_->fileName();
> if ($fname =~ /^\w+\/\w+\/$/) {
> $fname =~ s/^(\w+)\/(\w+)\/$/$1.$2/;
> return $fname;
> }
> }
> }
> return -1;
> }
>
> ## Ask the user for the best org name
> sub get_user_org
> {
> print "Enter preffered org:> ";
> my $line = <STDIN>;
> chomp($line);
> return $line;
> }
>
> ## Make a guess about the org name from the module name
> sub guess_org
> {
> my ($module) = @_;
> my $org = $module;
> $org=~s/spring.*/springframework/;
> $org=~s/hiber.*/hibernate/;
> $org=~s/standard.*/taglibs/;
> $org=~s/jstl.*/taglibs/;
> $org=~s/persistence.*/sun/;
> $org=~s/jta.*/sun/;
> $org=~s/servlet-api.*/tomcat/;
> $org=~s/ojdbc.*/oracle/;
> $org=~s/cglib.*/cglib/;
> $org=~s/javassist.*/jboss/;
> $org=~s/([^.]*)-\d.*jar/$1/;
> $org=~s/(.*)\.jar/$1/;
> return $org;
> }
>
> ## Search the maven repo for a module matching this jar
> ## If found, return the module's organization
> ## if none, or more than one, are found, ask the user
> sub maven_search_org
> {
> my ($module) = @_;
>
> my $ua = LWP::UserAgent->new;
> $ua->agent('Maven Repo Search');
>
> my $service=
> 'http://maven.ozacc.com/search?format=xml&type=jar&keyword=';
> my $url=URI->new($service . $module);
> my $req=HTTP::Request->new;
> $req->method('GET');
> $req->uri($url);
> my $resp = $ua->request($req)->content;
>
> my $org = &parse_resp( $module, $resp );
> return $org;
> }
>
> ## Parse the XML response from the Maven search service
> sub parse_resp
> {
> my ($module, $data) = @_;
>
> my $xp = XML::XPath->new(xml => $data);
>
> my $nodeset = $xp->find('//groupId'); # find all paragraphs
>
> # Get rid of duplicates
> my %uniq=();
> foreach my $node ($nodeset->get_nodelist) {
> $uniq{ $xp->getNodeText( $node ) } = 1;
> }
> my @keys = (sort keys %uniq);
>
> if ($#keys == 0) {
> return ($keys[0]);
> } elsif ($#keys == -1) {
> print "Found no module matching $module in Maven repo\n";
> return -1;
> } else {
> print "Found multiple possible orgs for $module in Maven repo:\n";
> foreach (@keys) {
> print " $_\n";
> }
> return -1;
> }
> }
>
> ## (obsolete) Write ant task and ivy.xml conf lines to create repo
> sub write_ivy_conf
> {
> my ($jar, $org, $module, $ver) = @_;
> my $line = $pre . $org . "\" module=\"" . $module . "\"
> revision=\"" . $ver . "\"". $post . "\n";
> system("mkdir -p $rep/$org/$module/jars");
> link "$indir/$jar", "$rep/$org/$module/jars/$jar";
> print $line;
> print "<dependency org=\"" . $org . "\" name=\"" . $module . "\"
> rev=\"" . $ver . "\"/>\n";
> }
>
>
--
View this message in context: http://www.nabble.com/private-repository-MANIFEST.MF-guessing-the-right-repository-structure-tp22687302p22690777.html
Sent from the ivy-user mailing list archive at Nabble.com.
Re: private repository MANIFEST.MF guessing the right repository
structure
Posted by Kirby Files <kf...@masergy.com>.
Francisco,
Francisco Peredo wrote on 03/24/2009 02:56 PM:
> We download stuff at our houses, and then copy them in usb portable storage
> and bring them to work.
>
[...]
> So, we want to upload our WEB-INF\lib .jar files (for hibernate, seam,
> spring, commons, etc) in to a shared repository to a server inside our
> intranet...
>
[...]
> I guess that to do that we need to use the task ivy:publish... Am I right?
> But since we have hundreds of files, we would like to be able to do it in
> single shot, something like
>
> <ivy:publis-as-repository source-dir="WEB-INF\lib\">
Well, ivy:install makes more sense to me, as you don't have an ivy.xml
file for each module.
ivy:install is good for moving things from one repo to another. The
source repo could be the ibiblio default repo, the IvyRoundup packager
resolver, or a local filesystem.
You could take another approach, and just use your home bandwidth to
install all of your required jars (if you knew their org/module/rev)
to a local filesystem resolver on a USB stick, using ibiblio as the
source repo.
>
> Is it possible to automate this with Ivy? or do I have to write my own
> ivy:publis-as-repository ant task?
Well, it's probably not elegant, but when faced with the same problem,
I wrote the following perl script, parse_jars.pl, which will use ant
with a default target like this:
<target name="installjar"
depends="init-ivy"
description="--> install jar to shared repo">
<ivy:install
settingsRef="masergy.settings"
organisation="${org}"
module="${module}"
revision="${rev}"
type="jar"
from="${localjar.resolver}"
to="${to.resolver}" />
</target>
My ivysettings.xml includes:
<filesystem name="localjars">
<artifact pattern="${install.path}/[artifact]-[revision].[ext]" />
<artifact pattern="${install.path}/[artifact].[ext]" />
</filesystem>
The script will attempt to divine the correct org/module using 4
techniques: (1) the ozacc Maven search API; (2) pattern-matching on
the module name; (3) the top-level of the jar packages; (4) if all
else fails. just ask the user.
Note the script requires all jars to follow the naming convention
module-ver.jar. Too few jars have relevant and consistent metadata
within their MANIFEST to do otherwise.
Thanks,
---
Kirby Files
Software Architect
Masergy Communications
kfiles@masergy.com
8< ------------------------ parse_jars.pl ---------------------- >8
#!/usr/bin/perl
# Looks for jars in the incoming directory, attemps to parse the
# needed ivy info from the filename, and if successful, publishes
# the jar to the Masergy ivrep repository.
# If using linux with package mgmt, dependencies are:
# perl-libwww-perl
# perl-XML-Xpath
# perl-Archive-Zip
use LWP::UserAgent;
use XML::XPath;
use XML::XPath::XMLParser;
use Archive::Zip;
use Cwd;
use strict qw(vars);
use vars qw($indir $ant $rep $pre $post);
$indir = shift(@ARGV);
if ($indir !~ m#^/#) {
my $cwd = getcwd();
$indir =~ s/^/$cwd\//;
print "CWD: $indir\n";
}
$ant = "/usr/local/ant/bin/ant";
$rep = "newjars";
$pre="<ivy:install settingsRef=\"masergy.settings\" organisation=\"";
$post=" from=\"\${from.resolver}\" to=\"\${to.resolver}\" />";
open (DIR, "ls -1 $indir|") || die "could not open $indir\n";
while (<DIR>) {
my ($org, $module, $jar, $ver);
next unless /jar/;
chomp;
my $file = $indir . "/" . $_;
$org=$_;
$module=$_;
$jar=$_;
$ver=$_;
$module=~s/^(.*)\.jar/$1/;
$module=~s/([^.]*)-\d.*/$1/;
$ver=~s/^[^.]*-([\d.]+.*)\.jar/$1/;
$ver="unknown" unless $ver =~ /\d/;
## Figure out the org
## Ask the user, if there's ambiguity
$org = &maven_search_org( $module );
if (-1 == $org) {
$org = &guess_org($module);
print "Org guessed from module name:\n $org\n";
$org = &classes_org($file);
print "Org guessed from jar package naming:\n $org\n"
unless (-1 == $org);
do {
$org = &get_user_org();
} while ($org =~ /^$/);
}
if ($ver eq "unknown" || ($module !~ /^\w+$/) || $org !~ /\w+/) {
print "Skipping jar $jar:\n";
print " $org / $module / $ver\n";
next;
} else {
&publish_jar($jar, $org, $module, $ver);
}
#&write_ivy_conf($jar, $org, $module, $ver);
}
## Use ivy::install to publish the jar via sftp to ivyrep
sub publish_jar
{
my ($jar, $org, $module, $ver) = @_;
print "Uploading $org / $module / $jar ... ";
my $resp = `$ant -Dorg=$org -Dmodule=$module -Drev=$ver
-Dinstall.path=$indir 2>&1`;
if ($resp =~ /failed/i) {
print STDERR "Error uploading $jar:\n", $resp;
print "\n";
} else {
print "Done.\n";
}
}
## Find a directory in the jar to propose as the org
## searches for any dirs with two path elements
sub classes_org
{
my ($file) = @_;
my $somezip = Archive::Zip->new();
unless ( $somezip->read( $file ) == AZ_OK ) {
die "Error reading jarfile $file";
}
my @members = $somezip->members();
my @dirnames = ();
foreach (@members) {
if ($_->isDirectory()) {
my $fname = $_->fileName();
if ($fname =~ /^\w+\/\w+\/$/) {
$fname =~ s/^(\w+)\/(\w+)\/$/$1.$2/;
return $fname;
}
}
}
return -1;
}
## Ask the user for the best org name
sub get_user_org
{
print "Enter preffered org:> ";
my $line = <STDIN>;
chomp($line);
return $line;
}
## Make a guess about the org name from the module name
sub guess_org
{
my ($module) = @_;
my $org = $module;
$org=~s/spring.*/springframework/;
$org=~s/hiber.*/hibernate/;
$org=~s/standard.*/taglibs/;
$org=~s/jstl.*/taglibs/;
$org=~s/persistence.*/sun/;
$org=~s/jta.*/sun/;
$org=~s/servlet-api.*/tomcat/;
$org=~s/ojdbc.*/oracle/;
$org=~s/cglib.*/cglib/;
$org=~s/javassist.*/jboss/;
$org=~s/([^.]*)-\d.*jar/$1/;
$org=~s/(.*)\.jar/$1/;
return $org;
}
## Search the maven repo for a module matching this jar
## If found, return the module's organization
## if none, or more than one, are found, ask the user
sub maven_search_org
{
my ($module) = @_;
my $ua = LWP::UserAgent->new;
$ua->agent('Maven Repo Search');
my $service=
'http://maven.ozacc.com/search?format=xml&type=jar&keyword=';
my $url=URI->new($service . $module);
my $req=HTTP::Request->new;
$req->method('GET');
$req->uri($url);
my $resp = $ua->request($req)->content;
my $org = &parse_resp( $module, $resp );
return $org;
}
## Parse the XML response from the Maven search service
sub parse_resp
{
my ($module, $data) = @_;
my $xp = XML::XPath->new(xml => $data);
my $nodeset = $xp->find('//groupId'); # find all paragraphs
# Get rid of duplicates
my %uniq=();
foreach my $node ($nodeset->get_nodelist) {
$uniq{ $xp->getNodeText( $node ) } = 1;
}
my @keys = (sort keys %uniq);
if ($#keys == 0) {
return ($keys[0]);
} elsif ($#keys == -1) {
print "Found no module matching $module in Maven repo\n";
return -1;
} else {
print "Found multiple possible orgs for $module in Maven repo:\n";
foreach (@keys) {
print " $_\n";
}
return -1;
}
}
## (obsolete) Write ant task and ivy.xml conf lines to create repo
sub write_ivy_conf
{
my ($jar, $org, $module, $ver) = @_;
my $line = $pre . $org . "\" module=\"" . $module . "\"
revision=\"" . $ver . "\"". $post . "\n";
system("mkdir -p $rep/$org/$module/jars");
link "$indir/$jar", "$rep/$org/$module/jars/$jar";
print $line;
print "<dependency org=\"" . $org . "\" name=\"" . $module . "\"
rev=\"" . $ver . "\"/>\n";
}