You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Matthieu Moy <ma...@grenoble-inp.fr> on 2013/10/29 10:38:01 UTC
System-wide, read-only repository
Hi,
I have a setup with many users (200 students), each of them having a limited $HOME. I'm looking for a way to provide them the most common plugins and dependencies and save disk space.
Is there a way to have a system-wide (e.g. /usr/share/maven/... or so) repository, where the sysadmin could download stuff that the students wouldn't have to download and store in their $HOME ?
I found many solutions using a proxy, that would allow me to download from the intranet, but these solutions still require a local repository in $HOME/.m2/, so this saves bandwidth but not disk space. I also saw some people advising to set localRepository in their pom.xml, but this does not seem to be a good idea (not thread-safe), and doesn't really solve my issue, as I would like the system-wide repository to be read-only (managed by root only), and to come in addition to $HOME/.m2/, not as a replacement.
Thanks in advance,
--
Matthieu Moy
http://www-verimag.imag.fr/~moy/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: System-wide, read-only repository
Posted by Stephen Connolly <st...@gmail.com>.
If you can control the version of Maven they run, you could create a aether
plugin that would let you use a tiered set of repositories... at least if I
understand aether correctly... then by putting that in the $MAVEN_HOME/lib
and doing some magic you might be able to at least share the read-only
part... but you would be neck deep in aether to write such a thing... and I
would advise checking with Benjamin to confirm that such a thing is
possible before starting down that road
On 30 October 2013 09:18, Matthieu Moy <Ma...@grenoble-inp.fr> wrote:
> Barrie Treloar <ba...@gmail.com> writes:
>
> > On 29 October 2013 23:56, Lyons, Roy <Ro...@cmegroup.com> wrote:
> >> Unfortunately, you will always have something in $HOME/.m2/repository
> >> because that's how maven works.
> >>
> >> Can I suggest perhaps that you use zfs for deduplication in /home?
> >> Otherwise, you can add something like
> >
> > Or give them more disk space - isn't this stuff meant to be cheap
> now-a-days?
>
> Local disk space is cheap. NFS-shared, RAID & backed-up disk space, less
> so. I can live with a few Gb of waste, but I was just wondering whether
> we could do any better.
>
> > If they dont have direct access to the internet,
>
> They do.
>
> Thanks,
>
> --
> Matthieu Moy
> http://www-verimag.imag.fr/~moy/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: System-wide, read-only repository
Posted by "Lyons, Roy" <Ro...@cmegroup.com>.
gah, resending with the link this time.
http://openbenchmarking.org/result/1304257-FO-MERGE786390
On 10/30/13 11:21 AM, "Lyons, Roy" <Ro...@cmegroup.com> wrote:
>I am trying to get my unix admins to use it for us. here are some
>benchmarks I have seen.
>
>
>On 10/30/13 10:51 AM, "Curtis Rueden" <ct...@wisc.edu> wrote:
>
>>Hi Roy,
>>
>>> zfs has built in de-duplication.
>>
>>ZFS sounds awesome in theory but have you actually tried it? If so, how
>>is
>>it working for you? In particular, how is the performance?
>>
>>-Curtis
>>
>>
>>On Wed, Oct 30, 2013 at 10:43 AM, Lyons, Roy <Ro...@cmegroup.com>
>>wrote:
>>
>>> :) but like I said, you wouldnt worry about the space if it was all on
>>> zfs. zfs has built in de-duplication. you could have 2000 local maven
>>> repos and probably not fill your disk since most of it has to do with
>>> duplicate jars and such.
>>>
>>> On 10/30/13 10:37 AM, "Curtis Rueden" <ct...@wisc.edu> wrote:
>>>
>>> >Hi all,
>>> >
>>> >There is plenty of room for improvement regarding reuse of Maven's
>>>local
>>> >repository cache. Releases in particular are supposed to be immutable
>>>so
>>> >once they are downloaded they could go into a read-only tier as
>>>suggested
>>> >by Stephen. Inventing such a scheme to reuse large portions of the
>>>repo
>>> >cache would be of great benefit to the Maven community.
>>> >
>>> >E.g.: the recommended CIS strategy is for every job to use its own
>>>local
>>> >repo cache, which becomes very large. My Jenkins has dozens of Maven
>>>build
>>> >jobs and I cannot afford the bloat; my Jenkins backups are huge enough
>>> >already. So what I do instead is limit my Maven Jenkins node to a
>>>single
>>> >executor, which is a real waste on a 16 core machine. Much better
>>>would be
>>> >if the jobs could share the bulk of the repo cache.
>>> >
>>> >So it's definitely an itch, but not quite itchy enough for anyone to
>>> >scratch yet...
>>> >
>>> >Regards,
>>> >Curtis
>>> > On Oct 30, 2013 8:35 AM, "Mark H. Wood" <mw...@iupui.edu> wrote:
>>> >
>>> >> On Wed, Oct 30, 2013 at 10:18:49AM +0100, Matthieu Moy wrote:
>>> >> > Barrie Treloar <ba...@gmail.com> writes:
>>> >> >
>>> >> > > On 29 October 2013 23:56, Lyons, Roy <Ro...@cmegroup.com>
>>> wrote:
>>> >> > >> Unfortunately, you will always have something in
>>> >>$HOME/.m2/repository
>>> >> > >> because that's how maven works.
>>> >> > >>
>>> >> > >> Can I suggest perhaps that you use zfs for deduplication in
>>>/home?
>>> >> > >> Otherwise, you can add something like
>>> >> > >
>>> >> > > Or give them more disk space - isn't this stuff meant to be
>>>cheap
>>> >> now-a-days?
>>> >> >
>>> >> > Local disk space is cheap. NFS-shared, RAID & backed-up disk
>>>space,
>>> >>less
>>> >> > so. I can live with a few Gb of waste, but I was just wondering
>>> >>whether
>>> >> > we could do any better.
>>> >>
>>> >> Disks are cheap. But not free. Running the procurement gantlet is
>>> >> not free. Downtime to install new storage is not free. Lord knows
>>> >> that additional backup tapes are not free, not even cheap. Longer
>>> >> backup windows are not free. Throwing storage at the problem is
>>>often
>>> >> a reasonable choice, but it's also reasonable to always ask if there
>>> >> isn't a better way.
>>> >>
>>> >> Sorry, I've been aching to write that for a long time....
>>> >>
>>> >> --
>>> >> Mark H. Wood, Lead System Programmer mwood@IUPUI.Edu
>>> >> Machines should not be friendly. Machines should be obedient.
>>> >>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
>>> For additional commands, e-mail: users-help@maven.apache.org
>>>
>>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: System-wide, read-only repository
Posted by "Lyons, Roy" <Ro...@cmegroup.com>.
I am trying to get my unix admins to use it for us. here are some
benchmarks I have seen.
On 10/30/13 10:51 AM, "Curtis Rueden" <ct...@wisc.edu> wrote:
>Hi Roy,
>
>> zfs has built in de-duplication.
>
>ZFS sounds awesome in theory but have you actually tried it? If so, how is
>it working for you? In particular, how is the performance?
>
>-Curtis
>
>
>On Wed, Oct 30, 2013 at 10:43 AM, Lyons, Roy <Ro...@cmegroup.com>
>wrote:
>
>> :) but like I said, you wouldnt worry about the space if it was all on
>> zfs. zfs has built in de-duplication. you could have 2000 local maven
>> repos and probably not fill your disk since most of it has to do with
>> duplicate jars and such.
>>
>> On 10/30/13 10:37 AM, "Curtis Rueden" <ct...@wisc.edu> wrote:
>>
>> >Hi all,
>> >
>> >There is plenty of room for improvement regarding reuse of Maven's
>>local
>> >repository cache. Releases in particular are supposed to be immutable
>>so
>> >once they are downloaded they could go into a read-only tier as
>>suggested
>> >by Stephen. Inventing such a scheme to reuse large portions of the repo
>> >cache would be of great benefit to the Maven community.
>> >
>> >E.g.: the recommended CIS strategy is for every job to use its own
>>local
>> >repo cache, which becomes very large. My Jenkins has dozens of Maven
>>build
>> >jobs and I cannot afford the bloat; my Jenkins backups are huge enough
>> >already. So what I do instead is limit my Maven Jenkins node to a
>>single
>> >executor, which is a real waste on a 16 core machine. Much better
>>would be
>> >if the jobs could share the bulk of the repo cache.
>> >
>> >So it's definitely an itch, but not quite itchy enough for anyone to
>> >scratch yet...
>> >
>> >Regards,
>> >Curtis
>> > On Oct 30, 2013 8:35 AM, "Mark H. Wood" <mw...@iupui.edu> wrote:
>> >
>> >> On Wed, Oct 30, 2013 at 10:18:49AM +0100, Matthieu Moy wrote:
>> >> > Barrie Treloar <ba...@gmail.com> writes:
>> >> >
>> >> > > On 29 October 2013 23:56, Lyons, Roy <Ro...@cmegroup.com>
>> wrote:
>> >> > >> Unfortunately, you will always have something in
>> >>$HOME/.m2/repository
>> >> > >> because that's how maven works.
>> >> > >>
>> >> > >> Can I suggest perhaps that you use zfs for deduplication in
>>/home?
>> >> > >> Otherwise, you can add something like
>> >> > >
>> >> > > Or give them more disk space - isn't this stuff meant to be cheap
>> >> now-a-days?
>> >> >
>> >> > Local disk space is cheap. NFS-shared, RAID & backed-up disk space,
>> >>less
>> >> > so. I can live with a few Gb of waste, but I was just wondering
>> >>whether
>> >> > we could do any better.
>> >>
>> >> Disks are cheap. But not free. Running the procurement gantlet is
>> >> not free. Downtime to install new storage is not free. Lord knows
>> >> that additional backup tapes are not free, not even cheap. Longer
>> >> backup windows are not free. Throwing storage at the problem is
>>often
>> >> a reasonable choice, but it's also reasonable to always ask if there
>> >> isn't a better way.
>> >>
>> >> Sorry, I've been aching to write that for a long time....
>> >>
>> >> --
>> >> Mark H. Wood, Lead System Programmer mwood@IUPUI.Edu
>> >> Machines should not be friendly. Machines should be obedient.
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
>> For additional commands, e-mail: users-help@maven.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: System-wide, read-only repository
Posted by Curtis Rueden <ct...@wisc.edu>.
Hi Roy,
> zfs has built in de-duplication.
ZFS sounds awesome in theory but have you actually tried it? If so, how is
it working for you? In particular, how is the performance?
-Curtis
On Wed, Oct 30, 2013 at 10:43 AM, Lyons, Roy <Ro...@cmegroup.com> wrote:
> :) but like I said, you wouldnt worry about the space if it was all on
> zfs. zfs has built in de-duplication. you could have 2000 local maven
> repos and probably not fill your disk since most of it has to do with
> duplicate jars and such.
>
> On 10/30/13 10:37 AM, "Curtis Rueden" <ct...@wisc.edu> wrote:
>
> >Hi all,
> >
> >There is plenty of room for improvement regarding reuse of Maven's local
> >repository cache. Releases in particular are supposed to be immutable so
> >once they are downloaded they could go into a read-only tier as suggested
> >by Stephen. Inventing such a scheme to reuse large portions of the repo
> >cache would be of great benefit to the Maven community.
> >
> >E.g.: the recommended CIS strategy is for every job to use its own local
> >repo cache, which becomes very large. My Jenkins has dozens of Maven build
> >jobs and I cannot afford the bloat; my Jenkins backups are huge enough
> >already. So what I do instead is limit my Maven Jenkins node to a single
> >executor, which is a real waste on a 16 core machine. Much better would be
> >if the jobs could share the bulk of the repo cache.
> >
> >So it's definitely an itch, but not quite itchy enough for anyone to
> >scratch yet...
> >
> >Regards,
> >Curtis
> > On Oct 30, 2013 8:35 AM, "Mark H. Wood" <mw...@iupui.edu> wrote:
> >
> >> On Wed, Oct 30, 2013 at 10:18:49AM +0100, Matthieu Moy wrote:
> >> > Barrie Treloar <ba...@gmail.com> writes:
> >> >
> >> > > On 29 October 2013 23:56, Lyons, Roy <Ro...@cmegroup.com>
> wrote:
> >> > >> Unfortunately, you will always have something in
> >>$HOME/.m2/repository
> >> > >> because that's how maven works.
> >> > >>
> >> > >> Can I suggest perhaps that you use zfs for deduplication in /home?
> >> > >> Otherwise, you can add something like
> >> > >
> >> > > Or give them more disk space - isn't this stuff meant to be cheap
> >> now-a-days?
> >> >
> >> > Local disk space is cheap. NFS-shared, RAID & backed-up disk space,
> >>less
> >> > so. I can live with a few Gb of waste, but I was just wondering
> >>whether
> >> > we could do any better.
> >>
> >> Disks are cheap. But not free. Running the procurement gantlet is
> >> not free. Downtime to install new storage is not free. Lord knows
> >> that additional backup tapes are not free, not even cheap. Longer
> >> backup windows are not free. Throwing storage at the problem is often
> >> a reasonable choice, but it's also reasonable to always ask if there
> >> isn't a better way.
> >>
> >> Sorry, I've been aching to write that for a long time....
> >>
> >> --
> >> Mark H. Wood, Lead System Programmer mwood@IUPUI.Edu
> >> Machines should not be friendly. Machines should be obedient.
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: System-wide, read-only repository
Posted by Matthieu Moy <Ma...@grenoble-inp.fr>.
"Lyons, Roy" <Ro...@cmegroup.com> writes:
> :) but like I said, you wouldnt worry about the space if it was all on
> zfs. zfs has built in de-duplication. you could have 2000 local maven
> repos and probably not fill your disk since most of it has to do with
> duplicate jars and such.
Deduplication in filesystem is nice, but does not necessarily do all the
job. If your backup system does an rsync to another machine, then you
blow up the other machine's disk. If you have disk quotas, then you have
to decide which user will be accounted for the deduped space (most
likely both users).
--
Matthieu Moy
http://www-verimag.imag.fr/~moy/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: System-wide, read-only repository
Posted by "Lyons, Roy" <Ro...@cmegroup.com>.
:) but like I said, you wouldnt worry about the space if it was all on
zfs. zfs has built in de-duplication. you could have 2000 local maven
repos and probably not fill your disk since most of it has to do with
duplicate jars and such.
On 10/30/13 10:37 AM, "Curtis Rueden" <ct...@wisc.edu> wrote:
>Hi all,
>
>There is plenty of room for improvement regarding reuse of Maven's local
>repository cache. Releases in particular are supposed to be immutable so
>once they are downloaded they could go into a read-only tier as suggested
>by Stephen. Inventing such a scheme to reuse large portions of the repo
>cache would be of great benefit to the Maven community.
>
>E.g.: the recommended CIS strategy is for every job to use its own local
>repo cache, which becomes very large. My Jenkins has dozens of Maven build
>jobs and I cannot afford the bloat; my Jenkins backups are huge enough
>already. So what I do instead is limit my Maven Jenkins node to a single
>executor, which is a real waste on a 16 core machine. Much better would be
>if the jobs could share the bulk of the repo cache.
>
>So it's definitely an itch, but not quite itchy enough for anyone to
>scratch yet...
>
>Regards,
>Curtis
> On Oct 30, 2013 8:35 AM, "Mark H. Wood" <mw...@iupui.edu> wrote:
>
>> On Wed, Oct 30, 2013 at 10:18:49AM +0100, Matthieu Moy wrote:
>> > Barrie Treloar <ba...@gmail.com> writes:
>> >
>> > > On 29 October 2013 23:56, Lyons, Roy <Ro...@cmegroup.com> wrote:
>> > >> Unfortunately, you will always have something in
>>$HOME/.m2/repository
>> > >> because that's how maven works.
>> > >>
>> > >> Can I suggest perhaps that you use zfs for deduplication in /home?
>> > >> Otherwise, you can add something like
>> > >
>> > > Or give them more disk space - isn't this stuff meant to be cheap
>> now-a-days?
>> >
>> > Local disk space is cheap. NFS-shared, RAID & backed-up disk space,
>>less
>> > so. I can live with a few Gb of waste, but I was just wondering
>>whether
>> > we could do any better.
>>
>> Disks are cheap. But not free. Running the procurement gantlet is
>> not free. Downtime to install new storage is not free. Lord knows
>> that additional backup tapes are not free, not even cheap. Longer
>> backup windows are not free. Throwing storage at the problem is often
>> a reasonable choice, but it's also reasonable to always ask if there
>> isn't a better way.
>>
>> Sorry, I've been aching to write that for a long time....
>>
>> --
>> Mark H. Wood, Lead System Programmer mwood@IUPUI.Edu
>> Machines should not be friendly. Machines should be obedient.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: System-wide, read-only repository
Posted by Curtis Rueden <ct...@wisc.edu>.
Hi all,
There is plenty of room for improvement regarding reuse of Maven's local
repository cache. Releases in particular are supposed to be immutable so
once they are downloaded they could go into a read-only tier as suggested
by Stephen. Inventing such a scheme to reuse large portions of the repo
cache would be of great benefit to the Maven community.
E.g.: the recommended CIS strategy is for every job to use its own local
repo cache, which becomes very large. My Jenkins has dozens of Maven build
jobs and I cannot afford the bloat; my Jenkins backups are huge enough
already. So what I do instead is limit my Maven Jenkins node to a single
executor, which is a real waste on a 16 core machine. Much better would be
if the jobs could share the bulk of the repo cache.
So it's definitely an itch, but not quite itchy enough for anyone to
scratch yet...
Regards,
Curtis
On Oct 30, 2013 8:35 AM, "Mark H. Wood" <mw...@iupui.edu> wrote:
> On Wed, Oct 30, 2013 at 10:18:49AM +0100, Matthieu Moy wrote:
> > Barrie Treloar <ba...@gmail.com> writes:
> >
> > > On 29 October 2013 23:56, Lyons, Roy <Ro...@cmegroup.com> wrote:
> > >> Unfortunately, you will always have something in $HOME/.m2/repository
> > >> because that's how maven works.
> > >>
> > >> Can I suggest perhaps that you use zfs for deduplication in /home?
> > >> Otherwise, you can add something like
> > >
> > > Or give them more disk space - isn't this stuff meant to be cheap
> now-a-days?
> >
> > Local disk space is cheap. NFS-shared, RAID & backed-up disk space, less
> > so. I can live with a few Gb of waste, but I was just wondering whether
> > we could do any better.
>
> Disks are cheap. But not free. Running the procurement gantlet is
> not free. Downtime to install new storage is not free. Lord knows
> that additional backup tapes are not free, not even cheap. Longer
> backup windows are not free. Throwing storage at the problem is often
> a reasonable choice, but it's also reasonable to always ask if there
> isn't a better way.
>
> Sorry, I've been aching to write that for a long time....
>
> --
> Mark H. Wood, Lead System Programmer mwood@IUPUI.Edu
> Machines should not be friendly. Machines should be obedient.
>
Re: System-wide, read-only repository
Posted by "Mark H. Wood" <mw...@IUPUI.Edu>.
On Wed, Oct 30, 2013 at 10:18:49AM +0100, Matthieu Moy wrote:
> Barrie Treloar <ba...@gmail.com> writes:
>
> > On 29 October 2013 23:56, Lyons, Roy <Ro...@cmegroup.com> wrote:
> >> Unfortunately, you will always have something in $HOME/.m2/repository
> >> because that's how maven works.
> >>
> >> Can I suggest perhaps that you use zfs for deduplication in /home?
> >> Otherwise, you can add something like
> >
> > Or give them more disk space - isn't this stuff meant to be cheap now-a-days?
>
> Local disk space is cheap. NFS-shared, RAID & backed-up disk space, less
> so. I can live with a few Gb of waste, but I was just wondering whether
> we could do any better.
Disks are cheap. But not free. Running the procurement gantlet is
not free. Downtime to install new storage is not free. Lord knows
that additional backup tapes are not free, not even cheap. Longer
backup windows are not free. Throwing storage at the problem is often
a reasonable choice, but it's also reasonable to always ask if there
isn't a better way.
Sorry, I've been aching to write that for a long time....
--
Mark H. Wood, Lead System Programmer mwood@IUPUI.Edu
Machines should not be friendly. Machines should be obedient.
Re: System-wide, read-only repository
Posted by Barrie Treloar <ba...@gmail.com>.
On 13 November 2013 17:53, Matthieu Moy <Matthieu.Moy@grenoble-
> It wouldn't be that great. We have a fast internet connection, and our
> project doesn't have _that_ many dependencies.
Hehehe.
Isn't common adage that the first use of Maven downloads the Internet
(and the kitchen sink)?
> It's the students first contact with Maven (and with tons of other
> tools), so we really need to make it easy for them (and for the
> teacher's team too, indeed, we have a bunch of things to maintain, and
> reducing the list by one is nice). Keeping the default configuration as
> much as possible is a real gain here (nothing to configure, works at
> school, works on their laptop too).
>
> I'm not having a real problem to solve here, just trying to avoid waste.
Fair enough, I can understand that.
Actually there is good reason to install a Maven Repository Manager on
laptops. It means that you can work disconnected (as long as you have
already downloaded the artifacts previously) as the offline flag (-o)
doesn't always work that great. I guess the more inquisitive students
will find enough stuff on the internet that they can do this
themselves when they face this problem.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: System-wide, read-only repository
Posted by Ziga GREGORIC <zi...@gmail.com>.
I'd go for what Wayne is suggesting. Installing Maven Repository Manager on
laptops? hmmm... It will cause artifacts to be duplicated in .m2/repository
and in Nexus/Artifactory folder structure.
There's no way (IMHO) of forcing users not to change settings.xml - but if
provided template works for most cases, not many will see the need to
change anything. Having control over repos used or artifacts allowed can be
accomplished on a project level.
My suggestion is to set-up organisation-wide repository, proxy to central,
codehous, jboss, ..., then:
1) Route everything through it with <mirror> configured, together with a)
routes defined (in case of Nexus) and b) take extra care of <mirrorOf> (see
http://maven.apache.org/guides/mini/guide-mirror-settings.html - Adv.
Mirror Spec)
2) When setting repositories override <updatePolicy> daily (default) with
never for releases repositories. (I consider re-release of an artifact bad
idea anyway).
<repository>
<id>organisation-repo</id>
<url>https://repo.organisation.com/the/right/path</url>
<releases>
<enabled>true</enabled>
<updatePolicy>never</updatePolicy>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
Ziga
On Wed, Nov 13, 2013 at 4:23 PM, Wayne Fay <wa...@gmail.com> wrote:
> > It's the students first contact with Maven (and with tons of other
> > tools), so we really need to make it easy for them (and for the
> > teacher's team too, indeed, we have a bunch of things to maintain, and
> > reducing the list by one is nice). Keeping the default configuration as
> > much as possible is a real gain here (nothing to configure, works at
> > school, works on their laptop too).
>
> You could also offer your own packaging of Maven with a customized
> settings.xml etc to make things even easier on your students. This is
> typical in sufficiently large corporate environments.
>
> Wayne
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: System-wide, read-only repository
Posted by Wayne Fay <wa...@gmail.com>.
> It's the students first contact with Maven (and with tons of other
> tools), so we really need to make it easy for them (and for the
> teacher's team too, indeed, we have a bunch of things to maintain, and
> reducing the list by one is nice). Keeping the default configuration as
> much as possible is a real gain here (nothing to configure, works at
> school, works on their laptop too).
You could also offer your own packaging of Maven with a customized
settings.xml etc to make things even easier on your students. This is
typical in sufficiently large corporate environments.
Wayne
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: System-wide, read-only repository
Posted by Matthieu Moy <Ma...@grenoble-inp.fr>.
Barrie Treloar <ba...@gmail.com> writes:
> Your wiki doesn't include a Maven Repository Manager setup.
> Are you not going to set one up?
That's not planed, no.
> Reduction in your internet bandwidth would be the biggest gain for the
> organisation.
It wouldn't be that great. We have a fast internet connection, and our
project doesn't have _that_ many dependencies.
It's the students first contact with Maven (and with tons of other
tools), so we really need to make it easy for them (and for the
teacher's team too, indeed, we have a bunch of things to maintain, and
reducing the list by one is nice). Keeping the default configuration as
much as possible is a real gain here (nothing to configure, works at
school, works on their laptop too).
I'm not having a real problem to solve here, just trying to avoid waste.
--
Matthieu Moy
http://www-verimag.imag.fr/~moy/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: System-wide, read-only repository
Posted by Barrie Treloar <ba...@gmail.com>.
On 13 November 2013 04:46, Matthieu Moy <Ma...@grenoble-inp.fr> wrote:
> Barrie Treloar <ba...@gmail.com> writes:
>
>> When you decide on a solution, a write up of what you did would be
>> helpful for the archives and anyone in the future that has the same
>> needs as you.
>
> After a bit of testing, the speed difference is almost unnoticeable.
> I've documented a way to put the local repo in /var/tmp/ (it's french,
> but if anyone cares, it's here:
> http://ensiwiki.ensimag.fr/index.php/Questions_fr%C3%A9quentes_avec_Maven_pour_le_projet_GL ).
> Most students won't read the docs, they will eat our disk space and load
> the NFS server, but that's not a very big issue anyway.
Thanks for that.
Your wiki doesn't include a Maven Repository Manager setup.
Are you not going to set one up?
Reduction in your internet bandwidth would be the biggest gain for the
organisation.
Download speedup will be the biggest gain for the students, instead of
taking 10s of minutes it should be downloaded quite quickly.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: System-wide, read-only repository
Posted by Matthieu Moy <Ma...@grenoble-inp.fr>.
Barrie Treloar <ba...@gmail.com> writes:
> When you decide on a solution, a write up of what you did would be
> helpful for the archives and anyone in the future that has the same
> needs as you.
After a bit of testing, the speed difference is almost unnoticeable.
I've documented a way to put the local repo in /var/tmp/ (it's french,
but if anyone cares, it's here:
http://ensiwiki.ensimag.fr/index.php/Questions_fr%C3%A9quentes_avec_Maven_pour_le_projet_GL ).
Most students won't read the docs, they will eat our disk space and load
the NFS server, but that's not a very big issue anyway.
--
Matthieu Moy
http://www-verimag.imag.fr/~moy/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: System-wide, read-only repository
Posted by Barrie Treloar <ba...@gmail.com>.
When you decide on a solution, a write up of what you did would be
helpful for the archives and anyone in the future that has the same
needs as you.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: System-wide, read-only repository
Posted by Matthieu Moy <Ma...@grenoble-inp.fr>.
Barrie Treloar <ba...@gmail.com> writes:
> I'd put things the student's local Maven cache in
> /tmp/maven_cache/<userid>.
That's an option. I'd put it in /var/tmp/... instead, as it is less
likely to be a small partition or RAM-disk.
>>> If they dont have direct access to the internet,
>
> Then you need to make sure there is sufficient documentation on how
> they should configure Maven, if they are not connecting to their Maven
> Repository Manager it will be too slow.
> A simple FAQ "Why is Maven slow?" should fix that. And its a natural
> penalty for those who cant follow instructions.
Yes. And having it opt-in means that students will at worse ignore it.
> If you have advanced students then they might want to attempt to patch
> maven so the local cache is thread safe. They'd need to also think
> about how to handle failed/stalled downloads correctly so it doesn't
> block everyone else using the cache. But you still have the problem
> that people working on the same artifacts will be clobbering each
> others build output. So Stephen's suggestion of a "tiered" repository
> layout would be necessary. And as Curtis points out it is something
> that would be useful, especially for Jenkins, At the very least,
> writing up some thoughts on the problem/solutions would be helpful.
I propose another project to some students, where they contribute to a
free software of their choice. I guess I'll add Maven to the list this
year, then ;-).
--
Matthieu Moy
http://www-verimag.imag.fr/~moy/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: System-wide, read-only repository
Posted by Barrie Treloar <ba...@gmail.com>.
On 30 October 2013 19:48, Matthieu Moy <Ma...@grenoble-inp.fr> wrote:
> Local disk space is cheap. NFS-shared, RAID & backed-up disk space, less
> so. I can live with a few Gb of waste, but I was just wondering whether
> we could do any better.
As others have pointed out, its not possible to do better at the moment.
The students ~/.m2 does not need to be backed-up/NFS/RAID as
everything is generated. Local disk on each machine is fine. As long
as you have a Maven Repository Manager to cache the central artifacts,
otherwise it will be too slow to re-seed the local Maven cache (i.e.
~/.m2/repository).
I'd put things the student's local Maven cache in /tmp/maven_cache/<userid>.
It doesn't need to survive reboots.
Depending on your level of trust, you will need to write scripts to
scan these directories for unauthorized files being squirreled here.
If you are doing full release cycles then you will also need somewhere
for students to deploy their released artifacts. The local Maven
cache is not where you deploy these, so its fine for the local cache
to get removed as the released artifacts will be fetched from the
deployed location.
A side benefit of local disk is that you wont get NFS latency and your
builds will continue to work when NFS is flaky.
>> If they dont have direct access to the internet,
Then you need to make sure there is sufficient documentation on how
they should configure Maven, if they are not connecting to their Maven
Repository Manager it will be too slow.
A simple FAQ "Why is Maven slow?" should fix that. And its a natural
penalty for those who cant follow instructions.
If you have advanced students then they might want to attempt to patch
maven so the local cache is thread safe. They'd need to also think
about how to handle failed/stalled downloads correctly so it doesn't
block everyone else using the cache. But you still have the problem
that people working on the same artifacts will be clobbering each
others build output. So Stephen's suggestion of a "tiered" repository
layout would be necessary. And as Curtis points out it is something
that would be useful, especially for Jenkins, At the very least,
writing up some thoughts on the problem/solutions would be helpful.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: System-wide, read-only repository
Posted by Matthieu Moy <Ma...@grenoble-inp.fr>.
Barrie Treloar <ba...@gmail.com> writes:
> On 29 October 2013 23:56, Lyons, Roy <Ro...@cmegroup.com> wrote:
>> Unfortunately, you will always have something in $HOME/.m2/repository
>> because that's how maven works.
>>
>> Can I suggest perhaps that you use zfs for deduplication in /home?
>> Otherwise, you can add something like
>
> Or give them more disk space - isn't this stuff meant to be cheap now-a-days?
Local disk space is cheap. NFS-shared, RAID & backed-up disk space, less
so. I can live with a few Gb of waste, but I was just wondering whether
we could do any better.
> If they dont have direct access to the internet,
They do.
Thanks,
--
Matthieu Moy
http://www-verimag.imag.fr/~moy/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: System-wide, read-only repository
Posted by Barrie Treloar <ba...@gmail.com>.
On 29 October 2013 23:56, Lyons, Roy <Ro...@cmegroup.com> wrote:
> Unfortunately, you will always have something in $HOME/.m2/repository
> because that's how maven works.
>
> Can I suggest perhaps that you use zfs for deduplication in /home?
> Otherwise, you can add something like
Or give them more disk space - isn't this stuff meant to be cheap now-a-days?
If they dont have direct access to the internet, then you can setup a
Maven Repository Manager, to vet/restrict what artifacts they use.
My ~/.m2/repository is 1.2GB but I would expect that to be an upper
limit considering the cruft I have accumulated in there.
Probably time to delete it and reseed from the local Maven Repository Manager,
So back of the envelope calculations say 200 * 1.2GB = 240GB.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: System-wide, read-only repository
Posted by "Lyons, Roy" <Ro...@cmegroup.com>.
Unfortunately, you will always have something in $HOME/.m2/repository
because that's how maven works.
Can I suggest perhaps that you use zfs for deduplication in /home?
Otherwise, you can add something like
-Dmaven.repo.local=/tmp/${USER}_repository (with a mkdir -p in their
profile or something to make the directory ahead of time). I believe this
can be added to MAVEN_OPTS and it would work...
Either way, with 200 students, I would recommend the zfs since there will
be LOTS of duplicated jars.
Making a single local cache is a very bad idea. there isnt any locking,
so they would clobber each other and get errors. One student would be
accessing the jar as part of compilation and the other will be
mid-overwriting the jar. its messy. Even messier if they change their
umask, and noone can read or write to the location.
So, thats my 2 bits on the topic.
On 10/29/13 4:38 AM, "Matthieu Moy" <ma...@grenoble-inp.fr> wrote:
>Hi,
>
>I have a setup with many users (200 students), each of them having a
>limited $HOME. I'm looking for a way to provide them the most common
>plugins and dependencies and save disk space.
>
>Is there a way to have a system-wide (e.g. /usr/share/maven/... or so)
>repository, where the sysadmin could download stuff that the students
>wouldn't have to download and store in their $HOME ?
>
>I found many solutions using a proxy, that would allow me to download
>from the intranet, but these solutions still require a local repository
>in $HOME/.m2/, so this saves bandwidth but not disk space. I also saw
>some people advising to set localRepository in their pom.xml, but this
>does not seem to be a good idea (not thread-safe), and doesn't really
>solve my issue, as I would like the system-wide repository to be
>read-only (managed by root only), and to come in addition to $HOME/.m2/,
>not as a replacement.
>
>Thanks in advance,
>
>--
>Matthieu Moy
>http://www-verimag.imag.fr/~moy/
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
>For additional commands, e-mail: users-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org