You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Jason Pyeron <jp...@pdinc.us> on 2012/04/22 17:56:51 UTC
RE: [maven] Re: Mirroring repo1.maven.apache.org
> -----Original Message-----
> From: Brian Topping
> Sent: Sunday, April 22, 2012 11:32
>
> On Apr 22, 2012, at 10:26 AM, Jason Pyeron wrote:
>
> > 1. Is mirrors.ibiblio.org a good source for mirroring
> repo1.maven.apache.org?
> > 2. is there a strong reason not use rsync?
>
> Mirroring a repository like that is considered very bad form
> and will probably get your servers blacklisted.
This caught me off guard. Is that not the point of ibiblio.org supporting rsync?
We have a daily rsync set up with them for other projects.
>
> I don't have the benefit of other posts you may have made on
> this subject,
No other posts on this topic.
> but unless you are hosting half the known world
> of developers, why wouldn't you just use a repository?
The mirror team has bandwith management infrastructure inplace. They did not
want to modify their system to support Nexus for Nexus sake. We have a nexus
server on our dev lan already. Our dev lan does not have direct access to the
internet.
> It
> makes things so much simpler with your builds since real
> world projects generally do not pull from just central.
It was listed on the bandwidth report.
> Having a half-dozen repositories in your build is a great way
> to have it constantly slow (and even slower at times when any
> one of those repositories is offline). Caching those
> repositories through Nexus insulates you from their downtime,
> without having to soak their bandwidth for files you will never use.
I agree that the Nexus pull only when needed is nice. But there are other
concerns too. The real question is there a strong reason for not using rsync
other than use Nexus.
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
- -
- Jason Pyeron PD Inc. http://www.pdinc.us -
- Principal Consultant 10 West 24th Street #100 -
- +1 (443) 269-1555 x333 Baltimore, Maryland 21218 -
- -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: [maven] Re: Mirroring repo1.maven.apache.org
Posted by Brian Topping <to...@codehaus.org>.
Hi Jason,
I don't know every detail, but my memory of it is the details that I gave you. You can find more in the list archives. There's a lot of history on things, such as why the repository directory structure moved from flat to hierarchical (the CPU costs of generating index.html pages was large). Ibiblio was the master host for central for many years, and they finally had to push back because the bandwidth use was so high. Sonatype took over primary hosting sometime after that.
It stands to reason that the 80/20 rule applies very well here, as most projects only use one version of a given library and do not change very often. Downloading 30 other versions of it, as well as 30 versions of hundreds of other projects that aren't ever used by a site (including sources) are what caused the general prohibition on doing this. All it takes is a few sites globally rsyncing and the aggregate bandwidth required to host central goes up by orders of magnitude. It's not a good use of scarce resources to cater to a few groups that can't get their IT department to install Nexus.
I'm sure there's some community policy on when it's okay for rsyncing ibiblio, and that's probably why it's still there.
Cheers, Brian
On Apr 22, 2012, at 11:56 AM, Jason Pyeron wrote:
>> -----Original Message-----
>> From: Brian Topping
>> Sent: Sunday, April 22, 2012 11:32
>>
>> On Apr 22, 2012, at 10:26 AM, Jason Pyeron wrote:
>>
>>> 1. Is mirrors.ibiblio.org a good source for mirroring
>> repo1.maven.apache.org?
>>> 2. is there a strong reason not use rsync?
>>
>> Mirroring a repository like that is considered very bad form
>> and will probably get your servers blacklisted.
>
> This caught me off guard. Is that not the point of ibiblio.org supporting rsync?
> We have a daily rsync set up with them for other projects.
>
>>
>> I don't have the benefit of other posts you may have made on
>> this subject,
>
> No other posts on this topic.
>
>> but unless you are hosting half the known world
>> of developers, why wouldn't you just use a repository?
>
> The mirror team has bandwith management infrastructure inplace. They did not
> want to modify their system to support Nexus for Nexus sake. We have a nexus
> server on our dev lan already. Our dev lan does not have direct access to the
> internet.
>
>> It
>> makes things so much simpler with your builds since real
>> world projects generally do not pull from just central.
>
> It was listed on the bandwidth report.
>
>> Having a half-dozen repositories in your build is a great way
>> to have it constantly slow (and even slower at times when any
>> one of those repositories is offline). Caching those
>> repositories through Nexus insulates you from their downtime,
>> without having to soak their bandwidth for files you will never use.
>
> I agree that the Nexus pull only when needed is nice. But there are other
> concerns too. The real question is there a strong reason for not using rsync
> other than use Nexus.
>
>
> --
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> - -
> - Jason Pyeron PD Inc. http://www.pdinc.us -
> - Principal Consultant 10 West 24th Street #100 -
> - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 -
> - -
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> This message is copyright PD Inc, subject to license 20080407P00.
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: [maven] Re: Mirroring repo1.maven.apache.org
Posted by Stephen Connolly <st...@gmail.com>.
And in the interests of keeping ASF vendor neutral,
There are at leaste three repository managers equally capable of providing
95% of what people need (they disagree on the remaining 5% either being
needed or how to solve)
In alphabetical order, so as not to imply any preference:
Archivia by Apache
Artifactory by JFrog
Nexus by Sonatype
I have used all three in different situations, and it's swings and
roundabouts for the advanced features. I would not reject any of them out
of hand based on my experience with them.
On 24 April 2012 02:23, Andrew Hughes <ah...@gmail.com> wrote:
> You asked...
>
> I agree that the Nexus pull only when needed is nice. But there are other
> concerns too. The real question is there a strong reason for not using
> rsync
> other than use Nexus.
>
>
> Yes.
>
> 1. I use nexus so that I limit the bandwidth I share with YOU (from
> hosted repositories). No one likes a 'road hog' :)
> 2. Nexus will allow you to proxy/mirror a lot more than one repository,
> it will also allow you to place rules on repositories and additional
> configuration.
> 3. Nexus will be useful as it provides you with your own maven
> repository to deploy/release your own artifacts/projects too.
> 4. Nexus can aggregate a bunch of repositories into "one" virtual
> repository, this will make life easier for your developers as they only
> ever need 1 repository.
> 5. Nexus can provide you with a place to "upload"/"deploy" artifacts
> that are not in a public repository so that they can be shared with your
> team members.
> 6. Because its "the standard", if you want to re-invent the wheel that's
> cool, it'll do very much the same job. Alternatively there is a bunch of
> documentation out there about nexus and its application, setup, usage,
> settings.xml e.t.c.... Just in case you get hit by a bus or win the
> lottery
> next week you'll be a lot less vulnerable.
>
>
> Hope that helps :)
>
>
>
> On Mon, Apr 23, 2012 at 1:26 AM, Jason Pyeron <jp...@pdinc.us> wrote:
>
> > > -----Original Message-----
> > > From: Brian Topping
> > > Sent: Sunday, April 22, 2012 11:32
> > >
> > > On Apr 22, 2012, at 10:26 AM, Jason Pyeron wrote:
> > >
> > > > 1. Is mirrors.ibiblio.org a good source for mirroring
> > > repo1.maven.apache.org?
> > > > 2. is there a strong reason not use rsync?
> > >
> > > Mirroring a repository like that is considered very bad form
> > > and will probably get your servers blacklisted.
> >
> > This caught me off guard. Is that not the point of ibiblio.orgsupporting
> > rsync?
> > We have a daily rsync set up with them for other projects.
> >
> > >
> > > I don't have the benefit of other posts you may have made on
> > > this subject,
> >
> > No other posts on this topic.
> >
> > > but unless you are hosting half the known world
> > > of developers, why wouldn't you just use a repository?
> >
> > The mirror team has bandwith management infrastructure inplace. They did
> > not
> > want to modify their system to support Nexus for Nexus sake. We have a
> > nexus
> > server on our dev lan already. Our dev lan does not have direct access to
> > the
> > internet.
> >
> > > It
> > > makes things so much simpler with your builds since real
> > > world projects generally do not pull from just central.
> >
> > It was listed on the bandwidth report.
> >
> > > Having a half-dozen repositories in your build is a great way
> > > to have it constantly slow (and even slower at times when any
> > > one of those repositories is offline). Caching those
> > > repositories through Nexus insulates you from their downtime,
> > > without having to soak their bandwidth for files you will never use.
> >
> > I agree that the Nexus pull only when needed is nice. But there are other
> > concerns too. The real question is there a strong reason for not using
> > rsync
> > other than use Nexus.
> >
> >
> > --
> > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> > - -
> > - Jason Pyeron PD Inc. http://www.pdinc.us -
> > - Principal Consultant 10 West 24th Street #100 -
> > - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 -
> > - -
> > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> > This message is copyright PD Inc, subject to license 20080407P00.
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > For additional commands, e-mail: users-help@maven.apache.org
> >
> >
>
Re: [maven] Re: Mirroring repo1.maven.apache.org
Posted by Brian Topping <to...@codehaus.org>.
On Apr 23, 2012, at 9:23 PM, Andrew Hughes wrote:
> 2. Nexus will allow you to proxy/mirror a lot more than one repository,
> it will also allow you to place rules on repositories and additional
> configuration.
This is worth noting in a large / paranoid corporate environment from a couple of different perspectives:
a) Without rules in place, it's possible to see what artifacts are being used throughout the company just by looking at the downloaded artifacts. (The results may surprise you!)
b) With rules in place, it's easy to do things like limit any new artifacts from being used that are not approved. Developers can't be stopped from downloading artifacts from around the interwebs, but if they can't add them to the repository you control, they will break the build by changing POMs to reference them.
For the ultra-paranoid, putting the POMs under 24x7 change control keeps any changes to the repositories from being checked in. I did this on a job at a big phone company last year.
Alternatively, Maven has a unique HTTP User-Agent, and it would be easy for corporate security to configure firewalls to reject any outside access to the Maven UA except via Nexus. This would allow the POMs to remain unlocked, but any references to new repositories from the corporate LAN would be rejected (regardless of the source of the project or whether the POM was under change control). Again, it's not that you are trying to stop one person from changing their UA, but stop the majority of people from accidentally downloading malware from a rogue repo after they check out tainted source.
It's hard to know what "other concerns" the original poster had, but maybe this provides some ideas if it is about security.
HTH, B
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: [maven] Re: Mirroring repo1.maven.apache.org
Posted by Andrew Hughes <ah...@gmail.com>.
You asked...
I agree that the Nexus pull only when needed is nice. But there are other
concerns too. The real question is there a strong reason for not using rsync
other than use Nexus.
Yes.
1. I use nexus so that I limit the bandwidth I share with YOU (from
hosted repositories). No one likes a 'road hog' :)
2. Nexus will allow you to proxy/mirror a lot more than one repository,
it will also allow you to place rules on repositories and additional
configuration.
3. Nexus will be useful as it provides you with your own maven
repository to deploy/release your own artifacts/projects too.
4. Nexus can aggregate a bunch of repositories into "one" virtual
repository, this will make life easier for your developers as they only
ever need 1 repository.
5. Nexus can provide you with a place to "upload"/"deploy" artifacts
that are not in a public repository so that they can be shared with your
team members.
6. Because its "the standard", if you want to re-invent the wheel that's
cool, it'll do very much the same job. Alternatively there is a bunch of
documentation out there about nexus and its application, setup, usage,
settings.xml e.t.c.... Just in case you get hit by a bus or win the lottery
next week you'll be a lot less vulnerable.
Hope that helps :)
On Mon, Apr 23, 2012 at 1:26 AM, Jason Pyeron <jp...@pdinc.us> wrote:
> > -----Original Message-----
> > From: Brian Topping
> > Sent: Sunday, April 22, 2012 11:32
> >
> > On Apr 22, 2012, at 10:26 AM, Jason Pyeron wrote:
> >
> > > 1. Is mirrors.ibiblio.org a good source for mirroring
> > repo1.maven.apache.org?
> > > 2. is there a strong reason not use rsync?
> >
> > Mirroring a repository like that is considered very bad form
> > and will probably get your servers blacklisted.
>
> This caught me off guard. Is that not the point of ibiblio.org supporting
> rsync?
> We have a daily rsync set up with them for other projects.
>
> >
> > I don't have the benefit of other posts you may have made on
> > this subject,
>
> No other posts on this topic.
>
> > but unless you are hosting half the known world
> > of developers, why wouldn't you just use a repository?
>
> The mirror team has bandwith management infrastructure inplace. They did
> not
> want to modify their system to support Nexus for Nexus sake. We have a
> nexus
> server on our dev lan already. Our dev lan does not have direct access to
> the
> internet.
>
> > It
> > makes things so much simpler with your builds since real
> > world projects generally do not pull from just central.
>
> It was listed on the bandwidth report.
>
> > Having a half-dozen repositories in your build is a great way
> > to have it constantly slow (and even slower at times when any
> > one of those repositories is offline). Caching those
> > repositories through Nexus insulates you from their downtime,
> > without having to soak their bandwidth for files you will never use.
>
> I agree that the Nexus pull only when needed is nice. But there are other
> concerns too. The real question is there a strong reason for not using
> rsync
> other than use Nexus.
>
>
> --
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> - -
> - Jason Pyeron PD Inc. http://www.pdinc.us -
> - Principal Consultant 10 West 24th Street #100 -
> - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 -
> - -
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> This message is copyright PD Inc, subject to license 20080407P00.
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>