You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by "Brian E. Fox" <br...@reply.infinity.nu> on 2008/11/25 16:27:08 UTC
[Public service announcement] mirrors of Central and considerate repo use
The central repo has been undergoing significant load lately, most
likely the result of people crawling and attempting to download all 70GB
of it. I'd like to point out the availability of additional mirrors that
you can use to increase your download performance and reduce the load on
central. These mirrors are updated daily right after central pulls in
all the new artifacts, so they are as fresh as central. Find the list of
mirrors here[1].
Also, for those of you with multiple developers in a single location not
using a repo manager, what are you waiting for? There are plenty to
choose from and numerous reasons to do so, particularly the ability to
isolate yourself from transitive network outages or slowdowns. You can
read more about why and the existing repo managers at [2],[3] and a
comparison grid here at [4]
Also, if you are using a repo manager, make sure to correctly configure
it to not make requests to public repositories for your internal
artifacts. Otherwise you are making a ton of requests for things that
will never be found, increasing the bandwidth usage on your side and
ours. Also it gives away potentially sensitive information as someone
could scrape the logs and figure out what you're up to internally based
on these requests. Nexus lets you configure routing rules so you could
exclude com.yourcompany.* from central. Archiva has similar
functionality, I'm not sure about Artifactory.
Please be a polite repository user. These repos and mirrors are provided
free of charge but there is a real cost behind providing this bandwidth.
Scraping the entire repository starves other users and increases the
cost of providing these repositories to the community. If the mirrors
start getting abused then we may find less geographical redundancy as a
result. There is never a reason to download the world if you have a
repository manager that is caching everything you need locally.
[1] http://docs.codehaus.org/display/MAVENUSER/Mirrors+Repositories
[2] http://maven.apache.org/repository-management.html
[3]
http://books.sonatype.com/maven-book/reference/repository-manager.html
[4]
http://docs.codehaus.org/display/MAVENUSER/Maven+Repository+Manager+Feat
ure+Matrix
Brian Fox
Apache Maven PMC
http://blogs.sonatype.com/people/brian
RE: [Public service announcement] mirrors of Central and considerate repo use
Posted by "Brian E. Fox" <br...@reply.infinity.nu>.
This is an interesting concept we'd have to persue. If you want to know,
email me directly and I can look you up and let you know what I see.
We will probably start publishing a list of abusers in a hall of shame
;-)
-----Original Message-----
From: bmathus@gmail.com [mailto:bmathus@gmail.com] On Behalf Of Baptiste
MATHUS
Sent: Tuesday, November 25, 2008 1:54 PM
To: Maven Users List
Subject: Re: [Public service announcement] mirrors of Central and
considerate repo use
2008/11/25 Brian E. Fox <br...@reply.infinity.nu>
> The central repo has been undergoing significant load lately,
<snip />
> Also, if you are using a repo manager, make sure to correctly
configure
> it to not make requests to public repositories for your internal
> artifacts. Otherwise you are making a ton of requests for things that
> will never be found, increasing the bandwidth usage on your side and
> ours. Also it gives away potentially sensitive information as someone
> could scrape the logs and figure out what you're up to internally
based
> on these requests. Nexus lets you configure routing rules so you could
> exclude com.yourcompany.* from central. Archiva has similar
> functionality, I'm not sure about Artifactory.
>
Hi Brian,
We've configured an internal maven repository manager some months ago
for
now. So, our requests should not be too high on the repo1.maven.org. But
as
we might have missed something in the configuration, wouldn't be
possible to
access logs from central in some way?
I mean, I know there would some privacy/security problems to do that
without
any restrictions. But maybe those logs could be filtered by the
requesting
(public) IP? This way, it would be possible for us to know if our hit
count
is acceptable or not according to say some typical/acceptable value you
would provide?
I know you all already have enough work with maven not to look for new
tasks
:-). Just wanting to detect my potential MRM leakages :-) (or inside
developers that would directly hit central without going through our
internal repo/proxy...).
Cheers.
--
Baptiste <Batmat> MATHUS - http://batmat.net
Save a tree,
Eat a beaver!
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: [Public service announcement] mirrors of Central and considerate repo use
Posted by Baptiste MATHUS <ml...@batmat.net>.
2008/11/25 Brian E. Fox <br...@reply.infinity.nu>
> The central repo has been undergoing significant load lately,
<snip />
> Also, if you are using a repo manager, make sure to correctly configure
> it to not make requests to public repositories for your internal
> artifacts. Otherwise you are making a ton of requests for things that
> will never be found, increasing the bandwidth usage on your side and
> ours. Also it gives away potentially sensitive information as someone
> could scrape the logs and figure out what you're up to internally based
> on these requests. Nexus lets you configure routing rules so you could
> exclude com.yourcompany.* from central. Archiva has similar
> functionality, I'm not sure about Artifactory.
>
Hi Brian,
We've configured an internal maven repository manager some months ago for
now. So, our requests should not be too high on the repo1.maven.org. But as
we might have missed something in the configuration, wouldn't be possible to
access logs from central in some way?
I mean, I know there would some privacy/security problems to do that without
any restrictions. But maybe those logs could be filtered by the requesting
(public) IP? This way, it would be possible for us to know if our hit count
is acceptable or not according to say some typical/acceptable value you
would provide?
I know you all already have enough work with maven not to look for new tasks
:-). Just wanting to detect my potential MRM leakages :-) (or inside
developers that would directly hit central without going through our
internal repo/proxy...).
Cheers.
--
Baptiste <Batmat> MATHUS - http://batmat.net
Save a tree,
Eat a beaver!
Re: [Public service announcement] mirrors of Central and considerate repo use
Posted by Wim Deblauwe <wi...@gmail.com>.
+1 for that
2008/11/29 Paul Benedict <pb...@apache.org>
> I think Maven should download and configure itself with a list of
> mirrors on first execution. Why leave this up to the users? I bet most
> will likely not care to change.
>
> Paul
>
> On Sat, Nov 29, 2008 at 8:33 AM, Wendy Smoak <ws...@gmail.com> wrote:
> > On Sat, Nov 29, 2008 at 5:16 AM, Alex Athanasopoulos
> > <al...@gmail.com> wrote:
> >
> >> One of the most annoying problems I ran into with maven was when I setup
> a
> >> central mirror containing my old local repository, deleted my local
> >> repository, and then tried to rebuild it by doing a build.
> >
> > A local repo can't be used as a remote repo... while the directory
> > structure looks the same, the metadata files are different.
> >
> > --
> > Wendy
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > For additional commands, e-mail: users-help@maven.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: [Public service announcement] mirrors of Central and considerate repo use
Posted by Paul Benedict <pb...@apache.org>.
I think Maven should download and configure itself with a list of
mirrors on first execution. Why leave this up to the users? I bet most
will likely not care to change.
Paul
On Sat, Nov 29, 2008 at 8:33 AM, Wendy Smoak <ws...@gmail.com> wrote:
> On Sat, Nov 29, 2008 at 5:16 AM, Alex Athanasopoulos
> <al...@gmail.com> wrote:
>
>> One of the most annoying problems I ran into with maven was when I setup a
>> central mirror containing my old local repository, deleted my local
>> repository, and then tried to rebuild it by doing a build.
>
> A local repo can't be used as a remote repo... while the directory
> structure looks the same, the metadata files are different.
>
> --
> Wendy
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: [Public service announcement] mirrors of Central and considerate repo use
Posted by Alex Athanasopoulos <al...@gmail.com>.
Thanks. This sheds some light to my darkness, and may explain why my
home-made maven mirror wasn't working properly.
Alex
On Sat, Nov 29, 2008 at 4:33 PM, Wendy Smoak <ws...@gmail.com> wrote:
> On Sat, Nov 29, 2008 at 5:16 AM, Alex Athanasopoulos
> <al...@gmail.com> wrote:
>
> > One of the most annoying problems I ran into with maven was when I setup
> a
> > central mirror containing my old local repository, deleted my local
> > repository, and then tried to rebuild it by doing a build.
>
> A local repo can't be used as a remote repo... while the directory
> structure looks the same, the metadata files are different.
>
> --
> Wendy
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: [Public service announcement] mirrors of Central and considerate repo use
Posted by Wendy Smoak <ws...@gmail.com>.
On Sat, Nov 29, 2008 at 5:16 AM, Alex Athanasopoulos
<al...@gmail.com> wrote:
> One of the most annoying problems I ran into with maven was when I setup a
> central mirror containing my old local repository, deleted my local
> repository, and then tried to rebuild it by doing a build.
A local repo can't be used as a remote repo... while the directory
structure looks the same, the metadata files are different.
--
Wendy
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: [Public service announcement] mirrors of Central and considerate repo use
Posted by Wendy Smoak <ws...@gmail.com>.
On Sat, Nov 29, 2008 at 6:48 AM, Torsten Werner
<ma...@googlemail.com> wrote:
> me too! ;-) And I like to have a minimal zip file: just one version of
> every core plugin and one version every direct or indirect dependency
> - not 4 versions of commons-collections, 4 versions of
> commons-logging, 3 versions of maven-plugin-plugin, etc.
You could get pretty close with the Assembly plugin's ability to build
a repository [1]. However, the problem I always run into is that,
like all Maven builds, it chooses *one* version of each dependency.
That means if you try to create a download with all the core plugins,
and the site plugin wants foo-1.0.jar, but the resources plugin wants
foo-2.0.jar, the resulting repository won't be complete. You'd have
to build a repository for each plugin, and merge them all together.
[1] http://maven.apache.org/plugins/maven-assembly-plugin/examples/single/using-repositories.html
--
Wendy
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: [Public service announcement] mirrors of Central and considerate repo use
Posted by Torsten Werner <ma...@googlemail.com>.
Hi,
On Sat, Nov 29, 2008 at 1:16 PM, Alex Athanasopoulos
<al...@gmail.com> wrote:
> I wish that I could download a zip file containing the subset of central
> containing maven's core plugins and their dependencies.
me too! ;-) And I like to have a minimal zip file: just one version of
every core plugin and one version every direct or indirect dependency
- not 4 versions of commons-collections, 4 versions of
commons-logging, 3 versions of maven-plugin-plugin, etc.
Cheers,
Torsten
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: [Public service announcement] mirrors of Central and considerate repo use
Posted by Alex Athanasopoulos <al...@gmail.com>.
This touches on my pet peeve about maven.
I wish that I could download a zip file containing the subset of central
containing maven's core plugins and their dependencies. I would then gladly
use it as my central mirror and work productively on my projects without
having to worry about central.
I also wish that Maven did not access a central online repository by
default, so I wouldn't have to go through the extra effort of preventing it
from doing so. When Maven is installed "out of the box", it could be
configured to use its own self-contained repository that came with the
installation. If I want non-core plugins or other things, then I could tell
maven to use an online repository.
In other words, I want an integration-tested self-contained maven release.
Is that a lot to ask for from a dependency management tool? I don't have
that now, so I have to do my own version managemnet of all maven plugins
that I use. When I try to use a plugin or a plugin feature that I haven't
used before, I have to go through a sometimes painful process of figuring
out which is the version of the plugin that works and is consistent with all
my existing projects and plugins. I've more than once used a plugin version
mentioned in the documentation, only to find out that it's buggy or
obsolete and I need to use a newer version (or sometimes even a completely
different groupId/artifactId).
Can maven, the tool, be separated from central, the grand repository of java
libraries? I don't want central. If I want library X for my projects, I
can go to library X's website, download X, and add it to my repository. If
library X wants to be maven-friendly, it can include a script that lets me
add all its jar files to my repository easily, so I don't have to add them
one by one manually.
Wouldn't this reduce traffic to maven central?
But maven doesn't come in a box. Instead, it lets its plugins download
whatever dependencies they want from central. It seems that it even gives
them the freedom to specify a range of versions, or any random version they
find, so that what a build does is unpredictable and dependent on the
vagaries of a moving target central repository. It's amazing that it works
most of the time, but it fails too often for me.
One of the most annoying problems I ran into with maven was when I setup a
central mirror containing my old local repository, deleted my local
repository, and then tried to rebuild it by doing a build. mvn went on to
happily copy things from my central mirror, until it couldn't resolve a
dependency. It has something to do with a "RELEASE" version of a plugin not
being found. I have aparently gotten a central artifact at some point in
time that has an integration problem with other artifacts and ends up using
an artifact version that it doesn't explicitly ask for. I believe it's
related to the site plugin, which I have since stopped trying to use. In
the end, I went back to copying my central mirror to my local repository
(something easily done with svn, which I use as my repository manager).
-Alex
On Tue, Nov 25, 2008 at 5:27 PM, Brian E. Fox <br...@reply.infinity.nu>wrote:
> The central repo has been undergoing significant load lately, most
> likely the result of people crawling and attempting to download all 70GB
> of it. I'd like to point out the availability of additional mirrors that
> you can use to increase your download performance and reduce the load on
> central. These mirrors are updated daily right after central pulls in
> all the new artifacts, so they are as fresh as central. Find the list of
> mirrors here[1].
>
>
>
> Also, for those of you with multiple developers in a single location not
> using a repo manager, what are you waiting for? There are plenty to
> choose from and numerous reasons to do so, particularly the ability to
> isolate yourself from transitive network outages or slowdowns. You can
> read more about why and the existing repo managers at [2],[3] and a
> comparison grid here at [4]
>
>
>
> Also, if you are using a repo manager, make sure to correctly configure
> it to not make requests to public repositories for your internal
> artifacts. Otherwise you are making a ton of requests for things that
> will never be found, increasing the bandwidth usage on your side and
> ours. Also it gives away potentially sensitive information as someone
> could scrape the logs and figure out what you're up to internally based
> on these requests. Nexus lets you configure routing rules so you could
> exclude com.yourcompany.* from central. Archiva has similar
> functionality, I'm not sure about Artifactory.
>
>
>
> Please be a polite repository user. These repos and mirrors are provided
> free of charge but there is a real cost behind providing this bandwidth.
> Scraping the entire repository starves other users and increases the
> cost of providing these repositories to the community. If the mirrors
> start getting abused then we may find less geographical redundancy as a
> result. There is never a reason to download the world if you have a
> repository manager that is caching everything you need locally.
>
>
>
> [1] http://docs.codehaus.org/display/MAVENUSER/Mirrors+Repositories
>
> [2] http://maven.apache.org/repository-management.html
>
> [3]
> http://books.sonatype.com/maven-book/reference/repository-manager.html
>
> [4]
> http://docs.codehaus.org/display/MAVENUSER/Maven+Repository+Manager+Feat
> ure+Matrix
>
>
>
> Brian Fox
>
> Apache Maven PMC
>
> http://blogs.sonatype.com/people/brian
>
>
>
>
>
>