You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Benson Margulies <bi...@gmail.com> on 2012/12/22 23:00:49 UTC
Getting Maven component and plugin releases to /dist -- space
https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus-releases/inventory.sizes
contains my attempt to inventory all the Maven releases that are on
repository.apache.org but are not on /dist.
The bottom line, literally, is 86221717 bytes -- not counting the
detached signature files.
Infra@, please let us know if this poses a problem; we won't actually
check anything into /dist before hearing from you (and taking care of
some organizational matters of our own).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Getting Maven component and plugin releases to /dist -- space
Posted by Mark Thomas <ma...@apache.org>.
On 22/12/2012 22:00, Benson Margulies wrote:
> https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus-releases/inventory.sizes
>
> contains my attempt to inventory all the Maven releases that are on
> repository.apache.org but are not on /dist.
>
> The bottom line, literally, is 86221717 bytes -- not counting the
> detached signature files.
>
> Infra@, please let us know if this poses a problem; we won't actually
> check anything into /dist before hearing from you (and taking care of
> some organizational matters of our own).
Only the latest version needs to go on /dist.
All versions need to be on archive.a.o
~86MB is not a significant amount. The current size of /dist is 10s of GB.
It might be simplest to copy everything to /dist, wait for it to copy to
the archives and then delete all but the latest versions from /dist.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Getting Maven component and plugin releases to /dist -- space
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Gavin McDonald wrote on Sun, Dec 23, 2012 at 08:41:22 +1030:
>
>
> > -----Original Message-----
> > From: Benson Margulies [mailto:bimargulies@gmail.com]
> > Sent: Sunday, 23 December 2012 8:31 AM
> > To: Maven Developers List
> > Cc: infrastructure@apache.org
> > Subject: Getting Maven component and plugin releases to /dist -- space
> >
> > https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus-
> > releases/inventory.sizes
> >
> > contains my attempt to inventory all the Maven releases that are on
> > repository.apache.org but are not on /dist.
> >
> > The bottom line, literally, is 86221717 bytes -- not counting the detached
> > signature files.
> >
> > Infra@, please let us know if this poses a problem; we won't actually check
> > anything into /dist before hearing from you (and taking care of some
> > organizational matters of our own).
>
> I see beta versions in there, they don’t belong in dist.
Why? If they have 3 +1's and the rest of the legal ducks in a row, they
can go on dist/. But contrarily, .0 versions that haven't passed voting
(eg, serf 0.5.0) don't belong on dist/.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Getting Maven component and plugin releases to /dist -- space
Posted by Brian Fox <br...@infinity.nu>.
On Sun, Dec 23, 2012 at 10:26 AM, Henk P. Penning <pe...@uu.nl> wrote:
> On Sun, 23 Dec 2012, Benson Margulies wrote:
>
> Date: Sun, 23 Dec 2012 15:26:18 +0100
>> From: Benson Margulies <bi...@gmail.com>
>> To: Henk P. Penning <pe...@uu.nl>
>> Cc: Maven Developers List <de...@maven.apache.org>,
>> "infrastructure@apache.org" <in...@apache.org>
>>
>> Subject: Re: Getting Maven component and plugin releases to /dist -- space
>>
>
> Hi Benson,
>
> excellent.
>
> As far as I can see, everything can go directly into
> "archive.a.o/dist/maven/".
>
> The only point of having stuff in /dist/ is that projects
> can refer users/downloaders to that stuff on a mirror.
> Since there will be no such pointers to the stuff you want
> to add, there is no point in having it on the mirrors, and
> therefore in /dist/.
>
I would completely agree with that. But it's not what the policy says. It
says it must go to dist. Do we really mean archives for everything and dist
only for things that are linked from webpages so they can be mirrored?
That's completely sensible.
Re: Getting Maven component and plugin releases to /dist -- space
Posted by "Henk P. Penning" <pe...@uu.nl>.
On Sun, 23 Dec 2012, Benson Margulies wrote:
> Date: Sun, 23 Dec 2012 15:26:18 +0100
> From: Benson Margulies <bi...@gmail.com>
> To: Henk P. Penning <pe...@uu.nl>
> Cc: Maven Developers List <de...@maven.apache.org>,
> "infrastructure@apache.org" <in...@apache.org>
> Subject: Re: Getting Maven component and plugin releases to /dist -- space
Hi Benson,
excellent.
As far as I can see, everything can go directly into
"archive.a.o/dist/maven/".
The only point of having stuff in /dist/ is that projects
can refer users/downloaders to that stuff on a mirror.
Since there will be no such pointers to the stuff you want
to add, there is no point in having it on the mirrors, and
therefore in /dist/.
Remember that /dist/ is only a cache for the 'recent' stuff
in "archive.a.o". Nothing in /dist/ is there permanently.
If you would only put 'your' stuff in archive.a.o, it would avoid
the problem of cleaning up in "/dist/maven/{plugins,components)".
Actually, I am opposed to the "automated" addition of stuff to
/dist/, if the removal or that stuff isn't "automated" by the
same mechanism too ; "cleanup" is as important as "adding stuff".
[ privately I still wonder why this stuff has to be archive.a.o'ed ;
if this stuff, why not everything else ; if not everything,
why this stuff? If these artefact are special, why doesn't
it suffice to just have a list of them somewhere?
]
HPP
> If you look in https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus-releases,
> you will find two files, inventory.archive and inventory.latest. These
> list the release archives that, as far as I can tell, make up Maven
> PMC releases that (a) were voted, and (b) were never put on /dist. The
> 'latest' file contains current releases, and the 'archive' file
> contains not-current releases. So, my intention is to push the files
> listed in the 'latest' file to /dist, and the others directly to
> archive. As for 'where' on /dist, my proposal to the Maven PMC is
> going to be to have two subdirectories of the Maven /dist: 'plugins'
> and 'components'. However, the Maven PMC is still in the early phases
> of confronting the requirement to do this at all, so it will be a few
> days, I expect, before there's a consensus. I'd be grateful if you
> have anything to add here; as you know, I fell into this project by
> discovering the hard way that it had to be done. This hasn't been my
> historical area of expertise.
>
> All these files currently reside on repository.apache.org. The Nexus
> configuration there forces each file to be accompanied by valid
> signatures when you deploy them there. The python program I've written
> (which you can also find there) just pulls metadata from Nexus. The
> next python program will actually pull files, including their
> accompanying detached signatures. Since the Maven project releases
> something about once a week, this next script will be driven by the
> Nexus metadata, not by the files. So the inventory is likely to creep
> up a bit by the time we go live.
>
> It wouldn't hurt to run a verification pass on the signatures before
> actually pushing these files live. Have you got a script that examines
> an entire directory? I suspect that you do.
>
> Regards,
>
> Benson
>
------------------------------------------------------------ _
Henk P. Penning, ICT-beta R Uithof WISK-412 _/ \_
Faculty of Science, Utrecht University T +31 30 253 4106 / \_/ \
Budapestlaan 6, 3584CD Utrecht, NL F +31 30 253 4553 \_/ \_/
http://www.staff.science.uu.nl/~penni101/ M penning@uu.nl \_/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Getting Maven component and plugin releases to /dist -- space
Posted by Benson Margulies <bi...@gmail.com>.
Henk,
I'm sorry that I'm effectively top-posting; for some reason gmail
doesn't want to set up interlinear commentary on your message. If I
miss something, please let me know.
If you look in https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus-releases,
you will find two files, inventory.archive and inventory.latest. These
list the release archives that, as far as I can tell, make up Maven
PMC releases that (a) were voted, and (b) were never put on /dist. The
'latest' file contains current releases, and the 'archive' file
contains not-current releases. So, my intention is to push the files
listed in the 'latest' file to /dist, and the others directly to
archive. As for 'where' on /dist, my proposal to the Maven PMC is
going to be to have two subdirectories of the Maven /dist: 'plugins'
and 'components'. However, the Maven PMC is still in the early phases
of confronting the requirement to do this at all, so it will be a few
days, I expect, before there's a consensus. I'd be grateful if you
have anything to add here; as you know, I fell into this project by
discovering the hard way that it had to be done. This hasn't been my
historical area of expertise.
All these files currently reside on repository.apache.org. The Nexus
configuration there forces each file to be accompanied by valid
signatures when you deploy them there. The python program I've written
(which you can also find there) just pulls metadata from Nexus. The
next python program will actually pull files, including their
accompanying detached signatures. Since the Maven project releases
something about once a week, this next script will be driven by the
Nexus metadata, not by the files. So the inventory is likely to creep
up a bit by the time we go live.
It wouldn't hurt to run a verification pass on the signatures before
actually pushing these files live. Have you got a script that examines
an entire directory? I suspect that you do.
Regards,
Benson
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Getting Maven component and plugin releases to /dist -- space
Posted by "Henk P. Penning" <pe...@uu.nl>.
On Sun, 23 Dec 2012, Benson Margulies wrote:
> Date: Sun, 23 Dec 2012 14:46:57 +0100
> From: Benson Margulies <bi...@gmail.com>
> To: "infrastructure@apache.org" <in...@apache.org>,
> Maven Developers List <de...@maven.apache.org>
> Subject: Re: Getting Maven component and plugin releases to /dist -- space
>
> As far as I can determine, there should never be / should never have
> been one of these items pushed to repository.apache.org except after a
> full release vote. If a vote wasn't passed, the staging repo should
> have been dropped, not released. As far as I am concerned, this whole
> set are full, voted, releases that should have originally gone to
> /dist (and thence to archive), and the only sorting out is between
> dist and archive.
>
> So, as I see it, the Maven PMC has now given infrastructure@ all the
> information you need about the disk space demands of this bit of
> retroactive release management. With the holiday upcoming, I'll wait
> until 28 December for further comments or objections before
> proceeding.
Hi Benson,
sorry for being dense ; ..., but do you have a list handy
showing which files you want to add to /dist/ and where?
from -> to
... -> dist/...
Either all of them (including the ones really only destined
for archive.a.o), or only the 'real', 'recent' stuff, or both.
Just curious, are the .md5's and .asc's on these to-be-added
artifacts included and checked for validity ?
Thanks a lot, regards,
HPP
> On Sat, Dec 22, 2012 at 7:43 PM, Benson Margulies <bi...@gmail.com> wrote:
>> Releases to go directly to the archive: Total size 64852152
>>
>> Releases to go to the live /dist: Total size 21369565
>>
>> The details are in additional files checked into the same directory.
>>
>>
>> On Sat, Dec 22, 2012 at 6:23 PM, Benson Margulies <bi...@gmail.com> wrote:
>>> On Sat, Dec 22, 2012 at 5:11 PM, Gavin McDonald <ga...@16degrees.com.au> wrote:
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Benson Margulies [mailto:bimargulies@gmail.com]
>>>>> Sent: Sunday, 23 December 2012 8:31 AM
>>>>> To: Maven Developers List
>>>>> Cc: infrastructure@apache.org
>>>>> Subject: Getting Maven component and plugin releases to /dist -- space
>>>>>
>>>>> https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus-
>>>>> releases/inventory.sizes
>>>>>
>>>>> contains my attempt to inventory all the Maven releases that are on
>>>>> repository.apache.org but are not on /dist.
>>>>>
>>>>> The bottom line, literally, is 86221717 bytes -- not counting the detached
>>>>> signature files.
>>>>>
>>>>> Infra@, please let us know if this poses a problem; we won't actually check
>>>>> anything into /dist before hearing from you (and taking care of some
>>>>> organizational matters of our own).
>>>>
>>>> I see beta versions in there, they don’t belong in dist.
>>>
>>> Yes they do, they were formal, voted, legal, Apache releases, that
>>> just happened to have 'beta' in their names.
>>>
>>>> I see multiple versions of the same product in there. Only the LATEST version of each
>>>> released product belongs in dist. (And certainly no need to be bothering mirrors
>>>> with outdated and since replaced versions.)
>>>
>>> The 'archive' section of the policy needs to be complied with. If you
>>> want to distinguish the disk space of the two, I can.
>>>
>>>
>>>>
>>>> You should recompile your list and total size based on removing the above.
>>>
>>> I can certainly arrange subtraction of the items that need to just to
>>> the archive.
>>>
>>>>
>>>> We'll see what the situation is after that.
>>>>
>>>> Gav...
>>>>
>
------------------------------------------------------------ _
Henk P. Penning, ICT-beta R Uithof WISK-412 _/ \_
Faculty of Science, Utrecht University T +31 30 253 4106 / \_/ \
Budapestlaan 6, 3584CD Utrecht, NL F +31 30 253 4553 \_/ \_/
http://www.staff.science.uu.nl/~penni101/ M penning@uu.nl \_/
Re: Getting Maven component and plugin releases to /dist -- space
Posted by Benson Margulies <bi...@gmail.com>.
As far as I can determine, there should never be / should never have
been one of these items pushed to repository.apache.org except after a
full release vote. If a vote wasn't passed, the staging repo should
have been dropped, not released. As far as I am concerned, this whole
set are full, voted, releases that should have originally gone to
/dist (and thence to archive), and the only sorting out is between
dist and archive.
So, as I see it, the Maven PMC has now given infrastructure@ all the
information you need about the disk space demands of this bit of
retroactive release management. With the holiday upcoming, I'll wait
until 28 December for further comments or objections before
proceeding.
On Sat, Dec 22, 2012 at 7:43 PM, Benson Margulies <bi...@gmail.com> wrote:
> Releases to go directly to the archive: Total size 64852152
>
> Releases to go to the live /dist: Total size 21369565
>
> The details are in additional files checked into the same directory.
>
>
> On Sat, Dec 22, 2012 at 6:23 PM, Benson Margulies <bi...@gmail.com> wrote:
>> On Sat, Dec 22, 2012 at 5:11 PM, Gavin McDonald <ga...@16degrees.com.au> wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Benson Margulies [mailto:bimargulies@gmail.com]
>>>> Sent: Sunday, 23 December 2012 8:31 AM
>>>> To: Maven Developers List
>>>> Cc: infrastructure@apache.org
>>>> Subject: Getting Maven component and plugin releases to /dist -- space
>>>>
>>>> https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus-
>>>> releases/inventory.sizes
>>>>
>>>> contains my attempt to inventory all the Maven releases that are on
>>>> repository.apache.org but are not on /dist.
>>>>
>>>> The bottom line, literally, is 86221717 bytes -- not counting the detached
>>>> signature files.
>>>>
>>>> Infra@, please let us know if this poses a problem; we won't actually check
>>>> anything into /dist before hearing from you (and taking care of some
>>>> organizational matters of our own).
>>>
>>> I see beta versions in there, they don’t belong in dist.
>>
>> Yes they do, they were formal, voted, legal, Apache releases, that
>> just happened to have 'beta' in their names.
>>
>>> I see multiple versions of the same product in there. Only the LATEST version of each
>>> released product belongs in dist. (And certainly no need to be bothering mirrors
>>> with outdated and since replaced versions.)
>>
>> The 'archive' section of the policy needs to be complied with. If you
>> want to distinguish the disk space of the two, I can.
>>
>>
>>>
>>> You should recompile your list and total size based on removing the above.
>>
>> I can certainly arrange subtraction of the items that need to just to
>> the archive.
>>
>>>
>>> We'll see what the situation is after that.
>>>
>>> Gav...
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Getting Maven component and plugin releases to /dist -- space
Posted by Benson Margulies <bi...@gmail.com>.
Releases to go directly to the archive: Total size 64852152
Releases to go to the live /dist: Total size 21369565
The details are in additional files checked into the same directory.
On Sat, Dec 22, 2012 at 6:23 PM, Benson Margulies <bi...@gmail.com> wrote:
> On Sat, Dec 22, 2012 at 5:11 PM, Gavin McDonald <ga...@16degrees.com.au> wrote:
>>
>>
>>> -----Original Message-----
>>> From: Benson Margulies [mailto:bimargulies@gmail.com]
>>> Sent: Sunday, 23 December 2012 8:31 AM
>>> To: Maven Developers List
>>> Cc: infrastructure@apache.org
>>> Subject: Getting Maven component and plugin releases to /dist -- space
>>>
>>> https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus-
>>> releases/inventory.sizes
>>>
>>> contains my attempt to inventory all the Maven releases that are on
>>> repository.apache.org but are not on /dist.
>>>
>>> The bottom line, literally, is 86221717 bytes -- not counting the detached
>>> signature files.
>>>
>>> Infra@, please let us know if this poses a problem; we won't actually check
>>> anything into /dist before hearing from you (and taking care of some
>>> organizational matters of our own).
>>
>> I see beta versions in there, they don’t belong in dist.
>
> Yes they do, they were formal, voted, legal, Apache releases, that
> just happened to have 'beta' in their names.
>
>> I see multiple versions of the same product in there. Only the LATEST version of each
>> released product belongs in dist. (And certainly no need to be bothering mirrors
>> with outdated and since replaced versions.)
>
> The 'archive' section of the policy needs to be complied with. If you
> want to distinguish the disk space of the two, I can.
>
>
>>
>> You should recompile your list and total size based on removing the above.
>
> I can certainly arrange subtraction of the items that need to just to
> the archive.
>
>>
>> We'll see what the situation is after that.
>>
>> Gav...
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Getting Maven component and plugin releases to /dist -- space
Posted by Benson Margulies <bi...@gmail.com>.
On Sat, Dec 22, 2012 at 5:11 PM, Gavin McDonald <ga...@16degrees.com.au> wrote:
>
>
>> -----Original Message-----
>> From: Benson Margulies [mailto:bimargulies@gmail.com]
>> Sent: Sunday, 23 December 2012 8:31 AM
>> To: Maven Developers List
>> Cc: infrastructure@apache.org
>> Subject: Getting Maven component and plugin releases to /dist -- space
>>
>> https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus-
>> releases/inventory.sizes
>>
>> contains my attempt to inventory all the Maven releases that are on
>> repository.apache.org but are not on /dist.
>>
>> The bottom line, literally, is 86221717 bytes -- not counting the detached
>> signature files.
>>
>> Infra@, please let us know if this poses a problem; we won't actually check
>> anything into /dist before hearing from you (and taking care of some
>> organizational matters of our own).
>
> I see beta versions in there, they don’t belong in dist.
Yes they do, they were formal, voted, legal, Apache releases, that
just happened to have 'beta' in their names.
> I see multiple versions of the same product in there. Only the LATEST version of each
> released product belongs in dist. (And certainly no need to be bothering mirrors
> with outdated and since replaced versions.)
The 'archive' section of the policy needs to be complied with. If you
want to distinguish the disk space of the two, I can.
>
> You should recompile your list and total size based on removing the above.
I can certainly arrange subtraction of the items that need to just to
the archive.
>
> We'll see what the situation is after that.
>
> Gav...
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
RE: Getting Maven component and plugin releases to /dist -- space
Posted by Gavin McDonald <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Benson Margulies [mailto:bimargulies@gmail.com]
> Sent: Sunday, 23 December 2012 8:31 AM
> To: Maven Developers List
> Cc: infrastructure@apache.org
> Subject: Getting Maven component and plugin releases to /dist -- space
>
> https://svn.apache.org/repos/asf/maven/project/tools/scrape-nexus-
> releases/inventory.sizes
>
> contains my attempt to inventory all the Maven releases that are on
> repository.apache.org but are not on /dist.
>
> The bottom line, literally, is 86221717 bytes -- not counting the detached
> signature files.
>
> Infra@, please let us know if this poses a problem; we won't actually check
> anything into /dist before hearing from you (and taking care of some
> organizational matters of our own).
I see beta versions in there, they don’t belong in dist.
I see multiple versions of the same product in there. Only the LATEST version of each
released product belongs in dist. (And certainly no need to be bothering mirrors
with outdated and since replaced versions.)
You should recompile your list and total size based on removing the above.
We'll see what the situation is after that.
Gav...
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org