You are viewing a plain text version of this content. The canonical link for it is here.
Posted to infrastructure-dev@apache.org by Paul Querna <pa...@querna.org> on 2009/09/04 09:55:01 UTC
How to best store /dist/?
I would like to store all of `dist` in a subversion repository.
Not the main subversion repository, but a new subversion repo dedicated to it.
Random order thoughts on it:
- This enables multi-datacenter mirroring of the root of our mirror
network by using SvnPubSub/SvnWcSub.
- Changes could be nearly instantly on all our root rsync mirrors,
rather than the extra 1-2 hour latency we have today.
- Moves root of mirrors from minotaur to eris -- this would still be
a single point of failure for releasing, but if SVN is down on eris,
you shouldn't be doing a release anyways, releases are no longer tied
to a file system with 2200+ shell acounts.
- If it grew too big, we could always reset() the whole thing once a
year with no harm.
- To make it possible to audit and watch changes in /dist/
- To also serve for /dev/dist/ for those projects that do that
currently, enabling projects like APR to hosted completely from
SvnWcSub[1]
- It would not need to be 'public' like the other repos, it is only
for use of our own internal mirroring and auditing systems.
- Normal committers would almost never need it, only release
managers, infrastructure people, and our mirroring system.
- With SVN sparse/limited depth working copies, RMs don't need to
checkout very much data at all to add/play with things.
yes/no/maybe?
Thanks,
Paul
[1] - SvnWcSub configuration for APR would be something like:
svn co https://svn.apache.org/repos/asf/apr/site/trunk/docs
/x1/www/apr.apache.org
svn co https://svn.apache.org/repos/dist/dev/apr/
/x1/www/apr.apache.org/dev/dist
With the following paths in config.json:
["/x1/www/apr.apache.org/", "/x1/www/apr.apache.org/dev/dist"]
Re: How to best store /dist/?
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
Paul Querna wrote:
> I would like to store all of `dist` in a subversion repository.
> Not the main subversion repository, but a new subversion repo dedicated to it.
E.g. svn.apache.org/repos/dist/ ? ok, one interesting side benefit...
> Random order thoughts on it:
>
> - If it grew too big, we could always reset() the whole thing once a
> year with no harm.
Speaking of 'big'... archive.apache.org/dist/... ? Since it's nothing to
svn cp the artifacts to archive and is zero copy, this seems like a win
if the size doesn't exceed svn and filesystem capacity.
> - It would not need to be 'public' like the other repos, it is only
> for use of our own internal mirroring and auditing systems.
Why not allow external mirrors to access it (perhaps by specific
arrangement)? They might appreciate the same benefits. Of course we
could continue to offer rsync, etc.
> - With SVN sparse/limited depth working copies, RMs don't need to
> checkout very much data at all to add/play with things.
For reference, at httpd and apr, the follow repositories are the /dist/...
contents, modulo the actual release artifacts.
http://svn.apache.org/repos/asf/apr/site/trunk/dist/
http://svn.apache.org/repos/asf/httpd/site/trunk/dist/
so in fact, we already essentially use this idea.
Re: How to best store /dist/?
Posted by Graham Leggett <mi...@sharp.fm>.
Paul Querna wrote:
> I would like to store all of `dist` in a subversion repository.
>
> Not the main subversion repository, but a new subversion repo dedicated to it.
+1, great idea.
Regards,
Graham
--
Re: How to best store /dist/?
Posted by Felix Meschberger <fm...@gmail.com>.
Hi,
Intriguing idea, and let me add another point:
* makes pushing releases to dist a bit easier by being able
to use the same tool as we use in our everyday work (svn)
Regards
Felix
Paul Querna schrieb:
> I would like to store all of `dist` in a subversion repository.
>
> Not the main subversion repository, but a new subversion repo dedicated to it.
>
> Random order thoughts on it:
>
> - This enables multi-datacenter mirroring of the root of our mirror
> network by using SvnPubSub/SvnWcSub.
>
> - Changes could be nearly instantly on all our root rsync mirrors,
> rather than the extra 1-2 hour latency we have today.
>
> - Moves root of mirrors from minotaur to eris -- this would still be
> a single point of failure for releasing, but if SVN is down on eris,
> you shouldn't be doing a release anyways, releases are no longer tied
> to a file system with 2200+ shell acounts.
>
> - If it grew too big, we could always reset() the whole thing once a
> year with no harm.
>
> - To make it possible to audit and watch changes in /dist/
>
> - To also serve for /dev/dist/ for those projects that do that
> currently, enabling projects like APR to hosted completely from
> SvnWcSub[1]
>
> - It would not need to be 'public' like the other repos, it is only
> for use of our own internal mirroring and auditing systems.
>
> - Normal committers would almost never need it, only release
> managers, infrastructure people, and our mirroring system.
>
> - With SVN sparse/limited depth working copies, RMs don't need to
> checkout very much data at all to add/play with things.
>
> yes/no/maybe?
>
> Thanks,
>
> Paul
>
> [1] - SvnWcSub configuration for APR would be something like:
> svn co https://svn.apache.org/repos/asf/apr/site/trunk/docs
> /x1/www/apr.apache.org
> svn co https://svn.apache.org/repos/dist/dev/apr/
> /x1/www/apr.apache.org/dev/dist
> With the following paths in config.json:
> ["/x1/www/apr.apache.org/", "/x1/www/apr.apache.org/dev/dist"]
>
Re: How to best store /dist/?
Posted by Justin Mason <jm...@jmason.org>.
a big (although non-binding) +1. great idea.
--j.
On Fri, Sep 4, 2009 at 08:55, Paul Querna<pa...@querna.org> wrote:
> I would like to store all of `dist` in a subversion repository.
>
> Not the main subversion repository, but a new subversion repo dedicated to it.
>
> Random order thoughts on it:
>
> - This enables multi-datacenter mirroring of the root of our mirror
> network by using SvnPubSub/SvnWcSub.
>
> - Changes could be nearly instantly on all our root rsync mirrors,
> rather than the extra 1-2 hour latency we have today.
>
> - Moves root of mirrors from minotaur to eris -- this would still be
> a single point of failure for releasing, but if SVN is down on eris,
> you shouldn't be doing a release anyways, releases are no longer tied
> to a file system with 2200+ shell acounts.
>
> - If it grew too big, we could always reset() the whole thing once a
> year with no harm.
>
> - To make it possible to audit and watch changes in /dist/
>
> - To also serve for /dev/dist/ for those projects that do that
> currently, enabling projects like APR to hosted completely from
> SvnWcSub[1]
>
> - It would not need to be 'public' like the other repos, it is only
> for use of our own internal mirroring and auditing systems.
>
> - Normal committers would almost never need it, only release
> managers, infrastructure people, and our mirroring system.
>
> - With SVN sparse/limited depth working copies, RMs don't need to
> checkout very much data at all to add/play with things.
>
> yes/no/maybe?
>
> Thanks,
>
> Paul
>
> [1] - SvnWcSub configuration for APR would be something like:
> svn co https://svn.apache.org/repos/asf/apr/site/trunk/docs
> /x1/www/apr.apache.org
> svn co https://svn.apache.org/repos/dist/dev/apr/
> /x1/www/apr.apache.org/dev/dist
> With the following paths in config.json:
> ["/x1/www/apr.apache.org/", "/x1/www/apr.apache.org/dev/dist"]
>
>
--
--j.
Re: How to best store /dist/?
Posted by Tony Stevenson <to...@pc-tony.com>.
On 4 Sep 2009, at 16:20, Justin Erenkrantz wrote:
> On Fri, Sep 4, 2009 at 12:55 AM, Paul Querna<pa...@querna.org> wrote:
>> I would like to store all of `dist` in a subversion repository.
>>
>> Not the main subversion repository, but a new subversion repo
>> dedicated to it.
>
> If we're going to do this (and I'm skeptical), I'd rather see us
> dedicate a new machine to hosting the dist repos rather than reusing
> eris - even if it means buying a new machine.
>
We could add it to nyx, with the new array. ;)
/me makes a hasty retreat.
Cheers,
Tony
--------------------------------------------
Tony Stevenson
tony@pc-tony.com - pctony@apache.org
pctony@freenode.net - tony@caret.cam.ac.uk
http://blog.pc-tony.com
1024D/51047D66 ECAF DC55 C608 5E82 0B5E
3359 C9C7 924E 5104 7D66
--------------------------------------------
Re: How to best store /dist/?
Posted by Norman Maurer <no...@apache.org>.
Why not use loki ?
Bye,
Norman
2009/9/4 Justin Erenkrantz <ju...@erenkrantz.com>:
> On Fri, Sep 4, 2009 at 12:55 AM, Paul Querna<pa...@querna.org> wrote:
>> I would like to store all of `dist` in a subversion repository.
>>
>> Not the main subversion repository, but a new subversion repo dedicated to it.
>
> If we're going to do this (and I'm skeptical), I'd rather see us
> dedicate a new machine to hosting the dist repos rather than reusing
> eris - even if it means buying a new machine.
>
> My $.02. -- justin
>
Re: How to best store /dist/?
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Fri, Sep 4, 2009 at 12:55 AM, Paul Querna<pa...@querna.org> wrote:
> I would like to store all of `dist` in a subversion repository.
>
> Not the main subversion repository, but a new subversion repo dedicated to it.
If we're going to do this (and I'm skeptical), I'd rather see us
dedicate a new machine to hosting the dist repos rather than reusing
eris - even if it means buying a new machine.
My $.02. -- justin
Re: How to best store /dist/?
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Fri, Sep 4, 2009 at 9:55 AM, Paul Querna<pa...@querna.org> wrote:
> I would like to store all of `dist` in a subversion repository.
Sounds good.
It would be quite useful if the "dist" repository was organized and
accessible so that release managers could also use it for staging
release candidates for review during the release vote. This way the
release could be deployed with a single svn move if the vote passes.
BR,
Jukka Zitting