Re: Problems updating via rsync

["Henk P. Penning" <he...@cs.uu.nl>]

On Mon, 1 Oct 2001, jason andrade wrote:

> Date: Mon, 1 Oct 2001 07:16:32 +1000 (EST)
> From: jason andrade <ja...@dstc.edu.au>
> To: mirrors@apache.org
> Subject: Re: Problems updating via rsync
> Sender: mirrors-return-371-henkp=cs.uu.nl@apache.org
> 
> On Sun, 30 Sep 2001, Henk P. Penning wrote:
> 
> > > Ohh no, Apache guys is there attention on this problem ?
> > 
> >   -- The 'dist' archive is a big mess now: current stuff, old stuff,
> >      devel stuff; it's all there. It appears, disappears,
> >      appears again.
> > 
> >   -- there is not a byte of guidance or documentation
> > 
> >   -- the server/connection is very, very slow.
> > 
> >   Isn't there someone with an ounce of brains to organise this ?
> 
> 
> a few of us have been asking this for more than a year now - perhaps
> not quite like that.  andrew kenna has come on board recently and
> started to help - i think that the apache group (well, brian and ask
> who seem to reply more often than most on this list) just don't have
> enough time to address this.  there appear to only be around half
> a dozen `active' mirrors in terms of commenting on the list.  does
> that mean the consensus is that the majority think the current setup
> is fine and the rest of us are wasting time debating it.

  I've tried to keep my mouth shut for a long time; I agreed
  with the effort that were made to improve the situation;
  even tried to calm down those who were pushing too hard;
  but now I've had it.

  The stupid errors in de new index page did it for me;
  clearly those in charge are clueless as to the needs
  of mirror sites.
  All the mirrors have been completely useless for years;
  now someone improves the index pages and fucks up (again).

> i'd personally like to see:
> 
> 
> o apache web site which allows mirroring of *all* sub projects either
>   as apache.localmirror.foo/project/  or  project.apache.localmirror.foo.
> 
> i've given up on really wanting the former - i'll even setup the 10 or 15
> virtual servers needed for the latter now :-/.  it's more work/less
> automatic than how apache previously used to work, but if that is what
> is needed, so be it.

  Well, actually your wish is granted. The new crud is all in
  'dist/jakarta/'. So let's consider that issue decided.

  Besides, if the projects want something else, how hard would it
  be to set up one authorative template mirror site? I mean,
  set up a box, collect the right stuff in the right place and
  mirror everything from there! I would give the job to the same
  guys that run 'http://www.apache.org/' and ask them to make
  it work (look ma, no politics).

  I would consider it unacceptable if a mirror would have to set
  up 10 to 15 virtual hosts for just one project.

> o decide if apache is going to stay as a ftp and web site all in one or
>   separate the functions out.  make the ftp site a pure distribution
>   system (no html files..) and have a separate web site that can be
>   mirrored in a much more coherent fashion.

  I would prefer a top division of

    -- dist/   stuff for distribution
    -- hist/   historical stuff
    -- devel/  stuff in development, beta

  It would make mirroring so easy. If the projects do not agree,
  I would like to see a dist/hist/devel split per project;
  The mirrors should 'exclude' 'hist' and 'devel' to avoid
  problems with bad mirrors.

  I see no reason why a proper index.html file per 'dist' (sub) directory 
  would be so hard to maintain.

> o similar to other projects, have a tiered mirroring system.  have a master
>   apache site that is only contactable by tier-1 mirrors and then use those
>   to propgate out regionally.

  Works fine for CPAN (perl stuff).

>   at the moment apache is a approximately a 4G archive.  it's mirrorable
>   by a lot of sites, but keeping track of all of this looks to have become
>   a nightmare.

  As I've said before, I would't mind if only full mirrors were allowed.
  I assume that a clean 'dist/' tree would take up much less.
  Moreover, 4 GB costs nothing, and an rsync check is quick enough.
  A template mirror box wold be cheap, easy to admin and can easily
  handle 200+ rsynced mirrors a day.

  Just my thoughts. I think I like the idea of a mirror template box.
  Let apache.org try to mirror themselves; they'll find out
  soon enough how set up things properly.

> -jason

  regards.

  Henk Penning

  PS mirror status: I've been busy all day to get my mirror into sync.
  It can't be done. and it's useless because of errors in 'index.html'.

Henk P. Penning, Dept of Computer Science, Utrecht University \__/  \
Padualaan 14, P.O. Box 80.089, 3508 TB Utrecht, The Netherlands. \__/
Telephone: +31-30-2534106, fax: 2513791, NIC-handle: HPP1 _/  \__/  \
News.answers http://www.cs.uu.nl/cgi-bin/faqwais     \__/  \__/  \__/

Re: Problems updating via rsync

[jason andrade <ja...@dstc.edu.au>]

On Mon, 1 Oct 2001, Henk P. Penning wrote:

>   I've tried to keep my mouth shut for a long time; I agreed
>   with the effort that were made to improve the situation;
>   even tried to calm down those who were pushing too hard;
>   but now I've had it.

shrug. i lost it a while ago, but there's only so much effort
i am willing to put into a project now, before i give up.

>   Well, actually your wish is granted. The new crud is all in
>   'dist/jakarta/'. So let's consider that issue decided.

not really.  these are the dist files (which users tell me they
can't even find/understand on our mirror and they just go back
to apache.org.   the dist/jakarta doesn't appear to be anything
like the jakarta.apache.org web pages.  i want it consistent
as a mirror, otherwise what's the point.

>   I would consider it unacceptable if a mirror would have to set
>   up 10 to 15 virtual hosts for just one project.

i used to, but my frustation level with this means i don't really
care - i'll now do whatever to get a working full apache mirror.

obviously needing to setup a separate vhost for each apache subproject
is less than sensible, but unless there is a publish system putting
them all under one tree, i don't see how to avoid that.

>   I would prefer a top division of
> 
>     -- dist/   stuff for distribution
>     -- hist/   historical stuff
>     -- devel/  stuff in development, beta
> 
>   It would make mirroring so easy. If the projects do not agree,
>   I would like to see a dist/hist/devel split per project;
>   The mirrors should 'exclude' 'hist' and 'devel' to avoid
>   problems with bad mirrors.

there should be no "projects do not agree".  it's a template and
people work within it.  

>   Works fine for CPAN (perl stuff).

coincidentally, ask is a CPAN developer, so i assume it's the same
system in use at both cpan and apache.org.  but for some reason at
apache, picking the closest mirror usually gets me mexico or somewhere
else (i'm in australia and maintain two local apache mirrors.. go figure)

>   Just my thoughts. I think I like the idea of a mirror template box.
>   Let apache.org try to mirror themselves; they'll find out
>   soon enough how set up things properly.

that's the whole point of setting up a master site - so that every other
site is a mirror, including www.apache.org and it can be rotated through
RR or have a www1.apache.org -> wwwX.apache.org.

>   PS mirror status: I've been busy all day to get my mirror into sync.
>   It can't be done. and it's useless because of errors in 'index.html'.

i started doing large amounts of search and replace with the html files
in the apache mirror to try and point things at actually locally mirrored
references rather than back to apache.org all the time.  i've given up -
there is no way to keep up with changes.


-jason