Re: sa-update mirroring: Coral broken

[Justin Mason <jm...@jmason.org>]

Theo Van Dinter writes:
> On Sat, Aug 12, 2006 at 02:40:43AM -0400, Daryl C. W. O'Shea wrote:
> > >For now, I removed them from the mirror list, so it's just my server left. 
> > >We
> > >should figure something out for this long term.
> > 
> > Short term, at least, I've got a server with a 1 Mbit up (8 Mbit down) 
> > static DSL connection with good connectivity that's pretty much idle if 
> > you find you need someone to share the load.
> 
> Thanks.  Right now it's not a problem, I'm more concerned with the single
> server than bandwidth usage.  It also makes it easier for me to figure out how
> many people are doing updates. ;)
> 
> On an aside, I noticed that someone aimed the 3.1 updates at the buildbot
> machine.  The current sa-update uses two different MIRRORED.BY files, one
> on the main site via minotaur, and the other from a mirror server -- the
> idea being we didn't want to always hit the main one when doing updates.
> However, this means we have 2 independent files to maintain.  Have to think
> about that a bit.

the two MIRRORED.BY files should be on the same machine, alright.
since the get uses If-Modified-Since, it should be fine bandwidth-wise.

It's fine to use the zone for hosting, too -- at least for now.

Perhaps we need to investigate using the Apache mirrors to distribute
updates?  There are a lot of issues involved there, though -- in
particular, since we generate two new (smallish) archives per day, that's
going to put a lot of load on archive.apache.org (where old files on
www.apache.org/dist are archived).   Also, the Apache mirrors are really
quite slow to update, when you're talking about the resolution required
for sa-updates.

the good news is that MIRRORED.BY means that we have a lot of flexibility
here.

--j.

Re: sa-update mirroring: Coral broken

[Theo Van Dinter <fe...@apache.org>]

On Sat, Aug 12, 2006 at 01:50:03PM +0100, Justin Mason wrote:
> the two MIRRORED.BY files should be on the same machine, alright.
> since the get uses If-Modified-Since, it should be fine bandwidth-wise.

I'm less concerned about bandwidth (though it adds up), and more about just
number of requests.  At the moment, I don't think there are any issues how we
do it.  If the full install base of SA starts doing updates, then that could
potentially impact a single server.

> It's fine to use the zone for hosting, too -- at least for now.

Well, that's debatable.  The zone machine seems to go up and down a lot
without notice.  For instance, it was down long enough overnight that I got a
ton of rsync errors in my inbox, and it cancelled our nightly mass-check.

So it could potentially get used as a non-reliable mirror, I guess.  I'm
getting to the point I'd rather just stop using it for important things.
<sigh>

> Perhaps we need to investigate using the Apache mirrors to distribute
> updates?  There are a lot of issues involved there, though -- in
> particular, since we generate two new (smallish) archives per day, that's
> going to put a lot of load on archive.apache.org (where old files on
> www.apache.org/dist are archived).   Also, the Apache mirrors are really
> quite slow to update, when you're talking about the resolution required
> for sa-updates.

Exactly, which are all reasons why we didn't go down that path. :)  For right
now this isn't a big issue for us.  As the use of updates increases and so on,
we'll want to look at expanding our mirror base.

-- 
Randomly Generated Tagline:
Above all else -- sky.