You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mirrors@apache.org by Joshua Slive <jo...@slive.ca> on 2003/01/20 16:52:23 UTC
Apache.org mirrors: how-to-mirror update
Hello mirrors,
I have made two small changes in the how-to-mirror document that you
should take note of:
1. We are now requesting that mirrors do an update-check at least once a
day. When using rsync this should not result in any substantial increase
in bandwidth usage, but it should allow us to more effectively use our
mirrors in distributing recent releases.
2. I have added a hint on how to deal with symlinks. Our distribution
directories make extensive use of symbolic links. Therefore it is highly
recommended that you use the "-l" option to rsync and have an Options line
that includes SymLinksIfOwnerMatch (or FollowSymLinks). If, for some
reason, you are not able to allow symlinks on your mirror, then please
change your rsync options to use "-L" instead of "-l". This will create a
copy of the linked file in place of the link.
Recent tests show that the majority of mirrors are already doing both of
these things, but it would be great if you could each make a quick check
to ensure you are included.
The full document is available at
http://www.apache.org/info/how-to-mirror.html
Thanks.
Joshua.
Re: Apache.org mirrors: how-to-mirror update
Posted by jason andrade <ja...@rtfmconsult.com>.
On Thu, 30 Jan 2003, Pier Fumagalli wrote:
> Yes... It is indeed dead and buried: look at http://java.apache.org/ and
> you'll see that all projects have been moved...
nod.. ok, so basically it's all moved into jakarta..
> > and i was/am going to ask about the nature of the other projects
> > (xml/perl/apr etc..). i've decided to ignore jakarta for now
> > since the 8G of space it seems to want is simply.. not sensible
> > compared to the rest of the apache project :-)
>
> In my book it's slightly over 4Gb.. But then I only mirror software
> distributions...
i was going by what the rsync motd said.
> http://nagoya.apache.org/dist/
this is sort out of sequence (since i think joshua and the others are
working on it already (?)) but are there plans to move the jakarta
dist stuff into the current dist tree ?
at the moment the jakarta directory is basically empty and there is
a .htaccess backup file which points redirects everything back to
www.apache.org
frankly i'm confused - i thought it would be relatively easy to migrate
all the download side of things into the dist tree that is exported to
mirrors (since they all seem to have a /dist in their respective web
trees already)
regards,
-jason
Re: Apache.org mirrors: how-to-mirror update
Posted by Pier Fumagalli <pi...@apache.org>.
On 29/1/03 8:07, "jason andrade" <ja...@rtfmconsult.com> wrote:
> On Wed, 29 Jan 2003, Pier Fumagalli wrote:
>
>> Hoy hoy hoy... JServ is still "my" baby... We got ONE download right
>> yesterday (one in the last 3 months, I believe) ! :-) :-)
>>
>> I fixed the perms...
>
> thanks pier.
>
>> Pier (yeah, mirroring Java.Apache.ORG is kinda weird!)
>
> is it actually a dead project ? i can't figure out otherwise why
> it can't be added to the list of mirrorable things..
Yes... It is indeed dead and buried: look at http://java.apache.org/ and
you'll see that all projects have been moved...
> and i was/am going to ask about the nature of the other projects
> (xml/perl/apr etc..). i've decided to ignore jakarta for now
> since the 8G of space it seems to want is simply.. not sensible
> compared to the rest of the apache project :-)
In my book it's slightly over 4Gb.. But then I only mirror software
distributions...
http://nagoya.apache.org/dist/
Pier
--
"Jakarta Tomcat is [...] it is produced by the Apache organisation under the
GNU public licence [...]" Emma Newby - Techincal Lead - SkyRock UK
Re: Apache.org mirrors: how-to-mirror update
Posted by jason andrade <ja...@rtfmconsult.com>.
On Wed, 29 Jan 2003, Pier Fumagalli wrote:
> Hoy hoy hoy... JServ is still "my" baby... We got ONE download right
> yesterday (one in the last 3 months, I believe) ! :-) :-)
>
> I fixed the perms...
thanks pier.
> Pier (yeah, mirroring Java.Apache.ORG is kinda weird!)
is it actually a dead project ? i can't figure out otherwise why
it can't be added to the list of mirrorable things..
and i was/am going to ask about the nature of the other projects
(xml/perl/apr etc..). i've decided to ignore jakarta for now
since the 8G of space it seems to want is simply.. not sensible
compared to the rest of the apache project :-)
regards,
-jason
Re: Apache.org mirrors: how-to-mirror update
Posted by Pier Fumagalli <pi...@betaversion.org>.
Joshua Slive wrote:
> On Wed, 29 Jan 2003, jason andrade wrote:
>
>>Any chance of getting these permissions fixed ?
>>
>>I am just following up on this from a few weeks ago.
>>
>>send_files failed to open //faq/maint: Permission denied
>>send_files failed to open //mail/icalendar-cvs/200103.gz: Permission denied
>>send_files failed to open //mail/jserv-bugdb/200108: Permission denied
>
>
> Ummm... OK. I'll forward this over to the infrastructure group. But why
> on earth are you mirroring java.apache.org? This is a WAY out of date
> project. It does little good to mirror it.
Hoy hoy hoy... JServ is still "my" baby... We got ONE download right
yesterday (one in the last 3 months, I believe) ! :-) :-)
I fixed the perms...
Pier (yeah, mirroring Java.Apache.ORG is kinda weird!)
Re: Apache.org mirrors: how-to-mirror update
Posted by jason andrade <ja...@rtfmconsult.com>.
On Tue, 28 Jan 2003, Joshua Slive wrote:
> OK. That's perfectly valid, and is the reason we make those available via
> rsync. Just wanted to make sure you know what you are doing.
i think i do, but it certainly doesn't hurt to get a second opinion (such
as from yourself :-) as things evolve and what originally made sense might
not anymore..
any chance of getting an update on how the mirroring side of apache is
going in the interim - and what the "plan" is for the direction we're
moving ?
from the feedback we get, our apache mirror users want to see:
o a replicated "apache.org" site which lets them browse things like documentation
o local downloads of apache source/binaries - the ability to "stay within" a mirror
once you have picked it.
one of the things that annoyed them was when they went to our mirror and found
themselves following a link that went back to apache.org and then it never
linked back. we've tried to address that but it's more of a structural issue
than anything we do with rewrites..
regards,
-jason
Re: Apache.org mirrors: how-to-mirror update
Posted by Joshua Slive <jo...@slive.ca>.
On Wed, 29 Jan 2003, jason andrade wrote:
> we have been mirroring some apache projects (even though only dist is
> pointed at us as a mirror) because we also do some of our own search
> stuff internally that people use and because that's also the only
> way we had in the past to get things like the apache documentation.
OK. That's perfectly valid, and is the reason we make those available via
rsync. Just wanted to make sure you know what you are doing.
Joshua.
Re: Apache.org mirrors: how-to-mirror update
Posted by jason andrade <ja...@rtfmconsult.com>.
On Tue, 28 Jan 2003, Joshua Slive wrote:
>
> On Wed, 29 Jan 2003, jason andrade wrote:
> > Any chance of getting these permissions fixed ?
> >
> > I am just following up on this from a few weeks ago.
> >
> > send_files failed to open //faq/maint: Permission denied
> > send_files failed to open //mail/icalendar-cvs/200103.gz: Permission denied
> > send_files failed to open //mail/jserv-bugdb/200108: Permission denied
>
> Ummm... OK. I'll forward this over to the infrastructure group. But why
> on earth are you mirroring java.apache.org? This is a WAY out of date
> project. It does little good to mirror it.
historial reasons. i will probably can it now.
we have been mirroring some apache projects (even though only dist is
pointed at us as a mirror) because we also do some of our own search
stuff internally that people use and because that's also the only
way we had in the past to get things like the apache documentation.
a lot of our users were quite pissed when they lost the local documentation
as part of a mirror so we've tried to accomodate them by mirroring parts
of the apache site they asked for..
i'm doing some cleanup now..
regards,
-jason
Re: Apache.org mirrors: how-to-mirror update
Posted by Joshua Slive <jo...@slive.ca>.
On Wed, 29 Jan 2003, jason andrade wrote:
> Any chance of getting these permissions fixed ?
>
> I am just following up on this from a few weeks ago.
>
> send_files failed to open //faq/maint: Permission denied
> send_files failed to open //mail/icalendar-cvs/200103.gz: Permission denied
> send_files failed to open //mail/jserv-bugdb/200108: Permission denied
Ummm... OK. I'll forward this over to the infrastructure group. But why
on earth are you mirroring java.apache.org? This is a WAY out of date
project. It does little good to mirror it.
Joshua.
Re: Apache.org mirrors: how-to-mirror update
Posted by jason andrade <ja...@rtfmconsult.com>.
On Mon, 20 Jan 2003, Joshua Slive wrote:
Hi Joshua,
Any chance of getting these permissions fixed ?
I am just following up on this from a few weeks ago.
send_files failed to open //faq/maint: Permission denied
send_files failed to open //mail/icalendar-cvs/200103.gz: Permission denied
send_files failed to open //mail/jserv-bugdb/200108: Permission denied
i believe it's from the java side of things.
regards,
-jason
Re: Apache.org mirrors: how-to-mirror update
Posted by jason andrade <ja...@rtfmconsult.com>.
On Sat, 22 Feb 2003, Joshua Slive wrote:
> We can check the server, but we really have no control over the clients.
> We consider the rsync repository to be a public resource, so it is
> difficult to control client access.
which is fair enough but i wasn't suggesting enforcing this - i was
just asking to get a message sent out to mirrors to please upgrade
and make it something for new mirrors to check.
also to add something to the motd (please make sure you are using
the latest version of rsync 2.5.5 blah..)
> I don't think increasing the limit will do any good. When a screwed up
> client comes along, it will eat up any number of available connections.
> Increasing the connections would just increase the damage that the bad
> client does to the server.
i figured that - sometimes when you have more slots (and a lower timeout)
it helps alleviate the pain.
> Having other rsync locations is certainly possible. The only reason I
> haven't personally pushed this is that it increases the complexity of the
> mirroring system, and I like simple things. It would also mean denying
> general rsync access to www.apache.org. But this may be the way to go.
i would limit rsync access to www.apache.org to mirrors only - but i was
more suggesting that you have two sets of mirror shares - one which
any mirror can access (and has a certain user limit) and another which
only some mirrors can access (host acl list).
regards,
-jason
Re: Apache.org mirrors: how-to-mirror update
Posted by Joshua Slive <jo...@slive.ca>.
On Sun, 23 Feb 2003, jason andrade wrote:
> get all the people using rsync to update to the latest version, e.g rsync
> 2.5.5 and make sure that is also running on the server.
We can check the server, but we really have no control over the clients.
We consider the rsync repository to be a public resource, so it is
difficult to control client access.
> also consider lowering the timeout, perhaps to 600
I'll pass that suggestion on to the admin.
> lastly, please consider making two different sharesets available - one
> for "general" mirrors and the other for 'tier1' mirrors who can also
> onfeed to others in their region.
>
> lastly, there are a huge number of apache mirrors - much more than
> most other projects that i know of so the default limit might be
> quite low if all of these are syncing from a single site (the apache
> master site)
I don't think increasing the limit will do any good. When a screwed up
client comes along, it will eat up any number of available connections.
Increasing the connections would just increase the damage that the bad
client does to the server.
Having other rsync locations is certainly possible. The only reason I
haven't personally pushed this is that it increases the complexity of the
mirroring system, and I like simple things. It would also mean denying
general rsync access to www.apache.org. But this may be the way to go.
Thanks for your ideas.
Joshua.
Re: Apache.org mirrors: how-to-mirror update
Posted by jason andrade <ja...@rtfmconsult.com>.
On Sat, 22 Feb 2003, Joshua Slive wrote:
> We cleared this earlier in the day and blocked the offending party, then
> another one came in and blocked everything up. It's getting pretty
> annoying.
it is indeed.
> If anyone has any ideas on what is causing this or how to avoid it, they
> would be welcome.
get all the people using rsync to update to the latest version, e.g rsync
2.5.5 and make sure that is also running on the server.
also consider lowering the timeout, perhaps to 600
lastly, please consider making two different sharesets available - one
for "general" mirrors and the other for 'tier1' mirrors who can also
onfeed to others in their region.
lastly, there are a huge number of apache mirrors - much more than
most other projects that i know of so the default limit might be
quite low if all of these are syncing from a single site (the apache
master site)
regards,
-jason
Re: Apache.org mirrors: how-to-mirror update
Posted by Joshua Slive <jo...@slive.ca>.
On Sun, 23 Feb 2003, Yusmar Yahaya wrote:
> i think it probably relates back to apache.org's "how to mirror apache"
> guide page, where it states that you have to pull the repository between
> 8am GMT till noon GMT.
>
> from the error messages i've got lately, it was during that time frame.
> perhaps if each region were given a time period to rsync, then you can
> roughly predict the traffic size. e.g. each region should rsync between
> 8am-noon at their own timezone, not 8am-noon GMT.
No, but perhaps I haven't been clear about the problem. This isn't a
question of www.apache.org being over capacity. This is a question of a
single client coming in and making multiple rsync connections until it
fills up all sockets. I have no idea why the clients are doing this, but
it probably is some very badly programmed script. The weird thing is that
is happening over and over again from multiple clients all over the world.
Joshua.
Re: Mirror not added?
Posted by Joshua Slive <jo...@slive.ca>.
On Sun, 23 Feb 2003, Neo Wee Teck wrote:
> Hi
>
> I haven seen my mirror in the official list. Please check :)
>
> http://apache.en.com.sg
>
> Singapore Mirror
>
> Maintained by : Neo Wee Teck
> Update : 6hrs interval daily
>
> Email : slash@en.com.sg
Got it. Thanks.
Joshua.
Re: I can't update
Posted by Joshua Slive <jo...@slive.ca>.
On Thu, 27 Feb 2003, Ivan Canizares wrote:
> > Hi
> >
> > I can;t update the packages from apache.org, with this command
> >
> > rsync -rtlzv --delete --exclude incoming www.apache.org::apache-dist
> > /ftp/pub/apache/dist
> >
> > the crontab say this:
> > produced the following output:
> >
> > rsync: failed to connect to www.apache.org: Connection timed out
> > rsync error: error in socket IO (code 10) at clientserver.c(83)
I'm not sure how I can help you here. Other mirror do not seem to be
having this problem. Could it be something on your local network?
Joshua.
Re: I can't update
Posted by Joshua Slive <jo...@slive.ca>.
This is getting irritating. You've sent this message at least three
times, and received at least two replies. What's going on?
On Thu, 27 Feb 2003, Ivan Canizares wrote:
> > Hi
> >
> > I can;t update the packages from apache.org, with this command
> >
> > rsync -rtlzv --delete --exclude incoming www.apache.org::apache-dist
> > /ftp/pub/apache/dist
> >
> > the crontab say this:
> > produced the following output:
> >
> > rsync: failed to connect to www.apache.org: Connection timed out
> > rsync error: error in socket IO (code 10) at clientserver.c(83)
> >
> > Can you help me
> > Bye
> > Ivan Canizares
> >
>
>
Re: I can't update
Posted by Ivan Canizares <ti...@ula.ve>.
> Hi
>
> I can;t update the packages from apache.org, with this command
>
> rsync -rtlzv --delete --exclude incoming www.apache.org::apache-dist
> /ftp/pub/apache/dist
>
> the crontab say this:
> produced the following output:
>
> rsync: failed to connect to www.apache.org: Connection timed out
> rsync error: error in socket IO (code 10) at clientserver.c(83)
>
> Can you help me
> Bye
> Ivan Canizares
>
Re: I can't update
Posted by jason andrade <ja...@rtfmconsult.com>.
On Tue, 25 Feb 2003, Ivan Canizares wrote:
> Hi
>
> I can;t update the packages from apache.org, with this command
>
> rsync -rtlzv --delete --exclude incoming www.apache.org::apache-dist
> /ftp/pub/apache/dist
>
> the crontab say this:
> produced the following output:
>
> failed to connect to www.apache.org - Connection timed out
for a start, turn off compression. instead try
rsync -rltvH --stats --delete --max-delete=1000 --exclude="incoming/" www.apache.org::apache-dist/ /ftp/pub/apache/dist/
i would also recommend you pick a closer mirror if possible, to sync from
other than www.apache.org (but perhaps there are no other mirrors in
.ve)
regards,
-jason
I can't update
Posted by Ivan Canizares <ti...@ula.ve>.
Hi
I can;t update the packages from apache.org, with this command
rsync -rtlzv --delete --exclude incoming www.apache.org::apache-dist
/ftp/pub/apache/dist
the crontab say this:
produced the following output:
failed to connect to www.apache.org - Connection timed out
Can you help me
Bye
Ivan Canizares
Re: Mirror not added?
Posted by Joshua Slive <jo...@slive.ca>.
On Tue, 25 Feb 2003, Yusmar Yahaya wrote:
> loc : Kuala Lumpur, Malaysia (+8GMT)
>
> URL : http://mirror.leafbug.org/apache-dist
> (rsync www.apache.org::apache-dist)
>
> freq. : twice daily at 8am GMT and 8pm GMT (4:30am MYT & 4:30pm MYT)
>
> link : relocated from 1.5bps SDSL to 2mbps dedicated leasedline
> yesterday. UPS backed, mirrored storage disk.
>
> svc date : since 14 Feb 2003.
Got it. Thanks.
Joshua.
Re: Mirror not added?
Posted by Yusmar Yahaya <yu...@boolean.pc.my>.
On Mon, 24 Feb 2003, Joshua Slive wrote:
>
> On Sun, 23 Feb 2003, Neo Wee Teck wrote:
> > Hi
> > I haven seen my mirror in the official list. Please check :)
> > http://apache.en.com.sg
>
> If nobody beats me to it, I'll try to get to this soon.
>
And when you're updating the list, I hope you could please update mine
into that as well? thanks, as my server is not yet updated.
BTW, here's the info again.
loc : Kuala Lumpur, Malaysia (+8GMT)
URL : http://mirror.leafbug.org/apache-dist
(rsync www.apache.org::apache-dist)
freq. : twice daily at 8am GMT and 8pm GMT (4:30am MYT & 4:30pm MYT)
link : relocated from 1.5bps SDSL to 2mbps dedicated leasedline
yesterday. UPS backed, mirrored storage disk.
svc date : since 14 Feb 2003.
Thank you.
-- _______________________
~Yusmar Yahaya yus@boolean.pc.my
http://leafbug.org/~yus
Re: Mirror not added?
Posted by Joshua Slive <jo...@slive.ca>.
On Sun, 23 Feb 2003, Neo Wee Teck wrote:
> Hi
>
> I haven seen my mirror in the official list. Please check :)
>
> http://apache.en.com.sg
>
If nobody beats me to it, I'll try to get to this soon.
Joshua.
Mirror not added?
Posted by Neo Wee Teck <sl...@en.com.sg>.
Hi
I haven seen my mirror in the official list. Please check :)
http://apache.en.com.sg
Singapore Mirror
Maintained by : Neo Wee Teck
Update : 6hrs interval daily
Email : slash@en.com.sg
Re: Apache.org mirrors: how-to-mirror update
Posted by Yusmar Yahaya <yu...@boolean.pc.my>.
On Sat, 22 Feb 2003, Joshua Slive wrote:
> We cleared this earlier in the day and blocked the offending party, then
> another one came in and blocked everything up. It's getting pretty
> annoying.
>
> If anyone has any ideas on what is causing this or how to avoid it, they
> would be welcome.
>
i think it probably relates back to apache.org's "how to mirror apache"
guide page, where it states that you have to pull the repository between
8am GMT till noon GMT.
from the error messages i've got lately, it was during that time frame.
perhaps if each region were given a time period to rsync, then you can
roughly predict the traffic size. e.g. each region should rsync between
8am-noon at their own timezone, not 8am-noon GMT.
my 2cents.
-- _______________________
~Yusmar Yahaya yus@boolean.pc.my
http://leafbug.org/~yus
Re: Apache.org mirrors: how-to-mirror update
Posted by Joshua Slive <jo...@slive.ca>.
We cleared this earlier in the day and blocked the offending party, then
another one came in and blocked everything up. It's getting pretty
annoying.
If anyone has any ideas on what is causing this or how to avoid it, they
would be welcome.
Anyway, I passed this on to the admins and they will deal with it.
Joshua.
On Sat, 22 Feb 2003, Haesu wrote:
> I am noticing the same issue right now...
>
> We are unable to retrieve the latest updates for apache projects.. We are
> going stale.. :-/ I am getting connection full from the rsync server..
>
> Best regards,
> -hc
>
>
> On Sat, 22 Feb 2003, jason andrade wrote:
>
> > On Mon, 20 Jan 2003, Joshua Slive wrote:
> >
> > Hi Joshua,
> >
> > It looks like the apache master mirror is full again and needs to
> > be kicked (and/or people idling on it need to be kicked..)
> >
> > @ERROR: max connections (25) reached - try again later
> > rsync: connection unexpectedly closed (68 bytes read so far)
> > rsync error: error in rsync protocol data stream (code 12) at io.c(151)
> >
> >
> > Since we can't connect to update we are obviously going to go stale :-/
> >
> > regards,
> >
> > -jason
> >
>
Re: Apache.org mirrors: how-to-mirror update
Posted by Haesu <ha...@towardex.com>.
I am noticing the same issue right now...
We are unable to retrieve the latest updates for apache projects.. We are
going stale.. :-/ I am getting connection full from the rsync server..
Best regards,
-hc
On Sat, 22 Feb 2003, jason andrade wrote:
> On Mon, 20 Jan 2003, Joshua Slive wrote:
>
> Hi Joshua,
>
> It looks like the apache master mirror is full again and needs to
> be kicked (and/or people idling on it need to be kicked..)
>
> @ERROR: max connections (25) reached - try again later
> rsync: connection unexpectedly closed (68 bytes read so far)
> rsync error: error in rsync protocol data stream (code 12) at io.c(151)
>
>
> Since we can't connect to update we are obviously going to go stale :-/
>
> regards,
>
> -jason
>
Re: Apache.org mirrors: how-to-mirror update
Posted by jason andrade <ja...@rtfmconsult.com>.
On Mon, 20 Jan 2003, Joshua Slive wrote:
Hi Joshua,
It looks like the apache master mirror is full again and needs to
be kicked (and/or people idling on it need to be kicked..)
@ERROR: max connections (25) reached - try again later
rsync: connection unexpectedly closed (68 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(151)
Since we can't connect to update we are obviously going to go stale :-/
regards,
-jason