You are viewing a plain text version of this content. The canonical link for it is here.
Posted to infrastructure-dev@apache.org by Daniel Spiewak <dj...@apache.org> on 2009/06/16 18:18:29 UTC
More GitHub Sync Lag
Project: Apache Buildr. It seems that the Git syncing is working fine for
the Apache mirror, but the GitHub clone thereof is almost a dozen commits
outdated. The latest commit on the Apache mirror is
ff6519290d08b5dbf469be8284ccd1899e5873e3 (r785038). However, the latest on
GitHub is f726a65f42f8ecd4afe36569f448506183fc1d0a (r783152). Any idea why
this would be happening?
Daniel
Re: More GitHub Sync Lag
Posted by Matthieu Riou <ma...@gmail.com>.
On Wed, Jun 17, 2009 at 4:44 AM, Tony Stevenson <to...@pc-tony.com> wrote:
> On the 11th, I banned 91.121.27.60, for hitting us with over 970k hits on
> the 10th, and by the 11th there were another 600k + hits.
> Totaling 1.5m hits in just over 24 hours. That is a lot.
>
> 0.27.121.91.in-addr.arpa domain name pointer node02.antelink.net.
>
> If this is them, we need to address their hit count. Perhaps scheduling,
> or throttling connection limits.
>
Doesn't look like it's them, antelink.net looks fairly unrelated to Github.
Matthieu
>
>
> Tony
>
>
>
>
> On 17 Jun 2009, at 12:26, Jukka Zitting wrote:
>
> Hi,
>>
>> On Wed, Jun 17, 2009 at 1:13 PM, Gavin<ga...@16degrees.com.au> wrote:
>>
>>> Please also check their IP range - it may be possible they've been banned
>>> due to excessive bandwidth?
>>>
>>
>> Hmm, that might be possible, though the git pulls they do shouldn't
>> require *that* much bandwidth. The only recent bigger bandwidth use
>> would have been them pulling the new Axis1 mirror that's about 1.2GB
>> in size, but that should have happened already well before the 10th.
>>
>> PS. The issue I filed can be seen in:
>>
>> http://support.github.com/discussions/repos/921-githubcomapache-mirrors-not-updated-since-june-10th
>>
>> BR,
>>
>> Jukka Zitting
>>
>>
>
>
>
> Cheers,
> Tony
>
>
> --------------------------------------------
> Tony Stevenson
>
> tony@pc-tony.com - pctony@apache.org
> pctony@freenode.net - tony@caret.cam.ac.uk
>
> http://blog.pc-tony.com
>
> 1024D/51047D66 ECAF DC55 C608 5E82 0B5E
> 3359 C9C7 924E 5104 7D66
> --------------------------------------------
>
>
>
>
>
>
Re: More GitHub Sync Lag
Posted by Daniel Spiewak <dj...@gmail.com>.
Well, way to go, Tony! ;-)
This may seem like an obvious question, but why aren't they just going off
the commit mailing list like everyone else? That would give them almost
zero lag time on syncing, and avoid unnecessary traffic on Apache's servers.
Daniel
On Wed, Jun 17, 2009 at 6:44 AM, Tony Stevenson <to...@pc-tony.com> wrote:
> On the 11th, I banned 91.121.27.60, for hitting us with over 970k hits on
> the 10th, and by the 11th there were another 600k + hits.
> Totaling 1.5m hits in just over 24 hours. That is a lot.
>
> 0.27.121.91.in-addr.arpa domain name pointer node02.antelink.net.
>
> If this is them, we need to address their hit count. Perhaps scheduling,
> or throttling connection limits.
>
>
> Tony
>
>
>
>
> On 17 Jun 2009, at 12:26, Jukka Zitting wrote:
>
> Hi,
>>
>> On Wed, Jun 17, 2009 at 1:13 PM, Gavin<ga...@16degrees.com.au> wrote:
>>
>>> Please also check their IP range - it may be possible they've been banned
>>> due to excessive bandwidth?
>>>
>>
>> Hmm, that might be possible, though the git pulls they do shouldn't
>> require *that* much bandwidth. The only recent bigger bandwidth use
>> would have been them pulling the new Axis1 mirror that's about 1.2GB
>> in size, but that should have happened already well before the 10th.
>>
>> PS. The issue I filed can be seen in:
>>
>> http://support.github.com/discussions/repos/921-githubcomapache-mirrors-not-updated-since-june-10th
>>
>> BR,
>>
>> Jukka Zitting
>>
>>
>
>
>
> Cheers,
> Tony
>
>
> --------------------------------------------
> Tony Stevenson
>
> tony@pc-tony.com - pctony@apache.org
> pctony@freenode.net - tony@caret.cam.ac.uk
>
> http://blog.pc-tony.com
>
> 1024D/51047D66 ECAF DC55 C608 5E82 0B5E
> 3359 C9C7 924E 5104 7D66
> --------------------------------------------
>
>
>
>
>
>
Re: More GitHub Sync Lag
Posted by Tony Stevenson <to...@pc-tony.com>.
On the 11th, I banned 91.121.27.60, for hitting us with over 970k hits
on the 10th, and by the 11th there were another 600k + hits.
Totaling 1.5m hits in just over 24 hours. That is a lot.
0.27.121.91.in-addr.arpa domain name pointer node02.antelink.net.
If this is them, we need to address their hit count. Perhaps
scheduling, or throttling connection limits.
Tony
On 17 Jun 2009, at 12:26, Jukka Zitting wrote:
> Hi,
>
> On Wed, Jun 17, 2009 at 1:13 PM, Gavin<ga...@16degrees.com.au> wrote:
>> Please also check their IP range - it may be possible they've been
>> banned
>> due to excessive bandwidth?
>
> Hmm, that might be possible, though the git pulls they do shouldn't
> require *that* much bandwidth. The only recent bigger bandwidth use
> would have been them pulling the new Axis1 mirror that's about 1.2GB
> in size, but that should have happened already well before the 10th.
>
> PS. The issue I filed can be seen in:
> http://support.github.com/discussions/repos/921-githubcomapache-mirrors-not-updated-since-june-10th
>
> BR,
>
> Jukka Zitting
>
Cheers,
Tony
--------------------------------------------
Tony Stevenson
tony@pc-tony.com - pctony@apache.org
pctony@freenode.net - tony@caret.cam.ac.uk
http://blog.pc-tony.com
1024D/51047D66 ECAF DC55 C608 5E82 0B5E
3359 C9C7 924E 5104 7D66
--------------------------------------------
Re: More GitHub Sync Lag
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Thu, Jun 25, 2009 at 7:36 PM, Daniel Spiewak<dj...@gmail.com> wrote:
> Forgive my ignorance, but wouldn't the better approach to GitHub syncing be
> if your Git mirror script pushed not only to git.apache.org but also
> GitHub? As far as your script is concerned, it's just another push that
> happens per commit, and as far as GitHub is concerned, it's just another
> worry-free repository.
Yes, something like that would be pretty trivial to set up on our side.
When I discussed this arrangement with the Github guys they preferred
a pull mechanism as (AFAIUI) they have some other mirroring setups
with similar scripting. Push-based updates were mentioned as something
we might want to do once everything is running smoothly.
I'm sure the Github people have their hands full with all sorts of
stuff, so I've tried to avoid pushing too much of our needs on their
plate. I assume they do their resourcing based on user demand more
than on upstream pressure, so the more people are cloning the
github.com/apache mirrors and asking for speedier updates, the more
inclined they'll be to ask for push updates from us. :-)
BR,
Jukka Zitting
Re: More GitHub Sync Lag
Posted by Daniel Spiewak <dj...@gmail.com>.
Forgive my ignorance, but wouldn't the better approach to GitHub syncing be
if your Git mirror script pushed not only to git.apache.org but also
GitHub? As far as your script is concerned, it's just another push that
happens per commit, and as far as GitHub is concerned, it's just another
worry-free repository.
Daniel
On Thu, Jun 25, 2009 at 9:13 AM, Jukka Zitting <ju...@gmail.com>wrote:
> Hi,
>
> On Wed, Jun 17, 2009 at 1:26 PM, Jukka Zitting<ju...@gmail.com>
> wrote:
> > On Wed, Jun 17, 2009 at 1:13 PM, Gavin<ga...@16degrees.com.au> wrote:
> >> Please also check their IP range - it may be possible they've been
> banned
> >> due to excessive bandwidth?
> >
> > Hmm, that might be possible, though the git pulls they do shouldn't
> > require *that* much bandwidth. The only recent bigger bandwidth use
> > would have been them pulling the new Axis1 mirror that's about 1.2GB
> > in size, but that should have happened already well before the 10th.
>
> Confirmed, the sync lag had nothing to do with excess bandwidth usage.
> Github needed to disable the syncing for a while due to scale issues
> at their side.
>
> I'm not sure when they're able to resume pulling from us. Meanwhile
> people can simply pull from git.apache.org for the latest updates.
>
> BR,
>
> Jukka Zitting
>
Re: More GitHub Sync Lag
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Wed, Jun 17, 2009 at 1:26 PM, Jukka Zitting<ju...@gmail.com> wrote:
> On Wed, Jun 17, 2009 at 1:13 PM, Gavin<ga...@16degrees.com.au> wrote:
>> Please also check their IP range - it may be possible they've been banned
>> due to excessive bandwidth?
>
> Hmm, that might be possible, though the git pulls they do shouldn't
> require *that* much bandwidth. The only recent bigger bandwidth use
> would have been them pulling the new Axis1 mirror that's about 1.2GB
> in size, but that should have happened already well before the 10th.
Confirmed, the sync lag had nothing to do with excess bandwidth usage.
Github needed to disable the syncing for a while due to scale issues
at their side.
I'm not sure when they're able to resume pulling from us. Meanwhile
people can simply pull from git.apache.org for the latest updates.
BR,
Jukka Zitting
Re: More GitHub Sync Lag
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Wed, Jun 17, 2009 at 1:13 PM, Gavin<ga...@16degrees.com.au> wrote:
> Please also check their IP range - it may be possible they've been banned
> due to excessive bandwidth?
Hmm, that might be possible, though the git pulls they do shouldn't
require *that* much bandwidth. The only recent bigger bandwidth use
would have been them pulling the new Axis1 mirror that's about 1.2GB
in size, but that should have happened already well before the 10th.
PS. The issue I filed can be seen in:
http://support.github.com/discussions/repos/921-githubcomapache-mirrors-not-updated-since-june-10th
BR,
Jukka Zitting
Re: More GitHub Sync Lag
Posted by Paul Querna <pa...@querna.org>.
On Wed, Jun 17, 2009 at 4:13 AM, Gavin<ga...@16degrees.com.au> wrote:
>
>
>> -----Original Message-----
>> From: Jukka Zitting [mailto:jukka.zitting@gmail.com]
>> Sent: Wednesday, 17 June 2009 8:58 PM
>> To: infrastructure-dev@apache.org
>> Subject: Re: More GitHub Sync Lag
>>
>> Hi,
>>
>> On Tue, Jun 16, 2009 at 6:18 PM, Daniel Spiewak<dj...@apache.org>
>> wrote:
>> > Project: Apache Buildr. It seems that the Git syncing is working fine
>> for
>> > the Apache mirror, but the GitHub clone thereof is almost a dozen
>> commits
>> > outdated. The latest commit on the Apache mirror is
>> > ff6519290d08b5dbf469be8284ccd1899e5873e3 (r785038). However, the latest
>> on
>> > GitHub is f726a65f42f8ecd4afe36569f448506183fc1d0a (r783152). Any idea
>> why
>> > this would be happening?
>>
>> It looks like quite a few of the mirrors on Github stopped syncing on
>> June 10th. Everything should be OK on our side; I'll ask the Github
>> guys about this.
>
> Please also check their IP range - it may be possible they've been banned
> due to excessive bandwidth?
>
we haven't banned anyone for bandwidth.
we generally only firewall on eris.apache.org, the actual svn server,
and not at any router level.
RE: More GitHub Sync Lag
Posted by Gavin <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Jukka Zitting [mailto:jukka.zitting@gmail.com]
> Sent: Wednesday, 17 June 2009 8:58 PM
> To: infrastructure-dev@apache.org
> Subject: Re: More GitHub Sync Lag
>
> Hi,
>
> On Tue, Jun 16, 2009 at 6:18 PM, Daniel Spiewak<dj...@apache.org>
> wrote:
> > Project: Apache Buildr. It seems that the Git syncing is working fine
> for
> > the Apache mirror, but the GitHub clone thereof is almost a dozen
> commits
> > outdated. The latest commit on the Apache mirror is
> > ff6519290d08b5dbf469be8284ccd1899e5873e3 (r785038). However, the latest
> on
> > GitHub is f726a65f42f8ecd4afe36569f448506183fc1d0a (r783152). Any idea
> why
> > this would be happening?
>
> It looks like quite a few of the mirrors on Github stopped syncing on
> June 10th. Everything should be OK on our side; I'll ask the Github
> guys about this.
Please also check their IP range - it may be possible they've been banned
due to excessive bandwidth?
Gav...
>
> BR,
>
> Jukka Zitting
Re: More GitHub Sync Lag
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Jun 16, 2009 at 6:18 PM, Daniel Spiewak<dj...@apache.org> wrote:
> Project: Apache Buildr. It seems that the Git syncing is working fine for
> the Apache mirror, but the GitHub clone thereof is almost a dozen commits
> outdated. The latest commit on the Apache mirror is
> ff6519290d08b5dbf469be8284ccd1899e5873e3 (r785038). However, the latest on
> GitHub is f726a65f42f8ecd4afe36569f448506183fc1d0a (r783152). Any idea why
> this would be happening?
It looks like quite a few of the mirrors on Github stopped syncing on
June 10th. Everything should be OK on our side; I'll ask the Github
guys about this.
BR,
Jukka Zitting