You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@netbeans.apache.org by Bertrand Delacretaz <bd...@apache.org> on 2019/05/14 13:55:22 UTC
Users banned from apache.org due to downloading updates.xml.gz
Hi,
I think something similar happened a few weeks ago already, see here:
https://lists.apache.org/thread.html/97f0619de82a630b6ec7d56f1ac5ab34dd85939ae1ff8dcf01434ee9@%3Cusers.infra.apache.org%3E
NetBeans users going through a company router have been banned from
*.apache.org due to too many requests in a short amount of time to
this URL:
http://www.apache.org/dyn/closer.lua?action=download&filename=incubator/netbeans/incubating-netbeans/incubating-11.0/nbms/updates.xml.gz
It might be worth discussing with infra how to avoid this - my
understanding (to be verified) is that it's the amount of requests to
closer.lua that's problematic, even if the download themselves go to
download mirrors.
-Bertrand
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Neil C Smith <ne...@apache.org>.
On Fri, 17 May 2019 at 21:04, Antonio <an...@vieiro.net> wrote:
> Downloading updates.xml.gz from n.a.o and the rest of NBMs through n.a.o
> > netbeans-vm.a.o > closer.lua should be easy.
I meant the bit where you mentioned getting the mirror for the first
module and then downloading the rest directly from that mirror,
bypassing closer.lua
updates.xml.gz can be on netbeans-vm.a.o
> That's the question! How well are we doing with statistics? Would it be
> possible/better for the IDE to send periodic reports (including
> OS/modules/etc.) to netbeans-vm.a.o for later statistics analysis? Or is
> this banned by the ASF?
No idea, but we could still track catalog access. But given people
can currently only download a full IDE, and we're not making much use
of updates, at least on the IDE side there doesn't seem much to track
via NBMs?
Best wishes,
Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Antonio <an...@vieiro.net>.
Hi,
El 17/05/2019 a las 21:53, Neil C Smith escribió:
> [...]
> Sounds like a plan, although also like it will complicate the VM
Downloading updates.xml.gz from n.a.o and the rest of NBMs through n.a.o
> netbeans-vm.a.o > closer.lua should be easy.
> configuration quite a bit? Given current release process do we really need
> to track individual nbm downloads?
That's the question! How well are we doing with statistics? Would it be
possible/better for the IDE to send periodic reports (including
OS/modules/etc.) to netbeans-vm.a.o for later statistics analysis? Or is
this banned by the ASF?
Kind regards,
Antonio
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Neil C Smith <ne...@apache.org>.
On Fri, 17 May 2019, 20:24 Antonio, <an...@vieiro.net> wrote:
> What they suggest is to try to find a closest mirror for the first
> module (using closer.lua) and then keep on using that mirror directly
> for the rest of modules (even if we redirect through n.a.o for each of
> them to keep track of downloads).
>
Sounds like a plan, although also like it will complicate the VM
configuration quite a bit? Given current release process do we really need
to track individual nbm downloads?
We do still have an issue with n.a.o to VM redirect as far as I can tell.
Any chance you or someone else can try a platform download using the Ant
harness and the UC? Maybe it's an issue at this end, but don't think it is.
Best wishes,
Neil
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Neil C Smith <ne...@apache.org>.
On Fri, 17 May 2019, 20:32 Antonio, <an...@vieiro.net> wrote:
> Forgot to mention that they recommend the updates.xml.gz _NOT_ being
> downloaded through closer.lua.
>
> I don't know if Jiri wants to keep track of statistics on this file,
> though.
>
Well, my assumption from the other thread was that it was intended to be
downloaded from the VM, not through closer.lua, but still with statistics?
Best wishes,
Neil
>
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Antonio <an...@vieiro.net>.
Hi again,
Forgot to mention that they recommend the updates.xml.gz _NOT_ being
downloaded through closer.lua.
I don't know if Jiri wants to keep track of statistics on this file, though.
Cheers,
Antonio
El 17/05/2019 a las 21:24, Antonio escribió:
> Hi all,
>
> Talked to infra about this. They confirm this is not problem at the
> moment [1].
>
> There's some room for improvement, though. I think we're doing the
> redirection _for each_ module (to keep track of the requested modules)
> and then end up in the "closer.lua" script for each module.
>
> What they suggest is to try to find a closest mirror for the first
> module (using closer.lua) and then keep on using that mirror directly
> for the rest of modules (even if we redirect through n.a.o for each of
> them to keep track of downloads).
>
> Cheers,
> Antonio
>
> [1]
> https://the-asf.slack.com/archives/CBX4TSBQ8/p1558119078154600
>
> [2]
> http://www.apache.org/dev/infra-ban.html
>
>
> El 14/05/2019 a las 15:55, Bertrand Delacretaz escribió:
>> Hi,
>>
>> I think something similar happened a few weeks ago already, see here:
>>
>> https://lists.apache.org/thread.html/97f0619de82a630b6ec7d56f1ac5ab34dd85939ae1ff8dcf01434ee9@%3Cusers.infra.apache.org%3E
>>
>>
>> NetBeans users going through a company router have been banned from
>> *.apache.org due to too many requests in a short amount of time to
>> this URL:
>>
>> http://www.apache.org/dyn/closer.lua?action=download&filename=incubator/netbeans/incubating-netbeans/incubating-11.0/nbms/updates.xml.gz
>>
>>
>> It might be worth discussing with infra how to avoid this - my
>> understanding (to be verified) is that it's the amount of requests to
>> closer.lua that's problematic, even if the download themselves go to
>> download mirrors.
>>
>> -Bertrand
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
>> For additional commands, e-mail: dev-help@netbeans.apache.org
>>
>> For further information about the NetBeans mailing lists, visit:
>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>>
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Antonio <an...@vieiro.net>.
Hi all,
Talked to infra about this. They confirm this is not problem at the
moment [1].
There's some room for improvement, though. I think we're doing the
redirection _for each_ module (to keep track of the requested modules)
and then end up in the "closer.lua" script for each module.
What they suggest is to try to find a closest mirror for the first
module (using closer.lua) and then keep on using that mirror directly
for the rest of modules (even if we redirect through n.a.o for each of
them to keep track of downloads).
Cheers,
Antonio
[1]
https://the-asf.slack.com/archives/CBX4TSBQ8/p1558119078154600
[2]
http://www.apache.org/dev/infra-ban.html
El 14/05/2019 a las 15:55, Bertrand Delacretaz escribió:
> Hi,
>
> I think something similar happened a few weeks ago already, see here:
>
> https://lists.apache.org/thread.html/97f0619de82a630b6ec7d56f1ac5ab34dd85939ae1ff8dcf01434ee9@%3Cusers.infra.apache.org%3E
>
> NetBeans users going through a company router have been banned from
> *.apache.org due to too many requests in a short amount of time to
> this URL:
>
> http://www.apache.org/dyn/closer.lua?action=download&filename=incubator/netbeans/incubating-netbeans/incubating-11.0/nbms/updates.xml.gz
>
> It might be worth discussing with infra how to avoid this - my
> understanding (to be verified) is that it's the amount of requests to
> closer.lua that's problematic, even if the download themselves go to
> download mirrors.
>
> -Bertrand
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Antonio <an...@vieiro.net>.
Hi Bertrand,
This happened before in March:
https://issues.apache.org/jira/browse/NETBEANS-2204, it would be good to
reopen the issue and see if the customer is the same.
Back in march it seems this was a problem with a single user. Is it the
same case now?
Anyway maybe we want to take a deeper look at how we fetch that
information, and try to cache it somewhere...
Cheers,
Antonio
El 14/05/2019 a las 15:55, Bertrand Delacretaz escribió:
> Hi,
>
> I think something similar happened a few weeks ago already, see here:
>
> https://lists.apache.org/thread.html/97f0619de82a630b6ec7d56f1ac5ab34dd85939ae1ff8dcf01434ee9@%3Cusers.infra.apache.org%3E
>
> NetBeans users going through a company router have been banned from
> *.apache.org due to too many requests in a short amount of time to
> this URL:
>
> http://www.apache.org/dyn/closer.lua?action=download&filename=incubator/netbeans/incubating-netbeans/incubating-11.0/nbms/updates.xml.gz
>
> It might be worth discussing with infra how to avoid this - my
> understanding (to be verified) is that it's the amount of requests to
> closer.lua that's problematic, even if the download themselves go to
> download mirrors.
>
> -Bertrand
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Sat, May 18, 2019 at 6:50 PM Tim Boudreau <ni...@gmail.com> wrote:
> ...This kind of thing is truly a job for a CDN...
It looks like ASF Infra is starting to use CDNs like
https://issues.apache.org/jira/browse/INFRA-18180 for example.
That's certainly something to take into account when talking to
them...but someone needs to do that talking, ideally in such a jira
ticket starting with the NetBeans requirements for these update
related downloads.
-Bertrand
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Tim Boudreau <ni...@gmail.com>.
Couple of things:
1. I don't know if the IDE makes use of HTTP ETag and Last-Modified
headers and the outbound If-None-Match and If-Modified-Since headers, much
less Cache Control headers, but it certainly *should* and would probably be
pretty easy to modify if it doesn't. That would trim the HTTP conversation
down to 20-40 bytes for 99.999% of requests, where nothing has changed. No
idea if the Apache infrastructure distinguishes between a download versus
sending a 304 Not Modified response to say "your existing copy is still the
latest", but it should. If everyone is constantly downloading the actual
file when nothing has changed, that's a bug, and judging from these
problems, a serious one.
2. This kind of thing is truly a job for a CDN. In a recent gig, we used
CloudFront in front of serving streaming video chopped into thousands of
files for HLS streaming. It was shockingly cheap (I remember Akamai's
prices way back when). That's how this really ought to be handled. It
also seems like something where, the Apache Foundation being a non-profit,
a deal might be reached where they "donate" CDN hosting of small things
like that, which gets them a tax write-off.
Or we could self-fund it - I'd be willing to kick in $100 toward that,
which would probably buy a few months of hosting (if the client side
actually makes use of HTTP caching headers so we don't blow through what we
paid for pointlessly) - if a few other folks were willing to pitch in too.
-Tim
On Tue, May 14, 2019 at 9:55 AM Bertrand Delacretaz <bd...@apache.org>
wrote:
> Hi,
>
> I think something similar happened a few weeks ago already, see here:
>
>
> https://lists.apache.org/thread.html/97f0619de82a630b6ec7d56f1ac5ab34dd85939ae1ff8dcf01434ee9@%3Cusers.infra.apache.org%3E
>
> NetBeans users going through a company router have been banned from
> *.apache.org due to too many requests in a short amount of time to
> this URL:
>
>
> http://www.apache.org/dyn/closer.lua?action=download&filename=incubator/netbeans/incubating-netbeans/incubating-11.0/nbms/updates.xml.gz
>
> It might be worth discussing with infra how to avoid this - my
> understanding (to be verified) is that it's the amount of requests to
> closer.lua that's problematic, even if the download themselves go to
> download mirrors.
>
> -Bertrand
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
>
--
http://timboudreau.com
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Neil C Smith <ne...@apache.org>.
On Tue, 14 May 2019 at 16:04, Bertrand Delacretaz
<bd...@apache.org> wrote:
> I'm not sure if http://netbeans-vm.apache.org/uc/11.0/updates.xml.gz
> is reliable enough to be used for that,
Well, it's currently more reliable than the alternative which fails
every time due to the relative links.
However, I've just done another verbose ant build to get a log, and
the issue I have isn't quite the same. The main catalog redirects to
the VM, which then redirects via closer.lua But it's the first
redirect that seems to fail after a while, so it's another issue in
how we're handling updates redirects i think?!
eg.
[get:org.netbeans.modules.editor.guards]
http://netbeans.apache.org/nb/updates/11.0/ide/org-netbeans-modules-editor-guards.nbm
moved to http://netbeans-vm.apache.org/uc/11.0/ide/org-netbeans-modules-editor-guards.nbm
[get:org.netbeans.modules.editor.guards]
http://netbeans-vm.apache.org/uc/11.0/ide/org-netbeans-modules-editor-guards.nbm
moved to http://www.apache.org/dyn/closer.lua?action=download&filename=incubator/netbeans/incubating-netbeans/incubating-11.0/nbms/ide/org-netbeans-modules-editor-guards.nbm
and later
[get:org.netbeans.modules.xml.xam] Getting:
http://netbeans.apache.org/nb/updates/11.0/ide/org-netbeans-modules-xml-xam.nbm
[get:org.netbeans.modules.xml.xam] To:
/tmp/org-netbeans-modules-xml-xam9083219336849970974.nbm
[get:org.netbeans.modules.xml.xam] Error getting
http://netbeans.apache.org/nb/updates/11.0/ide/org-netbeans-modules-xml-xam.nbm
to /tmp/org-netbeans-modules-xml-xam9083219336849970974.nbm
Fails at a different point each time.
Best wishes,
Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Neil C Smith <ne...@apache.org>.
On Wed, 15 May 2019 at 10:59, Bertrand Delacretaz
<bd...@apache.org> wrote:
> On Wed, May 15, 2019 at 10:19 AM Neil C Smith <ne...@apache.org> wrote:
> > ...use absolute rather than relative URLs in the catalog.xml and point
> > directly to closer.lua?...
>
> I suppose that might trigger bans as described in my original post here.
It should hit a lot less than currently, though - it seems that's what
we're already doing, just behind a layer of redirects. And it would
at least keep the catalog itself out of it.
> I think this really needs to be coordinated with ASF infra, a ticket
> at http://issues.apache.org/jira/browse/INFRA might be best for that -
Agreed, and ideally by someone who really knows the intricacies of
what the update center code can handle without code changes.
Best wishes,
Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Wed, May 15, 2019 at 10:19 AM Neil C Smith <ne...@apache.org> wrote:
> ...use absolute rather than relative URLs in the catalog.xml and point
> directly to closer.lua?...
I suppose that might trigger bans as described in my original post here.
I think this really needs to be coordinated with ASF infra, a ticket
at http://issues.apache.org/jira/browse/INFRA might be best for that -
state NetBeans' needs and see what they suggest. I don't know enough
about our infra to comment meaningfully.
-Bertrand
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Neil C Smith <ne...@apache.org>.
On Wed, 15 May 2019 at 02:50, Wade Chandler <wa...@apache.org> wrote:
> On Tue, May 14, 2019, 11:04 Bertrand Delacretaz <bd...@apache.org>
> wrote:
> > I'm not sure if http://netbeans-vm.apache.org/uc/11.0/updates.xml.gz
> > is reliable enough to be used for that, a CDN seems more appropriate.
...
> Indeed, and I assume if everyone went to the VM it will eventually just
> crash out on sheer number of connections.
Except that everything is going via the VM first anyway as far as I
can tell?! Check the log I posted, or #7 at
https://cwiki.apache.org/confluence/display/NETBEANS/Apache+NetBeans+Release+README
It seems my problem actually isn't quite the same, but something being
limited in the netbeans.apache.org -> netbeans-vm.apache.org redirect
process. It might be the solution is the same though?
What about -
* rethinking where the catalog.xml is hosted, ideally somewhere
without redirects, or at least without a query in the URL? (assume
this might be a caching issue?)
* use absolute rather than relative URLs in the catalog.xml and point
directly to closer.lua? That way hosting of catalog and NBMs can be
separated? And we can update the catalog.xml itself rather than the
redirect when archiving releases?
Best wishes,
Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Wade Chandler <wa...@apache.org>.
On Tue, May 14, 2019, 11:04 Bertrand Delacretaz <bd...@apache.org>
wrote:
> On Tue, May 14, 2019 at 4:27 PM Neil C Smith <ne...@apache.org>
> wrote:
> > ...I've had to switch from the URL in the IDE
> > directly to the VM to get this process to complete without an error
> > (eg.
> https://github.com/praxis-live/praxis-live/blob/nb11/nbproject/platform.properties#L337
> ...
>
> I'm not sure if http://netbeans-vm.apache.org/uc/11.0/updates.xml.gz
> is reliable enough to be used for that, a CDN seems more appropriate.
>
> The Apache download mirrors do form a CDN, but it looks like the
> bottleneck (or at least what triggers bans) is the closer.lua routing
> script, maybe that's where some work is needed.
>
Indeed, and I assume if everyone went to the VM it will eventually just
crash out on sheer number of connections.
Wade
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Tue, May 14, 2019 at 4:27 PM Neil C Smith <ne...@apache.org> wrote:
> ...I've had to switch from the URL in the IDE
> directly to the VM to get this process to complete without an error
> (eg. https://github.com/praxis-live/praxis-live/blob/nb11/nbproject/platform.properties#L337 ...
I'm not sure if http://netbeans-vm.apache.org/uc/11.0/updates.xml.gz
is reliable enough to be used for that, a CDN seems more appropriate.
The Apache download mirrors do form a CDN, but it looks like the
bottleneck (or at least what triggers bans) is the closer.lua routing
script, maybe that's where some work is needed.
-Bertrand
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Users banned from apache.org due to downloading updates.xml.gz
Posted by Neil C Smith <ne...@apache.org>.
Hi,
On Tue, 14 May 2019 at 14:55, Bertrand Delacretaz
<bd...@apache.org> wrote:
> It might be worth discussing with infra how to avoid this - my
> understanding (to be verified) is that it's the amount of requests to
> closer.lua that's problematic, even if the download themselves go to
> download mirrors.
I raised something similar here in a thread on redirects a few weeks
ago. It also affects downloading a harness for RCP applications using
autoupdate.catalog.url I've had to switch from the URL in the IDE
directly to the VM to get this process to complete without an error
(eg. https://github.com/praxis-live/praxis-live/blob/nb11/nbproject/platform.properties#L337
)
The nb-javac catalog uses full URLs rather than relative paths. I
wonder whether that change needs to be made in the main UC catalog
too? Otherwise not just the catalog itself, but every download, hits
that redirect IIRC.
Best wishes,
Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists