Fwd: [Notice] Rate limiting in effect on JIRA, BZ, Moin

[Sean Busbey <bu...@cloudera.com.INVALID>]

heads up, since this could impact Release Doc Maker. I haven't done a
pass of the code yet. Will try to find time this week.

---------- Forwarded message ---------
From: Daniel Gruno <hu...@apache.org>
Date: Sun, Jan 27, 2019 at 11:10 AM
Subject: [Notice] Rate limiting in effect on JIRA, BZ, Moin
To: users@infra.apache.org <us...@infra.apache.org>


Hi folks,

Over the past few days we have implemented rate limiting on selected
services across the ASF; starting with JIRA, BugZilla and Moin Wiki.

IF YOU ARE A NORMAL USER:
#########################
This very likely will never affect you, and you can go about your
business just like normal :) If you DO experience errors or 429 (rate
limited) response codes, please do let us know so we can address it.


IF YOU ARE A ROBOT/CYBORG/COMPUTRON:
####################################
There are now limits in place for how much CPU time you can use, varying
from service to service. If you get limited, you will receive a HTTP 429
response instead of the normal 200, and a short text blob will explain
that you have crossed our resource limits and have been rate-limited. It
will also explain why, and when you can expect to be unblocked again
(generally within two minutes time). Scrapers, bots etc using our
services should check for a 429 response code and act accordingly (or
just slow down the discovery pace in general, as that benefits all of
us).

Rate limits are applied across IP blocks to discourage distributed
abuse, thus if you have 1.2.3.4 abusing a service, 1.2.3.5 would
potentially also be affected by the rate limits till they expire.

Later this year, we will be rolling out rate limits on more services,
and we encourage people automating tasks to honor the 429 responses
across all ASF services.

With regards, Daniel on behalf of ASF Infra.



-- 
busbey

Re: [Notice] Rate limiting in effect on JIRA, BZ, Moin

[Allen Wittenauer <aw...@effectivemachines.com.INVALID>]

I really hate how INFRA just drops this stuff without any warning.  I wish they’d work with us for once. Anyway, this is going to totally destroy precommit-admin.  It needs a complete rewrite: it’s highly inefficient and doesn’t work with pipeline jobs. But it’s going to take time and I know that right now I personally don’t have any.

A few weeks ago (when I did have time) I was toying around with the jenkins<->lira plugin for patch submission.  It works by sending the full JIRA key (e.g., YETUS-1) but it doesn’t appear to be possible to limit it to just attachment changes (wether this is on the Jenkins side or the JIRA side, I’m unsure).  As a result of those experiments, I’ve been thinking that it would be worthwhile to make a backwards incompatible precommit-admin that does a few things differently:

	* always sends the full key as JIRA_ISSUE_KEY to be compatible with the web hook and with multibranch pipelines (since there will likely only be one job for multiple JIRA projects, e.g., Hadoop)
	* dumps out a file that lists what issue should be triggered and includes some sample Jenkinsfile groovy code to read it and talk to pipelines[1]
	* can use JIRA_ISSUE_KEY as input to determine if that job should be processed; this eliminates the 10 minute wait(!)
	* without JIRA_ISSUE_KEY, go ahead and query JIRA (via REST calls rather than the search xml API); this deals with the “Jenkins was down” problem

Configuration would be very different than what we have today, obviously. (Minimally, need a project -> job mapping so that a HDFS-xxx JIRA issue triggers the Hadoop multibranch pipeline job.)  It will probably also put us on track to EOL the various ‘Precommit-XXX-admin’ jobs.  Given the forced march to Github, that may not necessarily be a bad thing.

1 - Jenkins pipelines don’t support token auth for some reason.  This completely breaks the ASF model and I can’t see a way to unbreak it easily.  Dynamic job triggering pretty much requires something plugged into the Jenkins java code. :( :(  


> On Jan 28, 2019, at 8:00 AM, Sean Busbey <bu...@cloudera.com.INVALID> wrote:
> 
> heads up, since this could impact Release Doc Maker. I haven't done a
> pass of the code yet. Will try to find time this week.
> 
> ---------- Forwarded message ---------
> From: Daniel Gruno <hu...@apache.org>
> Date: Sun, Jan 27, 2019 at 11:10 AM
> Subject: [Notice] Rate limiting in effect on JIRA, BZ, Moin
> To: users@infra.apache.org <us...@infra.apache.org>
> 
> 
> Hi folks,
> 
> Over the past few days we have implemented rate limiting on selected
> services across the ASF; starting with JIRA, BugZilla and Moin Wiki.
> 
> IF YOU ARE A NORMAL USER:
> #########################
> This very likely will never affect you, and you can go about your
> business just like normal :) If you DO experience errors or 429 (rate
> limited) response codes, please do let us know so we can address it.
> 
> 
> IF YOU ARE A ROBOT/CYBORG/COMPUTRON:
> ####################################
> There are now limits in place for how much CPU time you can use, varying
> from service to service. If you get limited, you will receive a HTTP 429
> response instead of the normal 200, and a short text blob will explain
> that you have crossed our resource limits and have been rate-limited. It
> will also explain why, and when you can expect to be unblocked again
> (generally within two minutes time). Scrapers, bots etc using our
> services should check for a 429 response code and act accordingly (or
> just slow down the discovery pace in general, as that benefits all of
> us).
> 
> Rate limits are applied across IP blocks to discourage distributed
> abuse, thus if you have 1.2.3.4 abusing a service, 1.2.3.5 would
> potentially also be affected by the rate limits till they expire.
> 
> Later this year, we will be rolling out rate limits on more services,
> and we encourage people automating tasks to honor the 429 responses
> across all ASF services.
> 
> With regards, Daniel on behalf of ASF Infra.
> 
> 
> 
> -- 
> busbey