You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Pierre Villard (Jira)" <ji...@apache.org> on 2025/03/11 16:06:00 UTC
[jira] [Commented] (NIFI-14355) Authenticated GitHub Registry Client hitting rate limit (5000 reqs per hour)
[ https://issues.apache.org/jira/browse/NIFI-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17934263#comment-17934263 ]
Pierre Villard commented on NIFI-14355:
---------------------------------------
For each versioned process group in every NiFi node, we would check every minute if there is a new version available:
[https://github.com/apache/nifi/blob/main/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L1082]
I think the best approach is to make this a property of nifi.properties so that it can be configured to a higher number.
> Authenticated GitHub Registry Client hitting rate limit (5000 reqs per hour)
> ----------------------------------------------------------------------------
>
> Key: NIFI-14355
> URL: https://issues.apache.org/jira/browse/NIFI-14355
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Flow Versioning, NiFi Registry
> Affects Versions: 2.2.0
> Environment: Not relevant
> Reporter: Andrea Di Stefano
> Priority: Major
> Attachments: image-2025-03-11-16-47-39-522.png
>
>
> Hello.
> We are using the GitHub Registry Client with a Personal Authentication Token (PAT) and constantly hitting the API rate limit. The rate limit, [as stated in the GitHub docs|https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28#primary-rate-limit-for-authenticated-users], is 5000 requests per hour.
> We have a pipeline with 53 instances of 3 versioned process groups, which are part of a bigger pipeline (~1.1k components).
> I uploaded a related stack trace [here|https://pastebin.com/R0tTw4bP].
> Calling the GitHub API with Postman, we saw that the response headers include data about the rate limit, current usage, and reset time in Unix epoch format, see the attached screenshot for an example:
> !image-2025-03-11-16-47-39-522.png!
> Is NiFi periodically polling the GitHub API in background?
> Can we somehow limit the number of calls?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)