[jira] [Updated] (NIFI-14837) Performance improvement GitHub Registry Client

["Pierre Villard (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-14837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pierre Villard updated NIFI-14837:
----------------------------------
    Description: 
I'm using the GitHub Registry Client in my NiFi instance. I have about 50 process groups that are versioned. Every process group matches a versioned flow that may have tens of commits.

When I want to change version, the current implementation will list all commits, and {*}for each commit{*}, will make an API call to GiHub in order to retrieve some specific informations (commit message, commit date, etc).

This is extremely ineffective and changing the version a flow ends up taking a very long time. For some cases with many commits, I cannot change version because the call in the NiFi UI would time out before the backend has sent back the full list of commits with all of the information.

This becomes very not friendly and barely usable. This will also impact the API rate limits a lot.

This Jira is to introduce multiple improvements that are making all of this MUCH better.
 * The GitHub client being used is initialized with an optional OkHttp client cache (see [https://hub4j.github.io/github-api/)]

 
{quote}This library comes with a pluggable connector to use different HTTP client implementations through {{{}HttpConnector{}}}. In particular, this means you can use [OkHttp|https://square.github.io/okhttp/], so we can make use of its HTTP response cache. Making a conditional request against the GitHub API and receiving a 304 response [does not count against the rate limit|https://docs.github.com/en/rest/overview/resources-in-the-rest-api?apiVersion=2022-11-28#conditional-requests].
{quote}
 
 * Adding a LRU cache to the client with a fixed size of 1000 commits maximum in order to keep an internal cache of commit SHA to commit details.
 * Expose a property to limit the number of commits retrieved. The client does not ensure a chronological order but guarantees a topological ordering. So it should be chronological except in some specific edge cases like rebase, merge commits, cherry-pick, commits with manual dates, etc. However, this is very unlikely to happen with a normal usage of the client. Regardless the default is to retrieve all commits like it is right now.
 * Adding a Rate Abuse Limit Handler to log an error when abusing the API limits. 

  was:
I'm using the GitHub Registry Client in my NiFi instance. I have about 50 process groups that are versioned. Every process group matches a versioned flow that may have tens of commits.

When I want to change version, the current implementation will list all commits, and {*}for each commit{*}, will make an API call to GiHub in order to retrieve some specific informations (commit message, commit date, etc).

This is extremely ineffective and changing the version a flow ends up taking a very long time. For some cases with many commits, I cannot change version because the call in the NiFi UI would time out before the backend has sent back the full list of commits with all of the information.

This becomes very not friendly and barely usable. This will also impact the API rate limits a lot.

This Jira is to introduce multiple improvements that are making all of this MUCH better.
 * The GitHub client being used is initialized with an optional OkHttp client cache (see [https://hub4j.github.io/github-api/)]

{quote}This library comes with a pluggable connector to use different HTTP client implementations through {{{}HttpConnector{}}}. In particular, this means you can use [OkHttp|https://square.github.io/okhttp/], so we can make use of its HTTP response cache. Making a conditional request against the GitHub API and receiving a 304 response [does not count against the rate limit|https://docs.github.com/en/rest/overview/resources-in-the-rest-api?apiVersion=2022-11-28#conditional-requests].
{quote} * Adding a LRU cache to the client with a fixed size of 1000 commits maximum in order to keep an internal cache of commit SHA to commit details.
 * Expose a property to limit the number of commits retrieved. The client does not ensure a chronological order but guarantees a topological ordering. So it should be chronological except in some specific edge cases like rebase, merge commits, cherry-pick, commits with manual dates, etc. However, this is very unlikely to happen with a normal usage of the client. Regardless the default is to retrieve all commits like it is right now.
 * Adding a Rate Abuse Limit Handler to log an error when abusing the API limits. 


> Performance improvement GitHub Registry Client
> ----------------------------------------------
>
>                 Key: NIFI-14837
>                 URL: https://issues.apache.org/jira/browse/NIFI-14837
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>            Reporter: Pierre Villard
>            Assignee: Pierre Villard
>            Priority: Major
>
> I'm using the GitHub Registry Client in my NiFi instance. I have about 50 process groups that are versioned. Every process group matches a versioned flow that may have tens of commits.
> When I want to change version, the current implementation will list all commits, and {*}for each commit{*}, will make an API call to GiHub in order to retrieve some specific informations (commit message, commit date, etc).
> This is extremely ineffective and changing the version a flow ends up taking a very long time. For some cases with many commits, I cannot change version because the call in the NiFi UI would time out before the backend has sent back the full list of commits with all of the information.
> This becomes very not friendly and barely usable. This will also impact the API rate limits a lot.
> This Jira is to introduce multiple improvements that are making all of this MUCH better.
>  * The GitHub client being used is initialized with an optional OkHttp client cache (see [https://hub4j.github.io/github-api/)]
>  
> {quote}This library comes with a pluggable connector to use different HTTP client implementations through {{{}HttpConnector{}}}. In particular, this means you can use [OkHttp|https://square.github.io/okhttp/], so we can make use of its HTTP response cache. Making a conditional request against the GitHub API and receiving a 304 response [does not count against the rate limit|https://docs.github.com/en/rest/overview/resources-in-the-rest-api?apiVersion=2022-11-28#conditional-requests].
> {quote}
>  
>  * Adding a LRU cache to the client with a fixed size of 1000 commits maximum in order to keep an internal cache of commit SHA to commit details.
>  * Expose a property to limit the number of commits retrieved. The client does not ensure a chronological order but guarantees a topological ordering. So it should be chronological except in some specific edge cases like rebase, merge commits, cherry-pick, commits with manual dates, etc. However, this is very unlikely to happen with a normal usage of the client. Regardless the default is to retrieve all commits like it is right now.
>  * Adding a Rate Abuse Limit Handler to log an error when abusing the API limits. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)