[jira] [Updated] (NIFI-16391) GitLabRepositoryClient: limit commit listing to the first page instead of paging the full history

["Jannik Rebmann (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jannik Rebmann updated NIFI-16391:
----------------------------------
    Description: 
h4. Summary

{{GitLabRepositoryClient.getCommits(path, branch)}} uses the gitlab4j {{List}} overload, which internally delegates to {{{}Pager.all(){}}}. As a result, every call pages the *entire commit history* for the path (the client sets {{{}per_page=100{}}}). The commit listing should be bounded to the first page using a small, fixed page size ({{{}COMMIT_PAGE_SIZE{}}}).
h4. Environment / Observation
 * Self-hosted GitLab, many versioned process groups.
 * High volume of GitLab REST API calls ({{{}repository/commits{}}}) originating from NiFi.
 * Each individual {{getCommits(...)}} call is itself several page requests, because the full history is paged through.

h4. Root Cause

The use of {{Pager.all()}} in the GitLab client causes full-history paging on every call. The complete history is not required to determine the latest version or the version listing; the relevant commits are on the first page.
h4. Proposed Solution
 # In {{{}GitLabRepositoryClient.getCommits(path, branch){}}}, replace the {{{}Pager.all(){}}}-based overload with a call bounded to the first page.
 # Introduce a {{COMMIT_PAGE_SIZE}} constant with a small value (hard-coded initially; promotion to a property is an optional follow-up).
 # This bounds both the number of page requests per call and the payload size.

h4. Notes / Relationship
 * Mirrors the listing limit from {*}NIFI-14837 / PR #10186{*}, which was applied only to {{{}GitHubRepositoryClient{}}}; GitLab does not have this limit yet.
 * *Out of scope:* This issue covers only the bounding of the commit listing in the GitLab client. Removing the per-process-group multiplier (TTL cache in {{{}AbstractGitFlowRegistryClient{}}}) and the optional {{(commitSha, path)}} content cache are *not* part of this ticket.
 * The SHA→commit-detail cache from PR #10186 is not relevant for GitLab, because the gitlab4j commit listing returns fully-populated {{Commit}} objects.

h4. References

NIFI-14837, NIFI-16359, PR [https://github.com/apache/nifi/pull/10186]

  was:
h4. Summary
{{GitLabRepositoryClient.getCommits(path, branch)}} uses the gitlab4j {{List}} overload, which internally delegates to {{{}Pager.all(){}}}. As a result, every call pages the *entire commit history* for the path (the client sets {{{}per_page=100{}}}). The commit listing should be bounded to the first page using a small, fixed page size ({{{}COMMIT_PAGE_SIZE{}}}).
h4. Environment / Observation
 * Self-hosted GitLab, many versioned process groups.
 * High volume of GitLab REST API calls ({{{}repository/commits{}}}) originating from NiFi.
 * Each individual {{getCommits(...)}} call is itself several page requests, because the full history is paged through.

h4. Root Cause
The use of {{Pager.all()}} in the GitLab client causes full-history paging on every call. The complete history is not required to determine the latest version or the version listing; the relevant commits are on the first page.
h4. Proposed Solution
 # In {{{}GitLabRepositoryClient.getCommits(path, branch){}}}, replace the {{{}Pager.all(){}}}-based overload with a call bounded to the first page.
 # Introduce a {{COMMIT_PAGE_SIZE}} constant with a small value (hard-coded initially; promotion to a property is an optional follow-up).
 # This bounds both the number of page requests per call and the payload size.

h4. Notes / Relationship
 * Mirrors the listing limit from {*}NIFI-14837 / PR #10186{*}, which was applied only to {{{}GitHubRepositoryClient{}}}; GitLab does not have this limit yet.
 * *Out of scope:* This issue covers only the bounding of the commit listing in the GitLab client. Removing the per-process-group multiplier (TTL cache in {{{}AbstractGitFlowRegistryClient{}}}) and the optional {{(commitSha, path)}} content cache are *not* part of this ticket.
 * The SHA→commit-detail cache from PR #10186 is not relevant for GitLab, because the gitlab4j commit listing returns fully-populated {{Commit}} objects.

h4. References
NIFI-14837, PR [https://github.com/apache/nifi/pull/10186]


> GitLabRepositoryClient: limit commit listing to the first page instead of paging the full history
> -------------------------------------------------------------------------------------------------
>
>                 Key: NIFI-16391
>                 URL: https://issues.apache.org/jira/browse/NIFI-16391
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Flow Versioning
>    Affects Versions: 2.12.0
>            Reporter: Jannik Rebmann
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> h4. Summary
> {{GitLabRepositoryClient.getCommits(path, branch)}} uses the gitlab4j {{List}} overload, which internally delegates to {{{}Pager.all(){}}}. As a result, every call pages the *entire commit history* for the path (the client sets {{{}per_page=100{}}}). The commit listing should be bounded to the first page using a small, fixed page size ({{{}COMMIT_PAGE_SIZE{}}}).
> h4. Environment / Observation
>  * Self-hosted GitLab, many versioned process groups.
>  * High volume of GitLab REST API calls ({{{}repository/commits{}}}) originating from NiFi.
>  * Each individual {{getCommits(...)}} call is itself several page requests, because the full history is paged through.
> h4. Root Cause
> The use of {{Pager.all()}} in the GitLab client causes full-history paging on every call. The complete history is not required to determine the latest version or the version listing; the relevant commits are on the first page.
> h4. Proposed Solution
>  # In {{{}GitLabRepositoryClient.getCommits(path, branch){}}}, replace the {{{}Pager.all(){}}}-based overload with a call bounded to the first page.
>  # Introduce a {{COMMIT_PAGE_SIZE}} constant with a small value (hard-coded initially; promotion to a property is an optional follow-up).
>  # This bounds both the number of page requests per call and the payload size.
> h4. Notes / Relationship
>  * Mirrors the listing limit from {*}NIFI-14837 / PR #10186{*}, which was applied only to {{{}GitHubRepositoryClient{}}}; GitLab does not have this limit yet.
>  * *Out of scope:* This issue covers only the bounding of the commit listing in the GitLab client. Removing the per-process-group multiplier (TTL cache in {{{}AbstractGitFlowRegistryClient{}}}) and the optional {{(commitSha, path)}} content cache are *not* part of this ticket.
>  * The SHA→commit-detail cache from PR #10186 is not relevant for GitLab, because the gitlab4j commit listing returns fully-populated {{Commit}} objects.
> h4. References
> NIFI-14837, NIFI-16359, PR [https://github.com/apache/nifi/pull/10186]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)