[jira] [Created] (NIFI-16474) Nested versioned flows are fetched from the Flow Registry an exponential number of times when loading a flow's contents

["Marc Pellmann (Jira)" <ji...@apache.org>]

Marc Pellmann created NIFI-16474:
------------------------------------

             Summary: Nested versioned flows are fetched from the Flow Registry an exponential number of times when loading a flow's contents
                 Key: NIFI-16474
                 URL: https://issues.apache.org/jira/browse/NIFI-16474
             Project: Apache NiFi
          Issue Type: Bug
    Affects Versions: 2.12.0
            Reporter: Marc Pellmann


When NiFi loads a versioned flow together with its nested versioned flows, each nested flow is fetched from the Flow Registry Client many times instead of once. This happens through StandardFlowRegistryClientNode.getFlowContents(..., fetchRemoteFlows = true). The number of fetches grows exponentially with nesting depth.

 

This affects every operation that resolves the full snapshot, for example:
 * GET /process-groups/\{id}/local-modifications (StandardNiFiServiceFacade.getLocalModifications)
 * putting an existing process group under version control (PUT /process-groups/\{id} with versionControlInformation)
 * importing a versioned flow into the canvas

 

The Git-based registry clients (GitHub, GitLab, Bitbucket, Azure DevOps) talk to a hosted REST API. With them, a single request takes minutes and issues hundreds of API calls.

 

*Cause 1: duplicated call in getFlowContents*
{code:java}
for (final VersionedProcessGroup child : contents.getProcessGroups()) {
    final Map<String, VersionedParameterContext> childParameterContexts = populateVersionedContentsRecursively(context, child, snapshotContainer);
    
    for (final Map.Entry<String, VersionedParameterContext> childParameterContext : childParameterContexts.entrySet()) {
        flowSnapshot.getParameterContexts().putIfAbsent(childParameterContext.getKey(), childParameterContext.getValue());
    }
    populateVersionedContentsRecursively(context, child, snapshotContainer);  // <-- second, identical call
} {code}
populateVersionedContentsRecursively is called twice for every child. The result of the second call is discarded. The line was introduced with NIFI-11464.

 

*Cause 2: nested flows are resolved twice per level*

For a group that has VersionedFlowCoordinates, populateVersionedContentsRecursively calls fetchFlowContents(context, coordinates, true). That goes through clientNode.getFlowContents(..., fetchRemoteFlows = true), which already resolves every nested flow below that group. The group's contents are then copied over, and the method recurses into group.getProcessGroups() again. Those children still carry their coordinates, so every nested flow below is fetched again.

Combining both causes, reading the code suggests that a flow at nesting depth d is fetched on the order of 2·3^(d−1) times. If the same flow occurs several times in the tree, that number is multiplied again. Nested contents are also added to the FlowSnapshotContainer repeatedly.

 

*Cause 3: every fetch also lists the file's commit history (Git-based clients)*

AbstractGitFlowRegistryClient.getFlowContents calls getLatestVersion on every fetch, only to set RegisteredFlowSnapshot.isLatest. That lists the commit history of the snapshot file (GitRepositoryClient.getCommits), which may span several pages. Each fetch therefore costs at least two REST calls: content plus history. 

 

*Observed*

A flow with three levels of nesting, where one nested flow is used three times, on Bitbucket Cloud. A single GET /process-groups/\{id}/local-modifications ran for more than 4.5 minutes. During that time it fetched individual nested flows dozens of times (e.g. Nested.json 44 times, still running). One request thread produced more than 1,200 Bitbucket client log entries. Earlier, putting the same group under version control fetched the deepest nested flows 108 times each and took ~35 s.

 

*Impact*
 * getLocalModifications holds the flow read lock (NiFiServiceFacadeLock.proceedWithReadLock) for the whole time. Every write request waits behind it, e.g. deleting the very process group. 
 * Synchronous REST calls such as starting version control and importing a flow can outlast client read timeouts. The operation still completes on the server, and a client retry then fails with 409 ("process group is already under version control") or creates a duplicate group.
 * The hosted Git APIs see hundreds of calls per user action, which risks rate limiting.

 

*Steps to reproduce*
 #  Configure a Git-based Flow Registry Client, e.g. the Bitbucket or GitHub client.
 #  Version flows B and C. Version flow A so that it contains B, and B contains C. Optionally reference B more than once inside A.
 #  Import A into the canvas, or call GET /nifi-api/process-groups/\{id}/local-modifications on it.
 #  Enable DEBUG logging for the registry client. Each nested snapshot file is requested many times, and the count grows with depth.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)