[jira] [Created] (NIFI-15889) Cache Parameter Context name mapping to fix O(N_PG × N_PC^2) startup synchronization

["Xinyu Wang (Jira)" <ji...@apache.org>]

Xinyu Wang created NIFI-15889:
---------------------------------

             Summary: Cache Parameter Context name mapping to fix O(N_PG × N_PC^2) startup synchronization
                 Key: NIFI-15889
                 URL: https://issues.apache.org/jira/browse/NIFI-15889
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Core Framework
    Affects Versions: 2.5.0
            Reporter: Xinyu Wang


On NiFi startup, VersionedFlowSynchronizer synchronizes all child Process Groups, and for each PG it iterates the versioned Parameter Contexts and calls StandardParameterContextManager.getParameterContextNameMapping(). That method rebuilds the entire name→PC HashMap from a stream collect on every call, giving the synchronization an effective complexity of:

  O(N_PG × N_PC^2)

  For flows with thousands of Parameter Contexts the rebuild dominates startup time and turns a minute-scale boot into an hour-scale one.

  Affected code

  nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/parameter/StandardParameterContextManager.java

  @Override
  public Map<String, ParameterContext> getParameterContextNameMapping() {
      return parameterContexts.values().stream()
              .collect(Collectors.toMap(ParameterContext::getName, Function.identity()));
  }

  Hot caller: StandardVersionedComponentSynchronizer#createParameterContextWithoutReferences, which is invoked from addProcessGroup for every child PG. The same name mapping is rebuilt N_PG times even though the underlying parameterContexts map barely changes between calls.

  Proposed fix

  Maintain a name index incrementally and cache an unmodifiable view; reads become O(1), and the unique-name check at insert becomes O(1) too.

  private final Map<String, ParameterContext> parameterContexts = new HashMap<>();
  private final Map<String, ParameterContext> nameIndex = new HashMap<>();
  private volatile Map<String, ParameterContext> cachedNameMapping = Collections.emptyMap();

  @Override
  public synchronized void addParameterContext(final ParameterContext pc) {
      Objects.requireNonNull(pc);

      final ParameterContext existingById = parameterContexts.get(pc.getIdentifier());
      if (existingById != null && !(existingById instanceof ReferenceOnlyParameterContext)) {
          throw new IllegalStateException("Cannot add Parameter Context because another Parameter Context already exists with the same ID");
      }

      final ParameterContext existingByName = nameIndex.get(pc.getName());
      if (existingByName != null && existingByName != existingById) {
          throw new IllegalStateException("Cannot add Parameter Context because another Parameter Context already exists with the name '" + pc + "'");
      }

      // When replacing a ReferenceOnlyParameterContext placeholder, its name
      // differs from the real PC name, so its stale entry must be removed
      // from nameIndex first.
      if (existingById instanceof ReferenceOnlyParameterContext) {
          nameIndex.remove(existingById.getName(), existingById);
      }

      parameterContexts.put(pc.getIdentifier(), pc);
      nameIndex.put(pc.getName(), pc);
      cachedNameMapping = Collections.unmodifiableMap(new HashMap<>(nameIndex));
  }

  @Override
  public synchronized ParameterContext removeParameterContext(final String id) {
      Objects.requireNonNull(id);
      final ParameterContext removed = parameterContexts.remove(id);
      if (removed != null) {
          nameIndex.remove(removed.getName(), removed);
          cachedNameMapping = Collections.unmodifiableMap(new HashMap<>(nameIndex));
      }
      return removed;
  }

  @Override
  public Map<String, ParameterContext> getParameterContextNameMapping() {
      return cachedNameMapping;
  }

  Single file, ~25 lines. Complexity changes from O(N_PG × N_PC^2) to O(N_PG × N_PC).

  Subtle case: ReferenceOnlyParameterContext placeholder

  ReferenceOnlyParameterContext is used to break the chicken-and-egg cycle when Parameter Contexts inherit from each other. Its name is "Reference-Only Parameter Context [<id>]", which is different from the real PC's name. When addParameterContext is called to replace a placeholder with the real PC, the placeholder's old name must be explicitly removed from nameIndex; otherwise VersionedFlowSynchronizer.removeMissingParameterContexts walks the cached name map, sees "Reference-Only Parameter Context [...]" not in the proposed name list, looks up its ID, and calls removeParameterContext — deleting the real PC.

  Backwards compatibility

  - Public method signature is unchanged.
  - Returned map is now an unmodifiable view — callers that mutated it would already have been buggy.
  - All mutators remain synchronized.
  - Existing unique-name and unique-ID invariants are preserved.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)