[jira] [Created] (NIP-11) Introduce Connector extension point

["Mark Payne (Jira)" <ji...@apache.org>]

Mark Payne created NIP-11:
-----------------------------

             Summary: Introduce Connector extension point
                 Key: NIP-11
                 URL: https://issues.apache.org/jira/browse/NIP-11
             Project: NiFi Improvement Proposal
          Issue Type: New Feature
            Reporter: Mark Payne


h1. Motivation

The Apache NiFi ecosystem currently caters primarily to Data Engineers and Software Engineers, who possess the technical expertise required to navigate its learning curve. While NiFi offers powerful capabilities for creating and reusing versioned data flows, these flows often necessitate substantial modifications, particularly when adapting to diverse source/destination system authentication strategies. A consistent demand has emerged from less technical users for a more "self-service" solution, alongside requests for automated use cases where users can adjust a few properties without undertaking significant flow alterations. To address these challenges and broaden NiFi's accessibility, this proposal introduces a new "Connector" concept.
h1. Scope

The introduction of the Connector abstraction represents a significant undertaking that will necessitate extensive modifications across the NiFi architecture. This initiative will require substantial additions to the existing NiFi API and Framework. Furthermore, it will necessitate updates to the flow.json definition and the various Versioned object models to accommodate the new abstraction. Additionally, the user interface will require updates to provide a cohesive and intuitive experience for configuring and managing Connectors.
h1. Description

This proposal aims to introduce a new "Connector" abstraction and extension point within Apache NiFi. A Connector will encapsulate a complete NiFi flow, represented as a Process Group. This new entity will be responsible for the lifecycle management of its constituent components, including their starting and stopping. Crucially, a Connector will also be responsible for dynamically updating its encapsulated flow based on the configuration it exposes to the user. This dynamic update capability will encompass modifying Processor and Controller Service properties, and potentially even adding or removing components within the flow.

 

Unlike a Processor, which typically presents a flat list of configurable properties, a Connector will feature a structured collection of PropertyGroups. This organizational approach will enable developers to logically separate different areas of configuration, such as distinct sections for "Source configuration," "Destination configuration," "Transformations," and "Enrichments." This structured approach enhances clarity and ease of use for users.

 

Use of Custom UIs, or even more tailored reusable UI components could be used in order to provide a clearer, more usable configuration for selecting source entities, as well. For example, a UI might offer the ability to choose which remote resources should be polled for data whereas we typically rely on domain specific languages such as glob patterns or regular expressions. While this is feasible at a Processor level, it can be made more reusable and consistent among components with Connectors, and that information can be used to more intelligently configure the flow for the user.

 

Furthermore, the Connector abstraction will provide a robust foundation for more safely upgrading data flows from one version to another. This is achieved by enabling the execution of custom code to gracefully migrate configurations and flows. This capability extends beyond individual component updates, potentially allowing for complex operations such as migrating state between different components, which is not feasible at the individual component level.

 

Finally, the Connector concept lays the groundwork for building future capabilities that are inherently meaningful at the flow level, as opposed to the single component (Processor or Controller Service) level. Examples of such capabilities include comprehensive validation of an entire flow, gathering holistic metrics that provide a more complete operational picture, or providing metrics specifically tailored for intelligent scaling decisions.

 
h1. Expected API

The NiFi API will be impacted significantly. While it’s difficult to entirely suss out the full API, an initial implementation can be found at [https://github.com/markap14/nifi-api/tree/connectors/src/main/java/org/apache/nifi/components/connector]

 

Given the scope, the API will be noted as experimental and may change in a non-backward compatible manner between minor versions until the API has been more fully validated through developing concrete implementations.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)