You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Pierre Villard <pi...@gmail.com> on 2026/02/19 19:21:02 UTC
[DISCUSS] Releasing initial Connectors support
Hi everyone,
I wanted to bring up a discussion around releasing the initial
implementation of the Connectors feature that was introduced in NIP-11
[1]. The initial discussion happened on the mailing list back in
August 2025 [2], followed by a vote in September 2025 to start the
implementation work [3]. Since then, significant progress has been
made over the past few months and we believe the feature is ready for
an initial release.
As a reminder, the Connector concept introduces a new extension point
in Apache NiFi that encapsulates a complete NiFi dataflow behind a
higher-level, guided configuration experience. Rather than exposing
the full complexity of a flow to the user, a Connector presents a
structured set of Configuration Steps and Property Groups (for
example: "Source Configuration", "Destination Configuration",
"Transformations") that allow a user to configure an end-to-end data
pipeline without needing to understand or manipulate the underlying
flow directly. Under the hood, the Connector is responsible for
dynamically building and reconfiguring its encapsulated flow based on
the user's configuration choices. This includes managing the lifecycle
(starting/stopping) of all components within the flow. The goal is to
significantly simplify the user experience for common data movement
use cases, making NiFi more accessible to a broader audience while
also streamlining operations for experienced practitioners.
Given the scope of the feature, the API is marked as experimental and
may change in a non-backward compatible manner between minor versions
until the API has been more fully validated through developing
concrete Connector implementations.All of the development work
happened in a feature branch named NIFI-15258 in both the nifi-api [4]
and nifi [5] repositories. Here is a high-level summary of what was
implemented:
1. NiFi API - The Connector API has been introduced under
org.apache.nifi.components.connector. This includes the core Connector
interface and an AbstractConnector base class, the ConfigurationStep
and ConnectorPropertyGroup abstractions for structuring configuration,
ConnectorPropertyDescriptor for defining properties with dependencies
and allowable values, a FlowContext abstraction that provides both an
active flow and a working flow for configuration verification,
component facades (ProcessorFacade, ControllerServiceFacade,
ConnectionFacade, ProcessGroupFacade, ParameterContextFacade) for
Connectors to interact with their encapsulated flow, Secret references
for handling sensitive values, Asset support, ConnectorMethod for
exposing custom web endpoints, documentation writers for
auto-generating Connector documentation, and updates to the Versioned
flow model to accommodate Connectors. Comprehensive unit tests have
been added for the API.
2. NiFi Maven NAR Plugin - Support has been added for creating
Connector extension manifests in NAR files, so that the NiFi runtime
can discover and load Connector extensions. This change is already on
the main branch.
3. NiFi (Framework, REST API, UI, Tests) - The bulk of the
implementation resides here:
- Framework core: ConnectorNode and lifecycle management, connector
facades for standalone process groups, a connector configuration
repository, flow synchronization support for connectors, parameter
context and secrets management integration, and cluster replication
support.
- REST API: Full set of endpoints for connector CRUD operations,
lifecycle management (start/stop), configuration step retrieval and
updates, property group management, status reporting, data drainage,
connector actions, secrets, state management, verification, and
documentation.
- Frontend UI: Connectors are integrated into the flow designer, the
documentation browser, and access policies.
- Mock framework: A connector mock server and test runner to support
unit and integration testing of Connector implementations.
- Example connector: A Kafka-to-S3 connector implementation that
serves as a reference for building Connectors.
- CLI Toolkit: A ConnectorClient has been added for command-line
interaction with Connectors.
- System tests: 11 system test classes covering standalone and
clustered scenarios including CRUD, lifecycle, assets, data drainage,
parameter contexts, version resolution, and method marshalling.
Release plan:
We would like to proceed iteratively with three releases:
- NiFi Maven NAR Plugin - This has no dependency on the other
projects. The Connector manifest support is already on main so we can
get started with a release candidate right away. I volunteer to take
care of the release duties.
- NiFi API - A pull request [6] has been filed to merge the NIFI-15258
feature branch into main. Once merged, we will proceed with a release
candidate.
- NiFi - Once the NiFi API release is published, we will proceed with
the proper PRs (bumping the API version dependency and merging the
NIFI-15258 feature branch), and we will look at releasing a new
version of NiFi.
Please have a look at the feature branches and share your feedback.
Any comments, questions, or concerns are welcome.
Thanks,
Pierre
[1] https://issues.apache.org/jira/browse/NIP-11
[2] https://lists.apache.org/thread/77c7o4yf8lt9owpjsjb8gl2slk3f7mcf
[3] https://lists.apache.org/thread/skhn5kjkq6kfmrbqsshxs5348d968jvb
[4] https://github.com/apache/nifi-api/tree/NIFI-15258
[5] https://github.com/apache/nifi/tree/NIFI-15258
[6] https://github.com/apache/nifi-api/pull/70
Re: [DISCUSS] Releasing initial Connectors support
Posted by Kevin Doran <kd...@apache.org>.
I agree with what Pierre has said:
The initial release of the Connectors feature can and should use NiFi
2.x versions, and maintain full backwards compatibility for NiFi users
and admins that are not using the new Connectors functionality, it is
only additive and does not require a NiFi 3.
When we talk about experimental APIs that may see breaking changes
(without a major version update, ie NiFi 3), that is only for the new
interfaces and methods relating to Connectors. Existing APIs related
to traditional NiFi flows will not break on NiFi 2.x due to the
Connectors feature now or in the future, so only early-adopters of
Connectors might be impacted by changes to those experimental APIs.
I like this approach because it introduces new, useful functionality
for NiFi users that does not require disruptive changes such as
migrating to a new major version of NiFi, and on the dev side let's us
"get it right" and the dust to settle before we start work on NiFi 3.
I think we will learn a lot through the next few releases to inform
the best strategic direction for NiFi 3, connectors included, so I
don't think a major version bump is needed right now, but it could
make sense soon for many reasons.
Cheers,
Kevin
On Fri, Feb 20, 2026 at 6:18 AM Isha Lamboo via dev <de...@nifi.apache.org> wrote:
>
> Hi all,
>
> From the perspective of someone administering a number of NiFi deployments for customers, I'm very interested in testing and deploying this new feature, but bumping NiFi to version 3 would just mean a long delay as major version upgrades are treated very cautiously in many corporate environments, especially after the major work needed to move to NiFi 2 in some cases.
>
> If breaking changes are handled properly (ghosted Connectors perhaps) and don't break the entire NiFi canvas, I would rather introduce this with a regular minor update and offer it on trial basis to select users. Perhaps Connectors can have a separate authorization or be marked as restricted components to control their use?
>
> Kind regards,
>
> Isha
>
> -----Original Message-----
> From: Pierre Villard <pi...@gmail.com>
> Sent: Friday, 20 February 2026 11:25
> To: dev@nifi.apache.org
> Subject: Re: [DISCUSS] Releasing initial Connectors support
>
> Hi Csaba,
>
> Thanks for taking a look!
>
> While Connectors will indeed appear in the flow definition (flow.json.gz), this should not have any backward compatibility impact for users who are not using this new concept. The only breaking changes we anticipate would be limited to the Connector API and its related constructs (such as the Connector-specific portions of the flow definition and manifest) until we gain enough confidence through concrete implementations to mark the API as stable. Existing flows, processors, controller services, and other components would remain unaffected.
>
> Regarding a NiFi 3.0 release, I do not anticipate that the addition of the Connectors concept alone would motivate a major version bump, as it should not introduce breaking changes to the existing framework or user experience. That said, there are other topics on the horizon, such as a comprehensive review of the NiFi authorization model, that would likely require a new major release. But this is still very early and I would not expect anything like that any time soon. In any case, based on how things went with NiFi 2, any major version change would be subject to an extensive community discussion, as we know that major version upgrades always require significant time and resources, and many users are still on NiFi 1.x.
>
> This is, obviously, just my perspective and others may have additional thoughts.
>
> Thanks,
> Pierre
>
> Le ven. 20 févr. 2026 à 00:29, Csaba Bejan <be...@gmail.com> a écrit :
> >
> > Hi Pierre,
> >
> > Thanks for sharing all these details and thanks to all who worked on it.
> > This will be a great addition to the NiFi framework.
> >
> > I just did a quick scroll through and I saw you mentioned that the API
> > is experimental for now and can break going forward until it
> > stabilizes. I saw (and anticipate) Manifest and FlowDefinition
> > changes, are you expecting these to break as well in a non backward
> > compatible manner? I didn't do a proper review yet. I just saw that
> > the connector related constructs were introduced.
> > As this is a core nifi change would this mean a NiFi 3.0 major version
> > bump when it is complete or it is still too early for that discussion.
> >
> > Thanks,
> > Csaba
> >
> > On Thu, Feb 19, 2026 at 2:22 PM Pierre Villard
> > <pi...@gmail.com>
> > wrote:
> >
> > > Hi everyone,
> > >
> > > I wanted to bring up a discussion around releasing the initial
> > > implementation of the Connectors feature that was introduced in
> > > NIP-11 [1]. The initial discussion happened on the mailing list back
> > > in August 2025 [2], followed by a vote in September 2025 to start
> > > the implementation work [3]. Since then, significant progress has
> > > been made over the past few months and we believe the feature is
> > > ready for an initial release.
> > >
> > > As a reminder, the Connector concept introduces a new extension
> > > point in Apache NiFi that encapsulates a complete NiFi dataflow
> > > behind a higher-level, guided configuration experience. Rather than
> > > exposing the full complexity of a flow to the user, a Connector
> > > presents a structured set of Configuration Steps and Property Groups
> > > (for
> > > example: "Source Configuration", "Destination Configuration",
> > > "Transformations") that allow a user to configure an end-to-end data
> > > pipeline without needing to understand or manipulate the underlying
> > > flow directly. Under the hood, the Connector is responsible for
> > > dynamically building and reconfiguring its encapsulated flow based
> > > on the user's configuration choices. This includes managing the
> > > lifecycle
> > > (starting/stopping) of all components within the flow. The goal is
> > > to significantly simplify the user experience for common data
> > > movement use cases, making NiFi more accessible to a broader
> > > audience while also streamlining operations for experienced practitioners.
> > >
> > > Given the scope of the feature, the API is marked as experimental
> > > and may change in a non-backward compatible manner between minor
> > > versions until the API has been more fully validated through
> > > developing concrete Connector implementations.All of the development
> > > work happened in a feature branch named NIFI-15258 in both the
> > > nifi-api [4] and nifi [5] repositories. Here is a high-level summary
> > > of what was
> > > implemented:
> > >
> > > 1. NiFi API - The Connector API has been introduced under
> > > org.apache.nifi.components.connector. This includes the core
> > > Connector interface and an AbstractConnector base class, the
> > > ConfigurationStep and ConnectorPropertyGroup abstractions for
> > > structuring configuration, ConnectorPropertyDescriptor for defining
> > > properties with dependencies and allowable values, a FlowContext
> > > abstraction that provides both an active flow and a working flow for
> > > configuration verification, component facades (ProcessorFacade,
> > > ControllerServiceFacade, ConnectionFacade, ProcessGroupFacade,
> > > ParameterContextFacade) for Connectors to interact with their
> > > encapsulated flow, Secret references for handling sensitive values,
> > > Asset support, ConnectorMethod for exposing custom web endpoints,
> > > documentation writers for auto-generating Connector documentation,
> > > and updates to the Versioned flow model to accommodate Connectors.
> > > Comprehensive unit tests have been added for the API.
> > >
> > > 2. NiFi Maven NAR Plugin - Support has been added for creating
> > > Connector extension manifests in NAR files, so that the NiFi runtime
> > > can discover and load Connector extensions. This change is already
> > > on the main branch.
> > >
> > > 3. NiFi (Framework, REST API, UI, Tests) - The bulk of the
> > > implementation resides here:
> > > - Framework core: ConnectorNode and lifecycle management, connector
> > > facades for standalone process groups, a connector configuration
> > > repository, flow synchronization support for connectors, parameter
> > > context and secrets management integration, and cluster replication
> > > support.
> > > - REST API: Full set of endpoints for connector CRUD operations,
> > > lifecycle management (start/stop), configuration step retrieval and
> > > updates, property group management, status reporting, data drainage,
> > > connector actions, secrets, state management, verification, and
> > > documentation.
> > > - Frontend UI: Connectors are integrated into the flow designer, the
> > > documentation browser, and access policies.
> > > - Mock framework: A connector mock server and test runner to support
> > > unit and integration testing of Connector implementations.
> > > - Example connector: A Kafka-to-S3 connector implementation that
> > > serves as a reference for building Connectors.
> > > - CLI Toolkit: A ConnectorClient has been added for command-line
> > > interaction with Connectors.
> > > - System tests: 11 system test classes covering standalone and
> > > clustered scenarios including CRUD, lifecycle, assets, data
> > > drainage, parameter contexts, version resolution, and method marshalling.
> > >
> > > Release plan:
> > >
> > > We would like to proceed iteratively with three releases:
> > > - NiFi Maven NAR Plugin - This has no dependency on the other
> > > projects. The Connector manifest support is already on main so we
> > > can get started with a release candidate right away. I volunteer to
> > > take care of the release duties.
> > > - NiFi API - A pull request [6] has been filed to merge the
> > > NIFI-15258 feature branch into main. Once merged, we will proceed
> > > with a release candidate.
> > > - NiFi - Once the NiFi API release is published, we will proceed
> > > with the proper PRs (bumping the API version dependency and merging
> > > the
> > > NIFI-15258 feature branch), and we will look at releasing a new
> > > version of NiFi.
> > >
> > > Please have a look at the feature branches and share your feedback.
> > > Any comments, questions, or concerns are welcome.
> > >
> > > Thanks,
> > > Pierre
> > >
> > > [1]
> > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > > sues.apache.org%2Fjira%2Fbrowse%2FNIP-11&data=05%7C02%7Cisha.lamboo%
> > > 40virtualsciences.nl%7Cb725e0f96b1a4e85fda808de706aaa8c%7C21429da9e4
> > > ad45f99a6fcd126a64274b%7C0%7C0%7C639071800600952885%7CUnknown%7CTWFp
> > > bGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIs
> > > IkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=HGdGVE9GecIowtd9
> > > 9VyeB%2B4k9IRhSjfr8RQmDg72arY%3D&reserved=0
> > > [2]
> > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
> > > sts.apache.org%2Fthread%2F77c7o4yf8lt9owpjsjb8gl2slk3f7mcf&data=05%7
> > > C02%7Cisha.lamboo%40virtualsciences.nl%7Cb725e0f96b1a4e85fda808de706
> > > aaa8c%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C63907180060114245
> > > 6%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMC
> > > IsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdat
> > > a=QwXp5JlLFucTlbzBFsJqC1hVrb3LZoiC4EyY2ewAiQI%3D&reserved=0
> > > [3]
> > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
> > > sts.apache.org%2Fthread%2Fskhn5kjkq6kfmrbqsshxs5348d968jvb&data=05%7
> > > C02%7Cisha.lamboo%40virtualsciences.nl%7Cb725e0f96b1a4e85fda808de706
> > > aaa8c%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C63907180060115588
> > > 9%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMC
> > > IsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdat
> > > a=cErxoh20l%2BxcP76ObajMq81NqxC9QgIBihiVd3bYepI%3D&reserved=0
> > > [4]
> > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > > thub.com%2Fapache%2Fnifi-api%2Ftree%2FNIFI-15258&data=05%7C02%7Cisha
> > > .lamboo%40virtualsciences.nl%7Cb725e0f96b1a4e85fda808de706aaa8c%7C21
> > > 429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C639071800601169093%7CUnknow
> > > n%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJX
> > > aW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=lTxrayDJ
> > > b6buwy3XFgICzzL%2FtDizYB6qeCgsgn5pcfU%3D&reserved=0
> > > [5]
> > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > > thub.com%2Fapache%2Fnifi%2Ftree%2FNIFI-15258&data=05%7C02%7Cisha.lam
> > > boo%40virtualsciences.nl%7Cb725e0f96b1a4e85fda808de706aaa8c%7C21429d
> > > a9e4ad45f99a6fcd126a64274b%7C0%7C0%7C639071800601181591%7CUnknown%7C
> > > TWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4z
> > > MiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6krwrbZxn3ze
> > > Ujs92YUpB9%2F3H%2FZPikmzO7dq3wcD%2BBc%3D&reserved=0
> > > [6]
> > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > > thub.com%2Fapache%2Fnifi-api%2Fpull%2F70&data=05%7C02%7Cisha.lamboo%
> > > 40virtualsciences.nl%7Cb725e0f96b1a4e85fda808de706aaa8c%7C21429da9e4
> > > ad45f99a6fcd126a64274b%7C0%7C0%7C639071800601193701%7CUnknown%7CTWFp
> > > bGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIs
> > > IkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=iX6%2B%2BU%2FjGf
> > > uMfdf9Gp61MTYg%2B8sCcobJr84640Dvtg4%3D&reserved=0
> > >
RE: [DISCUSS] Releasing initial Connectors support
Posted by Isha Lamboo via dev <de...@nifi.apache.org>.
Hi all,
From the perspective of someone administering a number of NiFi deployments for customers, I'm very interested in testing and deploying this new feature, but bumping NiFi to version 3 would just mean a long delay as major version upgrades are treated very cautiously in many corporate environments, especially after the major work needed to move to NiFi 2 in some cases.
If breaking changes are handled properly (ghosted Connectors perhaps) and don't break the entire NiFi canvas, I would rather introduce this with a regular minor update and offer it on trial basis to select users. Perhaps Connectors can have a separate authorization or be marked as restricted components to control their use?
Kind regards,
Isha
-----Original Message-----
From: Pierre Villard <pi...@gmail.com>
Sent: Friday, 20 February 2026 11:25
To: dev@nifi.apache.org
Subject: Re: [DISCUSS] Releasing initial Connectors support
Hi Csaba,
Thanks for taking a look!
While Connectors will indeed appear in the flow definition (flow.json.gz), this should not have any backward compatibility impact for users who are not using this new concept. The only breaking changes we anticipate would be limited to the Connector API and its related constructs (such as the Connector-specific portions of the flow definition and manifest) until we gain enough confidence through concrete implementations to mark the API as stable. Existing flows, processors, controller services, and other components would remain unaffected.
Regarding a NiFi 3.0 release, I do not anticipate that the addition of the Connectors concept alone would motivate a major version bump, as it should not introduce breaking changes to the existing framework or user experience. That said, there are other topics on the horizon, such as a comprehensive review of the NiFi authorization model, that would likely require a new major release. But this is still very early and I would not expect anything like that any time soon. In any case, based on how things went with NiFi 2, any major version change would be subject to an extensive community discussion, as we know that major version upgrades always require significant time and resources, and many users are still on NiFi 1.x.
This is, obviously, just my perspective and others may have additional thoughts.
Thanks,
Pierre
Le ven. 20 févr. 2026 à 00:29, Csaba Bejan <be...@gmail.com> a écrit :
>
> Hi Pierre,
>
> Thanks for sharing all these details and thanks to all who worked on it.
> This will be a great addition to the NiFi framework.
>
> I just did a quick scroll through and I saw you mentioned that the API
> is experimental for now and can break going forward until it
> stabilizes. I saw (and anticipate) Manifest and FlowDefinition
> changes, are you expecting these to break as well in a non backward
> compatible manner? I didn't do a proper review yet. I just saw that
> the connector related constructs were introduced.
> As this is a core nifi change would this mean a NiFi 3.0 major version
> bump when it is complete or it is still too early for that discussion.
>
> Thanks,
> Csaba
>
> On Thu, Feb 19, 2026 at 2:22 PM Pierre Villard
> <pi...@gmail.com>
> wrote:
>
> > Hi everyone,
> >
> > I wanted to bring up a discussion around releasing the initial
> > implementation of the Connectors feature that was introduced in
> > NIP-11 [1]. The initial discussion happened on the mailing list back
> > in August 2025 [2], followed by a vote in September 2025 to start
> > the implementation work [3]. Since then, significant progress has
> > been made over the past few months and we believe the feature is
> > ready for an initial release.
> >
> > As a reminder, the Connector concept introduces a new extension
> > point in Apache NiFi that encapsulates a complete NiFi dataflow
> > behind a higher-level, guided configuration experience. Rather than
> > exposing the full complexity of a flow to the user, a Connector
> > presents a structured set of Configuration Steps and Property Groups
> > (for
> > example: "Source Configuration", "Destination Configuration",
> > "Transformations") that allow a user to configure an end-to-end data
> > pipeline without needing to understand or manipulate the underlying
> > flow directly. Under the hood, the Connector is responsible for
> > dynamically building and reconfiguring its encapsulated flow based
> > on the user's configuration choices. This includes managing the
> > lifecycle
> > (starting/stopping) of all components within the flow. The goal is
> > to significantly simplify the user experience for common data
> > movement use cases, making NiFi more accessible to a broader
> > audience while also streamlining operations for experienced practitioners.
> >
> > Given the scope of the feature, the API is marked as experimental
> > and may change in a non-backward compatible manner between minor
> > versions until the API has been more fully validated through
> > developing concrete Connector implementations.All of the development
> > work happened in a feature branch named NIFI-15258 in both the
> > nifi-api [4] and nifi [5] repositories. Here is a high-level summary
> > of what was
> > implemented:
> >
> > 1. NiFi API - The Connector API has been introduced under
> > org.apache.nifi.components.connector. This includes the core
> > Connector interface and an AbstractConnector base class, the
> > ConfigurationStep and ConnectorPropertyGroup abstractions for
> > structuring configuration, ConnectorPropertyDescriptor for defining
> > properties with dependencies and allowable values, a FlowContext
> > abstraction that provides both an active flow and a working flow for
> > configuration verification, component facades (ProcessorFacade,
> > ControllerServiceFacade, ConnectionFacade, ProcessGroupFacade,
> > ParameterContextFacade) for Connectors to interact with their
> > encapsulated flow, Secret references for handling sensitive values,
> > Asset support, ConnectorMethod for exposing custom web endpoints,
> > documentation writers for auto-generating Connector documentation,
> > and updates to the Versioned flow model to accommodate Connectors.
> > Comprehensive unit tests have been added for the API.
> >
> > 2. NiFi Maven NAR Plugin - Support has been added for creating
> > Connector extension manifests in NAR files, so that the NiFi runtime
> > can discover and load Connector extensions. This change is already
> > on the main branch.
> >
> > 3. NiFi (Framework, REST API, UI, Tests) - The bulk of the
> > implementation resides here:
> > - Framework core: ConnectorNode and lifecycle management, connector
> > facades for standalone process groups, a connector configuration
> > repository, flow synchronization support for connectors, parameter
> > context and secrets management integration, and cluster replication
> > support.
> > - REST API: Full set of endpoints for connector CRUD operations,
> > lifecycle management (start/stop), configuration step retrieval and
> > updates, property group management, status reporting, data drainage,
> > connector actions, secrets, state management, verification, and
> > documentation.
> > - Frontend UI: Connectors are integrated into the flow designer, the
> > documentation browser, and access policies.
> > - Mock framework: A connector mock server and test runner to support
> > unit and integration testing of Connector implementations.
> > - Example connector: A Kafka-to-S3 connector implementation that
> > serves as a reference for building Connectors.
> > - CLI Toolkit: A ConnectorClient has been added for command-line
> > interaction with Connectors.
> > - System tests: 11 system test classes covering standalone and
> > clustered scenarios including CRUD, lifecycle, assets, data
> > drainage, parameter contexts, version resolution, and method marshalling.
> >
> > Release plan:
> >
> > We would like to proceed iteratively with three releases:
> > - NiFi Maven NAR Plugin - This has no dependency on the other
> > projects. The Connector manifest support is already on main so we
> > can get started with a release candidate right away. I volunteer to
> > take care of the release duties.
> > - NiFi API - A pull request [6] has been filed to merge the
> > NIFI-15258 feature branch into main. Once merged, we will proceed
> > with a release candidate.
> > - NiFi - Once the NiFi API release is published, we will proceed
> > with the proper PRs (bumping the API version dependency and merging
> > the
> > NIFI-15258 feature branch), and we will look at releasing a new
> > version of NiFi.
> >
> > Please have a look at the feature branches and share your feedback.
> > Any comments, questions, or concerns are welcome.
> >
> > Thanks,
> > Pierre
> >
> > [1]
> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis
> > sues.apache.org%2Fjira%2Fbrowse%2FNIP-11&data=05%7C02%7Cisha.lamboo%
> > 40virtualsciences.nl%7Cb725e0f96b1a4e85fda808de706aaa8c%7C21429da9e4
> > ad45f99a6fcd126a64274b%7C0%7C0%7C639071800600952885%7CUnknown%7CTWFp
> > bGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIs
> > IkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=HGdGVE9GecIowtd9
> > 9VyeB%2B4k9IRhSjfr8RQmDg72arY%3D&reserved=0
> > [2]
> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
> > sts.apache.org%2Fthread%2F77c7o4yf8lt9owpjsjb8gl2slk3f7mcf&data=05%7
> > C02%7Cisha.lamboo%40virtualsciences.nl%7Cb725e0f96b1a4e85fda808de706
> > aaa8c%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C63907180060114245
> > 6%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMC
> > IsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdat
> > a=QwXp5JlLFucTlbzBFsJqC1hVrb3LZoiC4EyY2ewAiQI%3D&reserved=0
> > [3]
> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
> > sts.apache.org%2Fthread%2Fskhn5kjkq6kfmrbqsshxs5348d968jvb&data=05%7
> > C02%7Cisha.lamboo%40virtualsciences.nl%7Cb725e0f96b1a4e85fda808de706
> > aaa8c%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C63907180060115588
> > 9%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMC
> > IsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdat
> > a=cErxoh20l%2BxcP76ObajMq81NqxC9QgIBihiVd3bYepI%3D&reserved=0
> > [4]
> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > thub.com%2Fapache%2Fnifi-api%2Ftree%2FNIFI-15258&data=05%7C02%7Cisha
> > .lamboo%40virtualsciences.nl%7Cb725e0f96b1a4e85fda808de706aaa8c%7C21
> > 429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C639071800601169093%7CUnknow
> > n%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJX
> > aW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=lTxrayDJ
> > b6buwy3XFgICzzL%2FtDizYB6qeCgsgn5pcfU%3D&reserved=0
> > [5]
> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > thub.com%2Fapache%2Fnifi%2Ftree%2FNIFI-15258&data=05%7C02%7Cisha.lam
> > boo%40virtualsciences.nl%7Cb725e0f96b1a4e85fda808de706aaa8c%7C21429d
> > a9e4ad45f99a6fcd126a64274b%7C0%7C0%7C639071800601181591%7CUnknown%7C
> > TWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4z
> > MiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6krwrbZxn3ze
> > Ujs92YUpB9%2F3H%2FZPikmzO7dq3wcD%2BBc%3D&reserved=0
> > [6]
> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > thub.com%2Fapache%2Fnifi-api%2Fpull%2F70&data=05%7C02%7Cisha.lamboo%
> > 40virtualsciences.nl%7Cb725e0f96b1a4e85fda808de706aaa8c%7C21429da9e4
> > ad45f99a6fcd126a64274b%7C0%7C0%7C639071800601193701%7CUnknown%7CTWFp
> > bGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIs
> > IkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=iX6%2B%2BU%2FjGf
> > uMfdf9Gp61MTYg%2B8sCcobJr84640Dvtg4%3D&reserved=0
> >
Re: [DISCUSS] Releasing initial Connectors support
Posted by Pierre Villard <pi...@gmail.com>.
Hi Csaba,
Thanks for taking a look!
While Connectors will indeed appear in the flow definition
(flow.json.gz), this should not have any backward compatibility impact
for users who are not using this new concept. The only breaking
changes we anticipate would be limited to the Connector API and its
related constructs (such as the Connector-specific portions of the
flow definition and manifest) until we gain enough confidence through
concrete implementations to mark the API as stable. Existing flows,
processors, controller services, and other components would remain
unaffected.
Regarding a NiFi 3.0 release, I do not anticipate that the addition of
the Connectors concept alone would motivate a major version bump, as
it should not introduce breaking changes to the existing framework or
user experience. That said, there are other topics on the horizon,
such as a comprehensive review of the NiFi authorization model, that
would likely require a new major release. But this is still very early
and I would not expect anything like that any time soon. In any case,
based on how things went with NiFi 2, any major version change would
be subject to an extensive community discussion, as we know that major
version upgrades always require significant time and resources, and
many users are still on NiFi 1.x.
This is, obviously, just my perspective and others may have additional thoughts.
Thanks,
Pierre
Le ven. 20 févr. 2026 à 00:29, Csaba Bejan <be...@gmail.com> a écrit :
>
> Hi Pierre,
>
> Thanks for sharing all these details and thanks to all who worked on it.
> This will be a great addition to the NiFi framework.
>
> I just did a quick scroll through and I saw you mentioned that the API is
> experimental for now and can break going forward until it stabilizes. I saw
> (and anticipate) Manifest and FlowDefinition changes, are you expecting
> these to break as well in a non backward compatible manner? I didn't do a
> proper review yet. I just saw that the connector related constructs were
> introduced.
> As this is a core nifi change would this mean a NiFi 3.0 major version bump
> when it is complete or it is still too early for that discussion.
>
> Thanks,
> Csaba
>
> On Thu, Feb 19, 2026 at 2:22 PM Pierre Villard <pi...@gmail.com>
> wrote:
>
> > Hi everyone,
> >
> > I wanted to bring up a discussion around releasing the initial
> > implementation of the Connectors feature that was introduced in NIP-11
> > [1]. The initial discussion happened on the mailing list back in
> > August 2025 [2], followed by a vote in September 2025 to start the
> > implementation work [3]. Since then, significant progress has been
> > made over the past few months and we believe the feature is ready for
> > an initial release.
> >
> > As a reminder, the Connector concept introduces a new extension point
> > in Apache NiFi that encapsulates a complete NiFi dataflow behind a
> > higher-level, guided configuration experience. Rather than exposing
> > the full complexity of a flow to the user, a Connector presents a
> > structured set of Configuration Steps and Property Groups (for
> > example: "Source Configuration", "Destination Configuration",
> > "Transformations") that allow a user to configure an end-to-end data
> > pipeline without needing to understand or manipulate the underlying
> > flow directly. Under the hood, the Connector is responsible for
> > dynamically building and reconfiguring its encapsulated flow based on
> > the user's configuration choices. This includes managing the lifecycle
> > (starting/stopping) of all components within the flow. The goal is to
> > significantly simplify the user experience for common data movement
> > use cases, making NiFi more accessible to a broader audience while
> > also streamlining operations for experienced practitioners.
> >
> > Given the scope of the feature, the API is marked as experimental and
> > may change in a non-backward compatible manner between minor versions
> > until the API has been more fully validated through developing
> > concrete Connector implementations.All of the development work
> > happened in a feature branch named NIFI-15258 in both the nifi-api [4]
> > and nifi [5] repositories. Here is a high-level summary of what was
> > implemented:
> >
> > 1. NiFi API - The Connector API has been introduced under
> > org.apache.nifi.components.connector. This includes the core Connector
> > interface and an AbstractConnector base class, the ConfigurationStep
> > and ConnectorPropertyGroup abstractions for structuring configuration,
> > ConnectorPropertyDescriptor for defining properties with dependencies
> > and allowable values, a FlowContext abstraction that provides both an
> > active flow and a working flow for configuration verification,
> > component facades (ProcessorFacade, ControllerServiceFacade,
> > ConnectionFacade, ProcessGroupFacade, ParameterContextFacade) for
> > Connectors to interact with their encapsulated flow, Secret references
> > for handling sensitive values, Asset support, ConnectorMethod for
> > exposing custom web endpoints, documentation writers for
> > auto-generating Connector documentation, and updates to the Versioned
> > flow model to accommodate Connectors. Comprehensive unit tests have
> > been added for the API.
> >
> > 2. NiFi Maven NAR Plugin - Support has been added for creating
> > Connector extension manifests in NAR files, so that the NiFi runtime
> > can discover and load Connector extensions. This change is already on
> > the main branch.
> >
> > 3. NiFi (Framework, REST API, UI, Tests) - The bulk of the
> > implementation resides here:
> > - Framework core: ConnectorNode and lifecycle management, connector
> > facades for standalone process groups, a connector configuration
> > repository, flow synchronization support for connectors, parameter
> > context and secrets management integration, and cluster replication
> > support.
> > - REST API: Full set of endpoints for connector CRUD operations,
> > lifecycle management (start/stop), configuration step retrieval and
> > updates, property group management, status reporting, data drainage,
> > connector actions, secrets, state management, verification, and
> > documentation.
> > - Frontend UI: Connectors are integrated into the flow designer, the
> > documentation browser, and access policies.
> > - Mock framework: A connector mock server and test runner to support
> > unit and integration testing of Connector implementations.
> > - Example connector: A Kafka-to-S3 connector implementation that
> > serves as a reference for building Connectors.
> > - CLI Toolkit: A ConnectorClient has been added for command-line
> > interaction with Connectors.
> > - System tests: 11 system test classes covering standalone and
> > clustered scenarios including CRUD, lifecycle, assets, data drainage,
> > parameter contexts, version resolution, and method marshalling.
> >
> > Release plan:
> >
> > We would like to proceed iteratively with three releases:
> > - NiFi Maven NAR Plugin - This has no dependency on the other
> > projects. The Connector manifest support is already on main so we can
> > get started with a release candidate right away. I volunteer to take
> > care of the release duties.
> > - NiFi API - A pull request [6] has been filed to merge the NIFI-15258
> > feature branch into main. Once merged, we will proceed with a release
> > candidate.
> > - NiFi - Once the NiFi API release is published, we will proceed with
> > the proper PRs (bumping the API version dependency and merging the
> > NIFI-15258 feature branch), and we will look at releasing a new
> > version of NiFi.
> >
> > Please have a look at the feature branches and share your feedback.
> > Any comments, questions, or concerns are welcome.
> >
> > Thanks,
> > Pierre
> >
> > [1] https://issues.apache.org/jira/browse/NIP-11
> > [2] https://lists.apache.org/thread/77c7o4yf8lt9owpjsjb8gl2slk3f7mcf
> > [3] https://lists.apache.org/thread/skhn5kjkq6kfmrbqsshxs5348d968jvb
> > [4] https://github.com/apache/nifi-api/tree/NIFI-15258
> > [5] https://github.com/apache/nifi/tree/NIFI-15258
> > [6] https://github.com/apache/nifi-api/pull/70
> >
Re: [DISCUSS] Releasing initial Connectors support
Posted by Csaba Bejan <be...@gmail.com>.
Hi Pierre,
Thanks for sharing all these details and thanks to all who worked on it.
This will be a great addition to the NiFi framework.
I just did a quick scroll through and I saw you mentioned that the API is
experimental for now and can break going forward until it stabilizes. I saw
(and anticipate) Manifest and FlowDefinition changes, are you expecting
these to break as well in a non backward compatible manner? I didn't do a
proper review yet. I just saw that the connector related constructs were
introduced.
As this is a core nifi change would this mean a NiFi 3.0 major version bump
when it is complete or it is still too early for that discussion.
Thanks,
Csaba
On Thu, Feb 19, 2026 at 2:22 PM Pierre Villard <pi...@gmail.com>
wrote:
> Hi everyone,
>
> I wanted to bring up a discussion around releasing the initial
> implementation of the Connectors feature that was introduced in NIP-11
> [1]. The initial discussion happened on the mailing list back in
> August 2025 [2], followed by a vote in September 2025 to start the
> implementation work [3]. Since then, significant progress has been
> made over the past few months and we believe the feature is ready for
> an initial release.
>
> As a reminder, the Connector concept introduces a new extension point
> in Apache NiFi that encapsulates a complete NiFi dataflow behind a
> higher-level, guided configuration experience. Rather than exposing
> the full complexity of a flow to the user, a Connector presents a
> structured set of Configuration Steps and Property Groups (for
> example: "Source Configuration", "Destination Configuration",
> "Transformations") that allow a user to configure an end-to-end data
> pipeline without needing to understand or manipulate the underlying
> flow directly. Under the hood, the Connector is responsible for
> dynamically building and reconfiguring its encapsulated flow based on
> the user's configuration choices. This includes managing the lifecycle
> (starting/stopping) of all components within the flow. The goal is to
> significantly simplify the user experience for common data movement
> use cases, making NiFi more accessible to a broader audience while
> also streamlining operations for experienced practitioners.
>
> Given the scope of the feature, the API is marked as experimental and
> may change in a non-backward compatible manner between minor versions
> until the API has been more fully validated through developing
> concrete Connector implementations.All of the development work
> happened in a feature branch named NIFI-15258 in both the nifi-api [4]
> and nifi [5] repositories. Here is a high-level summary of what was
> implemented:
>
> 1. NiFi API - The Connector API has been introduced under
> org.apache.nifi.components.connector. This includes the core Connector
> interface and an AbstractConnector base class, the ConfigurationStep
> and ConnectorPropertyGroup abstractions for structuring configuration,
> ConnectorPropertyDescriptor for defining properties with dependencies
> and allowable values, a FlowContext abstraction that provides both an
> active flow and a working flow for configuration verification,
> component facades (ProcessorFacade, ControllerServiceFacade,
> ConnectionFacade, ProcessGroupFacade, ParameterContextFacade) for
> Connectors to interact with their encapsulated flow, Secret references
> for handling sensitive values, Asset support, ConnectorMethod for
> exposing custom web endpoints, documentation writers for
> auto-generating Connector documentation, and updates to the Versioned
> flow model to accommodate Connectors. Comprehensive unit tests have
> been added for the API.
>
> 2. NiFi Maven NAR Plugin - Support has been added for creating
> Connector extension manifests in NAR files, so that the NiFi runtime
> can discover and load Connector extensions. This change is already on
> the main branch.
>
> 3. NiFi (Framework, REST API, UI, Tests) - The bulk of the
> implementation resides here:
> - Framework core: ConnectorNode and lifecycle management, connector
> facades for standalone process groups, a connector configuration
> repository, flow synchronization support for connectors, parameter
> context and secrets management integration, and cluster replication
> support.
> - REST API: Full set of endpoints for connector CRUD operations,
> lifecycle management (start/stop), configuration step retrieval and
> updates, property group management, status reporting, data drainage,
> connector actions, secrets, state management, verification, and
> documentation.
> - Frontend UI: Connectors are integrated into the flow designer, the
> documentation browser, and access policies.
> - Mock framework: A connector mock server and test runner to support
> unit and integration testing of Connector implementations.
> - Example connector: A Kafka-to-S3 connector implementation that
> serves as a reference for building Connectors.
> - CLI Toolkit: A ConnectorClient has been added for command-line
> interaction with Connectors.
> - System tests: 11 system test classes covering standalone and
> clustered scenarios including CRUD, lifecycle, assets, data drainage,
> parameter contexts, version resolution, and method marshalling.
>
> Release plan:
>
> We would like to proceed iteratively with three releases:
> - NiFi Maven NAR Plugin - This has no dependency on the other
> projects. The Connector manifest support is already on main so we can
> get started with a release candidate right away. I volunteer to take
> care of the release duties.
> - NiFi API - A pull request [6] has been filed to merge the NIFI-15258
> feature branch into main. Once merged, we will proceed with a release
> candidate.
> - NiFi - Once the NiFi API release is published, we will proceed with
> the proper PRs (bumping the API version dependency and merging the
> NIFI-15258 feature branch), and we will look at releasing a new
> version of NiFi.
>
> Please have a look at the feature branches and share your feedback.
> Any comments, questions, or concerns are welcome.
>
> Thanks,
> Pierre
>
> [1] https://issues.apache.org/jira/browse/NIP-11
> [2] https://lists.apache.org/thread/77c7o4yf8lt9owpjsjb8gl2slk3f7mcf
> [3] https://lists.apache.org/thread/skhn5kjkq6kfmrbqsshxs5348d968jvb
> [4] https://github.com/apache/nifi-api/tree/NIFI-15258
> [5] https://github.com/apache/nifi/tree/NIFI-15258
> [6] https://github.com/apache/nifi-api/pull/70
>