You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by David Handermann <ex...@apache.org> on 2024/08/16 16:59:00 UTC

[DISCUSS] Introducing NiFi Improvement Proposals

Team,

As we wrap up remaining items for NiFi 2, we should consider how to
continue improving the quality and maintainability of the NiFi
ecosystem.

One primary focus of NiFi 2 has been the reduction of technical debt,
which involved the removal of numerous modules and thousands of lines
of code. In that process, it is worth highlighting that the core NiFi
API, and the NiFi Framework API, and the NiFi REST API have had
comparatively few breaking changes. This a testament to the quality of
the API design itself. The NiFi Version Schema and API Compatibility
[1] has provided a strong direction thus far.

With that background, we should consider adopting a more formal
process around changes that impact the fundamental API contracts that
NiFi provides. NiFi Feature Proposals [2] have provided elements of
this in the past, but did not include approval requirements. Kafka [3]
and Airflow [4] have more structured improvement proposal processes,
and that is what we should adopt going forward.

Part of moving in this direction requires identifying the areas that
would require going through the Improvement Proposal process itself.
At minimum, this should include the nifi-api [5] module. The
nifi-framework-api [6] is also worth consideration for inclusion in
this category. The NiFi REST API does not have quite the same level of
concern, but may warrant inclusion.

As part of this discussion, we should consider separating the nifi-api
module into its own repository, with its own versioning scheme. This
will provide a helpful distinction in terms of the scope of changes,
and allow the API to be released independently of the application,
providing strong version compatibility guarantees.

Based on feedback for the general idea, I would be glad to draft a
NiFi Improvement Proposal page, outlining the recommended steps in
more detail so we can come to consensus on how this should work.

Regards,
David Handermann

[1] https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility
[2] https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
[3] https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
[4] https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
[5] https://github.com/apache/nifi/tree/main/nifi-api
[6] https://github.com/apache/nifi/tree/main/nifi-framework-api

Re: [DISCUSS] Introducing NiFi Improvement Proposals

Posted by David Handermann <ex...@apache.org>.
Thanks again for the feedback, I will be sending out a vote thread soon!

Regards,
David Handermann

On Sat, Aug 31, 2024 at 12:09 PM Bryan Bende <bb...@gmail.com> wrote:
>
> Thanks David. The proposal looks good to me and I definitely like the idea
> of a separate NIP jira project, as well as separating nifi-api to its own
> repo.
>
> On Sat, Aug 31, 2024 at 10:36 AM Joe Witt <jo...@gmail.com> wrote:
>
> > Thanks David.
> >
> > Looks like a solid starting point.
> >
> >
> > On Fri, Aug 30, 2024 at 8:15 PM David Handermann <
> > exceptionfactory@apache.org> wrote:
> >
> > > Team,
> > >
> > > I have drafted an initial version of the NiFi Improvement Proposal
> > Process:
> > >
> > >
> > >
> > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Improvement+Proposal+Process
> > >
> > > The document outlines the scope of project components requiring a
> > > proposal, as well as the requirements for a proposal, and the review
> > > process.
> > >
> > > The general approach for review and approval is designed to be similar
> > > to the release process.
> > >
> > > The requirements for a proposal are similar to other project
> > > improvement proposals.
> > >
> > > I recommend creating a new Jira project for handling Improvement
> > > Proposals, as opposed to using Confluence pages.
> > >
> > > Please review the draft and I will look to call for a vote within the
> > next
> > > week.
> > >
> > > As mentioned earlier, with the nifi-api module being the core contract
> > > for internal and external extensions, I would like to include moving
> > > the nifi-api to a separate Git repository as part of the vote for
> > > instituting the Improvement Proposal Process. If this proves to be too
> > > much of a concern, we can divide the question and vote separately on
> > > the Improvement Proposal Process and moving the nifi-api.
> > >
> > > Thanks for the feedback!
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Wed, Aug 21, 2024 at 11:09 AM David Handermann
> > > <ex...@apache.org> wrote:
> > > >
> > > > Thanks for the replies!
> > > >
> > > > I will move forward with drafting a NiFi Improvements Proposal page to
> > > > keep the discussion going.
> > > >
> > > > Regards,
> > > > David Handermann
> > > >
> > > > On Tue, Aug 20, 2024 at 3:22 PM Mark Payne <ma...@hotmail.com>
> > wrote:
> > > > >
> > > > > David,
> > > > >
> > > > > +1 to all of this. Especially for guaranteeing compatibility and
> > > maintainability, I think instrumenting a more
> > > > > formal approach for updating the API is a step in the right
> > direction.
> > > The huge amount of purging, cleanup,
> > > > > and refactoring that has gone into 2.0 helps to highlight the
> > > importance of ensuring maintainability.
> > > > >
> > > > > Thanks
> > > > > -Mark
> > > > >
> > > > >
> > > > > > On Aug 20, 2024, at 2:00 AM, Joe Witt <jo...@gmail.com> wrote:
> > > > > >
> > > > > > David
> > > > > >
> > > > > > Yeah well written and good points.
> > > > > >
> > > > > > I dont think this says we would relax our existing commitment we
> > > have shown
> > > > > > for things such as the http based api but rather is calling out
> > > specific
> > > > > > things which we will make changes to even more formalized.
> > > > > >
> > > > > > To that end +1 on the concept of proposals and +1 to breaking out
> > the
> > > > > > nifi-api (fundamental designed extension points for flow
> > > development) in
> > > > > > particular.
> > > > > >
> > > > > > Thanks
> > > > > > Joe
> > > > > >
> > > > > > On Fri, Aug 16, 2024 at 8:20 PM Adam Taft <ad...@adamtaft.com>
> > wrote:
> > > > > >
> > > > > >> These are all really appreciated concepts, David. Thank you for
> > > putting the
> > > > > >> thoughts and time in on this!
> > > > > >>
> > > > > >> Regarding this:
> > > > > >>> The NiFi REST API does not have quite the same level of concern,
> > > but may
> > > > > >> warrant inclusion.
> > > > > >>
> > > > > >> I hear what you're saying. However, the REST API (from my
> > > > > >> observation/experience) has gathered quite a number of useful
> > tools
> > > and
> > > > > >> "hacks" for NiFi. Quite often, many different monitoring and
> > > alerting tools
> > > > > >> have been developed against the REST API by third parties and/or
> > > > > >> integrators of NiFi against their internal workflows. Having
> > stable
> > > API
> > > > > >> versioning in the REST API possibly makes just as much sense as
> > > having the
> > > > > >> same for the nifi-api itself. This is a prime entry point for
> > > extensions
> > > > > >> and other features developed alongside NiFi, maybe even the weird
> > > stuff
> > > > > >> that you can't do with the nifi-api directly.
> > > > > >>
> > > > > >> Food for thought of course, but I would hope that we can treat the
> > > REST API
> > > > > >> as a proper first-class citizen in terms of documented versioning.
> > > It turns
> > > > > >> out it's quite a useful means for interacting with a running NiFi
> > > instance.
> > > > > >>
> > > > > >> /Adam
> > > > > >>
> > > > > >> On Fri, Aug 16, 2024 at 9:59 AM David Handermann <
> > > > > >> exceptionfactory@apache.org> wrote:
> > > > > >>
> > > > > >>> Team,
> > > > > >>>
> > > > > >>> As we wrap up remaining items for NiFi 2, we should consider how
> > to
> > > > > >>> continue improving the quality and maintainability of the NiFi
> > > > > >>> ecosystem.
> > > > > >>>
> > > > > >>> One primary focus of NiFi 2 has been the reduction of technical
> > > debt,
> > > > > >>> which involved the removal of numerous modules and thousands of
> > > lines
> > > > > >>> of code. In that process, it is worth highlighting that the core
> > > NiFi
> > > > > >>> API, and the NiFi Framework API, and the NiFi REST API have had
> > > > > >>> comparatively few breaking changes. This a testament to the
> > > quality of
> > > > > >>> the API design itself. The NiFi Version Schema and API
> > > Compatibility
> > > > > >>> [1] has provided a strong direction thus far.
> > > > > >>>
> > > > > >>> With that background, we should consider adopting a more formal
> > > > > >>> process around changes that impact the fundamental API contracts
> > > that
> > > > > >>> NiFi provides. NiFi Feature Proposals [2] have provided elements
> > of
> > > > > >>> this in the past, but did not include approval requirements.
> > Kafka
> > > [3]
> > > > > >>> and Airflow [4] have more structured improvement proposal
> > > processes,
> > > > > >>> and that is what we should adopt going forward.
> > > > > >>>
> > > > > >>> Part of moving in this direction requires identifying the areas
> > > that
> > > > > >>> would require going through the Improvement Proposal process
> > > itself.
> > > > > >>> At minimum, this should include the nifi-api [5] module. The
> > > > > >>> nifi-framework-api [6] is also worth consideration for inclusion
> > in
> > > > > >>> this category. The NiFi REST API does not have quite the same
> > > level of
> > > > > >>> concern, but may warrant inclusion.
> > > > > >>>
> > > > > >>> As part of this discussion, we should consider separating the
> > > nifi-api
> > > > > >>> module into its own repository, with its own versioning scheme.
> > > This
> > > > > >>> will provide a helpful distinction in terms of the scope of
> > > changes,
> > > > > >>> and allow the API to be released independently of the
> > application,
> > > > > >>> providing strong version compatibility guarantees.
> > > > > >>>
> > > > > >>> Based on feedback for the general idea, I would be glad to draft
> > a
> > > > > >>> NiFi Improvement Proposal page, outlining the recommended steps
> > in
> > > > > >>> more detail so we can come to consensus on how this should work.
> > > > > >>>
> > > > > >>> Regards,
> > > > > >>> David Handermann
> > > > > >>>
> > > > > >>> [1]
> > > > > >>>
> > > > > >>
> > >
> > https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility
> > > > > >>> [2]
> > > > > >>>
> > > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
> > > > > >>> [3]
> > > > > >>>
> > > > > >>
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > >>> [4]
> > > > > >>>
> > > > > >>
> > >
> > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
> > > > > >>> [5] https://github.com/apache/nifi/tree/main/nifi-api
> > > > > >>> [6] https://github.com/apache/nifi/tree/main/nifi-framework-api
> > > > > >>>
> > > > > >>
> > > > >
> > >
> >

Re: [DISCUSS] Introducing NiFi Improvement Proposals

Posted by Bryan Bende <bb...@gmail.com>.
Thanks David. The proposal looks good to me and I definitely like the idea
of a separate NIP jira project, as well as separating nifi-api to its own
repo.

On Sat, Aug 31, 2024 at 10:36 AM Joe Witt <jo...@gmail.com> wrote:

> Thanks David.
>
> Looks like a solid starting point.
>
>
> On Fri, Aug 30, 2024 at 8:15 PM David Handermann <
> exceptionfactory@apache.org> wrote:
>
> > Team,
> >
> > I have drafted an initial version of the NiFi Improvement Proposal
> Process:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Improvement+Proposal+Process
> >
> > The document outlines the scope of project components requiring a
> > proposal, as well as the requirements for a proposal, and the review
> > process.
> >
> > The general approach for review and approval is designed to be similar
> > to the release process.
> >
> > The requirements for a proposal are similar to other project
> > improvement proposals.
> >
> > I recommend creating a new Jira project for handling Improvement
> > Proposals, as opposed to using Confluence pages.
> >
> > Please review the draft and I will look to call for a vote within the
> next
> > week.
> >
> > As mentioned earlier, with the nifi-api module being the core contract
> > for internal and external extensions, I would like to include moving
> > the nifi-api to a separate Git repository as part of the vote for
> > instituting the Improvement Proposal Process. If this proves to be too
> > much of a concern, we can divide the question and vote separately on
> > the Improvement Proposal Process and moving the nifi-api.
> >
> > Thanks for the feedback!
> >
> > Regards,
> > David Handermann
> >
> > On Wed, Aug 21, 2024 at 11:09 AM David Handermann
> > <ex...@apache.org> wrote:
> > >
> > > Thanks for the replies!
> > >
> > > I will move forward with drafting a NiFi Improvements Proposal page to
> > > keep the discussion going.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Tue, Aug 20, 2024 at 3:22 PM Mark Payne <ma...@hotmail.com>
> wrote:
> > > >
> > > > David,
> > > >
> > > > +1 to all of this. Especially for guaranteeing compatibility and
> > maintainability, I think instrumenting a more
> > > > formal approach for updating the API is a step in the right
> direction.
> > The huge amount of purging, cleanup,
> > > > and refactoring that has gone into 2.0 helps to highlight the
> > importance of ensuring maintainability.
> > > >
> > > > Thanks
> > > > -Mark
> > > >
> > > >
> > > > > On Aug 20, 2024, at 2:00 AM, Joe Witt <jo...@gmail.com> wrote:
> > > > >
> > > > > David
> > > > >
> > > > > Yeah well written and good points.
> > > > >
> > > > > I dont think this says we would relax our existing commitment we
> > have shown
> > > > > for things such as the http based api but rather is calling out
> > specific
> > > > > things which we will make changes to even more formalized.
> > > > >
> > > > > To that end +1 on the concept of proposals and +1 to breaking out
> the
> > > > > nifi-api (fundamental designed extension points for flow
> > development) in
> > > > > particular.
> > > > >
> > > > > Thanks
> > > > > Joe
> > > > >
> > > > > On Fri, Aug 16, 2024 at 8:20 PM Adam Taft <ad...@adamtaft.com>
> wrote:
> > > > >
> > > > >> These are all really appreciated concepts, David. Thank you for
> > putting the
> > > > >> thoughts and time in on this!
> > > > >>
> > > > >> Regarding this:
> > > > >>> The NiFi REST API does not have quite the same level of concern,
> > but may
> > > > >> warrant inclusion.
> > > > >>
> > > > >> I hear what you're saying. However, the REST API (from my
> > > > >> observation/experience) has gathered quite a number of useful
> tools
> > and
> > > > >> "hacks" for NiFi. Quite often, many different monitoring and
> > alerting tools
> > > > >> have been developed against the REST API by third parties and/or
> > > > >> integrators of NiFi against their internal workflows. Having
> stable
> > API
> > > > >> versioning in the REST API possibly makes just as much sense as
> > having the
> > > > >> same for the nifi-api itself. This is a prime entry point for
> > extensions
> > > > >> and other features developed alongside NiFi, maybe even the weird
> > stuff
> > > > >> that you can't do with the nifi-api directly.
> > > > >>
> > > > >> Food for thought of course, but I would hope that we can treat the
> > REST API
> > > > >> as a proper first-class citizen in terms of documented versioning.
> > It turns
> > > > >> out it's quite a useful means for interacting with a running NiFi
> > instance.
> > > > >>
> > > > >> /Adam
> > > > >>
> > > > >> On Fri, Aug 16, 2024 at 9:59 AM David Handermann <
> > > > >> exceptionfactory@apache.org> wrote:
> > > > >>
> > > > >>> Team,
> > > > >>>
> > > > >>> As we wrap up remaining items for NiFi 2, we should consider how
> to
> > > > >>> continue improving the quality and maintainability of the NiFi
> > > > >>> ecosystem.
> > > > >>>
> > > > >>> One primary focus of NiFi 2 has been the reduction of technical
> > debt,
> > > > >>> which involved the removal of numerous modules and thousands of
> > lines
> > > > >>> of code. In that process, it is worth highlighting that the core
> > NiFi
> > > > >>> API, and the NiFi Framework API, and the NiFi REST API have had
> > > > >>> comparatively few breaking changes. This a testament to the
> > quality of
> > > > >>> the API design itself. The NiFi Version Schema and API
> > Compatibility
> > > > >>> [1] has provided a strong direction thus far.
> > > > >>>
> > > > >>> With that background, we should consider adopting a more formal
> > > > >>> process around changes that impact the fundamental API contracts
> > that
> > > > >>> NiFi provides. NiFi Feature Proposals [2] have provided elements
> of
> > > > >>> this in the past, but did not include approval requirements.
> Kafka
> > [3]
> > > > >>> and Airflow [4] have more structured improvement proposal
> > processes,
> > > > >>> and that is what we should adopt going forward.
> > > > >>>
> > > > >>> Part of moving in this direction requires identifying the areas
> > that
> > > > >>> would require going through the Improvement Proposal process
> > itself.
> > > > >>> At minimum, this should include the nifi-api [5] module. The
> > > > >>> nifi-framework-api [6] is also worth consideration for inclusion
> in
> > > > >>> this category. The NiFi REST API does not have quite the same
> > level of
> > > > >>> concern, but may warrant inclusion.
> > > > >>>
> > > > >>> As part of this discussion, we should consider separating the
> > nifi-api
> > > > >>> module into its own repository, with its own versioning scheme.
> > This
> > > > >>> will provide a helpful distinction in terms of the scope of
> > changes,
> > > > >>> and allow the API to be released independently of the
> application,
> > > > >>> providing strong version compatibility guarantees.
> > > > >>>
> > > > >>> Based on feedback for the general idea, I would be glad to draft
> a
> > > > >>> NiFi Improvement Proposal page, outlining the recommended steps
> in
> > > > >>> more detail so we can come to consensus on how this should work.
> > > > >>>
> > > > >>> Regards,
> > > > >>> David Handermann
> > > > >>>
> > > > >>> [1]
> > > > >>>
> > > > >>
> >
> https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility
> > > > >>> [2]
> > > > >>>
> > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
> > > > >>> [3]
> > > > >>>
> > > > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > >>> [4]
> > > > >>>
> > > > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
> > > > >>> [5] https://github.com/apache/nifi/tree/main/nifi-api
> > > > >>> [6] https://github.com/apache/nifi/tree/main/nifi-framework-api
> > > > >>>
> > > > >>
> > > >
> >
>

Re: [DISCUSS] Introducing NiFi Improvement Proposals

Posted by Joe Witt <jo...@gmail.com>.
Thanks David.

Looks like a solid starting point.


On Fri, Aug 30, 2024 at 8:15 PM David Handermann <
exceptionfactory@apache.org> wrote:

> Team,
>
> I have drafted an initial version of the NiFi Improvement Proposal Process:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Improvement+Proposal+Process
>
> The document outlines the scope of project components requiring a
> proposal, as well as the requirements for a proposal, and the review
> process.
>
> The general approach for review and approval is designed to be similar
> to the release process.
>
> The requirements for a proposal are similar to other project
> improvement proposals.
>
> I recommend creating a new Jira project for handling Improvement
> Proposals, as opposed to using Confluence pages.
>
> Please review the draft and I will look to call for a vote within the next
> week.
>
> As mentioned earlier, with the nifi-api module being the core contract
> for internal and external extensions, I would like to include moving
> the nifi-api to a separate Git repository as part of the vote for
> instituting the Improvement Proposal Process. If this proves to be too
> much of a concern, we can divide the question and vote separately on
> the Improvement Proposal Process and moving the nifi-api.
>
> Thanks for the feedback!
>
> Regards,
> David Handermann
>
> On Wed, Aug 21, 2024 at 11:09 AM David Handermann
> <ex...@apache.org> wrote:
> >
> > Thanks for the replies!
> >
> > I will move forward with drafting a NiFi Improvements Proposal page to
> > keep the discussion going.
> >
> > Regards,
> > David Handermann
> >
> > On Tue, Aug 20, 2024 at 3:22 PM Mark Payne <ma...@hotmail.com> wrote:
> > >
> > > David,
> > >
> > > +1 to all of this. Especially for guaranteeing compatibility and
> maintainability, I think instrumenting a more
> > > formal approach for updating the API is a step in the right direction.
> The huge amount of purging, cleanup,
> > > and refactoring that has gone into 2.0 helps to highlight the
> importance of ensuring maintainability.
> > >
> > > Thanks
> > > -Mark
> > >
> > >
> > > > On Aug 20, 2024, at 2:00 AM, Joe Witt <jo...@gmail.com> wrote:
> > > >
> > > > David
> > > >
> > > > Yeah well written and good points.
> > > >
> > > > I dont think this says we would relax our existing commitment we
> have shown
> > > > for things such as the http based api but rather is calling out
> specific
> > > > things which we will make changes to even more formalized.
> > > >
> > > > To that end +1 on the concept of proposals and +1 to breaking out the
> > > > nifi-api (fundamental designed extension points for flow
> development) in
> > > > particular.
> > > >
> > > > Thanks
> > > > Joe
> > > >
> > > > On Fri, Aug 16, 2024 at 8:20 PM Adam Taft <ad...@adamtaft.com> wrote:
> > > >
> > > >> These are all really appreciated concepts, David. Thank you for
> putting the
> > > >> thoughts and time in on this!
> > > >>
> > > >> Regarding this:
> > > >>> The NiFi REST API does not have quite the same level of concern,
> but may
> > > >> warrant inclusion.
> > > >>
> > > >> I hear what you're saying. However, the REST API (from my
> > > >> observation/experience) has gathered quite a number of useful tools
> and
> > > >> "hacks" for NiFi. Quite often, many different monitoring and
> alerting tools
> > > >> have been developed against the REST API by third parties and/or
> > > >> integrators of NiFi against their internal workflows. Having stable
> API
> > > >> versioning in the REST API possibly makes just as much sense as
> having the
> > > >> same for the nifi-api itself. This is a prime entry point for
> extensions
> > > >> and other features developed alongside NiFi, maybe even the weird
> stuff
> > > >> that you can't do with the nifi-api directly.
> > > >>
> > > >> Food for thought of course, but I would hope that we can treat the
> REST API
> > > >> as a proper first-class citizen in terms of documented versioning.
> It turns
> > > >> out it's quite a useful means for interacting with a running NiFi
> instance.
> > > >>
> > > >> /Adam
> > > >>
> > > >> On Fri, Aug 16, 2024 at 9:59 AM David Handermann <
> > > >> exceptionfactory@apache.org> wrote:
> > > >>
> > > >>> Team,
> > > >>>
> > > >>> As we wrap up remaining items for NiFi 2, we should consider how to
> > > >>> continue improving the quality and maintainability of the NiFi
> > > >>> ecosystem.
> > > >>>
> > > >>> One primary focus of NiFi 2 has been the reduction of technical
> debt,
> > > >>> which involved the removal of numerous modules and thousands of
> lines
> > > >>> of code. In that process, it is worth highlighting that the core
> NiFi
> > > >>> API, and the NiFi Framework API, and the NiFi REST API have had
> > > >>> comparatively few breaking changes. This a testament to the
> quality of
> > > >>> the API design itself. The NiFi Version Schema and API
> Compatibility
> > > >>> [1] has provided a strong direction thus far.
> > > >>>
> > > >>> With that background, we should consider adopting a more formal
> > > >>> process around changes that impact the fundamental API contracts
> that
> > > >>> NiFi provides. NiFi Feature Proposals [2] have provided elements of
> > > >>> this in the past, but did not include approval requirements. Kafka
> [3]
> > > >>> and Airflow [4] have more structured improvement proposal
> processes,
> > > >>> and that is what we should adopt going forward.
> > > >>>
> > > >>> Part of moving in this direction requires identifying the areas
> that
> > > >>> would require going through the Improvement Proposal process
> itself.
> > > >>> At minimum, this should include the nifi-api [5] module. The
> > > >>> nifi-framework-api [6] is also worth consideration for inclusion in
> > > >>> this category. The NiFi REST API does not have quite the same
> level of
> > > >>> concern, but may warrant inclusion.
> > > >>>
> > > >>> As part of this discussion, we should consider separating the
> nifi-api
> > > >>> module into its own repository, with its own versioning scheme.
> This
> > > >>> will provide a helpful distinction in terms of the scope of
> changes,
> > > >>> and allow the API to be released independently of the application,
> > > >>> providing strong version compatibility guarantees.
> > > >>>
> > > >>> Based on feedback for the general idea, I would be glad to draft a
> > > >>> NiFi Improvement Proposal page, outlining the recommended steps in
> > > >>> more detail so we can come to consensus on how this should work.
> > > >>>
> > > >>> Regards,
> > > >>> David Handermann
> > > >>>
> > > >>> [1]
> > > >>>
> > > >>
> https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility
> > > >>> [2]
> > > >>>
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
> > > >>> [3]
> > > >>>
> > > >>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > >>> [4]
> > > >>>
> > > >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
> > > >>> [5] https://github.com/apache/nifi/tree/main/nifi-api
> > > >>> [6] https://github.com/apache/nifi/tree/main/nifi-framework-api
> > > >>>
> > > >>
> > >
>

Re: [DISCUSS] Introducing NiFi Improvement Proposals

Posted by David Handermann <ex...@apache.org>.
Team,

I have drafted an initial version of the NiFi Improvement Proposal Process:

https://cwiki.apache.org/confluence/display/NIFI/NiFi+Improvement+Proposal+Process

The document outlines the scope of project components requiring a
proposal, as well as the requirements for a proposal, and the review
process.

The general approach for review and approval is designed to be similar
to the release process.

The requirements for a proposal are similar to other project
improvement proposals.

I recommend creating a new Jira project for handling Improvement
Proposals, as opposed to using Confluence pages.

Please review the draft and I will look to call for a vote within the next week.

As mentioned earlier, with the nifi-api module being the core contract
for internal and external extensions, I would like to include moving
the nifi-api to a separate Git repository as part of the vote for
instituting the Improvement Proposal Process. If this proves to be too
much of a concern, we can divide the question and vote separately on
the Improvement Proposal Process and moving the nifi-api.

Thanks for the feedback!

Regards,
David Handermann

On Wed, Aug 21, 2024 at 11:09 AM David Handermann
<ex...@apache.org> wrote:
>
> Thanks for the replies!
>
> I will move forward with drafting a NiFi Improvements Proposal page to
> keep the discussion going.
>
> Regards,
> David Handermann
>
> On Tue, Aug 20, 2024 at 3:22 PM Mark Payne <ma...@hotmail.com> wrote:
> >
> > David,
> >
> > +1 to all of this. Especially for guaranteeing compatibility and maintainability, I think instrumenting a more
> > formal approach for updating the API is a step in the right direction. The huge amount of purging, cleanup,
> > and refactoring that has gone into 2.0 helps to highlight the importance of ensuring maintainability.
> >
> > Thanks
> > -Mark
> >
> >
> > > On Aug 20, 2024, at 2:00 AM, Joe Witt <jo...@gmail.com> wrote:
> > >
> > > David
> > >
> > > Yeah well written and good points.
> > >
> > > I dont think this says we would relax our existing commitment we have shown
> > > for things such as the http based api but rather is calling out specific
> > > things which we will make changes to even more formalized.
> > >
> > > To that end +1 on the concept of proposals and +1 to breaking out the
> > > nifi-api (fundamental designed extension points for flow development) in
> > > particular.
> > >
> > > Thanks
> > > Joe
> > >
> > > On Fri, Aug 16, 2024 at 8:20 PM Adam Taft <ad...@adamtaft.com> wrote:
> > >
> > >> These are all really appreciated concepts, David. Thank you for putting the
> > >> thoughts and time in on this!
> > >>
> > >> Regarding this:
> > >>> The NiFi REST API does not have quite the same level of concern, but may
> > >> warrant inclusion.
> > >>
> > >> I hear what you're saying. However, the REST API (from my
> > >> observation/experience) has gathered quite a number of useful tools and
> > >> "hacks" for NiFi. Quite often, many different monitoring and alerting tools
> > >> have been developed against the REST API by third parties and/or
> > >> integrators of NiFi against their internal workflows. Having stable API
> > >> versioning in the REST API possibly makes just as much sense as having the
> > >> same for the nifi-api itself. This is a prime entry point for extensions
> > >> and other features developed alongside NiFi, maybe even the weird stuff
> > >> that you can't do with the nifi-api directly.
> > >>
> > >> Food for thought of course, but I would hope that we can treat the REST API
> > >> as a proper first-class citizen in terms of documented versioning. It turns
> > >> out it's quite a useful means for interacting with a running NiFi instance.
> > >>
> > >> /Adam
> > >>
> > >> On Fri, Aug 16, 2024 at 9:59 AM David Handermann <
> > >> exceptionfactory@apache.org> wrote:
> > >>
> > >>> Team,
> > >>>
> > >>> As we wrap up remaining items for NiFi 2, we should consider how to
> > >>> continue improving the quality and maintainability of the NiFi
> > >>> ecosystem.
> > >>>
> > >>> One primary focus of NiFi 2 has been the reduction of technical debt,
> > >>> which involved the removal of numerous modules and thousands of lines
> > >>> of code. In that process, it is worth highlighting that the core NiFi
> > >>> API, and the NiFi Framework API, and the NiFi REST API have had
> > >>> comparatively few breaking changes. This a testament to the quality of
> > >>> the API design itself. The NiFi Version Schema and API Compatibility
> > >>> [1] has provided a strong direction thus far.
> > >>>
> > >>> With that background, we should consider adopting a more formal
> > >>> process around changes that impact the fundamental API contracts that
> > >>> NiFi provides. NiFi Feature Proposals [2] have provided elements of
> > >>> this in the past, but did not include approval requirements. Kafka [3]
> > >>> and Airflow [4] have more structured improvement proposal processes,
> > >>> and that is what we should adopt going forward.
> > >>>
> > >>> Part of moving in this direction requires identifying the areas that
> > >>> would require going through the Improvement Proposal process itself.
> > >>> At minimum, this should include the nifi-api [5] module. The
> > >>> nifi-framework-api [6] is also worth consideration for inclusion in
> > >>> this category. The NiFi REST API does not have quite the same level of
> > >>> concern, but may warrant inclusion.
> > >>>
> > >>> As part of this discussion, we should consider separating the nifi-api
> > >>> module into its own repository, with its own versioning scheme. This
> > >>> will provide a helpful distinction in terms of the scope of changes,
> > >>> and allow the API to be released independently of the application,
> > >>> providing strong version compatibility guarantees.
> > >>>
> > >>> Based on feedback for the general idea, I would be glad to draft a
> > >>> NiFi Improvement Proposal page, outlining the recommended steps in
> > >>> more detail so we can come to consensus on how this should work.
> > >>>
> > >>> Regards,
> > >>> David Handermann
> > >>>
> > >>> [1]
> > >>>
> > >> https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility
> > >>> [2]
> > >>> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
> > >>> [3]
> > >>>
> > >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > >>> [4]
> > >>>
> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
> > >>> [5] https://github.com/apache/nifi/tree/main/nifi-api
> > >>> [6] https://github.com/apache/nifi/tree/main/nifi-framework-api
> > >>>
> > >>
> >

Re: [DISCUSS] Introducing NiFi Improvement Proposals

Posted by David Handermann <ex...@apache.org>.
Thanks for the replies!

I will move forward with drafting a NiFi Improvements Proposal page to
keep the discussion going.

Regards,
David Handermann

On Tue, Aug 20, 2024 at 3:22 PM Mark Payne <ma...@hotmail.com> wrote:
>
> David,
>
> +1 to all of this. Especially for guaranteeing compatibility and maintainability, I think instrumenting a more
> formal approach for updating the API is a step in the right direction. The huge amount of purging, cleanup,
> and refactoring that has gone into 2.0 helps to highlight the importance of ensuring maintainability.
>
> Thanks
> -Mark
>
>
> > On Aug 20, 2024, at 2:00 AM, Joe Witt <jo...@gmail.com> wrote:
> >
> > David
> >
> > Yeah well written and good points.
> >
> > I dont think this says we would relax our existing commitment we have shown
> > for things such as the http based api but rather is calling out specific
> > things which we will make changes to even more formalized.
> >
> > To that end +1 on the concept of proposals and +1 to breaking out the
> > nifi-api (fundamental designed extension points for flow development) in
> > particular.
> >
> > Thanks
> > Joe
> >
> > On Fri, Aug 16, 2024 at 8:20 PM Adam Taft <ad...@adamtaft.com> wrote:
> >
> >> These are all really appreciated concepts, David. Thank you for putting the
> >> thoughts and time in on this!
> >>
> >> Regarding this:
> >>> The NiFi REST API does not have quite the same level of concern, but may
> >> warrant inclusion.
> >>
> >> I hear what you're saying. However, the REST API (from my
> >> observation/experience) has gathered quite a number of useful tools and
> >> "hacks" for NiFi. Quite often, many different monitoring and alerting tools
> >> have been developed against the REST API by third parties and/or
> >> integrators of NiFi against their internal workflows. Having stable API
> >> versioning in the REST API possibly makes just as much sense as having the
> >> same for the nifi-api itself. This is a prime entry point for extensions
> >> and other features developed alongside NiFi, maybe even the weird stuff
> >> that you can't do with the nifi-api directly.
> >>
> >> Food for thought of course, but I would hope that we can treat the REST API
> >> as a proper first-class citizen in terms of documented versioning. It turns
> >> out it's quite a useful means for interacting with a running NiFi instance.
> >>
> >> /Adam
> >>
> >> On Fri, Aug 16, 2024 at 9:59 AM David Handermann <
> >> exceptionfactory@apache.org> wrote:
> >>
> >>> Team,
> >>>
> >>> As we wrap up remaining items for NiFi 2, we should consider how to
> >>> continue improving the quality and maintainability of the NiFi
> >>> ecosystem.
> >>>
> >>> One primary focus of NiFi 2 has been the reduction of technical debt,
> >>> which involved the removal of numerous modules and thousands of lines
> >>> of code. In that process, it is worth highlighting that the core NiFi
> >>> API, and the NiFi Framework API, and the NiFi REST API have had
> >>> comparatively few breaking changes. This a testament to the quality of
> >>> the API design itself. The NiFi Version Schema and API Compatibility
> >>> [1] has provided a strong direction thus far.
> >>>
> >>> With that background, we should consider adopting a more formal
> >>> process around changes that impact the fundamental API contracts that
> >>> NiFi provides. NiFi Feature Proposals [2] have provided elements of
> >>> this in the past, but did not include approval requirements. Kafka [3]
> >>> and Airflow [4] have more structured improvement proposal processes,
> >>> and that is what we should adopt going forward.
> >>>
> >>> Part of moving in this direction requires identifying the areas that
> >>> would require going through the Improvement Proposal process itself.
> >>> At minimum, this should include the nifi-api [5] module. The
> >>> nifi-framework-api [6] is also worth consideration for inclusion in
> >>> this category. The NiFi REST API does not have quite the same level of
> >>> concern, but may warrant inclusion.
> >>>
> >>> As part of this discussion, we should consider separating the nifi-api
> >>> module into its own repository, with its own versioning scheme. This
> >>> will provide a helpful distinction in terms of the scope of changes,
> >>> and allow the API to be released independently of the application,
> >>> providing strong version compatibility guarantees.
> >>>
> >>> Based on feedback for the general idea, I would be glad to draft a
> >>> NiFi Improvement Proposal page, outlining the recommended steps in
> >>> more detail so we can come to consensus on how this should work.
> >>>
> >>> Regards,
> >>> David Handermann
> >>>
> >>> [1]
> >>>
> >> https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility
> >>> [2]
> >>> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
> >>> [3]
> >>>
> >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >>> [4]
> >>>
> >> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
> >>> [5] https://github.com/apache/nifi/tree/main/nifi-api
> >>> [6] https://github.com/apache/nifi/tree/main/nifi-framework-api
> >>>
> >>
>

Re: [DISCUSS] Introducing NiFi Improvement Proposals

Posted by Mark Payne <ma...@hotmail.com>.
David,

+1 to all of this. Especially for guaranteeing compatibility and maintainability, I think instrumenting a more
formal approach for updating the API is a step in the right direction. The huge amount of purging, cleanup,
and refactoring that has gone into 2.0 helps to highlight the importance of ensuring maintainability.

Thanks
-Mark


> On Aug 20, 2024, at 2:00 AM, Joe Witt <jo...@gmail.com> wrote:
> 
> David
> 
> Yeah well written and good points.
> 
> I dont think this says we would relax our existing commitment we have shown
> for things such as the http based api but rather is calling out specific
> things which we will make changes to even more formalized.
> 
> To that end +1 on the concept of proposals and +1 to breaking out the
> nifi-api (fundamental designed extension points for flow development) in
> particular.
> 
> Thanks
> Joe
> 
> On Fri, Aug 16, 2024 at 8:20 PM Adam Taft <ad...@adamtaft.com> wrote:
> 
>> These are all really appreciated concepts, David. Thank you for putting the
>> thoughts and time in on this!
>> 
>> Regarding this:
>>> The NiFi REST API does not have quite the same level of concern, but may
>> warrant inclusion.
>> 
>> I hear what you're saying. However, the REST API (from my
>> observation/experience) has gathered quite a number of useful tools and
>> "hacks" for NiFi. Quite often, many different monitoring and alerting tools
>> have been developed against the REST API by third parties and/or
>> integrators of NiFi against their internal workflows. Having stable API
>> versioning in the REST API possibly makes just as much sense as having the
>> same for the nifi-api itself. This is a prime entry point for extensions
>> and other features developed alongside NiFi, maybe even the weird stuff
>> that you can't do with the nifi-api directly.
>> 
>> Food for thought of course, but I would hope that we can treat the REST API
>> as a proper first-class citizen in terms of documented versioning. It turns
>> out it's quite a useful means for interacting with a running NiFi instance.
>> 
>> /Adam
>> 
>> On Fri, Aug 16, 2024 at 9:59 AM David Handermann <
>> exceptionfactory@apache.org> wrote:
>> 
>>> Team,
>>> 
>>> As we wrap up remaining items for NiFi 2, we should consider how to
>>> continue improving the quality and maintainability of the NiFi
>>> ecosystem.
>>> 
>>> One primary focus of NiFi 2 has been the reduction of technical debt,
>>> which involved the removal of numerous modules and thousands of lines
>>> of code. In that process, it is worth highlighting that the core NiFi
>>> API, and the NiFi Framework API, and the NiFi REST API have had
>>> comparatively few breaking changes. This a testament to the quality of
>>> the API design itself. The NiFi Version Schema and API Compatibility
>>> [1] has provided a strong direction thus far.
>>> 
>>> With that background, we should consider adopting a more formal
>>> process around changes that impact the fundamental API contracts that
>>> NiFi provides. NiFi Feature Proposals [2] have provided elements of
>>> this in the past, but did not include approval requirements. Kafka [3]
>>> and Airflow [4] have more structured improvement proposal processes,
>>> and that is what we should adopt going forward.
>>> 
>>> Part of moving in this direction requires identifying the areas that
>>> would require going through the Improvement Proposal process itself.
>>> At minimum, this should include the nifi-api [5] module. The
>>> nifi-framework-api [6] is also worth consideration for inclusion in
>>> this category. The NiFi REST API does not have quite the same level of
>>> concern, but may warrant inclusion.
>>> 
>>> As part of this discussion, we should consider separating the nifi-api
>>> module into its own repository, with its own versioning scheme. This
>>> will provide a helpful distinction in terms of the scope of changes,
>>> and allow the API to be released independently of the application,
>>> providing strong version compatibility guarantees.
>>> 
>>> Based on feedback for the general idea, I would be glad to draft a
>>> NiFi Improvement Proposal page, outlining the recommended steps in
>>> more detail so we can come to consensus on how this should work.
>>> 
>>> Regards,
>>> David Handermann
>>> 
>>> [1]
>>> 
>> https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility
>>> [2]
>>> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
>>> [3]
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>>> [4]
>>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
>>> [5] https://github.com/apache/nifi/tree/main/nifi-api
>>> [6] https://github.com/apache/nifi/tree/main/nifi-framework-api
>>> 
>> 


Re: [DISCUSS] Introducing NiFi Improvement Proposals

Posted by Mark Payne <ma...@hotmail.com>.
David,

+1 to all of this. Especially for guaranteeing compatibility and maintainability, I think instrumenting a more
formal approach for updating the API is a step in the right direction. The huge amount of purging, cleanup,
and refactoring that has gone into 2.0 helps to highlight the importance of ensuring maintainability.

Thanks
-Mark


> On Aug 20, 2024, at 2:00 AM, Joe Witt <jo...@gmail.com> wrote:
> 
> David
> 
> Yeah well written and good points.
> 
> I dont think this says we would relax our existing commitment we have shown
> for things such as the http based api but rather is calling out specific
> things which we will make changes to even more formalized.
> 
> To that end +1 on the concept of proposals and +1 to breaking out the
> nifi-api (fundamental designed extension points for flow development) in
> particular.
> 
> Thanks
> Joe
> 
> On Fri, Aug 16, 2024 at 8:20 PM Adam Taft <ad...@adamtaft.com> wrote:
> 
>> These are all really appreciated concepts, David. Thank you for putting the
>> thoughts and time in on this!
>> 
>> Regarding this:
>>> The NiFi REST API does not have quite the same level of concern, but may
>> warrant inclusion.
>> 
>> I hear what you're saying. However, the REST API (from my
>> observation/experience) has gathered quite a number of useful tools and
>> "hacks" for NiFi. Quite often, many different monitoring and alerting tools
>> have been developed against the REST API by third parties and/or
>> integrators of NiFi against their internal workflows. Having stable API
>> versioning in the REST API possibly makes just as much sense as having the
>> same for the nifi-api itself. This is a prime entry point for extensions
>> and other features developed alongside NiFi, maybe even the weird stuff
>> that you can't do with the nifi-api directly.
>> 
>> Food for thought of course, but I would hope that we can treat the REST API
>> as a proper first-class citizen in terms of documented versioning. It turns
>> out it's quite a useful means for interacting with a running NiFi instance.
>> 
>> /Adam
>> 
>> On Fri, Aug 16, 2024 at 9:59 AM David Handermann <
>> exceptionfactory@apache.org> wrote:
>> 
>>> Team,
>>> 
>>> As we wrap up remaining items for NiFi 2, we should consider how to
>>> continue improving the quality and maintainability of the NiFi
>>> ecosystem.
>>> 
>>> One primary focus of NiFi 2 has been the reduction of technical debt,
>>> which involved the removal of numerous modules and thousands of lines
>>> of code. In that process, it is worth highlighting that the core NiFi
>>> API, and the NiFi Framework API, and the NiFi REST API have had
>>> comparatively few breaking changes. This a testament to the quality of
>>> the API design itself. The NiFi Version Schema and API Compatibility
>>> [1] has provided a strong direction thus far.
>>> 
>>> With that background, we should consider adopting a more formal
>>> process around changes that impact the fundamental API contracts that
>>> NiFi provides. NiFi Feature Proposals [2] have provided elements of
>>> this in the past, but did not include approval requirements. Kafka [3]
>>> and Airflow [4] have more structured improvement proposal processes,
>>> and that is what we should adopt going forward.
>>> 
>>> Part of moving in this direction requires identifying the areas that
>>> would require going through the Improvement Proposal process itself.
>>> At minimum, this should include the nifi-api [5] module. The
>>> nifi-framework-api [6] is also worth consideration for inclusion in
>>> this category. The NiFi REST API does not have quite the same level of
>>> concern, but may warrant inclusion.
>>> 
>>> As part of this discussion, we should consider separating the nifi-api
>>> module into its own repository, with its own versioning scheme. This
>>> will provide a helpful distinction in terms of the scope of changes,
>>> and allow the API to be released independently of the application,
>>> providing strong version compatibility guarantees.
>>> 
>>> Based on feedback for the general idea, I would be glad to draft a
>>> NiFi Improvement Proposal page, outlining the recommended steps in
>>> more detail so we can come to consensus on how this should work.
>>> 
>>> Regards,
>>> David Handermann
>>> 
>>> [1]
>>> 
>> https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility
>>> [2]
>>> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
>>> [3]
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>>> [4]
>>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
>>> [5] https://github.com/apache/nifi/tree/main/nifi-api
>>> [6] https://github.com/apache/nifi/tree/main/nifi-framework-api
>>> 
>> 


Re: [DISCUSS] Introducing NiFi Improvement Proposals

Posted by Joe Witt <jo...@gmail.com>.
David

Yeah well written and good points.

I dont think this says we would relax our existing commitment we have shown
for things such as the http based api but rather is calling out specific
things which we will make changes to even more formalized.

To that end +1 on the concept of proposals and +1 to breaking out the
nifi-api (fundamental designed extension points for flow development) in
particular.

Thanks
Joe

On Fri, Aug 16, 2024 at 8:20 PM Adam Taft <ad...@adamtaft.com> wrote:

> These are all really appreciated concepts, David. Thank you for putting the
> thoughts and time in on this!
>
> Regarding this:
> > The NiFi REST API does not have quite the same level of concern, but may
> warrant inclusion.
>
> I hear what you're saying. However, the REST API (from my
> observation/experience) has gathered quite a number of useful tools and
> "hacks" for NiFi. Quite often, many different monitoring and alerting tools
> have been developed against the REST API by third parties and/or
> integrators of NiFi against their internal workflows. Having stable API
> versioning in the REST API possibly makes just as much sense as having the
> same for the nifi-api itself. This is a prime entry point for extensions
> and other features developed alongside NiFi, maybe even the weird stuff
> that you can't do with the nifi-api directly.
>
> Food for thought of course, but I would hope that we can treat the REST API
> as a proper first-class citizen in terms of documented versioning. It turns
> out it's quite a useful means for interacting with a running NiFi instance.
>
> /Adam
>
> On Fri, Aug 16, 2024 at 9:59 AM David Handermann <
> exceptionfactory@apache.org> wrote:
>
> > Team,
> >
> > As we wrap up remaining items for NiFi 2, we should consider how to
> > continue improving the quality and maintainability of the NiFi
> > ecosystem.
> >
> > One primary focus of NiFi 2 has been the reduction of technical debt,
> > which involved the removal of numerous modules and thousands of lines
> > of code. In that process, it is worth highlighting that the core NiFi
> > API, and the NiFi Framework API, and the NiFi REST API have had
> > comparatively few breaking changes. This a testament to the quality of
> > the API design itself. The NiFi Version Schema and API Compatibility
> > [1] has provided a strong direction thus far.
> >
> > With that background, we should consider adopting a more formal
> > process around changes that impact the fundamental API contracts that
> > NiFi provides. NiFi Feature Proposals [2] have provided elements of
> > this in the past, but did not include approval requirements. Kafka [3]
> > and Airflow [4] have more structured improvement proposal processes,
> > and that is what we should adopt going forward.
> >
> > Part of moving in this direction requires identifying the areas that
> > would require going through the Improvement Proposal process itself.
> > At minimum, this should include the nifi-api [5] module. The
> > nifi-framework-api [6] is also worth consideration for inclusion in
> > this category. The NiFi REST API does not have quite the same level of
> > concern, but may warrant inclusion.
> >
> > As part of this discussion, we should consider separating the nifi-api
> > module into its own repository, with its own versioning scheme. This
> > will provide a helpful distinction in terms of the scope of changes,
> > and allow the API to be released independently of the application,
> > providing strong version compatibility guarantees.
> >
> > Based on feedback for the general idea, I would be glad to draft a
> > NiFi Improvement Proposal page, outlining the recommended steps in
> > more detail so we can come to consensus on how this should work.
> >
> > Regards,
> > David Handermann
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility
> > [2]
> > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
> > [3]
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > [4]
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
> > [5] https://github.com/apache/nifi/tree/main/nifi-api
> > [6] https://github.com/apache/nifi/tree/main/nifi-framework-api
> >
>

Re: [DISCUSS] Introducing NiFi Improvement Proposals

Posted by Adam Taft <ad...@adamtaft.com>.
These are all really appreciated concepts, David. Thank you for putting the
thoughts and time in on this!

Regarding this:
> The NiFi REST API does not have quite the same level of concern, but may
warrant inclusion.

I hear what you're saying. However, the REST API (from my
observation/experience) has gathered quite a number of useful tools and
"hacks" for NiFi. Quite often, many different monitoring and alerting tools
have been developed against the REST API by third parties and/or
integrators of NiFi against their internal workflows. Having stable API
versioning in the REST API possibly makes just as much sense as having the
same for the nifi-api itself. This is a prime entry point for extensions
and other features developed alongside NiFi, maybe even the weird stuff
that you can't do with the nifi-api directly.

Food for thought of course, but I would hope that we can treat the REST API
as a proper first-class citizen in terms of documented versioning. It turns
out it's quite a useful means for interacting with a running NiFi instance.

/Adam

On Fri, Aug 16, 2024 at 9:59 AM David Handermann <
exceptionfactory@apache.org> wrote:

> Team,
>
> As we wrap up remaining items for NiFi 2, we should consider how to
> continue improving the quality and maintainability of the NiFi
> ecosystem.
>
> One primary focus of NiFi 2 has been the reduction of technical debt,
> which involved the removal of numerous modules and thousands of lines
> of code. In that process, it is worth highlighting that the core NiFi
> API, and the NiFi Framework API, and the NiFi REST API have had
> comparatively few breaking changes. This a testament to the quality of
> the API design itself. The NiFi Version Schema and API Compatibility
> [1] has provided a strong direction thus far.
>
> With that background, we should consider adopting a more formal
> process around changes that impact the fundamental API contracts that
> NiFi provides. NiFi Feature Proposals [2] have provided elements of
> this in the past, but did not include approval requirements. Kafka [3]
> and Airflow [4] have more structured improvement proposal processes,
> and that is what we should adopt going forward.
>
> Part of moving in this direction requires identifying the areas that
> would require going through the Improvement Proposal process itself.
> At minimum, this should include the nifi-api [5] module. The
> nifi-framework-api [6] is also worth consideration for inclusion in
> this category. The NiFi REST API does not have quite the same level of
> concern, but may warrant inclusion.
>
> As part of this discussion, we should consider separating the nifi-api
> module into its own repository, with its own versioning scheme. This
> will provide a helpful distinction in terms of the scope of changes,
> and allow the API to be released independently of the application,
> providing strong version compatibility guarantees.
>
> Based on feedback for the general idea, I would be glad to draft a
> NiFi Improvement Proposal page, outlining the recommended steps in
> more detail so we can come to consensus on how this should work.
>
> Regards,
> David Handermann
>
> [1]
> https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility
> [2]
> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
> [3]
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> [4]
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
> [5] https://github.com/apache/nifi/tree/main/nifi-api
> [6] https://github.com/apache/nifi/tree/main/nifi-framework-api
>