Thoughts on various ways to help drive NiFi and the community forward

[Joe Witt <jo...@apache.org>]

Team,

We are seeing an increase in the unique contributors on a monthly basis to
NiFi and a clear and steady increase in the overall contribution rate over
the past 1-2 years in particular.  This is awesome but we do have important
problems to cover and we need to make it easier to help newer contributors
know where to jump in and add the most value.

For sure people will contribute various things just because they have a
particular interest such as some new extension, etc..  But from a community
direction perspective there are things we need to continue to improve upon
and we want to help make the path to committership and PMC membership more
clear as a function of adding value to the community.

So not necessarily ordered, here are some key things I've observed that
need more attention and are among the most valuable things for the health
of the nifi system and community.

Dependency Management

This is a hugely important area as we have a massive amount of dependencies
and this work is just plain not that fun.  But if we want to present a
beautiful landscape we need to pull weeds.  We need to reduce dependencies
and keep things recent.  This is largely accomplished today by a very small
group of people.  We have to increase the contribution level here or
perhaps the automation level.  We have tooling that is helping now but
still needs more attention.

Vulnerability Management particular as it relates to dependencies

Related to dependency management but broader.  This one at times requires
us to do more than simply maintain dependencies.  It sometimes forces us to
change libraries and sometimes even makes maintaining
compatibility/configuration complicated.  These are among the most critical
and most difficult tasks to take on.  This is largely done today by an
extremely small group of people.

Maintenance of extensions

We have a ton of components in NiFi.  Obviously not all of them really do
require active maintenance but there is often a contribute and forget model
involved.  For us to keep these broad range of components under community
care they need active maintenance.  I already mentioned the dependency
factor above but the broader point here is that APIs/versions of systems
evolve.  We need to be staying up to date.  There are systems like Kafka,
Elastic, Mongo, Relational Databases, etc.. that we should always be
supporting the latest on.  These are among the most wildly popular sources
and destinations.  We do tend to fall behind here and this is powerfully
useful for people to contribute to.  These are the things our users
actually use NiFi for!

Maintenance of things other than core nifi

If you look at components ancillary to NiFi they receive far less attention
and effort trending toward them going away.  The NiFi registry is a great
example of this as NiFi MiNiFi Java for instance.  These are things which
are useful and indeed there is a decent user base in both cases.  But they
require more active maintenance than they get.  We need to evolve when it
comes to security methods offered for authn/authz and that takes focus.

Performance analysis and improvements targeted as a result

This is done very rarely as far as I can tell outside a couple of people
who I'm sure we could all name in a few seconds.  The codebase is large and
Java and other parts evolve.  Continually emphasizing faster and more
efficient operations is vital to NiFi's continued growth in the
community at large.  People who spend time running NiFi in a measured
manner with realistic flows and doing things like profiling for CPU,
memory, etc.. and finding bottlenecks and offering solutions - absolutely
can be high impact and valuable.  Even if not at a point where a confident
contribution can be made, the very act of doing this level of evaluation
and flagging concerns is valuable.

Improvements related to tests both unit and integration - but also for
users of NiFi

As part of our push to NiFi 2.0 we have been running these tests far more
often and more complete than ever before.  Github Actions now pretty much
does run all the things.  We've deleted a wild amount of junk tests that
would always fail or were spurious or required some complex local
access/setup and therefore nobody ever ran them.  We need to always improve
coverage but we want to make tests faster/more efficient and more
leveraging of things like test containers would be highly valuable.
Importantly though here I also mean to highlight we need to do a lot more
for users of NiFi that want to create automated tests for their flows.  We
should make it easy for them to provide a flow definition or perhaps a DSL
for their flow and write tests which would leverage actual nifi to validate
a given input results in an expected output.  This is something that would
add a ton of value for our user base but also lead into some really
powerful testing we can do against our own framework.

Development of blogs/videos/conference presentations/social media posts

Building great software by itself doesn't grow the community nearly as much
as if you build a strong developer advocacy game.  People that create these
various forms of blogs/videos and conference talks about NiFi, how it
works, and how to make it better are extremely valuable.  That is often
just as valuable or even more so than a PR review, code contribution, etc..

Engagement in the Slack/Mailing lists

Our community needs help.  They ask a lot of questions about how to use
NiFi, bugs they are hitting, etc.. Responding to these questions in a
timely and meaningful manner is hugely valuable to them and is also pretty
fun for the person who helps out.

Meaningful engagement acting as a pull request reviewer

This one is last but certainly not least!  We have a review-then-commit
model which means we need at least one committer or PMC member to review a
contribution before it gets merged.  Even if you're not a committer or PMC
member, a review is still super helpful and can save time and also puts you
on a great path to earning merit.  Reviews are a critical part of the code
and overall quality of NiFi.  'LGTM' is generally not that helpful and
while warranted for some trivial things we even see it on larger more
meaningful commits.  Not every review needs to be a back and forth affair
of code style inputs either.  There is a middle ground and you'll find it
as you engage in more reviews!    Code reviews are in far more short supply
than code contributions.  I encourage everyone to please do at least one
true pull request review for every contribution.  In general the goal
should be to contribute to more pull request reviews than new pull requests
created.  If you do so this by definition means the community is growing
and you're directly helping make that happen!

Would love to see additional inputs/missing categories/other thoughts.
Perhaps this turns into a wiki page that helps give more context to people
that want to find ways to add value and be on a path to committer and PMC
membership as well.

Thanks
Joe

Re: Thoughts on various ways to help drive NiFi and the community forward

[Matt Burgess <ma...@gmail.com>]

Mathew, certainly! I've been meaning to update the blog too. Email me at
mattyb149@apache.org and we'll get 'em up there :)

Thanks,
Matt

On Sat, Jun 15, 2024 at 2:46 PM Mathew Kiprop <ma...@gmail.com>
wrote:

> +1 to all subjects addressed Joe.
>
> I have been planning to improve developer advocacy around scripting
> components. The groovy scripts have been one of my best tools to build
> flows in NiFi (Rest API development , complex data structures
> transformation, controlled flows scheduling…..). I’ll take this opportunity
> to ask Mat Burges whether I can contribute some of my work to
> http://funnifi.blogspot.com/?m=1.
>
> Mathew
>
> On Sat, 15 Jun 2024 at 21:03, Joe Witt <jo...@apache.org> wrote:
>
> > Team,
> >
> > We are seeing an increase in the unique contributors on a monthly basis
> to
> > NiFi and a clear and steady increase in the overall contribution rate
> over
> > the past 1-2 years in particular.  This is awesome but we do have
> important
> > problems to cover and we need to make it easier to help newer
> contributors
> > know where to jump in and add the most value.
> >
> > For sure people will contribute various things just because they have a
> > particular interest such as some new extension, etc..  But from a
> community
> > direction perspective there are things we need to continue to improve
> upon
> > and we want to help make the path to committership and PMC membership
> more
> > clear as a function of adding value to the community.
> >
> > So not necessarily ordered, here are some key things I've observed that
> > need more attention and are among the most valuable things for the health
> > of the nifi system and community.
> >
> > Dependency Management
> >
> > This is a hugely important area as we have a massive amount of
> dependencies
> > and this work is just plain not that fun.  But if we want to present a
> > beautiful landscape we need to pull weeds.  We need to reduce
> dependencies
> > and keep things recent.  This is largely accomplished today by a very
> small
> > group of people.  We have to increase the contribution level here or
> > perhaps the automation level.  We have tooling that is helping now but
> > still needs more attention.
> >
> > Vulnerability Management particular as it relates to dependencies
> >
> > Related to dependency management but broader.  This one at times requires
> > us to do more than simply maintain dependencies.  It sometimes forces us
> to
> > change libraries and sometimes even makes maintaining
> > compatibility/configuration complicated.  These are among the most
> critical
> > and most difficult tasks to take on.  This is largely done today by an
> > extremely small group of people.
> >
> > Maintenance of extensions
> >
> > We have a ton of components in NiFi.  Obviously not all of them really do
> > require active maintenance but there is often a contribute and forget
> model
> > involved.  For us to keep these broad range of components under community
> > care they need active maintenance.  I already mentioned the dependency
> > factor above but the broader point here is that APIs/versions of systems
> > evolve.  We need to be staying up to date.  There are systems like Kafka,
> > Elastic, Mongo, Relational Databases, etc.. that we should always be
> > supporting the latest on.  These are among the most wildly popular
> sources
> > and destinations.  We do tend to fall behind here and this is powerfully
> > useful for people to contribute to.  These are the things our users
> > actually use NiFi for!
> >
> > Maintenance of things other than core nifi
> >
> > If you look at components ancillary to NiFi they receive far less
> attention
> > and effort trending toward them going away.  The NiFi registry is a great
> > example of this as NiFi MiNiFi Java for instance.  These are things which
> > are useful and indeed there is a decent user base in both cases.  But
> they
> > require more active maintenance than they get.  We need to evolve when it
> > comes to security methods offered for authn/authz and that takes focus.
> >
> > Performance analysis and improvements targeted as a result
> >
> > This is done very rarely as far as I can tell outside a couple of people
> > who I'm sure we could all name in a few seconds.  The codebase is large
> and
> > Java and other parts evolve.  Continually emphasizing faster and more
> > efficient operations is vital to NiFi's continued growth in the
> > community at large.  People who spend time running NiFi in a measured
> > manner with realistic flows and doing things like profiling for CPU,
> > memory, etc.. and finding bottlenecks and offering solutions - absolutely
> > can be high impact and valuable.  Even if not at a point where a
> confident
> > contribution can be made, the very act of doing this level of evaluation
> > and flagging concerns is valuable.
> >
> > Improvements related to tests both unit and integration - but also for
> > users of NiFi
> >
> > As part of our push to NiFi 2.0 we have been running these tests far more
> > often and more complete than ever before.  Github Actions now pretty much
> > does run all the things.  We've deleted a wild amount of junk tests that
> > would always fail or were spurious or required some complex local
> > access/setup and therefore nobody ever ran them.  We need to always
> improve
> > coverage but we want to make tests faster/more efficient and more
> > leveraging of things like test containers would be highly valuable.
> > Importantly though here I also mean to highlight we need to do a lot more
> > for users of NiFi that want to create automated tests for their flows.
> We
> > should make it easy for them to provide a flow definition or perhaps a
> DSL
> > for their flow and write tests which would leverage actual nifi to
> validate
> > a given input results in an expected output.  This is something that
> would
> > add a ton of value for our user base but also lead into some really
> > powerful testing we can do against our own framework.
> >
> > Development of blogs/videos/conference presentations/social media posts
> >
> > Building great software by itself doesn't grow the community nearly as
> much
> > as if you build a strong developer advocacy game.  People that create
> these
> > various forms of blogs/videos and conference talks about NiFi, how it
> > works, and how to make it better are extremely valuable.  That is often
> > just as valuable or even more so than a PR review, code contribution,
> etc..
> >
> > Engagement in the Slack/Mailing lists
> >
> > Our community needs help.  They ask a lot of questions about how to use
> > NiFi, bugs they are hitting, etc.. Responding to these questions in a
> > timely and meaningful manner is hugely valuable to them and is also
> pretty
> > fun for the person who helps out.
> >
> > Meaningful engagement acting as a pull request reviewer
> >
> > This one is last but certainly not least!  We have a review-then-commit
> > model which means we need at least one committer or PMC member to review
> a
> > contribution before it gets merged.  Even if you're not a committer or
> PMC
> > member, a review is still super helpful and can save time and also puts
> you
> > on a great path to earning merit.  Reviews are a critical part of the
> code
> > and overall quality of NiFi.  'LGTM' is generally not that helpful and
> > while warranted for some trivial things we even see it on larger more
> > meaningful commits.  Not every review needs to be a back and forth affair
> > of code style inputs either.  There is a middle ground and you'll find it
> > as you engage in more reviews!    Code reviews are in far more short
> supply
> > than code contributions.  I encourage everyone to please do at least one
> > true pull request review for every contribution.  In general the goal
> > should be to contribute to more pull request reviews than new pull
> requests
> > created.  If you do so this by definition means the community is growing
> > and you're directly helping make that happen!
> >
> > Would love to see additional inputs/missing categories/other thoughts.
> > Perhaps this turns into a wiki page that helps give more context to
> people
> > that want to find ways to add value and be on a path to committer and PMC
> > membership as well.
> >
> > Thanks
> > Joe
> >
>

Re: Thoughts on various ways to help drive NiFi and the community forward

[Mathew Kiprop <ma...@gmail.com>]

+1 to all subjects addressed Joe.

I have been planning to improve developer advocacy around scripting
components. The groovy scripts have been one of my best tools to build
flows in NiFi (Rest API development , complex data structures
transformation, controlled flows scheduling…..). I’ll take this opportunity
to ask Mat Burges whether I can contribute some of my work to
http://funnifi.blogspot.com/?m=1.

Mathew

On Sat, 15 Jun 2024 at 21:03, Joe Witt <jo...@apache.org> wrote:

> Team,
>
> We are seeing an increase in the unique contributors on a monthly basis to
> NiFi and a clear and steady increase in the overall contribution rate over
> the past 1-2 years in particular.  This is awesome but we do have important
> problems to cover and we need to make it easier to help newer contributors
> know where to jump in and add the most value.
>
> For sure people will contribute various things just because they have a
> particular interest such as some new extension, etc..  But from a community
> direction perspective there are things we need to continue to improve upon
> and we want to help make the path to committership and PMC membership more
> clear as a function of adding value to the community.
>
> So not necessarily ordered, here are some key things I've observed that
> need more attention and are among the most valuable things for the health
> of the nifi system and community.
>
> Dependency Management
>
> This is a hugely important area as we have a massive amount of dependencies
> and this work is just plain not that fun.  But if we want to present a
> beautiful landscape we need to pull weeds.  We need to reduce dependencies
> and keep things recent.  This is largely accomplished today by a very small
> group of people.  We have to increase the contribution level here or
> perhaps the automation level.  We have tooling that is helping now but
> still needs more attention.
>
> Vulnerability Management particular as it relates to dependencies
>
> Related to dependency management but broader.  This one at times requires
> us to do more than simply maintain dependencies.  It sometimes forces us to
> change libraries and sometimes even makes maintaining
> compatibility/configuration complicated.  These are among the most critical
> and most difficult tasks to take on.  This is largely done today by an
> extremely small group of people.
>
> Maintenance of extensions
>
> We have a ton of components in NiFi.  Obviously not all of them really do
> require active maintenance but there is often a contribute and forget model
> involved.  For us to keep these broad range of components under community
> care they need active maintenance.  I already mentioned the dependency
> factor above but the broader point here is that APIs/versions of systems
> evolve.  We need to be staying up to date.  There are systems like Kafka,
> Elastic, Mongo, Relational Databases, etc.. that we should always be
> supporting the latest on.  These are among the most wildly popular sources
> and destinations.  We do tend to fall behind here and this is powerfully
> useful for people to contribute to.  These are the things our users
> actually use NiFi for!
>
> Maintenance of things other than core nifi
>
> If you look at components ancillary to NiFi they receive far less attention
> and effort trending toward them going away.  The NiFi registry is a great
> example of this as NiFi MiNiFi Java for instance.  These are things which
> are useful and indeed there is a decent user base in both cases.  But they
> require more active maintenance than they get.  We need to evolve when it
> comes to security methods offered for authn/authz and that takes focus.
>
> Performance analysis and improvements targeted as a result
>
> This is done very rarely as far as I can tell outside a couple of people
> who I'm sure we could all name in a few seconds.  The codebase is large and
> Java and other parts evolve.  Continually emphasizing faster and more
> efficient operations is vital to NiFi's continued growth in the
> community at large.  People who spend time running NiFi in a measured
> manner with realistic flows and doing things like profiling for CPU,
> memory, etc.. and finding bottlenecks and offering solutions - absolutely
> can be high impact and valuable.  Even if not at a point where a confident
> contribution can be made, the very act of doing this level of evaluation
> and flagging concerns is valuable.
>
> Improvements related to tests both unit and integration - but also for
> users of NiFi
>
> As part of our push to NiFi 2.0 we have been running these tests far more
> often and more complete than ever before.  Github Actions now pretty much
> does run all the things.  We've deleted a wild amount of junk tests that
> would always fail or were spurious or required some complex local
> access/setup and therefore nobody ever ran them.  We need to always improve
> coverage but we want to make tests faster/more efficient and more
> leveraging of things like test containers would be highly valuable.
> Importantly though here I also mean to highlight we need to do a lot more
> for users of NiFi that want to create automated tests for their flows.  We
> should make it easy for them to provide a flow definition or perhaps a DSL
> for their flow and write tests which would leverage actual nifi to validate
> a given input results in an expected output.  This is something that would
> add a ton of value for our user base but also lead into some really
> powerful testing we can do against our own framework.
>
> Development of blogs/videos/conference presentations/social media posts
>
> Building great software by itself doesn't grow the community nearly as much
> as if you build a strong developer advocacy game.  People that create these
> various forms of blogs/videos and conference talks about NiFi, how it
> works, and how to make it better are extremely valuable.  That is often
> just as valuable or even more so than a PR review, code contribution, etc..
>
> Engagement in the Slack/Mailing lists
>
> Our community needs help.  They ask a lot of questions about how to use
> NiFi, bugs they are hitting, etc.. Responding to these questions in a
> timely and meaningful manner is hugely valuable to them and is also pretty
> fun for the person who helps out.
>
> Meaningful engagement acting as a pull request reviewer
>
> This one is last but certainly not least!  We have a review-then-commit
> model which means we need at least one committer or PMC member to review a
> contribution before it gets merged.  Even if you're not a committer or PMC
> member, a review is still super helpful and can save time and also puts you
> on a great path to earning merit.  Reviews are a critical part of the code
> and overall quality of NiFi.  'LGTM' is generally not that helpful and
> while warranted for some trivial things we even see it on larger more
> meaningful commits.  Not every review needs to be a back and forth affair
> of code style inputs either.  There is a middle ground and you'll find it
> as you engage in more reviews!    Code reviews are in far more short supply
> than code contributions.  I encourage everyone to please do at least one
> true pull request review for every contribution.  In general the goal
> should be to contribute to more pull request reviews than new pull requests
> created.  If you do so this by definition means the community is growing
> and you're directly helping make that happen!
>
> Would love to see additional inputs/missing categories/other thoughts.
> Perhaps this turns into a wiki page that helps give more context to people
> that want to find ways to add value and be on a path to committer and PMC
> membership as well.
>
> Thanks
> Joe
>