You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Matt Burgess <ma...@apache.org> on 2020/11/16 17:28:37 UTC
Past, Present, and Future of MiNiFi Java -> NiFi
All,
As you know, MiNiFi Java is a separate project and codebase than NiFi,
it's my understanding that this was done because MiNiFi Java was a
fledgling effort while NiFi was an established project, and trying to
do rapid development and releases on MiNiFi would cause some
interference with NiFi features, releases, etc.
Since then, MiNiFi Java has advanced in its own right, but lately
development and release cadence has slowed, and AFAICT much of the
effort is to bring NiFi capabilities into MiNiFi, exposing previously
ignored NiFi properties in MiNiFi's config.yml, etc.
I've begun work on getting MiNiFi Java back into the NiFi codebase,
under its own area with its own assembly, bootstrap, etc. This is
being tracked in Jira under MINIFI-422 [1].
The idea is to be able to build a MiNiFi assembly from NiFi and MiNiFi
components, thereby giving MiNiFi capabilities from NiFi without
having to port them to a separate project. Kerberos support is a
shining example of this, as are other features available via NiFi's
various Context objects (where MiNiFi currently has them stubbed out).
The first step was refactoring some NiFi NAR structure in order to
allow for a "headless" NiFi [2][3], the idea being that we could reuse
some existing framework components/NARs without being tied to the UI,
associated webapps, and Jetty itself.
I will create separate [DISCUSS] threads for each of the following,
but here are some things I'm looking at going forward:
1) Bring MiNiFi bootstrap and external command-and-control (C2) code
over to a MiNiFi area in the NiFi project structure. Also create a
MiNiFi assembly in that area for the purposes of generating release
artifacts when ready.
2) Bring over and update the C2 Server reference implementation, for
testing and to give some way to externally update config/flows in
MiNiFi. This may mean we don't need the change ingestors anymore, but
I will send a separate DISCUSS thread to see who's using them, if
they're important to keep, etc.
3) Discuss whether to bring over the config.yml mechanism for
defining/updating configuration and flow. If we eventually want to
share more between MiNiFi and NiFi, should we return to using
nifi.properties and flow.xml.gz rather than maintaining (and updating)
the code to expose nifi properties as human-readable YAML strings, to
include all the schemas and converters? This too will be a separate
[DISCUSS] thread.
4) Once all is determined and implemented, look at ways to share any
components between NiFi and MiNiFi. This might include refactoring the
C2 stuff into module(s) for use by NiFi [4], adding/enabling profiles
and/or assemblies to build various products (headless NiFi with C2,
MiNiFi with Elasticsearch NARs, e.g.)
I plan on starting on 1 soon, but am glad to discuss any of these
things or any other topics you'd like to bring up (for 2 and 3 let's
have the discussion in the forthcoming DISCUSS threads)
Regards,
Matt
[1] https://issues.apache.org/jira/browse/MINIFI-422
[2] https://issues.apache.org/jira/browse/NIFI-7592
[3] https://issues.apache.org/jira/browse/MINIFI-485
[4] https://issues.apache.org/jira/browse/NIFI-6813
Re: Past, Present, and Future of MiNiFi Java -> NiFi
Posted by Joe Witt <jo...@gmail.com>.
Likewise - thanks! Will be good to see this come back into core so they
can grow together. This along with the work happening for stateless nifi
will offer a lot of choices for users to use the most appropriate engine
for their use case while building flows against consistent flow
design/language and massive set of extensions!
On Mon, Nov 16, 2020 at 10:41 AM Pierre Villard <pi...@gmail.com>
wrote:
> +1 to all of this - I'm really looking forward to seeing more of this!
>
> Le lun. 16 nov. 2020 à 18:39, Kevin Doran <kd...@apache.org> a écrit :
>
> > Thanks for your attention and effort on this, Matt. I look forward to the
> > discuss threads, but at a high level, this sounds like a good and
> important
> > step for MiNiFi Java and C2.
> >
> > Cheers,
> > Kevin
> >
> > > On Nov 16, 2020, at 12:28, Matt Burgess <ma...@apache.org> wrote:
> > >
> > > All,
> > >
> > > As you know, MiNiFi Java is a separate project and codebase than NiFi,
> > > it's my understanding that this was done because MiNiFi Java was a
> > > fledgling effort while NiFi was an established project, and trying to
> > > do rapid development and releases on MiNiFi would cause some
> > > interference with NiFi features, releases, etc.
> > >
> > > Since then, MiNiFi Java has advanced in its own right, but lately
> > > development and release cadence has slowed, and AFAICT much of the
> > > effort is to bring NiFi capabilities into MiNiFi, exposing previously
> > > ignored NiFi properties in MiNiFi's config.yml, etc.
> > >
> > > I've begun work on getting MiNiFi Java back into the NiFi codebase,
> > > under its own area with its own assembly, bootstrap, etc. This is
> > > being tracked in Jira under MINIFI-422 [1].
> > >
> > > The idea is to be able to build a MiNiFi assembly from NiFi and MiNiFi
> > > components, thereby giving MiNiFi capabilities from NiFi without
> > > having to port them to a separate project. Kerberos support is a
> > > shining example of this, as are other features available via NiFi's
> > > various Context objects (where MiNiFi currently has them stubbed out).
> > >
> > > The first step was refactoring some NiFi NAR structure in order to
> > > allow for a "headless" NiFi [2][3], the idea being that we could reuse
> > > some existing framework components/NARs without being tied to the UI,
> > > associated webapps, and Jetty itself.
> > >
> > > I will create separate [DISCUSS] threads for each of the following,
> > > but here are some things I'm looking at going forward:
> > >
> > > 1) Bring MiNiFi bootstrap and external command-and-control (C2) code
> > > over to a MiNiFi area in the NiFi project structure. Also create a
> > > MiNiFi assembly in that area for the purposes of generating release
> > > artifacts when ready.
> > >
> > > 2) Bring over and update the C2 Server reference implementation, for
> > > testing and to give some way to externally update config/flows in
> > > MiNiFi. This may mean we don't need the change ingestors anymore, but
> > > I will send a separate DISCUSS thread to see who's using them, if
> > > they're important to keep, etc.
> > >
> > > 3) Discuss whether to bring over the config.yml mechanism for
> > > defining/updating configuration and flow. If we eventually want to
> > > share more between MiNiFi and NiFi, should we return to using
> > > nifi.properties and flow.xml.gz rather than maintaining (and updating)
> > > the code to expose nifi properties as human-readable YAML strings, to
> > > include all the schemas and converters? This too will be a separate
> > > [DISCUSS] thread.
> > >
> > > 4) Once all is determined and implemented, look at ways to share any
> > > components between NiFi and MiNiFi. This might include refactoring the
> > > C2 stuff into module(s) for use by NiFi [4], adding/enabling profiles
> > > and/or assemblies to build various products (headless NiFi with C2,
> > > MiNiFi with Elasticsearch NARs, e.g.)
> > >
> > > I plan on starting on 1 soon, but am glad to discuss any of these
> > > things or any other topics you'd like to bring up (for 2 and 3 let's
> > > have the discussion in the forthcoming DISCUSS threads)
> > >
> > > Regards,
> > > Matt
> > >
> > > [1] https://issues.apache.org/jira/browse/MINIFI-422
> > > [2] https://issues.apache.org/jira/browse/NIFI-7592
> > > [3] https://issues.apache.org/jira/browse/MINIFI-485
> > > [4] https://issues.apache.org/jira/browse/NIFI-6813
> >
> >
>
Re: Past, Present, and Future of MiNiFi Java -> NiFi
Posted by Pierre Villard <pi...@gmail.com>.
+1 to all of this - I'm really looking forward to seeing more of this!
Le lun. 16 nov. 2020 à 18:39, Kevin Doran <kd...@apache.org> a écrit :
> Thanks for your attention and effort on this, Matt. I look forward to the
> discuss threads, but at a high level, this sounds like a good and important
> step for MiNiFi Java and C2.
>
> Cheers,
> Kevin
>
> > On Nov 16, 2020, at 12:28, Matt Burgess <ma...@apache.org> wrote:
> >
> > All,
> >
> > As you know, MiNiFi Java is a separate project and codebase than NiFi,
> > it's my understanding that this was done because MiNiFi Java was a
> > fledgling effort while NiFi was an established project, and trying to
> > do rapid development and releases on MiNiFi would cause some
> > interference with NiFi features, releases, etc.
> >
> > Since then, MiNiFi Java has advanced in its own right, but lately
> > development and release cadence has slowed, and AFAICT much of the
> > effort is to bring NiFi capabilities into MiNiFi, exposing previously
> > ignored NiFi properties in MiNiFi's config.yml, etc.
> >
> > I've begun work on getting MiNiFi Java back into the NiFi codebase,
> > under its own area with its own assembly, bootstrap, etc. This is
> > being tracked in Jira under MINIFI-422 [1].
> >
> > The idea is to be able to build a MiNiFi assembly from NiFi and MiNiFi
> > components, thereby giving MiNiFi capabilities from NiFi without
> > having to port them to a separate project. Kerberos support is a
> > shining example of this, as are other features available via NiFi's
> > various Context objects (where MiNiFi currently has them stubbed out).
> >
> > The first step was refactoring some NiFi NAR structure in order to
> > allow for a "headless" NiFi [2][3], the idea being that we could reuse
> > some existing framework components/NARs without being tied to the UI,
> > associated webapps, and Jetty itself.
> >
> > I will create separate [DISCUSS] threads for each of the following,
> > but here are some things I'm looking at going forward:
> >
> > 1) Bring MiNiFi bootstrap and external command-and-control (C2) code
> > over to a MiNiFi area in the NiFi project structure. Also create a
> > MiNiFi assembly in that area for the purposes of generating release
> > artifacts when ready.
> >
> > 2) Bring over and update the C2 Server reference implementation, for
> > testing and to give some way to externally update config/flows in
> > MiNiFi. This may mean we don't need the change ingestors anymore, but
> > I will send a separate DISCUSS thread to see who's using them, if
> > they're important to keep, etc.
> >
> > 3) Discuss whether to bring over the config.yml mechanism for
> > defining/updating configuration and flow. If we eventually want to
> > share more between MiNiFi and NiFi, should we return to using
> > nifi.properties and flow.xml.gz rather than maintaining (and updating)
> > the code to expose nifi properties as human-readable YAML strings, to
> > include all the schemas and converters? This too will be a separate
> > [DISCUSS] thread.
> >
> > 4) Once all is determined and implemented, look at ways to share any
> > components between NiFi and MiNiFi. This might include refactoring the
> > C2 stuff into module(s) for use by NiFi [4], adding/enabling profiles
> > and/or assemblies to build various products (headless NiFi with C2,
> > MiNiFi with Elasticsearch NARs, e.g.)
> >
> > I plan on starting on 1 soon, but am glad to discuss any of these
> > things or any other topics you'd like to bring up (for 2 and 3 let's
> > have the discussion in the forthcoming DISCUSS threads)
> >
> > Regards,
> > Matt
> >
> > [1] https://issues.apache.org/jira/browse/MINIFI-422
> > [2] https://issues.apache.org/jira/browse/NIFI-7592
> > [3] https://issues.apache.org/jira/browse/MINIFI-485
> > [4] https://issues.apache.org/jira/browse/NIFI-6813
>
>
Re: Past, Present, and Future of MiNiFi Java -> NiFi
Posted by Kevin Doran <kd...@apache.org>.
Thanks for your attention and effort on this, Matt. I look forward to the discuss threads, but at a high level, this sounds like a good and important step for MiNiFi Java and C2.
Cheers,
Kevin
> On Nov 16, 2020, at 12:28, Matt Burgess <ma...@apache.org> wrote:
>
> All,
>
> As you know, MiNiFi Java is a separate project and codebase than NiFi,
> it's my understanding that this was done because MiNiFi Java was a
> fledgling effort while NiFi was an established project, and trying to
> do rapid development and releases on MiNiFi would cause some
> interference with NiFi features, releases, etc.
>
> Since then, MiNiFi Java has advanced in its own right, but lately
> development and release cadence has slowed, and AFAICT much of the
> effort is to bring NiFi capabilities into MiNiFi, exposing previously
> ignored NiFi properties in MiNiFi's config.yml, etc.
>
> I've begun work on getting MiNiFi Java back into the NiFi codebase,
> under its own area with its own assembly, bootstrap, etc. This is
> being tracked in Jira under MINIFI-422 [1].
>
> The idea is to be able to build a MiNiFi assembly from NiFi and MiNiFi
> components, thereby giving MiNiFi capabilities from NiFi without
> having to port them to a separate project. Kerberos support is a
> shining example of this, as are other features available via NiFi's
> various Context objects (where MiNiFi currently has them stubbed out).
>
> The first step was refactoring some NiFi NAR structure in order to
> allow for a "headless" NiFi [2][3], the idea being that we could reuse
> some existing framework components/NARs without being tied to the UI,
> associated webapps, and Jetty itself.
>
> I will create separate [DISCUSS] threads for each of the following,
> but here are some things I'm looking at going forward:
>
> 1) Bring MiNiFi bootstrap and external command-and-control (C2) code
> over to a MiNiFi area in the NiFi project structure. Also create a
> MiNiFi assembly in that area for the purposes of generating release
> artifacts when ready.
>
> 2) Bring over and update the C2 Server reference implementation, for
> testing and to give some way to externally update config/flows in
> MiNiFi. This may mean we don't need the change ingestors anymore, but
> I will send a separate DISCUSS thread to see who's using them, if
> they're important to keep, etc.
>
> 3) Discuss whether to bring over the config.yml mechanism for
> defining/updating configuration and flow. If we eventually want to
> share more between MiNiFi and NiFi, should we return to using
> nifi.properties and flow.xml.gz rather than maintaining (and updating)
> the code to expose nifi properties as human-readable YAML strings, to
> include all the schemas and converters? This too will be a separate
> [DISCUSS] thread.
>
> 4) Once all is determined and implemented, look at ways to share any
> components between NiFi and MiNiFi. This might include refactoring the
> C2 stuff into module(s) for use by NiFi [4], adding/enabling profiles
> and/or assemblies to build various products (headless NiFi with C2,
> MiNiFi with Elasticsearch NARs, e.g.)
>
> I plan on starting on 1 soon, but am glad to discuss any of these
> things or any other topics you'd like to bring up (for 2 and 3 let's
> have the discussion in the forthcoming DISCUSS threads)
>
> Regards,
> Matt
>
> [1] https://issues.apache.org/jira/browse/MINIFI-422
> [2] https://issues.apache.org/jira/browse/NIFI-7592
> [3] https://issues.apache.org/jira/browse/MINIFI-485
> [4] https://issues.apache.org/jira/browse/NIFI-6813