You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Marc Parisi <ph...@apache.org> on 2019/02/27 16:43:51 UTC
Opinions on Assembly target location
Hi ,
I've submitted a PR to Apache NiFi MiNiFi C++ that enables the project
to access and run NiFi processors. This requires that certain jars be
included to bootstrap that process long with a set of NARs from which we
will access the desired components. Effectively this means that a user can
run the current set of C++ processors alongside Java processors.
To facilitate this feature's use, I submitted a PR [1] to NiFi with that
assembles the appropriate JARs and a basic set of useful/necessary NARs.
I'd like opinions on where this should live. The options have been to place
it within Apache NiFi MiNIFi C++ ( specifically within the aforementioned
feature's build directory) as it only relates to that project and, perhaps
allowing us to more easily address problems...or as per the PR: placing it
in NiFi proper, hopefully building regression tests that notify us of
issues before a release.
I've been wavering on the options and want community opinion. The end
goal in either case is that if/when a user builds with this functionality
we can deliver a package that runs out of the box. This can be done with
either approach.
The net positive of placing it in NiFi as I see it: sense that it is an
official feature and perhaps notification before NiFi release or in
travis. Side effects include: questioning if this belongs here, hair loss,
and perhaps a sudden loss of appetite.
The net positive of isolating this within MiNIFi C++: cleaner end
solution, it's produced in the same place it's consumed, ability to notify
of problems in our CI and release process ( this could be viewed as a
negative too ? ).
TL;DR: Where feature Live?
Any insight/opinions are appreciated. Thanks!
[1] https://github.com/apache/nifi/pull/3330
Re: Opinions on Assembly target location
Posted by Marc Parisi <ph...@apache.org>.
Thank you for your replies. My goal was to have that shared ownership as
Bryan suggested; however, the simplicity of bootstrapping these efforts
within Apache NiFi MiNiFi C++, controlling dependencies, and overall ease
of use by maintainers has led me to close the PR and place this assembly in
MiNiFi C++. This enables the builder to package all necessary components
within the produced archive and have access to Java processors within
MiNiFI C++( they must purposefully enable this feature ). Just wanted to
close the loop.
Thank you for your consideration
On Wed, Feb 27, 2019 at 12:52 PM Bryan Bende <bb...@gmail.com> wrote:
> I think this situation is sort of analogous to some other scenarios we
> have run into, such as the integrations between NiFi and stream
> processing.
>
> For example, the storm and spark integration lives in NiFi's code
> base, and NiFi is responsible for determining if a new version of
> those APIs is available, updating the dependency, and resolving any
> necessary code changes.
>
> The Flink and Apex integration lives in each of those respective code
> bases, so if Flink changes all of their APIs, they will be forced to
> update the NiFi Flink source/sink in order to make their build pass.
>
> I can see value in either approach. If we consider the JNI stuff to be
> a more generic capability, then I think I lean towards NiFi owning it.
>
> Even with that, MiNiFi will still be dependent on some version of
> whatever JNI artifacts NiFi produces, so MiNiFi would still have
> ownership over when to change that dependency and ensure that it still
> works properly in a given release of MiNiFi.
>
> On Wed, Feb 27, 2019 at 12:02 PM Arpad Boda <ab...@hortonworks.com> wrote:
> >
> > Hi,
> >
> > In my opinion, there is another question here to be answered: what
> version of NiFi NARs do we expect to work in MiNiFi?
> >
> > I see two different approaches here:
> > -#1) MiNiFi is compatible with a given NiFi version, like 1.9.0, NARs in
> that release work properly in MiNiFi.
> > In this case the tests belong to MiNiFi and NiFi there becomes a
> dependency -> the tests are required to pass when upgrading the dependency.
> > -#2) MiNiFi is compatible with master. This means that whatever we
> introduce in NiFi shouldn't break, so the tests more belong to NiFi.
> >
> > According to my experience preventing the merge of a breaking change is
> easier than fixing later, which would make me vote for #2, however this
> approach has a side effect, too: introducing a breaking change (that MiNiFi
> needs to adapt) becomes a huge pain, it's much easier in #1.
> >
> > Regards,
> > Arpad
> >
> > On 27/02/2019, 17:44, "Marc Parisi" <ph...@apache.org> wrote:
> >
> > Hi ,
> > I've submitted a PR to Apache NiFi MiNiFi C++ that enables the
> project
> > to access and run NiFi processors. This requires that certain jars be
> > included to bootstrap that process long with a set of NARs from
> which we
> > will access the desired components. Effectively this means that a
> user can
> > run the current set of C++ processors alongside Java processors.
> > To facilitate this feature's use, I submitted a PR [1] to NiFi
> with that
> > assembles the appropriate JARs and a basic set of useful/necessary
> NARs.
> > I'd like opinions on where this should live. The options have been
> to place
> > it within Apache NiFi MiNIFi C++ ( specifically within the
> aforementioned
> > feature's build directory) as it only relates to that project and,
> perhaps
> > allowing us to more easily address problems...or as per the PR:
> placing it
> > in NiFi proper, hopefully building regression tests that notify us of
> > issues before a release.
> >
> > I've been wavering on the options and want community opinion. The
> end
> > goal in either case is that if/when a user builds with this
> functionality
> > we can deliver a package that runs out of the box. This can be done
> with
> > either approach.
> >
> > The net positive of placing it in NiFi as I see it: sense that
> it is an
> > official feature and perhaps notification before NiFi release or in
> > travis. Side effects include: questioning if this belongs here,
> hair loss,
> > and perhaps a sudden loss of appetite.
> >
> > The net positive of isolating this within MiNIFi C++: cleaner end
> > solution, it's produced in the same place it's consumed, ability to
> notify
> > of problems in our CI and release process ( this could be viewed as a
> > negative too ? ).
> >
> > TL;DR: Where feature Live?
> >
> > Any insight/opinions are appreciated. Thanks!
> >
> > [1] https://github.com/apache/nifi/pull/3330
> >
> >
>
Re: Opinions on Assembly target location
Posted by Bryan Bende <bb...@gmail.com>.
I think this situation is sort of analogous to some other scenarios we
have run into, such as the integrations between NiFi and stream
processing.
For example, the storm and spark integration lives in NiFi's code
base, and NiFi is responsible for determining if a new version of
those APIs is available, updating the dependency, and resolving any
necessary code changes.
The Flink and Apex integration lives in each of those respective code
bases, so if Flink changes all of their APIs, they will be forced to
update the NiFi Flink source/sink in order to make their build pass.
I can see value in either approach. If we consider the JNI stuff to be
a more generic capability, then I think I lean towards NiFi owning it.
Even with that, MiNiFi will still be dependent on some version of
whatever JNI artifacts NiFi produces, so MiNiFi would still have
ownership over when to change that dependency and ensure that it still
works properly in a given release of MiNiFi.
On Wed, Feb 27, 2019 at 12:02 PM Arpad Boda <ab...@hortonworks.com> wrote:
>
> Hi,
>
> In my opinion, there is another question here to be answered: what version of NiFi NARs do we expect to work in MiNiFi?
>
> I see two different approaches here:
> -#1) MiNiFi is compatible with a given NiFi version, like 1.9.0, NARs in that release work properly in MiNiFi.
> In this case the tests belong to MiNiFi and NiFi there becomes a dependency -> the tests are required to pass when upgrading the dependency.
> -#2) MiNiFi is compatible with master. This means that whatever we introduce in NiFi shouldn't break, so the tests more belong to NiFi.
>
> According to my experience preventing the merge of a breaking change is easier than fixing later, which would make me vote for #2, however this approach has a side effect, too: introducing a breaking change (that MiNiFi needs to adapt) becomes a huge pain, it's much easier in #1.
>
> Regards,
> Arpad
>
> On 27/02/2019, 17:44, "Marc Parisi" <ph...@apache.org> wrote:
>
> Hi ,
> I've submitted a PR to Apache NiFi MiNiFi C++ that enables the project
> to access and run NiFi processors. This requires that certain jars be
> included to bootstrap that process long with a set of NARs from which we
> will access the desired components. Effectively this means that a user can
> run the current set of C++ processors alongside Java processors.
> To facilitate this feature's use, I submitted a PR [1] to NiFi with that
> assembles the appropriate JARs and a basic set of useful/necessary NARs.
> I'd like opinions on where this should live. The options have been to place
> it within Apache NiFi MiNIFi C++ ( specifically within the aforementioned
> feature's build directory) as it only relates to that project and, perhaps
> allowing us to more easily address problems...or as per the PR: placing it
> in NiFi proper, hopefully building regression tests that notify us of
> issues before a release.
>
> I've been wavering on the options and want community opinion. The end
> goal in either case is that if/when a user builds with this functionality
> we can deliver a package that runs out of the box. This can be done with
> either approach.
>
> The net positive of placing it in NiFi as I see it: sense that it is an
> official feature and perhaps notification before NiFi release or in
> travis. Side effects include: questioning if this belongs here, hair loss,
> and perhaps a sudden loss of appetite.
>
> The net positive of isolating this within MiNIFi C++: cleaner end
> solution, it's produced in the same place it's consumed, ability to notify
> of problems in our CI and release process ( this could be viewed as a
> negative too ? ).
>
> TL;DR: Where feature Live?
>
> Any insight/opinions are appreciated. Thanks!
>
> [1] https://github.com/apache/nifi/pull/3330
>
>
Re: Opinions on Assembly target location
Posted by Arpad Boda <ab...@hortonworks.com>.
Hi,
In my opinion, there is another question here to be answered: what version of NiFi NARs do we expect to work in MiNiFi?
I see two different approaches here:
-#1) MiNiFi is compatible with a given NiFi version, like 1.9.0, NARs in that release work properly in MiNiFi.
In this case the tests belong to MiNiFi and NiFi there becomes a dependency -> the tests are required to pass when upgrading the dependency.
-#2) MiNiFi is compatible with master. This means that whatever we introduce in NiFi shouldn't break, so the tests more belong to NiFi.
According to my experience preventing the merge of a breaking change is easier than fixing later, which would make me vote for #2, however this approach has a side effect, too: introducing a breaking change (that MiNiFi needs to adapt) becomes a huge pain, it's much easier in #1.
Regards,
Arpad
On 27/02/2019, 17:44, "Marc Parisi" <ph...@apache.org> wrote:
Hi ,
I've submitted a PR to Apache NiFi MiNiFi C++ that enables the project
to access and run NiFi processors. This requires that certain jars be
included to bootstrap that process long with a set of NARs from which we
will access the desired components. Effectively this means that a user can
run the current set of C++ processors alongside Java processors.
To facilitate this feature's use, I submitted a PR [1] to NiFi with that
assembles the appropriate JARs and a basic set of useful/necessary NARs.
I'd like opinions on where this should live. The options have been to place
it within Apache NiFi MiNIFi C++ ( specifically within the aforementioned
feature's build directory) as it only relates to that project and, perhaps
allowing us to more easily address problems...or as per the PR: placing it
in NiFi proper, hopefully building regression tests that notify us of
issues before a release.
I've been wavering on the options and want community opinion. The end
goal in either case is that if/when a user builds with this functionality
we can deliver a package that runs out of the box. This can be done with
either approach.
The net positive of placing it in NiFi as I see it: sense that it is an
official feature and perhaps notification before NiFi release or in
travis. Side effects include: questioning if this belongs here, hair loss,
and perhaps a sudden loss of appetite.
The net positive of isolating this within MiNIFi C++: cleaner end
solution, it's produced in the same place it's consumed, ability to notify
of problems in our CI and release process ( this could be viewed as a
negative too ? ).
TL;DR: Where feature Live?
Any insight/opinions are appreciated. Thanks!
[1] https://github.com/apache/nifi/pull/3330