You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Javi Roman <jr...@gmail.com> on 2021/03/23 05:44:18 UTC
[DISCUSS] Processors Market
Hi!
I'm not sure whether this topic has been discussed in the past. I would
like to open a thread talking about the possibility of creating a kind of
market of NiFi Processors, for third-party processors.
I'm thinking about something similar to Kubernetes Helm hub [1], or
Kubernetes Operators Hub [2], but for NiFi processors (NAR bundles).
This repository could be managed by NiFi Registry in order to use
processors in different stages of development (preview, stable, official
... and so forth).
This is a way of slimming down the NiFi image and opening the development
to new users.
It would also be interesting to discuss if this "processors market" could
be hosted in ASF resources.
What do you think?
[1] https://artifacthub.io/
[2] https://operatorhub.io/
--
Javi Roman
Twitter: @javiromanrh
GitHub: github.com/javiroman
Linkedin: es.linkedin.com/in/javiroman
Big Data Blog: dataintensive.info
Re: [DISCUSS] Processors Market
Posted by ��scar Fern��ndez <os...@apache.org>.
I find this idea really interesting, this should be something to keep in mind for future NiFi development.
The main advantages of this approach would be to help the publication of new NiFi processors and to
slim the NiFi base image.
Maybe we can create a page in the NiFi wiki (confluence) to note the ideas related to the Processor Marketplace so that we can make it more accesible.
On 2021/03/26 15:29:59, Mark Bean <ma...@gmail.com> wrote:
> I like the idea of a marketplace and a reduced footprint for the NiFi
> application. One of the things to consider in the implementation is how a
> user gets information on components which are not installed, but are
> available in the marketplace. This is not only a matter of version, but
> also component selection. For example, a flow designer may rely on the
> documentation which includes all _installed_ components when selecting an
> appropriate one. This same "here are all your choices" documentation should
> still be available to the user, but with an indication of "installed" or
> "available".
>
> Another challenge is version compatibility. For example, if the API for a
> controller service changes in a given NiFi release, it could make all
> existing implementations incompatible. Or is it imagined that the API's
> would be in the marketplace as well rather than the application? This
> coul;d make the upgrade strategy a bit different and more challenging to
> manage. It's not insurmountable, but something to consider in the overall
> scheme.
>
> -Mark
>
> On Fri, Mar 26, 2021 at 11:11 AM Joe Witt <jo...@gmail.com> wrote:
>
> > Eric
> >
> > I think the intent overall is for us to get away from the notion of
> > components and them being tied to a given base image. With a
> > competent registry/artifact repository configuration we should be able
> > to get away from shipping nars with the nifi release at all. Rather
> > your versioned flow indicates exactly what it wants/needs. We'd
> > source precisely those artifacts from the artifact repository. You
> > then get to control as the user whenever you choose to update to a
> > newer version of a given component/etc..
> >
> > Thanks
> >
> > On Fri, Mar 26, 2021 at 8:08 AM Eric Secules <es...@gmail.com> wrote:
> > >
> > > First of all, this sounds like a great idea! As a user I would like a way
> > > to find processors developed by the larger community and also contribute.
> > >
> > > One of my concerns is the user experience around upgrading and deploying
> > a
> > > flow that was built with processors that have now been removed from the
> > > base image. I would want it to be easy and obvious that I need to
> > download
> > > those specific processors from the marketplace. Another concern is user
> > > trust. I can imagine that some users will be wary to download processors
> > > from the marketplace even if those same processors used to be included in
> > > the base image of NiFi and they didn't notice them before.
> > >
> > > On Wed., Mar. 24, 2021, 8:05 a.m. Joe Witt, <jo...@gmail.com> wrote:
> > >
> > > > Javi,
> > > >
> > > > Yes! This is a great idea. It is the sweat equity behind it that
> > > > needs to happen. I can guarantee you there is significant
> > > > interest/desire to do this or something like it.
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Mar 24, 2021 at 8:01 AM Russell Bateman <russ@windofkeltia.com
> > >
> > > > wrote:
> > > > >
> > > > > Javi,
> > > > >
> > > > > Don't despair. Could just be that folk are busy and haven't had time
> > to
> > > > > reflect upon it.
> > > > >
> > > > >
> > > > > On 3/23/21 11:55 PM, Javi Roman wrote:
> > > > > > I see that it has not been well received, I thought it would be a
> > good
> > > > idea
> > > > > > :-(
> > > > > >
> > > > > > --
> > > > > > Javi Roman
> > > > > >
> > > > > > Twitter: @javiromanrh
> > > > > > GitHub: github.com/javiroman
> > > > > > Linkedin: es.linkedin.com/in/javiroman
> > > > > > Big Data Blog: dataintensive.info
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 23, 2021 at 6:44 AM Javi Roman <
> > jroman.espinar@gmail.com>
> > > > wrote:
> > > > > >
> > > > > >> Hi!
> > > > > >>
> > > > > >> I'm not sure whether this topic has been discussed in the past. I
> > > > would
> > > > > >> like to open a thread talking about the possibility of creating a
> > > > kind of
> > > > > >> market of NiFi Processors, for third-party processors.
> > > > > >>
> > > > > >> I'm thinking about something similar to Kubernetes Helm hub [1],
> > or
> > > > > >> Kubernetes Operators Hub [2], but for NiFi processors (NAR
> > bundles).
> > > > > >>
> > > > > >> This repository could be managed by NiFi Registry in order to use
> > > > > >> processors in different stages of development (preview, stable,
> > > > official
> > > > > >> ... and so forth).
> > > > > >>
> > > > > >> This is a way of slimming down the NiFi image and opening the
> > > > development
> > > > > >> to new users.
> > > > > >>
> > > > > >> It would also be interesting to discuss if this "processors
> > market"
> > > > could
> > > > > >> be hosted in ASF resources.
> > > > > >>
> > > > > >> What do you think?
> > > > > >>
> > > > > >> [1] https://artifacthub.io/
> > > > > >> [2] https://operatorhub.io/
> > > > > >> --
> > > > > >> Javi Roman
> > > > > >>
> > > > > >> Twitter: @javiromanrh
> > > > > >> GitHub: github.com/javiroman
> > > > > >> Linkedin: es.linkedin.com/in/javiroman
> > > > > >> Big Data Blog: dataintensive.info
> > > > > >>
> > > > >
> > > >
> >
>
Re: [DISCUSS] Processors Market
Posted by Mark Bean <ma...@gmail.com>.
I like the idea of a marketplace and a reduced footprint for the NiFi
application. One of the things to consider in the implementation is how a
user gets information on components which are not installed, but are
available in the marketplace. This is not only a matter of version, but
also component selection. For example, a flow designer may rely on the
documentation which includes all _installed_ components when selecting an
appropriate one. This same "here are all your choices" documentation should
still be available to the user, but with an indication of "installed" or
"available".
Another challenge is version compatibility. For example, if the API for a
controller service changes in a given NiFi release, it could make all
existing implementations incompatible. Or is it imagined that the API's
would be in the marketplace as well rather than the application? This
coul;d make the upgrade strategy a bit different and more challenging to
manage. It's not insurmountable, but something to consider in the overall
scheme.
-Mark
On Fri, Mar 26, 2021 at 11:11 AM Joe Witt <jo...@gmail.com> wrote:
> Eric
>
> I think the intent overall is for us to get away from the notion of
> components and them being tied to a given base image. With a
> competent registry/artifact repository configuration we should be able
> to get away from shipping nars with the nifi release at all. Rather
> your versioned flow indicates exactly what it wants/needs. We'd
> source precisely those artifacts from the artifact repository. You
> then get to control as the user whenever you choose to update to a
> newer version of a given component/etc..
>
> Thanks
>
> On Fri, Mar 26, 2021 at 8:08 AM Eric Secules <es...@gmail.com> wrote:
> >
> > First of all, this sounds like a great idea! As a user I would like a way
> > to find processors developed by the larger community and also contribute.
> >
> > One of my concerns is the user experience around upgrading and deploying
> a
> > flow that was built with processors that have now been removed from the
> > base image. I would want it to be easy and obvious that I need to
> download
> > those specific processors from the marketplace. Another concern is user
> > trust. I can imagine that some users will be wary to download processors
> > from the marketplace even if those same processors used to be included in
> > the base image of NiFi and they didn't notice them before.
> >
> > On Wed., Mar. 24, 2021, 8:05 a.m. Joe Witt, <jo...@gmail.com> wrote:
> >
> > > Javi,
> > >
> > > Yes! This is a great idea. It is the sweat equity behind it that
> > > needs to happen. I can guarantee you there is significant
> > > interest/desire to do this or something like it.
> > >
> > > Thanks
> > >
> > > On Wed, Mar 24, 2021 at 8:01 AM Russell Bateman <russ@windofkeltia.com
> >
> > > wrote:
> > > >
> > > > Javi,
> > > >
> > > > Don't despair. Could just be that folk are busy and haven't had time
> to
> > > > reflect upon it.
> > > >
> > > >
> > > > On 3/23/21 11:55 PM, Javi Roman wrote:
> > > > > I see that it has not been well received, I thought it would be a
> good
> > > idea
> > > > > :-(
> > > > >
> > > > > --
> > > > > Javi Roman
> > > > >
> > > > > Twitter: @javiromanrh
> > > > > GitHub: github.com/javiroman
> > > > > Linkedin: es.linkedin.com/in/javiroman
> > > > > Big Data Blog: dataintensive.info
> > > > >
> > > > >
> > > > > On Tue, Mar 23, 2021 at 6:44 AM Javi Roman <
> jroman.espinar@gmail.com>
> > > wrote:
> > > > >
> > > > >> Hi!
> > > > >>
> > > > >> I'm not sure whether this topic has been discussed in the past. I
> > > would
> > > > >> like to open a thread talking about the possibility of creating a
> > > kind of
> > > > >> market of NiFi Processors, for third-party processors.
> > > > >>
> > > > >> I'm thinking about something similar to Kubernetes Helm hub [1],
> or
> > > > >> Kubernetes Operators Hub [2], but for NiFi processors (NAR
> bundles).
> > > > >>
> > > > >> This repository could be managed by NiFi Registry in order to use
> > > > >> processors in different stages of development (preview, stable,
> > > official
> > > > >> ... and so forth).
> > > > >>
> > > > >> This is a way of slimming down the NiFi image and opening the
> > > development
> > > > >> to new users.
> > > > >>
> > > > >> It would also be interesting to discuss if this "processors
> market"
> > > could
> > > > >> be hosted in ASF resources.
> > > > >>
> > > > >> What do you think?
> > > > >>
> > > > >> [1] https://artifacthub.io/
> > > > >> [2] https://operatorhub.io/
> > > > >> --
> > > > >> Javi Roman
> > > > >>
> > > > >> Twitter: @javiromanrh
> > > > >> GitHub: github.com/javiroman
> > > > >> Linkedin: es.linkedin.com/in/javiroman
> > > > >> Big Data Blog: dataintensive.info
> > > > >>
> > > >
> > >
>
Re: [DISCUSS] Processors Market
Posted by Joe Witt <jo...@gmail.com>.
Eric
I think the intent overall is for us to get away from the notion of
components and them being tied to a given base image. With a
competent registry/artifact repository configuration we should be able
to get away from shipping nars with the nifi release at all. Rather
your versioned flow indicates exactly what it wants/needs. We'd
source precisely those artifacts from the artifact repository. You
then get to control as the user whenever you choose to update to a
newer version of a given component/etc..
Thanks
On Fri, Mar 26, 2021 at 8:08 AM Eric Secules <es...@gmail.com> wrote:
>
> First of all, this sounds like a great idea! As a user I would like a way
> to find processors developed by the larger community and also contribute.
>
> One of my concerns is the user experience around upgrading and deploying a
> flow that was built with processors that have now been removed from the
> base image. I would want it to be easy and obvious that I need to download
> those specific processors from the marketplace. Another concern is user
> trust. I can imagine that some users will be wary to download processors
> from the marketplace even if those same processors used to be included in
> the base image of NiFi and they didn't notice them before.
>
> On Wed., Mar. 24, 2021, 8:05 a.m. Joe Witt, <jo...@gmail.com> wrote:
>
> > Javi,
> >
> > Yes! This is a great idea. It is the sweat equity behind it that
> > needs to happen. I can guarantee you there is significant
> > interest/desire to do this or something like it.
> >
> > Thanks
> >
> > On Wed, Mar 24, 2021 at 8:01 AM Russell Bateman <ru...@windofkeltia.com>
> > wrote:
> > >
> > > Javi,
> > >
> > > Don't despair. Could just be that folk are busy and haven't had time to
> > > reflect upon it.
> > >
> > >
> > > On 3/23/21 11:55 PM, Javi Roman wrote:
> > > > I see that it has not been well received, I thought it would be a good
> > idea
> > > > :-(
> > > >
> > > > --
> > > > Javi Roman
> > > >
> > > > Twitter: @javiromanrh
> > > > GitHub: github.com/javiroman
> > > > Linkedin: es.linkedin.com/in/javiroman
> > > > Big Data Blog: dataintensive.info
> > > >
> > > >
> > > > On Tue, Mar 23, 2021 at 6:44 AM Javi Roman <jr...@gmail.com>
> > wrote:
> > > >
> > > >> Hi!
> > > >>
> > > >> I'm not sure whether this topic has been discussed in the past. I
> > would
> > > >> like to open a thread talking about the possibility of creating a
> > kind of
> > > >> market of NiFi Processors, for third-party processors.
> > > >>
> > > >> I'm thinking about something similar to Kubernetes Helm hub [1], or
> > > >> Kubernetes Operators Hub [2], but for NiFi processors (NAR bundles).
> > > >>
> > > >> This repository could be managed by NiFi Registry in order to use
> > > >> processors in different stages of development (preview, stable,
> > official
> > > >> ... and so forth).
> > > >>
> > > >> This is a way of slimming down the NiFi image and opening the
> > development
> > > >> to new users.
> > > >>
> > > >> It would also be interesting to discuss if this "processors market"
> > could
> > > >> be hosted in ASF resources.
> > > >>
> > > >> What do you think?
> > > >>
> > > >> [1] https://artifacthub.io/
> > > >> [2] https://operatorhub.io/
> > > >> --
> > > >> Javi Roman
> > > >>
> > > >> Twitter: @javiromanrh
> > > >> GitHub: github.com/javiroman
> > > >> Linkedin: es.linkedin.com/in/javiroman
> > > >> Big Data Blog: dataintensive.info
> > > >>
> > >
> >
Re: [DISCUSS] Processors Market
Posted by Eric Secules <es...@gmail.com>.
First of all, this sounds like a great idea! As a user I would like a way
to find processors developed by the larger community and also contribute.
One of my concerns is the user experience around upgrading and deploying a
flow that was built with processors that have now been removed from the
base image. I would want it to be easy and obvious that I need to download
those specific processors from the marketplace. Another concern is user
trust. I can imagine that some users will be wary to download processors
from the marketplace even if those same processors used to be included in
the base image of NiFi and they didn't notice them before.
On Wed., Mar. 24, 2021, 8:05 a.m. Joe Witt, <jo...@gmail.com> wrote:
> Javi,
>
> Yes! This is a great idea. It is the sweat equity behind it that
> needs to happen. I can guarantee you there is significant
> interest/desire to do this or something like it.
>
> Thanks
>
> On Wed, Mar 24, 2021 at 8:01 AM Russell Bateman <ru...@windofkeltia.com>
> wrote:
> >
> > Javi,
> >
> > Don't despair. Could just be that folk are busy and haven't had time to
> > reflect upon it.
> >
> >
> > On 3/23/21 11:55 PM, Javi Roman wrote:
> > > I see that it has not been well received, I thought it would be a good
> idea
> > > :-(
> > >
> > > --
> > > Javi Roman
> > >
> > > Twitter: @javiromanrh
> > > GitHub: github.com/javiroman
> > > Linkedin: es.linkedin.com/in/javiroman
> > > Big Data Blog: dataintensive.info
> > >
> > >
> > > On Tue, Mar 23, 2021 at 6:44 AM Javi Roman <jr...@gmail.com>
> wrote:
> > >
> > >> Hi!
> > >>
> > >> I'm not sure whether this topic has been discussed in the past. I
> would
> > >> like to open a thread talking about the possibility of creating a
> kind of
> > >> market of NiFi Processors, for third-party processors.
> > >>
> > >> I'm thinking about something similar to Kubernetes Helm hub [1], or
> > >> Kubernetes Operators Hub [2], but for NiFi processors (NAR bundles).
> > >>
> > >> This repository could be managed by NiFi Registry in order to use
> > >> processors in different stages of development (preview, stable,
> official
> > >> ... and so forth).
> > >>
> > >> This is a way of slimming down the NiFi image and opening the
> development
> > >> to new users.
> > >>
> > >> It would also be interesting to discuss if this "processors market"
> could
> > >> be hosted in ASF resources.
> > >>
> > >> What do you think?
> > >>
> > >> [1] https://artifacthub.io/
> > >> [2] https://operatorhub.io/
> > >> --
> > >> Javi Roman
> > >>
> > >> Twitter: @javiromanrh
> > >> GitHub: github.com/javiroman
> > >> Linkedin: es.linkedin.com/in/javiroman
> > >> Big Data Blog: dataintensive.info
> > >>
> >
>
Re: [DISCUSS] Processors Market
Posted by Joe Witt <jo...@gmail.com>.
Javi,
Yes! This is a great idea. It is the sweat equity behind it that
needs to happen. I can guarantee you there is significant
interest/desire to do this or something like it.
Thanks
On Wed, Mar 24, 2021 at 8:01 AM Russell Bateman <ru...@windofkeltia.com> wrote:
>
> Javi,
>
> Don't despair. Could just be that folk are busy and haven't had time to
> reflect upon it.
>
>
> On 3/23/21 11:55 PM, Javi Roman wrote:
> > I see that it has not been well received, I thought it would be a good idea
> > :-(
> >
> > --
> > Javi Roman
> >
> > Twitter: @javiromanrh
> > GitHub: github.com/javiroman
> > Linkedin: es.linkedin.com/in/javiroman
> > Big Data Blog: dataintensive.info
> >
> >
> > On Tue, Mar 23, 2021 at 6:44 AM Javi Roman <jr...@gmail.com> wrote:
> >
> >> Hi!
> >>
> >> I'm not sure whether this topic has been discussed in the past. I would
> >> like to open a thread talking about the possibility of creating a kind of
> >> market of NiFi Processors, for third-party processors.
> >>
> >> I'm thinking about something similar to Kubernetes Helm hub [1], or
> >> Kubernetes Operators Hub [2], but for NiFi processors (NAR bundles).
> >>
> >> This repository could be managed by NiFi Registry in order to use
> >> processors in different stages of development (preview, stable, official
> >> ... and so forth).
> >>
> >> This is a way of slimming down the NiFi image and opening the development
> >> to new users.
> >>
> >> It would also be interesting to discuss if this "processors market" could
> >> be hosted in ASF resources.
> >>
> >> What do you think?
> >>
> >> [1] https://artifacthub.io/
> >> [2] https://operatorhub.io/
> >> --
> >> Javi Roman
> >>
> >> Twitter: @javiromanrh
> >> GitHub: github.com/javiroman
> >> Linkedin: es.linkedin.com/in/javiroman
> >> Big Data Blog: dataintensive.info
> >>
>
Re: [DISCUSS] Processors Market
Posted by Russell Bateman <ru...@windofkeltia.com>.
Javi,
Don't despair. Could just be that folk are busy and haven't had time to
reflect upon it.
On 3/23/21 11:55 PM, Javi Roman wrote:
> I see that it has not been well received, I thought it would be a good idea
> :-(
>
> --
> Javi Roman
>
> Twitter: @javiromanrh
> GitHub: github.com/javiroman
> Linkedin: es.linkedin.com/in/javiroman
> Big Data Blog: dataintensive.info
>
>
> On Tue, Mar 23, 2021 at 6:44 AM Javi Roman <jr...@gmail.com> wrote:
>
>> Hi!
>>
>> I'm not sure whether this topic has been discussed in the past. I would
>> like to open a thread talking about the possibility of creating a kind of
>> market of NiFi Processors, for third-party processors.
>>
>> I'm thinking about something similar to Kubernetes Helm hub [1], or
>> Kubernetes Operators Hub [2], but for NiFi processors (NAR bundles).
>>
>> This repository could be managed by NiFi Registry in order to use
>> processors in different stages of development (preview, stable, official
>> ... and so forth).
>>
>> This is a way of slimming down the NiFi image and opening the development
>> to new users.
>>
>> It would also be interesting to discuss if this "processors market" could
>> be hosted in ASF resources.
>>
>> What do you think?
>>
>> [1] https://artifacthub.io/
>> [2] https://operatorhub.io/
>> --
>> Javi Roman
>>
>> Twitter: @javiromanrh
>> GitHub: github.com/javiroman
>> Linkedin: es.linkedin.com/in/javiroman
>> Big Data Blog: dataintensive.info
>>
Re: [DISCUSS] Processors Market
Posted by Javi Roman <jr...@gmail.com>.
I see that it has not been well received, I thought it would be a good idea
:-(
--
Javi Roman
Twitter: @javiromanrh
GitHub: github.com/javiroman
Linkedin: es.linkedin.com/in/javiroman
Big Data Blog: dataintensive.info
On Tue, Mar 23, 2021 at 6:44 AM Javi Roman <jr...@gmail.com> wrote:
> Hi!
>
> I'm not sure whether this topic has been discussed in the past. I would
> like to open a thread talking about the possibility of creating a kind of
> market of NiFi Processors, for third-party processors.
>
> I'm thinking about something similar to Kubernetes Helm hub [1], or
> Kubernetes Operators Hub [2], but for NiFi processors (NAR bundles).
>
> This repository could be managed by NiFi Registry in order to use
> processors in different stages of development (preview, stable, official
> ... and so forth).
>
> This is a way of slimming down the NiFi image and opening the development
> to new users.
>
> It would also be interesting to discuss if this "processors market" could
> be hosted in ASF resources.
>
> What do you think?
>
> [1] https://artifacthub.io/
> [2] https://operatorhub.io/
> --
> Javi Roman
>
> Twitter: @javiromanrh
> GitHub: github.com/javiroman
> Linkedin: es.linkedin.com/in/javiroman
> Big Data Blog: dataintensive.info
>