[DISCUSS] Extension Registry

[Bryan Bende <bb...@gmail.com>]

All,

We've needed the elusive extension registry for quite some time now,
and with NiFi Registry I think we are in a good place to make some
progress in this area.

I've started looking into adding "extension bundles" to NiFi Registry
as the next type of versioned item, along side the existing versioned
flows, and I wanted to take a minute to outline how that approach
could work before getting too far into it.

Also, I'd like to focus this discussion on the design and
functionality of the extension registry, and not on how the community
is going to use it. Topics like hosting a central registry, changing
the build process, restructuring the git repo, releasing NARs, etc,
are all important topics, but first we need an extension registry
before we can do any of that :)

Here is a high-level description of what needs to be done...

NiFi Registry

- Add a new type of item called an extension bundle, where each bundle
can contain one ore extensions or APIs

- Support bundles for traditional NiFi (aka NARs) and also bundles for
MiNiFi CPP

- Ability to upload the binary artifact for a bundle and extract the
metadata about the bundle, and metadata about the extensions contained
in the bundle (more on this later)

- Provide a pluggable storage provider for saving the content of each
extension bundle so that we can have different implementations like
local fileysystem, S3, and other object stores

- Provide a REST API for listing and retrieving available bundles,
integrate this into the registry Java client and NiFi CLI

NAR Maven Plugin

- Generate a descriptor for each component in the NAR which will
provide information like the description, tags, restricted or not,
etc.

- These descriptors will be parsed by NiFi Registry when a NAR is
being uploaded so that NiFi Registry will know about the extensions
contained with in the NAR

NiFi

- Provide some type of extension manager experience where users can
search, browse, & install bundles that are available in any of the
registry clients that have been defined

- Introduce a new security policy to control which users are allowed
to access the extension manager

- Installing a bundle should load the NAR and make the extensions
available leveraging the recent work done to auto-load new NARs

- Importing versioned flows from registry should provide an easy way
to install bundles that are required by the flow, but missing from the
local NiFi instance


If anyone has any thoughts or concerns about this approach, please let me know.

Thanks,

Bryan

Re: [DISCUSS] Extension Registry

[Joe Witt <jo...@gmail.com>]

i'm not gonna lie - i googled elusive vs illusive.  Well played, Rick.
Well played.
On Tue, Nov 13, 2018 at 12:44 PM Richard St. John <rs...@gmail.com> wrote:
>
> Bryan,
>
> Good luck.  Godspeed, my friend.  As a hard-core NiFi user, I can’t wait for the illusive extension registry to be a reality.
>
> Rick.
>
> --
> Richard St. John, PhD
> Asymmetrik
> AGILITY | EXPERIENCE | RESULTS
> 308 Sentinel Drive, Suite 400
> Annapolis Junction, MD 20701
> On Nov 13, 2018, 12:37 PM -0500, Joe Witt <jo...@gmail.com>, wrote:
> > Bryan
> >
> > Very exciting to see this under way!!! We desperately have to get our
> > core nifi build size way down and make it far easier for people to
> > contribute new processors. We have a long line of extensions that
> > could be really useful/valuable and this will help unlock that while
> > improving the user experience tremendously.
> >
> > For the doc/concerns noted above we should also add/mention that the
> > relationship between nars (dependencies between nars for
> > apis/controller services/parent nars/etc..) we want to have a way to
> > manage/show that or a user experience for it. Maybe not a day-1 thing
> > but important to call out.
> >
> > Thanks!
> > Joe
> > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <bb...@gmail.com> wrote:
> > >
> > > All,
> > >
> > > We've needed the elusive extension registry for quite some time now,
> > > and with NiFi Registry I think we are in a good place to make some
> > > progress in this area.
> > >
> > > I've started looking into adding "extension bundles" to NiFi Registry
> > > as the next type of versioned item, along side the existing versioned
> > > flows, and I wanted to take a minute to outline how that approach
> > > could work before getting too far into it.
> > >
> > > Also, I'd like to focus this discussion on the design and
> > > functionality of the extension registry, and not on how the community
> > > is going to use it. Topics like hosting a central registry, changing
> > > the build process, restructuring the git repo, releasing NARs, etc,
> > > are all important topics, but first we need an extension registry
> > > before we can do any of that :)
> > >
> > > Here is a high-level description of what needs to be done...
> > >
> > > NiFi Registry
> > >
> > > - Add a new type of item called an extension bundle, where each bundle
> > > can contain one ore extensions or APIs
> > >
> > > - Support bundles for traditional NiFi (aka NARs) and also bundles for
> > > MiNiFi CPP
> > >
> > > - Ability to upload the binary artifact for a bundle and extract the
> > > metadata about the bundle, and metadata about the extensions contained
> > > in the bundle (more on this later)
> > >
> > > - Provide a pluggable storage provider for saving the content of each
> > > extension bundle so that we can have different implementations like
> > > local fileysystem, S3, and other object stores
> > >
> > > - Provide a REST API for listing and retrieving available bundles,
> > > integrate this into the registry Java client and NiFi CLI
> > >
> > > NAR Maven Plugin
> > >
> > > - Generate a descriptor for each component in the NAR which will
> > > provide information like the description, tags, restricted or not,
> > > etc.
> > >
> > > - These descriptors will be parsed by NiFi Registry when a NAR is
> > > being uploaded so that NiFi Registry will know about the extensions
> > > contained with in the NAR
> > >
> > > NiFi
> > >
> > > - Provide some type of extension manager experience where users can
> > > search, browse, & install bundles that are available in any of the
> > > registry clients that have been defined
> > >
> > > - Introduce a new security policy to control which users are allowed
> > > to access the extension manager
> > >
> > > - Installing a bundle should load the NAR and make the extensions
> > > available leveraging the recent work done to auto-load new NARs
> > >
> > > - Importing versioned flows from registry should provide an easy way
> > > to install bundles that are required by the flow, but missing from the
> > > local NiFi instance
> > >
> > >
> > > If anyone has any thoughts or concerns about this approach, please let me know.
> > >
> > > Thanks,
> > >
> > > Bryan

Re: [DISCUSS] Extension Registry

["Richard St. John" <rs...@gmail.com>]

Bryan,

Good luck.  Godspeed, my friend.  As a hard-core NiFi user, I can’t wait for the illusive extension registry to be a reality.

Rick.

--
Richard St. John, PhD
Asymmetrik
AGILITY | EXPERIENCE | RESULTS
308 Sentinel Drive, Suite 400
Annapolis Junction, MD 20701
On Nov 13, 2018, 12:37 PM -0500, Joe Witt <jo...@gmail.com>, wrote:
> Bryan
>
> Very exciting to see this under way!!! We desperately have to get our
> core nifi build size way down and make it far easier for people to
> contribute new processors. We have a long line of extensions that
> could be really useful/valuable and this will help unlock that while
> improving the user experience tremendously.
>
> For the doc/concerns noted above we should also add/mention that the
> relationship between nars (dependencies between nars for
> apis/controller services/parent nars/etc..) we want to have a way to
> manage/show that or a user experience for it. Maybe not a day-1 thing
> but important to call out.
>
> Thanks!
> Joe
> On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <bb...@gmail.com> wrote:
> >
> > All,
> >
> > We've needed the elusive extension registry for quite some time now,
> > and with NiFi Registry I think we are in a good place to make some
> > progress in this area.
> >
> > I've started looking into adding "extension bundles" to NiFi Registry
> > as the next type of versioned item, along side the existing versioned
> > flows, and I wanted to take a minute to outline how that approach
> > could work before getting too far into it.
> >
> > Also, I'd like to focus this discussion on the design and
> > functionality of the extension registry, and not on how the community
> > is going to use it. Topics like hosting a central registry, changing
> > the build process, restructuring the git repo, releasing NARs, etc,
> > are all important topics, but first we need an extension registry
> > before we can do any of that :)
> >
> > Here is a high-level description of what needs to be done...
> >
> > NiFi Registry
> >
> > - Add a new type of item called an extension bundle, where each bundle
> > can contain one ore extensions or APIs
> >
> > - Support bundles for traditional NiFi (aka NARs) and also bundles for
> > MiNiFi CPP
> >
> > - Ability to upload the binary artifact for a bundle and extract the
> > metadata about the bundle, and metadata about the extensions contained
> > in the bundle (more on this later)
> >
> > - Provide a pluggable storage provider for saving the content of each
> > extension bundle so that we can have different implementations like
> > local fileysystem, S3, and other object stores
> >
> > - Provide a REST API for listing and retrieving available bundles,
> > integrate this into the registry Java client and NiFi CLI
> >
> > NAR Maven Plugin
> >
> > - Generate a descriptor for each component in the NAR which will
> > provide information like the description, tags, restricted or not,
> > etc.
> >
> > - These descriptors will be parsed by NiFi Registry when a NAR is
> > being uploaded so that NiFi Registry will know about the extensions
> > contained with in the NAR
> >
> > NiFi
> >
> > - Provide some type of extension manager experience where users can
> > search, browse, & install bundles that are available in any of the
> > registry clients that have been defined
> >
> > - Introduce a new security policy to control which users are allowed
> > to access the extension manager
> >
> > - Installing a bundle should load the NAR and make the extensions
> > available leveraging the recent work done to auto-load new NARs
> >
> > - Importing versioned flows from registry should provide an easy way
> > to install bundles that are required by the flow, but missing from the
> > local NiFi instance
> >
> >
> > If anyone has any thoughts or concerns about this approach, please let me know.
> >
> > Thanks,
> >
> > Bryan

Re: [DISCUSS] Extension Registry

[Andy LoPresto <al...@gmail.com>]

From a security perspective, I’d like to see attention paid to the following topics:

* verifiable checksums over extensions/bundles to ensure the code being deployed from a remote system is trusted
* fuzzy hashing (complementary) to provide “change factor” on new version of the same extension/bundle
* cryptographic signatures from a trusted entity on extensions/bundles to allow for verification even over untrusted transmission
* mechanisms to verify the available signatures automatically within NiFi rather than requiring external/manual verification 
* the same mutual authentication via TLS we provide for other services
* the ability to configure multiple registries to allow for redundancy, load balancing, mitigation of denial of service, etc. 

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Nov 14, 2018, at 06:55, Andrew Grande <ap...@gmail.com> wrote:
> 
> I would like to see a clear separation of blob and metadata storage. Most
> often you'd see some object storage being already distributed and
> replicated, let's think about an easy way to backup or migrate the metadata
> between registry instances.
> 
> Andrew
> 
>> On Tue, Nov 13, 2018, 11:32 AM Michael Moser <mo...@gmail.com> wrote:
>> 
>> I have thought about this in the past, too.  Here's a scenario where I
>> could never really lay down an approach I was happy with.
>> 
>> Consider that a NiFi user searches the NiFi registry and finds the PutMongo
>> processor.  Registry knows the PutMongo processor is in the
>> nifi-mongodb-nar, and through its Nar-Dependency-Id that it has a
>> dependency on a controller service interface in
>> nifi-mongodb-client-service-api-nar.  Great, the user can then download and
>> install those two nars.  How would we then suggest that the user also needs
>> a MongoDBClientService controller service implementation, such as that in
>> the nifi-mongodb-services-nar?
>> 
>> I'm not looking for an answer now, of course, but I just wanted to feed the
>> discussion.
>> 
>> Thanks,
>> -- Mike
>> 
>> 
>>> On Tue, Nov 13, 2018 at 1:34 PM Bryan Bende <bb...@gmail.com> wrote:
>>> 
>>> Mark,
>>> 
>>> I think there are a couple of ways that could be solved, but
>>> ultimately it would be up to how the users choose to setup/manage the
>>> registry, or registries.
>>> 
>>> The NiFi Registry security model is based around permissions to
>>> buckets (read/write), and all versioned items belong to a bucket. So
>>> you could think of each bucket as a mini extension repository for
>>> which you can control access to specific users and groups, so there
>>> could be a bucket of extensions for each of the NiFi instances in your
>>> example. There can also be multiple registry instances registered with
>>> a given NiFi so extensions can be pulled from multiple registries.
>>> 
>>> The extensions an instance needs is based on the flows it is running,
>>> the flows already have specific bundle coordinates for every component
>>> in the flow. You can think of it similar to Maven where you declared a
>>> dependency on library foo and the build goes out and gets it for you,
>>> in this case it is a flow that declares a dependency on a bundle.
>>> 
>>> Mike,
>>> 
>>> Bundles would need to be uploaded to NiFi Registry (to a specific
>>> bucket) as part of some TBD release process. At a minimum I was
>>> envisioning NiFi CLI commands that can be pointed to a file or a
>>> directory and upload the given bundles to registry. There could be
>>> other options as well, possibly through a Maven plugin to release
>>> directly into registry, or possibly to have a type of extension in
>>> NiFi Registry that actually points to an external location, i.e. all
>>> the NARs that end up in Maven central could somehow be imported into
>>> NiFi Registry, but with pointers back to the content which actually
>>> comes from Maven central. Lot of things to figure out here.
>>> 
>>> -Bryan
>>>> On Tue, Nov 13, 2018 at 1:16 PM Joe Witt <jo...@gmail.com> wrote:
>>>> 
>>>> Group selection based on tag names for bundles could probably do that.
>>>> Meaning it could be a sorting/filtering mechanism in the NiFi/Registry
>>>> interface perhaps.  Will be good to consider that UX as that
>>>> progresses.
>>>> 
>>>> As far as the different environments NiFi instances would certainly be
>>>> able to load only referenced Nars for versioned flows so you'll get
>>>> the optimal set (at runtime) automatically.  Very powerful.
>>>> On Tue, Nov 13, 2018 at 1:12 PM Mark Bean <ma...@gmail.com>
>> wrote:
>>>>> 
>>>>> Joe,
>>>>> 
>>>>> I envision the Registry being able to provide a subset of NARs
>>> required for
>>>>> a specific NiFi instance. The user may have a relatively small set of
>>> NARs
>>>>> required for a NiFi used for basic routing/distribution, and a
>>> different
>>>>> more extensive set of NARs required for a more robust NiFi instance
>>> which
>>>>> performs various forms of processing/transformations. The grouping I
>> am
>>>>> describing would be a way to select multiple NARs required for a
>>> specific
>>>>> NiFi instance.
>>>>> 
>>>>> Expanding the scenario a little farther, suppose an integration/test
>>>>> environment defines the group. Then, the production environment can
>>> use the
>>>>> group definition to pull (or ensure it possesses) only the relevant
>>> NARs
>>>>> necessary.
>>>>> 
>>>>> -Mark
>>>>> 
>>>>>> On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <jo...@gmail.com> wrote:
>>>>>> 
>>>>>> Mark
>>>>>> 
>>>>>> Can you describe your use case from the user perspective both for
>> the
>>>>>> entity that would upload the items and demarcate them as a group as
>>>>>> well as the user that would consume those bundles?
>>>>>> 
>>>>>> I ask because the point here is that nars are themselves a 'group'
>> in
>>>>>> that they are a logical/contained grouping of extensions.  These
>> can
>>>>>> have relationships to other nars as we know.  And flows are
>> designed
>>>>>> against specific components that come from certain nars for which
>> we
>>>>>> know the precise coordinates.  When importing flows that depend on
>>>>>> these the system will be able to automatically acquire all that it
>>>>>> needs.
>>>>>> 
>>>>>> Thanks
>>>>>> On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <ma...@gmail.com>
>>> wrote:
>>>>>>> 
>>>>>>> I would like to see a "group" capability in the Registry for NAR
>>> bundles.
>>>>>>> Then, users can create their own customized grouping of relevant
>>> NARs.
>>>>>>> Managing bundles one at a time will become tedious.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Mark
>>>>>>> 
>>>>>>> On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <jo...@gmail.com>
>>> wrote:
>>>>>>> 
>>>>>>>> Sivaprasanna - yes absolutely.  That is the core point I think
>> of
>>>>>>>> Bryan's first bullet under the 'NiFi' section.
>>>>>>>> On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <
>>>>>> sivaprasanna246@gmail.com>
>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> One quick question. With the extension registry, my
>>> understanding is
>>>>>> that
>>>>>>>>> we would try to reduce the overall NiFi size by offloading
>>> certain
>>>>>>>> existing
>>>>>>>>> NAR bundles to the extension registry. So are we expecting
>> the
>>>>>> extension
>>>>>>>>> registry to also come up with the ability/mechanism that lets
>>> an
>>>>>> user to
>>>>>>>>> download these bundles ?
>>>>>>>>> 
>>>>>>>>> -
>>>>>>>>> Sivaprasanna
>>>>>>>>> 
>>>>>>>>> On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <
>> joe.witt@gmail.com>
>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Bryan
>>>>>>>>>> 
>>>>>>>>>> Very exciting to see this under way!!!  We desperately have
>>> to get
>>>>>> our
>>>>>>>>>> core nifi build size way down and make it far easier for
>>> people to
>>>>>>>>>> contribute new processors.  We have a long line of
>>> extensions that
>>>>>>>>>> could be really useful/valuable and this will help unlock
>>> that
>>>>>> while
>>>>>>>>>> improving the user experience tremendously.
>>>>>>>>>> 
>>>>>>>>>> For the doc/concerns noted above we should also add/mention
>>> that
>>>>>> the
>>>>>>>>>> relationship between nars (dependencies between nars for
>>>>>>>>>> apis/controller services/parent nars/etc..) we want to have
>>> a way
>>>>>> to
>>>>>>>>>> manage/show that or a user experience for it.  Maybe not a
>>> day-1
>>>>>> thing
>>>>>>>>>> but important to call out.
>>>>>>>>>> 
>>>>>>>>>> Thanks!
>>>>>>>>>> Joe
>>>>>>>>>> On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <
>>> bbende@gmail.com>
>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> All,
>>>>>>>>>>> 
>>>>>>>>>>> We've needed the elusive extension registry for quite
>> some
>>> time
>>>>>> now,
>>>>>>>>>>> and with NiFi Registry I think we are in a good place to
>>> make
>>>>>> some
>>>>>>>>>>> progress in this area.
>>>>>>>>>>> 
>>>>>>>>>>> I've started looking into adding "extension bundles" to
>>> NiFi
>>>>>> Registry
>>>>>>>>>>> as the next type of versioned item, along side the
>> existing
>>>>>> versioned
>>>>>>>>>>> flows, and I wanted to take a minute to outline how that
>>> approach
>>>>>>>>>>> could work before getting too far into it.
>>>>>>>>>>> 
>>>>>>>>>>> Also, I'd like to focus this discussion on the design and
>>>>>>>>>>> functionality of the extension registry, and not on how
>> the
>>>>>> community
>>>>>>>>>>> is going to use it. Topics like hosting a central
>> registry,
>>>>>> changing
>>>>>>>>>>> the build process, restructuring the git repo, releasing
>>> NARs,
>>>>>> etc,
>>>>>>>>>>> are all important topics, but first we need an extension
>>> registry
>>>>>>>>>>> before we can do any of that :)
>>>>>>>>>>> 
>>>>>>>>>>> Here is a high-level description of what needs to be
>>> done...
>>>>>>>>>>> 
>>>>>>>>>>> NiFi Registry
>>>>>>>>>>> 
>>>>>>>>>>> - Add a new type of item called an extension bundle,
>> where
>>> each
>>>>>>>> bundle
>>>>>>>>>>> can contain one ore extensions or APIs
>>>>>>>>>>> 
>>>>>>>>>>> - Support bundles for traditional NiFi (aka NARs) and
>> also
>>>>>> bundles
>>>>>>>> for
>>>>>>>>>>> MiNiFi CPP
>>>>>>>>>>> 
>>>>>>>>>>> - Ability to upload the binary artifact for a bundle and
>>> extract
>>>>>> the
>>>>>>>>>>> metadata about the bundle, and metadata about the
>>> extensions
>>>>>>>> contained
>>>>>>>>>>> in the bundle (more on this later)
>>>>>>>>>>> 
>>>>>>>>>>> - Provide a pluggable storage provider for saving the
>>> content of
>>>>>> each
>>>>>>>>>>> extension bundle so that we can have different
>>> implementations
>>>>>> like
>>>>>>>>>>> local fileysystem, S3, and other object stores
>>>>>>>>>>> 
>>>>>>>>>>> - Provide a REST API for listing and retrieving available
>>>>>> bundles,
>>>>>>>>>>> integrate this into the registry Java client and NiFi CLI
>>>>>>>>>>> 
>>>>>>>>>>> NAR Maven Plugin
>>>>>>>>>>> 
>>>>>>>>>>> - Generate a descriptor for each component in the NAR
>>> which will
>>>>>>>>>>> provide information like the description, tags,
>> restricted
>>> or
>>>>>> not,
>>>>>>>>>>> etc.
>>>>>>>>>>> 
>>>>>>>>>>> - These descriptors will be parsed by NiFi Registry when
>> a
>>> NAR is
>>>>>>>>>>> being uploaded so that NiFi Registry will know about the
>>>>>> extensions
>>>>>>>>>>> contained with in the NAR
>>>>>>>>>>> 
>>>>>>>>>>> NiFi
>>>>>>>>>>> 
>>>>>>>>>>> - Provide some type of extension manager experience where
>>> users
>>>>>> can
>>>>>>>>>>> search, browse, & install bundles that are available in
>>> any of
>>>>>> the
>>>>>>>>>>> registry clients that have been defined
>>>>>>>>>>> 
>>>>>>>>>>> - Introduce a new security policy to control which users
>>> are
>>>>>> allowed
>>>>>>>>>>> to access the extension manager
>>>>>>>>>>> 
>>>>>>>>>>> - Installing a bundle should load the NAR and make the
>>> extensions
>>>>>>>>>>> available leveraging the recent work done to auto-load
>> new
>>> NARs
>>>>>>>>>>> 
>>>>>>>>>>> - Importing versioned flows from registry should provide
>>> an easy
>>>>>> way
>>>>>>>>>>> to install bundles that are required by the flow, but
>>> missing
>>>>>> from
>>>>>>>> the
>>>>>>>>>>> local NiFi instance
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> If anyone has any thoughts or concerns about this
>> approach,
>>>>>> please
>>>>>>>> let
>>>>>>>>>> me know.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> 
>>>>>>>>>>> Bryan
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>> 
>> 

Re: [DISCUSS] Extension Registry

[Andrew Grande <ap...@gmail.com>]

I would like to see a clear separation of blob and metadata storage. Most
often you'd see some object storage being already distributed and
replicated, let's think about an easy way to backup or migrate the metadata
between registry instances.

Andrew

On Tue, Nov 13, 2018, 11:32 AM Michael Moser <mo...@gmail.com> wrote:

> I have thought about this in the past, too.  Here's a scenario where I
> could never really lay down an approach I was happy with.
>
> Consider that a NiFi user searches the NiFi registry and finds the PutMongo
> processor.  Registry knows the PutMongo processor is in the
> nifi-mongodb-nar, and through its Nar-Dependency-Id that it has a
> dependency on a controller service interface in
> nifi-mongodb-client-service-api-nar.  Great, the user can then download and
> install those two nars.  How would we then suggest that the user also needs
> a MongoDBClientService controller service implementation, such as that in
> the nifi-mongodb-services-nar?
>
> I'm not looking for an answer now, of course, but I just wanted to feed the
> discussion.
>
> Thanks,
> -- Mike
>
>
> On Tue, Nov 13, 2018 at 1:34 PM Bryan Bende <bb...@gmail.com> wrote:
>
> > Mark,
> >
> > I think there are a couple of ways that could be solved, but
> > ultimately it would be up to how the users choose to setup/manage the
> > registry, or registries.
> >
> > The NiFi Registry security model is based around permissions to
> > buckets (read/write), and all versioned items belong to a bucket. So
> > you could think of each bucket as a mini extension repository for
> > which you can control access to specific users and groups, so there
> > could be a bucket of extensions for each of the NiFi instances in your
> > example. There can also be multiple registry instances registered with
> > a given NiFi so extensions can be pulled from multiple registries.
> >
> > The extensions an instance needs is based on the flows it is running,
> > the flows already have specific bundle coordinates for every component
> > in the flow. You can think of it similar to Maven where you declared a
> > dependency on library foo and the build goes out and gets it for you,
> > in this case it is a flow that declares a dependency on a bundle.
> >
> > Mike,
> >
> > Bundles would need to be uploaded to NiFi Registry (to a specific
> > bucket) as part of some TBD release process. At a minimum I was
> > envisioning NiFi CLI commands that can be pointed to a file or a
> > directory and upload the given bundles to registry. There could be
> > other options as well, possibly through a Maven plugin to release
> > directly into registry, or possibly to have a type of extension in
> > NiFi Registry that actually points to an external location, i.e. all
> > the NARs that end up in Maven central could somehow be imported into
> > NiFi Registry, but with pointers back to the content which actually
> > comes from Maven central. Lot of things to figure out here.
> >
> > -Bryan
> > On Tue, Nov 13, 2018 at 1:16 PM Joe Witt <jo...@gmail.com> wrote:
> > >
> > > Group selection based on tag names for bundles could probably do that.
> > > Meaning it could be a sorting/filtering mechanism in the NiFi/Registry
> > > interface perhaps.  Will be good to consider that UX as that
> > > progresses.
> > >
> > > As far as the different environments NiFi instances would certainly be
> > > able to load only referenced Nars for versioned flows so you'll get
> > > the optimal set (at runtime) automatically.  Very powerful.
> > > On Tue, Nov 13, 2018 at 1:12 PM Mark Bean <ma...@gmail.com>
> wrote:
> > > >
> > > > Joe,
> > > >
> > > > I envision the Registry being able to provide a subset of NARs
> > required for
> > > > a specific NiFi instance. The user may have a relatively small set of
> > NARs
> > > > required for a NiFi used for basic routing/distribution, and a
> > different
> > > > more extensive set of NARs required for a more robust NiFi instance
> > which
> > > > performs various forms of processing/transformations. The grouping I
> am
> > > > describing would be a way to select multiple NARs required for a
> > specific
> > > > NiFi instance.
> > > >
> > > > Expanding the scenario a little farther, suppose an integration/test
> > > > environment defines the group. Then, the production environment can
> > use the
> > > > group definition to pull (or ensure it possesses) only the relevant
> > NARs
> > > > necessary.
> > > >
> > > > -Mark
> > > >
> > > > On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <jo...@gmail.com> wrote:
> > > >
> > > > > Mark
> > > > >
> > > > > Can you describe your use case from the user perspective both for
> the
> > > > > entity that would upload the items and demarcate them as a group as
> > > > > well as the user that would consume those bundles?
> > > > >
> > > > > I ask because the point here is that nars are themselves a 'group'
> in
> > > > > that they are a logical/contained grouping of extensions.  These
> can
> > > > > have relationships to other nars as we know.  And flows are
> designed
> > > > > against specific components that come from certain nars for which
> we
> > > > > know the precise coordinates.  When importing flows that depend on
> > > > > these the system will be able to automatically acquire all that it
> > > > > needs.
> > > > >
> > > > > Thanks
> > > > > On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <ma...@gmail.com>
> > wrote:
> > > > > >
> > > > > > I would like to see a "group" capability in the Registry for NAR
> > bundles.
> > > > > > Then, users can create their own customized grouping of relevant
> > NARs.
> > > > > > Managing bundles one at a time will become tedious.
> > > > > >
> > > > > > Thanks,
> > > > > > Mark
> > > > > >
> > > > > > On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <jo...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Sivaprasanna - yes absolutely.  That is the core point I think
> of
> > > > > > > Bryan's first bullet under the 'NiFi' section.
> > > > > > > On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <
> > > > > sivaprasanna246@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > One quick question. With the extension registry, my
> > understanding is
> > > > > that
> > > > > > > > we would try to reduce the overall NiFi size by offloading
> > certain
> > > > > > > existing
> > > > > > > > NAR bundles to the extension registry. So are we expecting
> the
> > > > > extension
> > > > > > > > registry to also come up with the ability/mechanism that lets
> > an
> > > > > user to
> > > > > > > > download these bundles ?
> > > > > > > >
> > > > > > > > -
> > > > > > > > Sivaprasanna
> > > > > > > >
> > > > > > > > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <
> joe.witt@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Bryan
> > > > > > > > >
> > > > > > > > > Very exciting to see this under way!!!  We desperately have
> > to get
> > > > > our
> > > > > > > > > core nifi build size way down and make it far easier for
> > people to
> > > > > > > > > contribute new processors.  We have a long line of
> > extensions that
> > > > > > > > > could be really useful/valuable and this will help unlock
> > that
> > > > > while
> > > > > > > > > improving the user experience tremendously.
> > > > > > > > >
> > > > > > > > > For the doc/concerns noted above we should also add/mention
> > that
> > > > > the
> > > > > > > > > relationship between nars (dependencies between nars for
> > > > > > > > > apis/controller services/parent nars/etc..) we want to have
> > a way
> > > > > to
> > > > > > > > > manage/show that or a user experience for it.  Maybe not a
> > day-1
> > > > > thing
> > > > > > > > > but important to call out.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > > Joe
> > > > > > > > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <
> > bbende@gmail.com>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > All,
> > > > > > > > > >
> > > > > > > > > > We've needed the elusive extension registry for quite
> some
> > time
> > > > > now,
> > > > > > > > > > and with NiFi Registry I think we are in a good place to
> > make
> > > > > some
> > > > > > > > > > progress in this area.
> > > > > > > > > >
> > > > > > > > > > I've started looking into adding "extension bundles" to
> > NiFi
> > > > > Registry
> > > > > > > > > > as the next type of versioned item, along side the
> existing
> > > > > versioned
> > > > > > > > > > flows, and I wanted to take a minute to outline how that
> > approach
> > > > > > > > > > could work before getting too far into it.
> > > > > > > > > >
> > > > > > > > > > Also, I'd like to focus this discussion on the design and
> > > > > > > > > > functionality of the extension registry, and not on how
> the
> > > > > community
> > > > > > > > > > is going to use it. Topics like hosting a central
> registry,
> > > > > changing
> > > > > > > > > > the build process, restructuring the git repo, releasing
> > NARs,
> > > > > etc,
> > > > > > > > > > are all important topics, but first we need an extension
> > registry
> > > > > > > > > > before we can do any of that :)
> > > > > > > > > >
> > > > > > > > > > Here is a high-level description of what needs to be
> > done...
> > > > > > > > > >
> > > > > > > > > > NiFi Registry
> > > > > > > > > >
> > > > > > > > > > - Add a new type of item called an extension bundle,
> where
> > each
> > > > > > > bundle
> > > > > > > > > > can contain one ore extensions or APIs
> > > > > > > > > >
> > > > > > > > > > - Support bundles for traditional NiFi (aka NARs) and
> also
> > > > > bundles
> > > > > > > for
> > > > > > > > > > MiNiFi CPP
> > > > > > > > > >
> > > > > > > > > > - Ability to upload the binary artifact for a bundle and
> > extract
> > > > > the
> > > > > > > > > > metadata about the bundle, and metadata about the
> > extensions
> > > > > > > contained
> > > > > > > > > > in the bundle (more on this later)
> > > > > > > > > >
> > > > > > > > > > - Provide a pluggable storage provider for saving the
> > content of
> > > > > each
> > > > > > > > > > extension bundle so that we can have different
> > implementations
> > > > > like
> > > > > > > > > > local fileysystem, S3, and other object stores
> > > > > > > > > >
> > > > > > > > > > - Provide a REST API for listing and retrieving available
> > > > > bundles,
> > > > > > > > > > integrate this into the registry Java client and NiFi CLI
> > > > > > > > > >
> > > > > > > > > > NAR Maven Plugin
> > > > > > > > > >
> > > > > > > > > > - Generate a descriptor for each component in the NAR
> > which will
> > > > > > > > > > provide information like the description, tags,
> restricted
> > or
> > > > > not,
> > > > > > > > > > etc.
> > > > > > > > > >
> > > > > > > > > > - These descriptors will be parsed by NiFi Registry when
> a
> > NAR is
> > > > > > > > > > being uploaded so that NiFi Registry will know about the
> > > > > extensions
> > > > > > > > > > contained with in the NAR
> > > > > > > > > >
> > > > > > > > > > NiFi
> > > > > > > > > >
> > > > > > > > > > - Provide some type of extension manager experience where
> > users
> > > > > can
> > > > > > > > > > search, browse, & install bundles that are available in
> > any of
> > > > > the
> > > > > > > > > > registry clients that have been defined
> > > > > > > > > >
> > > > > > > > > > - Introduce a new security policy to control which users
> > are
> > > > > allowed
> > > > > > > > > > to access the extension manager
> > > > > > > > > >
> > > > > > > > > > - Installing a bundle should load the NAR and make the
> > extensions
> > > > > > > > > > available leveraging the recent work done to auto-load
> new
> > NARs
> > > > > > > > > >
> > > > > > > > > > - Importing versioned flows from registry should provide
> > an easy
> > > > > way
> > > > > > > > > > to install bundles that are required by the flow, but
> > missing
> > > > > from
> > > > > > > the
> > > > > > > > > > local NiFi instance
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > If anyone has any thoughts or concerns about this
> approach,
> > > > > please
> > > > > > > let
> > > > > > > > > me know.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Bryan
> > > > > > > > >
> > > > > > >
> > > > >
> >
>

Re: [DISCUSS] Extension Registry

[Mark Payne <ma...@hotmail.com>]

Mike,

Thanks for bringing that up. It's a good point that we need to keep in mind. As mentioned,
the NAR Maven Plugin will be generating the info necessary and putting it into the NAR. One
piece of information that it will gather is any Controller Service API's that are provided by a given
implementation. So the registry will have this information. So the technical details are not concerning.
We do, however, need to ensure that we provide a good User Experience for such a thing. Perhaps
when you click to create a new Controller Service from a 'Configure' dialog we should have a way to
jump to a 'remote registry browser' type of thing and download from there...


> On Nov 13, 2018, at 2:32 PM, Michael Moser <mo...@gmail.com> wrote:
> 
> I have thought about this in the past, too.  Here's a scenario where I
> could never really lay down an approach I was happy with.
> 
> Consider that a NiFi user searches the NiFi registry and finds the PutMongo
> processor.  Registry knows the PutMongo processor is in the
> nifi-mongodb-nar, and through its Nar-Dependency-Id that it has a
> dependency on a controller service interface in
> nifi-mongodb-client-service-api-nar.  Great, the user can then download and
> install those two nars.  How would we then suggest that the user also needs
> a MongoDBClientService controller service implementation, such as that in
> the nifi-mongodb-services-nar?
> 
> I'm not looking for an answer now, of course, but I just wanted to feed the
> discussion.
> 
> Thanks,
> -- Mike
> 
> 
> On Tue, Nov 13, 2018 at 1:34 PM Bryan Bende <bb...@gmail.com> wrote:
> 
>> Mark,
>> 
>> I think there are a couple of ways that could be solved, but
>> ultimately it would be up to how the users choose to setup/manage the
>> registry, or registries.
>> 
>> The NiFi Registry security model is based around permissions to
>> buckets (read/write), and all versioned items belong to a bucket. So
>> you could think of each bucket as a mini extension repository for
>> which you can control access to specific users and groups, so there
>> could be a bucket of extensions for each of the NiFi instances in your
>> example. There can also be multiple registry instances registered with
>> a given NiFi so extensions can be pulled from multiple registries.
>> 
>> The extensions an instance needs is based on the flows it is running,
>> the flows already have specific bundle coordinates for every component
>> in the flow. You can think of it similar to Maven where you declared a
>> dependency on library foo and the build goes out and gets it for you,
>> in this case it is a flow that declares a dependency on a bundle.
>> 
>> Mike,
>> 
>> Bundles would need to be uploaded to NiFi Registry (to a specific
>> bucket) as part of some TBD release process. At a minimum I was
>> envisioning NiFi CLI commands that can be pointed to a file or a
>> directory and upload the given bundles to registry. There could be
>> other options as well, possibly through a Maven plugin to release
>> directly into registry, or possibly to have a type of extension in
>> NiFi Registry that actually points to an external location, i.e. all
>> the NARs that end up in Maven central could somehow be imported into
>> NiFi Registry, but with pointers back to the content which actually
>> comes from Maven central. Lot of things to figure out here.
>> 
>> -Bryan
>> On Tue, Nov 13, 2018 at 1:16 PM Joe Witt <jo...@gmail.com> wrote:
>>> 
>>> Group selection based on tag names for bundles could probably do that.
>>> Meaning it could be a sorting/filtering mechanism in the NiFi/Registry
>>> interface perhaps.  Will be good to consider that UX as that
>>> progresses.
>>> 
>>> As far as the different environments NiFi instances would certainly be
>>> able to load only referenced Nars for versioned flows so you'll get
>>> the optimal set (at runtime) automatically.  Very powerful.
>>> On Tue, Nov 13, 2018 at 1:12 PM Mark Bean <ma...@gmail.com> wrote:
>>>> 
>>>> Joe,
>>>> 
>>>> I envision the Registry being able to provide a subset of NARs
>> required for
>>>> a specific NiFi instance. The user may have a relatively small set of
>> NARs
>>>> required for a NiFi used for basic routing/distribution, and a
>> different
>>>> more extensive set of NARs required for a more robust NiFi instance
>> which
>>>> performs various forms of processing/transformations. The grouping I am
>>>> describing would be a way to select multiple NARs required for a
>> specific
>>>> NiFi instance.
>>>> 
>>>> Expanding the scenario a little farther, suppose an integration/test
>>>> environment defines the group. Then, the production environment can
>> use the
>>>> group definition to pull (or ensure it possesses) only the relevant
>> NARs
>>>> necessary.
>>>> 
>>>> -Mark
>>>> 
>>>> On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <jo...@gmail.com> wrote:
>>>> 
>>>>> Mark
>>>>> 
>>>>> Can you describe your use case from the user perspective both for the
>>>>> entity that would upload the items and demarcate them as a group as
>>>>> well as the user that would consume those bundles?
>>>>> 
>>>>> I ask because the point here is that nars are themselves a 'group' in
>>>>> that they are a logical/contained grouping of extensions.  These can
>>>>> have relationships to other nars as we know.  And flows are designed
>>>>> against specific components that come from certain nars for which we
>>>>> know the precise coordinates.  When importing flows that depend on
>>>>> these the system will be able to automatically acquire all that it
>>>>> needs.
>>>>> 
>>>>> Thanks
>>>>> On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <ma...@gmail.com>
>> wrote:
>>>>>> 
>>>>>> I would like to see a "group" capability in the Registry for NAR
>> bundles.
>>>>>> Then, users can create their own customized grouping of relevant
>> NARs.
>>>>>> Managing bundles one at a time will become tedious.
>>>>>> 
>>>>>> Thanks,
>>>>>> Mark
>>>>>> 
>>>>>> On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <jo...@gmail.com>
>> wrote:
>>>>>> 
>>>>>>> Sivaprasanna - yes absolutely.  That is the core point I think of
>>>>>>> Bryan's first bullet under the 'NiFi' section.
>>>>>>> On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <
>>>>> sivaprasanna246@gmail.com>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> One quick question. With the extension registry, my
>> understanding is
>>>>> that
>>>>>>>> we would try to reduce the overall NiFi size by offloading
>> certain
>>>>>>> existing
>>>>>>>> NAR bundles to the extension registry. So are we expecting the
>>>>> extension
>>>>>>>> registry to also come up with the ability/mechanism that lets
>> an
>>>>> user to
>>>>>>>> download these bundles ?
>>>>>>>> 
>>>>>>>> -
>>>>>>>> Sivaprasanna
>>>>>>>> 
>>>>>>>> On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <jo...@gmail.com>
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Bryan
>>>>>>>>> 
>>>>>>>>> Very exciting to see this under way!!!  We desperately have
>> to get
>>>>> our
>>>>>>>>> core nifi build size way down and make it far easier for
>> people to
>>>>>>>>> contribute new processors.  We have a long line of
>> extensions that
>>>>>>>>> could be really useful/valuable and this will help unlock
>> that
>>>>> while
>>>>>>>>> improving the user experience tremendously.
>>>>>>>>> 
>>>>>>>>> For the doc/concerns noted above we should also add/mention
>> that
>>>>> the
>>>>>>>>> relationship between nars (dependencies between nars for
>>>>>>>>> apis/controller services/parent nars/etc..) we want to have
>> a way
>>>>> to
>>>>>>>>> manage/show that or a user experience for it.  Maybe not a
>> day-1
>>>>> thing
>>>>>>>>> but important to call out.
>>>>>>>>> 
>>>>>>>>> Thanks!
>>>>>>>>> Joe
>>>>>>>>> On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <
>> bbende@gmail.com>
>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> All,
>>>>>>>>>> 
>>>>>>>>>> We've needed the elusive extension registry for quite some
>> time
>>>>> now,
>>>>>>>>>> and with NiFi Registry I think we are in a good place to
>> make
>>>>> some
>>>>>>>>>> progress in this area.
>>>>>>>>>> 
>>>>>>>>>> I've started looking into adding "extension bundles" to
>> NiFi
>>>>> Registry
>>>>>>>>>> as the next type of versioned item, along side the existing
>>>>> versioned
>>>>>>>>>> flows, and I wanted to take a minute to outline how that
>> approach
>>>>>>>>>> could work before getting too far into it.
>>>>>>>>>> 
>>>>>>>>>> Also, I'd like to focus this discussion on the design and
>>>>>>>>>> functionality of the extension registry, and not on how the
>>>>> community
>>>>>>>>>> is going to use it. Topics like hosting a central registry,
>>>>> changing
>>>>>>>>>> the build process, restructuring the git repo, releasing
>> NARs,
>>>>> etc,
>>>>>>>>>> are all important topics, but first we need an extension
>> registry
>>>>>>>>>> before we can do any of that :)
>>>>>>>>>> 
>>>>>>>>>> Here is a high-level description of what needs to be
>> done...
>>>>>>>>>> 
>>>>>>>>>> NiFi Registry
>>>>>>>>>> 
>>>>>>>>>> - Add a new type of item called an extension bundle, where
>> each
>>>>>>> bundle
>>>>>>>>>> can contain one ore extensions or APIs
>>>>>>>>>> 
>>>>>>>>>> - Support bundles for traditional NiFi (aka NARs) and also
>>>>> bundles
>>>>>>> for
>>>>>>>>>> MiNiFi CPP
>>>>>>>>>> 
>>>>>>>>>> - Ability to upload the binary artifact for a bundle and
>> extract
>>>>> the
>>>>>>>>>> metadata about the bundle, and metadata about the
>> extensions
>>>>>>> contained
>>>>>>>>>> in the bundle (more on this later)
>>>>>>>>>> 
>>>>>>>>>> - Provide a pluggable storage provider for saving the
>> content of
>>>>> each
>>>>>>>>>> extension bundle so that we can have different
>> implementations
>>>>> like
>>>>>>>>>> local fileysystem, S3, and other object stores
>>>>>>>>>> 
>>>>>>>>>> - Provide a REST API for listing and retrieving available
>>>>> bundles,
>>>>>>>>>> integrate this into the registry Java client and NiFi CLI
>>>>>>>>>> 
>>>>>>>>>> NAR Maven Plugin
>>>>>>>>>> 
>>>>>>>>>> - Generate a descriptor for each component in the NAR
>> which will
>>>>>>>>>> provide information like the description, tags, restricted
>> or
>>>>> not,
>>>>>>>>>> etc.
>>>>>>>>>> 
>>>>>>>>>> - These descriptors will be parsed by NiFi Registry when a
>> NAR is
>>>>>>>>>> being uploaded so that NiFi Registry will know about the
>>>>> extensions
>>>>>>>>>> contained with in the NAR
>>>>>>>>>> 
>>>>>>>>>> NiFi
>>>>>>>>>> 
>>>>>>>>>> - Provide some type of extension manager experience where
>> users
>>>>> can
>>>>>>>>>> search, browse, & install bundles that are available in
>> any of
>>>>> the
>>>>>>>>>> registry clients that have been defined
>>>>>>>>>> 
>>>>>>>>>> - Introduce a new security policy to control which users
>> are
>>>>> allowed
>>>>>>>>>> to access the extension manager
>>>>>>>>>> 
>>>>>>>>>> - Installing a bundle should load the NAR and make the
>> extensions
>>>>>>>>>> available leveraging the recent work done to auto-load new
>> NARs
>>>>>>>>>> 
>>>>>>>>>> - Importing versioned flows from registry should provide
>> an easy
>>>>> way
>>>>>>>>>> to install bundles that are required by the flow, but
>> missing
>>>>> from
>>>>>>> the
>>>>>>>>>> local NiFi instance
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> If anyone has any thoughts or concerns about this approach,
>>>>> please
>>>>>>> let
>>>>>>>>> me know.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> 
>>>>>>>>>> Bryan
>>>>>>>>> 
>>>>>>> 
>>>>> 
>> 

Re: [DISCUSS] Extension Registry

[Michael Moser <mo...@gmail.com>]

I have thought about this in the past, too.  Here's a scenario where I
could never really lay down an approach I was happy with.

Consider that a NiFi user searches the NiFi registry and finds the PutMongo
processor.  Registry knows the PutMongo processor is in the
nifi-mongodb-nar, and through its Nar-Dependency-Id that it has a
dependency on a controller service interface in
nifi-mongodb-client-service-api-nar.  Great, the user can then download and
install those two nars.  How would we then suggest that the user also needs
a MongoDBClientService controller service implementation, such as that in
the nifi-mongodb-services-nar?

I'm not looking for an answer now, of course, but I just wanted to feed the
discussion.

Thanks,
-- Mike


On Tue, Nov 13, 2018 at 1:34 PM Bryan Bende <bb...@gmail.com> wrote:

> Mark,
>
> I think there are a couple of ways that could be solved, but
> ultimately it would be up to how the users choose to setup/manage the
> registry, or registries.
>
> The NiFi Registry security model is based around permissions to
> buckets (read/write), and all versioned items belong to a bucket. So
> you could think of each bucket as a mini extension repository for
> which you can control access to specific users and groups, so there
> could be a bucket of extensions for each of the NiFi instances in your
> example. There can also be multiple registry instances registered with
> a given NiFi so extensions can be pulled from multiple registries.
>
> The extensions an instance needs is based on the flows it is running,
> the flows already have specific bundle coordinates for every component
> in the flow. You can think of it similar to Maven where you declared a
> dependency on library foo and the build goes out and gets it for you,
> in this case it is a flow that declares a dependency on a bundle.
>
> Mike,
>
> Bundles would need to be uploaded to NiFi Registry (to a specific
> bucket) as part of some TBD release process. At a minimum I was
> envisioning NiFi CLI commands that can be pointed to a file or a
> directory and upload the given bundles to registry. There could be
> other options as well, possibly through a Maven plugin to release
> directly into registry, or possibly to have a type of extension in
> NiFi Registry that actually points to an external location, i.e. all
> the NARs that end up in Maven central could somehow be imported into
> NiFi Registry, but with pointers back to the content which actually
> comes from Maven central. Lot of things to figure out here.
>
> -Bryan
> On Tue, Nov 13, 2018 at 1:16 PM Joe Witt <jo...@gmail.com> wrote:
> >
> > Group selection based on tag names for bundles could probably do that.
> > Meaning it could be a sorting/filtering mechanism in the NiFi/Registry
> > interface perhaps.  Will be good to consider that UX as that
> > progresses.
> >
> > As far as the different environments NiFi instances would certainly be
> > able to load only referenced Nars for versioned flows so you'll get
> > the optimal set (at runtime) automatically.  Very powerful.
> > On Tue, Nov 13, 2018 at 1:12 PM Mark Bean <ma...@gmail.com> wrote:
> > >
> > > Joe,
> > >
> > > I envision the Registry being able to provide a subset of NARs
> required for
> > > a specific NiFi instance. The user may have a relatively small set of
> NARs
> > > required for a NiFi used for basic routing/distribution, and a
> different
> > > more extensive set of NARs required for a more robust NiFi instance
> which
> > > performs various forms of processing/transformations. The grouping I am
> > > describing would be a way to select multiple NARs required for a
> specific
> > > NiFi instance.
> > >
> > > Expanding the scenario a little farther, suppose an integration/test
> > > environment defines the group. Then, the production environment can
> use the
> > > group definition to pull (or ensure it possesses) only the relevant
> NARs
> > > necessary.
> > >
> > > -Mark
> > >
> > > On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <jo...@gmail.com> wrote:
> > >
> > > > Mark
> > > >
> > > > Can you describe your use case from the user perspective both for the
> > > > entity that would upload the items and demarcate them as a group as
> > > > well as the user that would consume those bundles?
> > > >
> > > > I ask because the point here is that nars are themselves a 'group' in
> > > > that they are a logical/contained grouping of extensions.  These can
> > > > have relationships to other nars as we know.  And flows are designed
> > > > against specific components that come from certain nars for which we
> > > > know the precise coordinates.  When importing flows that depend on
> > > > these the system will be able to automatically acquire all that it
> > > > needs.
> > > >
> > > > Thanks
> > > > On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <ma...@gmail.com>
> wrote:
> > > > >
> > > > > I would like to see a "group" capability in the Registry for NAR
> bundles.
> > > > > Then, users can create their own customized grouping of relevant
> NARs.
> > > > > Managing bundles one at a time will become tedious.
> > > > >
> > > > > Thanks,
> > > > > Mark
> > > > >
> > > > > On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <jo...@gmail.com>
> wrote:
> > > > >
> > > > > > Sivaprasanna - yes absolutely.  That is the core point I think of
> > > > > > Bryan's first bullet under the 'NiFi' section.
> > > > > > On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <
> > > > sivaprasanna246@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > One quick question. With the extension registry, my
> understanding is
> > > > that
> > > > > > > we would try to reduce the overall NiFi size by offloading
> certain
> > > > > > existing
> > > > > > > NAR bundles to the extension registry. So are we expecting the
> > > > extension
> > > > > > > registry to also come up with the ability/mechanism that lets
> an
> > > > user to
> > > > > > > download these bundles ?
> > > > > > >
> > > > > > > -
> > > > > > > Sivaprasanna
> > > > > > >
> > > > > > > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <jo...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Bryan
> > > > > > > >
> > > > > > > > Very exciting to see this under way!!!  We desperately have
> to get
> > > > our
> > > > > > > > core nifi build size way down and make it far easier for
> people to
> > > > > > > > contribute new processors.  We have a long line of
> extensions that
> > > > > > > > could be really useful/valuable and this will help unlock
> that
> > > > while
> > > > > > > > improving the user experience tremendously.
> > > > > > > >
> > > > > > > > For the doc/concerns noted above we should also add/mention
> that
> > > > the
> > > > > > > > relationship between nars (dependencies between nars for
> > > > > > > > apis/controller services/parent nars/etc..) we want to have
> a way
> > > > to
> > > > > > > > manage/show that or a user experience for it.  Maybe not a
> day-1
> > > > thing
> > > > > > > > but important to call out.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > Joe
> > > > > > > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <
> bbende@gmail.com>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > All,
> > > > > > > > >
> > > > > > > > > We've needed the elusive extension registry for quite some
> time
> > > > now,
> > > > > > > > > and with NiFi Registry I think we are in a good place to
> make
> > > > some
> > > > > > > > > progress in this area.
> > > > > > > > >
> > > > > > > > > I've started looking into adding "extension bundles" to
> NiFi
> > > > Registry
> > > > > > > > > as the next type of versioned item, along side the existing
> > > > versioned
> > > > > > > > > flows, and I wanted to take a minute to outline how that
> approach
> > > > > > > > > could work before getting too far into it.
> > > > > > > > >
> > > > > > > > > Also, I'd like to focus this discussion on the design and
> > > > > > > > > functionality of the extension registry, and not on how the
> > > > community
> > > > > > > > > is going to use it. Topics like hosting a central registry,
> > > > changing
> > > > > > > > > the build process, restructuring the git repo, releasing
> NARs,
> > > > etc,
> > > > > > > > > are all important topics, but first we need an extension
> registry
> > > > > > > > > before we can do any of that :)
> > > > > > > > >
> > > > > > > > > Here is a high-level description of what needs to be
> done...
> > > > > > > > >
> > > > > > > > > NiFi Registry
> > > > > > > > >
> > > > > > > > > - Add a new type of item called an extension bundle, where
> each
> > > > > > bundle
> > > > > > > > > can contain one ore extensions or APIs
> > > > > > > > >
> > > > > > > > > - Support bundles for traditional NiFi (aka NARs) and also
> > > > bundles
> > > > > > for
> > > > > > > > > MiNiFi CPP
> > > > > > > > >
> > > > > > > > > - Ability to upload the binary artifact for a bundle and
> extract
> > > > the
> > > > > > > > > metadata about the bundle, and metadata about the
> extensions
> > > > > > contained
> > > > > > > > > in the bundle (more on this later)
> > > > > > > > >
> > > > > > > > > - Provide a pluggable storage provider for saving the
> content of
> > > > each
> > > > > > > > > extension bundle so that we can have different
> implementations
> > > > like
> > > > > > > > > local fileysystem, S3, and other object stores
> > > > > > > > >
> > > > > > > > > - Provide a REST API for listing and retrieving available
> > > > bundles,
> > > > > > > > > integrate this into the registry Java client and NiFi CLI
> > > > > > > > >
> > > > > > > > > NAR Maven Plugin
> > > > > > > > >
> > > > > > > > > - Generate a descriptor for each component in the NAR
> which will
> > > > > > > > > provide information like the description, tags, restricted
> or
> > > > not,
> > > > > > > > > etc.
> > > > > > > > >
> > > > > > > > > - These descriptors will be parsed by NiFi Registry when a
> NAR is
> > > > > > > > > being uploaded so that NiFi Registry will know about the
> > > > extensions
> > > > > > > > > contained with in the NAR
> > > > > > > > >
> > > > > > > > > NiFi
> > > > > > > > >
> > > > > > > > > - Provide some type of extension manager experience where
> users
> > > > can
> > > > > > > > > search, browse, & install bundles that are available in
> any of
> > > > the
> > > > > > > > > registry clients that have been defined
> > > > > > > > >
> > > > > > > > > - Introduce a new security policy to control which users
> are
> > > > allowed
> > > > > > > > > to access the extension manager
> > > > > > > > >
> > > > > > > > > - Installing a bundle should load the NAR and make the
> extensions
> > > > > > > > > available leveraging the recent work done to auto-load new
> NARs
> > > > > > > > >
> > > > > > > > > - Importing versioned flows from registry should provide
> an easy
> > > > way
> > > > > > > > > to install bundles that are required by the flow, but
> missing
> > > > from
> > > > > > the
> > > > > > > > > local NiFi instance
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > If anyone has any thoughts or concerns about this approach,
> > > > please
> > > > > > let
> > > > > > > > me know.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Bryan
> > > > > > > >
> > > > > >
> > > >
>

Re: [DISCUSS] Extension Registry

[Bryan Bende <bb...@gmail.com>]

Mark,

I think there are a couple of ways that could be solved, but
ultimately it would be up to how the users choose to setup/manage the
registry, or registries.

The NiFi Registry security model is based around permissions to
buckets (read/write), and all versioned items belong to a bucket. So
you could think of each bucket as a mini extension repository for
which you can control access to specific users and groups, so there
could be a bucket of extensions for each of the NiFi instances in your
example. There can also be multiple registry instances registered with
a given NiFi so extensions can be pulled from multiple registries.

The extensions an instance needs is based on the flows it is running,
the flows already have specific bundle coordinates for every component
in the flow. You can think of it similar to Maven where you declared a
dependency on library foo and the build goes out and gets it for you,
in this case it is a flow that declares a dependency on a bundle.

Mike,

Bundles would need to be uploaded to NiFi Registry (to a specific
bucket) as part of some TBD release process. At a minimum I was
envisioning NiFi CLI commands that can be pointed to a file or a
directory and upload the given bundles to registry. There could be
other options as well, possibly through a Maven plugin to release
directly into registry, or possibly to have a type of extension in
NiFi Registry that actually points to an external location, i.e. all
the NARs that end up in Maven central could somehow be imported into
NiFi Registry, but with pointers back to the content which actually
comes from Maven central. Lot of things to figure out here.

-Bryan
On Tue, Nov 13, 2018 at 1:16 PM Joe Witt <jo...@gmail.com> wrote:
>
> Group selection based on tag names for bundles could probably do that.
> Meaning it could be a sorting/filtering mechanism in the NiFi/Registry
> interface perhaps.  Will be good to consider that UX as that
> progresses.
>
> As far as the different environments NiFi instances would certainly be
> able to load only referenced Nars for versioned flows so you'll get
> the optimal set (at runtime) automatically.  Very powerful.
> On Tue, Nov 13, 2018 at 1:12 PM Mark Bean <ma...@gmail.com> wrote:
> >
> > Joe,
> >
> > I envision the Registry being able to provide a subset of NARs required for
> > a specific NiFi instance. The user may have a relatively small set of NARs
> > required for a NiFi used for basic routing/distribution, and a different
> > more extensive set of NARs required for a more robust NiFi instance which
> > performs various forms of processing/transformations. The grouping I am
> > describing would be a way to select multiple NARs required for a specific
> > NiFi instance.
> >
> > Expanding the scenario a little farther, suppose an integration/test
> > environment defines the group. Then, the production environment can use the
> > group definition to pull (or ensure it possesses) only the relevant NARs
> > necessary.
> >
> > -Mark
> >
> > On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <jo...@gmail.com> wrote:
> >
> > > Mark
> > >
> > > Can you describe your use case from the user perspective both for the
> > > entity that would upload the items and demarcate them as a group as
> > > well as the user that would consume those bundles?
> > >
> > > I ask because the point here is that nars are themselves a 'group' in
> > > that they are a logical/contained grouping of extensions.  These can
> > > have relationships to other nars as we know.  And flows are designed
> > > against specific components that come from certain nars for which we
> > > know the precise coordinates.  When importing flows that depend on
> > > these the system will be able to automatically acquire all that it
> > > needs.
> > >
> > > Thanks
> > > On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <ma...@gmail.com> wrote:
> > > >
> > > > I would like to see a "group" capability in the Registry for NAR bundles.
> > > > Then, users can create their own customized grouping of relevant NARs.
> > > > Managing bundles one at a time will become tedious.
> > > >
> > > > Thanks,
> > > > Mark
> > > >
> > > > On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <jo...@gmail.com> wrote:
> > > >
> > > > > Sivaprasanna - yes absolutely.  That is the core point I think of
> > > > > Bryan's first bullet under the 'NiFi' section.
> > > > > On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <
> > > sivaprasanna246@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > One quick question. With the extension registry, my understanding is
> > > that
> > > > > > we would try to reduce the overall NiFi size by offloading certain
> > > > > existing
> > > > > > NAR bundles to the extension registry. So are we expecting the
> > > extension
> > > > > > registry to also come up with the ability/mechanism that lets an
> > > user to
> > > > > > download these bundles ?
> > > > > >
> > > > > > -
> > > > > > Sivaprasanna
> > > > > >
> > > > > > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <jo...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Bryan
> > > > > > >
> > > > > > > Very exciting to see this under way!!!  We desperately have to get
> > > our
> > > > > > > core nifi build size way down and make it far easier for people to
> > > > > > > contribute new processors.  We have a long line of extensions that
> > > > > > > could be really useful/valuable and this will help unlock that
> > > while
> > > > > > > improving the user experience tremendously.
> > > > > > >
> > > > > > > For the doc/concerns noted above we should also add/mention that
> > > the
> > > > > > > relationship between nars (dependencies between nars for
> > > > > > > apis/controller services/parent nars/etc..) we want to have a way
> > > to
> > > > > > > manage/show that or a user experience for it.  Maybe not a day-1
> > > thing
> > > > > > > but important to call out.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Joe
> > > > > > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <bb...@gmail.com>
> > > wrote:
> > > > > > > >
> > > > > > > > All,
> > > > > > > >
> > > > > > > > We've needed the elusive extension registry for quite some time
> > > now,
> > > > > > > > and with NiFi Registry I think we are in a good place to make
> > > some
> > > > > > > > progress in this area.
> > > > > > > >
> > > > > > > > I've started looking into adding "extension bundles" to NiFi
> > > Registry
> > > > > > > > as the next type of versioned item, along side the existing
> > > versioned
> > > > > > > > flows, and I wanted to take a minute to outline how that approach
> > > > > > > > could work before getting too far into it.
> > > > > > > >
> > > > > > > > Also, I'd like to focus this discussion on the design and
> > > > > > > > functionality of the extension registry, and not on how the
> > > community
> > > > > > > > is going to use it. Topics like hosting a central registry,
> > > changing
> > > > > > > > the build process, restructuring the git repo, releasing NARs,
> > > etc,
> > > > > > > > are all important topics, but first we need an extension registry
> > > > > > > > before we can do any of that :)
> > > > > > > >
> > > > > > > > Here is a high-level description of what needs to be done...
> > > > > > > >
> > > > > > > > NiFi Registry
> > > > > > > >
> > > > > > > > - Add a new type of item called an extension bundle, where each
> > > > > bundle
> > > > > > > > can contain one ore extensions or APIs
> > > > > > > >
> > > > > > > > - Support bundles for traditional NiFi (aka NARs) and also
> > > bundles
> > > > > for
> > > > > > > > MiNiFi CPP
> > > > > > > >
> > > > > > > > - Ability to upload the binary artifact for a bundle and extract
> > > the
> > > > > > > > metadata about the bundle, and metadata about the extensions
> > > > > contained
> > > > > > > > in the bundle (more on this later)
> > > > > > > >
> > > > > > > > - Provide a pluggable storage provider for saving the content of
> > > each
> > > > > > > > extension bundle so that we can have different implementations
> > > like
> > > > > > > > local fileysystem, S3, and other object stores
> > > > > > > >
> > > > > > > > - Provide a REST API for listing and retrieving available
> > > bundles,
> > > > > > > > integrate this into the registry Java client and NiFi CLI
> > > > > > > >
> > > > > > > > NAR Maven Plugin
> > > > > > > >
> > > > > > > > - Generate a descriptor for each component in the NAR which will
> > > > > > > > provide information like the description, tags, restricted or
> > > not,
> > > > > > > > etc.
> > > > > > > >
> > > > > > > > - These descriptors will be parsed by NiFi Registry when a NAR is
> > > > > > > > being uploaded so that NiFi Registry will know about the
> > > extensions
> > > > > > > > contained with in the NAR
> > > > > > > >
> > > > > > > > NiFi
> > > > > > > >
> > > > > > > > - Provide some type of extension manager experience where users
> > > can
> > > > > > > > search, browse, & install bundles that are available in any of
> > > the
> > > > > > > > registry clients that have been defined
> > > > > > > >
> > > > > > > > - Introduce a new security policy to control which users are
> > > allowed
> > > > > > > > to access the extension manager
> > > > > > > >
> > > > > > > > - Installing a bundle should load the NAR and make the extensions
> > > > > > > > available leveraging the recent work done to auto-load new NARs
> > > > > > > >
> > > > > > > > - Importing versioned flows from registry should provide an easy
> > > way
> > > > > > > > to install bundles that are required by the flow, but missing
> > > from
> > > > > the
> > > > > > > > local NiFi instance
> > > > > > > >
> > > > > > > >
> > > > > > > > If anyone has any thoughts or concerns about this approach,
> > > please
> > > > > let
> > > > > > > me know.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Bryan
> > > > > > >
> > > > >
> > >

Re: [DISCUSS] Extension Registry

[Joe Witt <jo...@gmail.com>]

Group selection based on tag names for bundles could probably do that.
Meaning it could be a sorting/filtering mechanism in the NiFi/Registry
interface perhaps.  Will be good to consider that UX as that
progresses.

As far as the different environments NiFi instances would certainly be
able to load only referenced Nars for versioned flows so you'll get
the optimal set (at runtime) automatically.  Very powerful.
On Tue, Nov 13, 2018 at 1:12 PM Mark Bean <ma...@gmail.com> wrote:
>
> Joe,
>
> I envision the Registry being able to provide a subset of NARs required for
> a specific NiFi instance. The user may have a relatively small set of NARs
> required for a NiFi used for basic routing/distribution, and a different
> more extensive set of NARs required for a more robust NiFi instance which
> performs various forms of processing/transformations. The grouping I am
> describing would be a way to select multiple NARs required for a specific
> NiFi instance.
>
> Expanding the scenario a little farther, suppose an integration/test
> environment defines the group. Then, the production environment can use the
> group definition to pull (or ensure it possesses) only the relevant NARs
> necessary.
>
> -Mark
>
> On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <jo...@gmail.com> wrote:
>
> > Mark
> >
> > Can you describe your use case from the user perspective both for the
> > entity that would upload the items and demarcate them as a group as
> > well as the user that would consume those bundles?
> >
> > I ask because the point here is that nars are themselves a 'group' in
> > that they are a logical/contained grouping of extensions.  These can
> > have relationships to other nars as we know.  And flows are designed
> > against specific components that come from certain nars for which we
> > know the precise coordinates.  When importing flows that depend on
> > these the system will be able to automatically acquire all that it
> > needs.
> >
> > Thanks
> > On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <ma...@gmail.com> wrote:
> > >
> > > I would like to see a "group" capability in the Registry for NAR bundles.
> > > Then, users can create their own customized grouping of relevant NARs.
> > > Managing bundles one at a time will become tedious.
> > >
> > > Thanks,
> > > Mark
> > >
> > > On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <jo...@gmail.com> wrote:
> > >
> > > > Sivaprasanna - yes absolutely.  That is the core point I think of
> > > > Bryan's first bullet under the 'NiFi' section.
> > > > On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <
> > sivaprasanna246@gmail.com>
> > > > wrote:
> > > > >
> > > > > One quick question. With the extension registry, my understanding is
> > that
> > > > > we would try to reduce the overall NiFi size by offloading certain
> > > > existing
> > > > > NAR bundles to the extension registry. So are we expecting the
> > extension
> > > > > registry to also come up with the ability/mechanism that lets an
> > user to
> > > > > download these bundles ?
> > > > >
> > > > > -
> > > > > Sivaprasanna
> > > > >
> > > > > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <jo...@gmail.com>
> > wrote:
> > > > >
> > > > > > Bryan
> > > > > >
> > > > > > Very exciting to see this under way!!!  We desperately have to get
> > our
> > > > > > core nifi build size way down and make it far easier for people to
> > > > > > contribute new processors.  We have a long line of extensions that
> > > > > > could be really useful/valuable and this will help unlock that
> > while
> > > > > > improving the user experience tremendously.
> > > > > >
> > > > > > For the doc/concerns noted above we should also add/mention that
> > the
> > > > > > relationship between nars (dependencies between nars for
> > > > > > apis/controller services/parent nars/etc..) we want to have a way
> > to
> > > > > > manage/show that or a user experience for it.  Maybe not a day-1
> > thing
> > > > > > but important to call out.
> > > > > >
> > > > > > Thanks!
> > > > > > Joe
> > > > > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <bb...@gmail.com>
> > wrote:
> > > > > > >
> > > > > > > All,
> > > > > > >
> > > > > > > We've needed the elusive extension registry for quite some time
> > now,
> > > > > > > and with NiFi Registry I think we are in a good place to make
> > some
> > > > > > > progress in this area.
> > > > > > >
> > > > > > > I've started looking into adding "extension bundles" to NiFi
> > Registry
> > > > > > > as the next type of versioned item, along side the existing
> > versioned
> > > > > > > flows, and I wanted to take a minute to outline how that approach
> > > > > > > could work before getting too far into it.
> > > > > > >
> > > > > > > Also, I'd like to focus this discussion on the design and
> > > > > > > functionality of the extension registry, and not on how the
> > community
> > > > > > > is going to use it. Topics like hosting a central registry,
> > changing
> > > > > > > the build process, restructuring the git repo, releasing NARs,
> > etc,
> > > > > > > are all important topics, but first we need an extension registry
> > > > > > > before we can do any of that :)
> > > > > > >
> > > > > > > Here is a high-level description of what needs to be done...
> > > > > > >
> > > > > > > NiFi Registry
> > > > > > >
> > > > > > > - Add a new type of item called an extension bundle, where each
> > > > bundle
> > > > > > > can contain one ore extensions or APIs
> > > > > > >
> > > > > > > - Support bundles for traditional NiFi (aka NARs) and also
> > bundles
> > > > for
> > > > > > > MiNiFi CPP
> > > > > > >
> > > > > > > - Ability to upload the binary artifact for a bundle and extract
> > the
> > > > > > > metadata about the bundle, and metadata about the extensions
> > > > contained
> > > > > > > in the bundle (more on this later)
> > > > > > >
> > > > > > > - Provide a pluggable storage provider for saving the content of
> > each
> > > > > > > extension bundle so that we can have different implementations
> > like
> > > > > > > local fileysystem, S3, and other object stores
> > > > > > >
> > > > > > > - Provide a REST API for listing and retrieving available
> > bundles,
> > > > > > > integrate this into the registry Java client and NiFi CLI
> > > > > > >
> > > > > > > NAR Maven Plugin
> > > > > > >
> > > > > > > - Generate a descriptor for each component in the NAR which will
> > > > > > > provide information like the description, tags, restricted or
> > not,
> > > > > > > etc.
> > > > > > >
> > > > > > > - These descriptors will be parsed by NiFi Registry when a NAR is
> > > > > > > being uploaded so that NiFi Registry will know about the
> > extensions
> > > > > > > contained with in the NAR
> > > > > > >
> > > > > > > NiFi
> > > > > > >
> > > > > > > - Provide some type of extension manager experience where users
> > can
> > > > > > > search, browse, & install bundles that are available in any of
> > the
> > > > > > > registry clients that have been defined
> > > > > > >
> > > > > > > - Introduce a new security policy to control which users are
> > allowed
> > > > > > > to access the extension manager
> > > > > > >
> > > > > > > - Installing a bundle should load the NAR and make the extensions
> > > > > > > available leveraging the recent work done to auto-load new NARs
> > > > > > >
> > > > > > > - Importing versioned flows from registry should provide an easy
> > way
> > > > > > > to install bundles that are required by the flow, but missing
> > from
> > > > the
> > > > > > > local NiFi instance
> > > > > > >
> > > > > > >
> > > > > > > If anyone has any thoughts or concerns about this approach,
> > please
> > > > let
> > > > > > me know.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Bryan
> > > > > >
> > > >
> >

Re: [DISCUSS] Extension Registry

[Mark Bean <ma...@gmail.com>]

Joe,

I envision the Registry being able to provide a subset of NARs required for
a specific NiFi instance. The user may have a relatively small set of NARs
required for a NiFi used for basic routing/distribution, and a different
more extensive set of NARs required for a more robust NiFi instance which
performs various forms of processing/transformations. The grouping I am
describing would be a way to select multiple NARs required for a specific
NiFi instance.

Expanding the scenario a little farther, suppose an integration/test
environment defines the group. Then, the production environment can use the
group definition to pull (or ensure it possesses) only the relevant NARs
necessary.

-Mark

On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <jo...@gmail.com> wrote:

> Mark
>
> Can you describe your use case from the user perspective both for the
> entity that would upload the items and demarcate them as a group as
> well as the user that would consume those bundles?
>
> I ask because the point here is that nars are themselves a 'group' in
> that they are a logical/contained grouping of extensions.  These can
> have relationships to other nars as we know.  And flows are designed
> against specific components that come from certain nars for which we
> know the precise coordinates.  When importing flows that depend on
> these the system will be able to automatically acquire all that it
> needs.
>
> Thanks
> On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <ma...@gmail.com> wrote:
> >
> > I would like to see a "group" capability in the Registry for NAR bundles.
> > Then, users can create their own customized grouping of relevant NARs.
> > Managing bundles one at a time will become tedious.
> >
> > Thanks,
> > Mark
> >
> > On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <jo...@gmail.com> wrote:
> >
> > > Sivaprasanna - yes absolutely.  That is the core point I think of
> > > Bryan's first bullet under the 'NiFi' section.
> > > On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <
> sivaprasanna246@gmail.com>
> > > wrote:
> > > >
> > > > One quick question. With the extension registry, my understanding is
> that
> > > > we would try to reduce the overall NiFi size by offloading certain
> > > existing
> > > > NAR bundles to the extension registry. So are we expecting the
> extension
> > > > registry to also come up with the ability/mechanism that lets an
> user to
> > > > download these bundles ?
> > > >
> > > > -
> > > > Sivaprasanna
> > > >
> > > > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <jo...@gmail.com>
> wrote:
> > > >
> > > > > Bryan
> > > > >
> > > > > Very exciting to see this under way!!!  We desperately have to get
> our
> > > > > core nifi build size way down and make it far easier for people to
> > > > > contribute new processors.  We have a long line of extensions that
> > > > > could be really useful/valuable and this will help unlock that
> while
> > > > > improving the user experience tremendously.
> > > > >
> > > > > For the doc/concerns noted above we should also add/mention that
> the
> > > > > relationship between nars (dependencies between nars for
> > > > > apis/controller services/parent nars/etc..) we want to have a way
> to
> > > > > manage/show that or a user experience for it.  Maybe not a day-1
> thing
> > > > > but important to call out.
> > > > >
> > > > > Thanks!
> > > > > Joe
> > > > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <bb...@gmail.com>
> wrote:
> > > > > >
> > > > > > All,
> > > > > >
> > > > > > We've needed the elusive extension registry for quite some time
> now,
> > > > > > and with NiFi Registry I think we are in a good place to make
> some
> > > > > > progress in this area.
> > > > > >
> > > > > > I've started looking into adding "extension bundles" to NiFi
> Registry
> > > > > > as the next type of versioned item, along side the existing
> versioned
> > > > > > flows, and I wanted to take a minute to outline how that approach
> > > > > > could work before getting too far into it.
> > > > > >
> > > > > > Also, I'd like to focus this discussion on the design and
> > > > > > functionality of the extension registry, and not on how the
> community
> > > > > > is going to use it. Topics like hosting a central registry,
> changing
> > > > > > the build process, restructuring the git repo, releasing NARs,
> etc,
> > > > > > are all important topics, but first we need an extension registry
> > > > > > before we can do any of that :)
> > > > > >
> > > > > > Here is a high-level description of what needs to be done...
> > > > > >
> > > > > > NiFi Registry
> > > > > >
> > > > > > - Add a new type of item called an extension bundle, where each
> > > bundle
> > > > > > can contain one ore extensions or APIs
> > > > > >
> > > > > > - Support bundles for traditional NiFi (aka NARs) and also
> bundles
> > > for
> > > > > > MiNiFi CPP
> > > > > >
> > > > > > - Ability to upload the binary artifact for a bundle and extract
> the
> > > > > > metadata about the bundle, and metadata about the extensions
> > > contained
> > > > > > in the bundle (more on this later)
> > > > > >
> > > > > > - Provide a pluggable storage provider for saving the content of
> each
> > > > > > extension bundle so that we can have different implementations
> like
> > > > > > local fileysystem, S3, and other object stores
> > > > > >
> > > > > > - Provide a REST API for listing and retrieving available
> bundles,
> > > > > > integrate this into the registry Java client and NiFi CLI
> > > > > >
> > > > > > NAR Maven Plugin
> > > > > >
> > > > > > - Generate a descriptor for each component in the NAR which will
> > > > > > provide information like the description, tags, restricted or
> not,
> > > > > > etc.
> > > > > >
> > > > > > - These descriptors will be parsed by NiFi Registry when a NAR is
> > > > > > being uploaded so that NiFi Registry will know about the
> extensions
> > > > > > contained with in the NAR
> > > > > >
> > > > > > NiFi
> > > > > >
> > > > > > - Provide some type of extension manager experience where users
> can
> > > > > > search, browse, & install bundles that are available in any of
> the
> > > > > > registry clients that have been defined
> > > > > >
> > > > > > - Introduce a new security policy to control which users are
> allowed
> > > > > > to access the extension manager
> > > > > >
> > > > > > - Installing a bundle should load the NAR and make the extensions
> > > > > > available leveraging the recent work done to auto-load new NARs
> > > > > >
> > > > > > - Importing versioned flows from registry should provide an easy
> way
> > > > > > to install bundles that are required by the flow, but missing
> from
> > > the
> > > > > > local NiFi instance
> > > > > >
> > > > > >
> > > > > > If anyone has any thoughts or concerns about this approach,
> please
> > > let
> > > > > me know.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Bryan
> > > > >
> > >
>

Re: [DISCUSS] Extension Registry

[Mike Thomsen <mi...@gmail.com>]

What are the plans for enabling the registry/NiFi to pull NARs and deploy
them? Off the top of my head, piggybacking off of the Maven libraries/repos
might be a good way to go there if there are no other plans.

Private repositories should be baked in from the start here. Ideal would be
something where DevOps could have Jenkins set up a job that builds a
bundle, "deploys it" and the extension registry is informed that a new
NAR/set of NARs are there. If we go with hot-swappable ones, might also be
a good idea to borrow from Maven there by allowing different snapshot
builds to coexist so users can tell the Registry to swap one build for
another.

Mike

On Tue, Nov 13, 2018 at 1:00 PM Joe Witt <jo...@gmail.com> wrote:

> Mark
>
> Can you describe your use case from the user perspective both for the
> entity that would upload the items and demarcate them as a group as
> well as the user that would consume those bundles?
>
> I ask because the point here is that nars are themselves a 'group' in
> that they are a logical/contained grouping of extensions.  These can
> have relationships to other nars as we know.  And flows are designed
> against specific components that come from certain nars for which we
> know the precise coordinates.  When importing flows that depend on
> these the system will be able to automatically acquire all that it
> needs.
>
> Thanks
> On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <ma...@gmail.com> wrote:
> >
> > I would like to see a "group" capability in the Registry for NAR bundles.
> > Then, users can create their own customized grouping of relevant NARs.
> > Managing bundles one at a time will become tedious.
> >
> > Thanks,
> > Mark
> >
> > On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <jo...@gmail.com> wrote:
> >
> > > Sivaprasanna - yes absolutely.  That is the core point I think of
> > > Bryan's first bullet under the 'NiFi' section.
> > > On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <
> sivaprasanna246@gmail.com>
> > > wrote:
> > > >
> > > > One quick question. With the extension registry, my understanding is
> that
> > > > we would try to reduce the overall NiFi size by offloading certain
> > > existing
> > > > NAR bundles to the extension registry. So are we expecting the
> extension
> > > > registry to also come up with the ability/mechanism that lets an
> user to
> > > > download these bundles ?
> > > >
> > > > -
> > > > Sivaprasanna
> > > >
> > > > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <jo...@gmail.com>
> wrote:
> > > >
> > > > > Bryan
> > > > >
> > > > > Very exciting to see this under way!!!  We desperately have to get
> our
> > > > > core nifi build size way down and make it far easier for people to
> > > > > contribute new processors.  We have a long line of extensions that
> > > > > could be really useful/valuable and this will help unlock that
> while
> > > > > improving the user experience tremendously.
> > > > >
> > > > > For the doc/concerns noted above we should also add/mention that
> the
> > > > > relationship between nars (dependencies between nars for
> > > > > apis/controller services/parent nars/etc..) we want to have a way
> to
> > > > > manage/show that or a user experience for it.  Maybe not a day-1
> thing
> > > > > but important to call out.
> > > > >
> > > > > Thanks!
> > > > > Joe
> > > > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <bb...@gmail.com>
> wrote:
> > > > > >
> > > > > > All,
> > > > > >
> > > > > > We've needed the elusive extension registry for quite some time
> now,
> > > > > > and with NiFi Registry I think we are in a good place to make
> some
> > > > > > progress in this area.
> > > > > >
> > > > > > I've started looking into adding "extension bundles" to NiFi
> Registry
> > > > > > as the next type of versioned item, along side the existing
> versioned
> > > > > > flows, and I wanted to take a minute to outline how that approach
> > > > > > could work before getting too far into it.
> > > > > >
> > > > > > Also, I'd like to focus this discussion on the design and
> > > > > > functionality of the extension registry, and not on how the
> community
> > > > > > is going to use it. Topics like hosting a central registry,
> changing
> > > > > > the build process, restructuring the git repo, releasing NARs,
> etc,
> > > > > > are all important topics, but first we need an extension registry
> > > > > > before we can do any of that :)
> > > > > >
> > > > > > Here is a high-level description of what needs to be done...
> > > > > >
> > > > > > NiFi Registry
> > > > > >
> > > > > > - Add a new type of item called an extension bundle, where each
> > > bundle
> > > > > > can contain one ore extensions or APIs
> > > > > >
> > > > > > - Support bundles for traditional NiFi (aka NARs) and also
> bundles
> > > for
> > > > > > MiNiFi CPP
> > > > > >
> > > > > > - Ability to upload the binary artifact for a bundle and extract
> the
> > > > > > metadata about the bundle, and metadata about the extensions
> > > contained
> > > > > > in the bundle (more on this later)
> > > > > >
> > > > > > - Provide a pluggable storage provider for saving the content of
> each
> > > > > > extension bundle so that we can have different implementations
> like
> > > > > > local fileysystem, S3, and other object stores
> > > > > >
> > > > > > - Provide a REST API for listing and retrieving available
> bundles,
> > > > > > integrate this into the registry Java client and NiFi CLI
> > > > > >
> > > > > > NAR Maven Plugin
> > > > > >
> > > > > > - Generate a descriptor for each component in the NAR which will
> > > > > > provide information like the description, tags, restricted or
> not,
> > > > > > etc.
> > > > > >
> > > > > > - These descriptors will be parsed by NiFi Registry when a NAR is
> > > > > > being uploaded so that NiFi Registry will know about the
> extensions
> > > > > > contained with in the NAR
> > > > > >
> > > > > > NiFi
> > > > > >
> > > > > > - Provide some type of extension manager experience where users
> can
> > > > > > search, browse, & install bundles that are available in any of
> the
> > > > > > registry clients that have been defined
> > > > > >
> > > > > > - Introduce a new security policy to control which users are
> allowed
> > > > > > to access the extension manager
> > > > > >
> > > > > > - Installing a bundle should load the NAR and make the extensions
> > > > > > available leveraging the recent work done to auto-load new NARs
> > > > > >
> > > > > > - Importing versioned flows from registry should provide an easy
> way
> > > > > > to install bundles that are required by the flow, but missing
> from
> > > the
> > > > > > local NiFi instance
> > > > > >
> > > > > >
> > > > > > If anyone has any thoughts or concerns about this approach,
> please
> > > let
> > > > > me know.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Bryan
> > > > >
> > >
>

Re: [DISCUSS] Extension Registry

[Joe Witt <jo...@gmail.com>]

Mark

Can you describe your use case from the user perspective both for the
entity that would upload the items and demarcate them as a group as
well as the user that would consume those bundles?

I ask because the point here is that nars are themselves a 'group' in
that they are a logical/contained grouping of extensions.  These can
have relationships to other nars as we know.  And flows are designed
against specific components that come from certain nars for which we
know the precise coordinates.  When importing flows that depend on
these the system will be able to automatically acquire all that it
needs.

Thanks
On Tue, Nov 13, 2018 at 12:57 PM Mark Bean <ma...@gmail.com> wrote:
>
> I would like to see a "group" capability in the Registry for NAR bundles.
> Then, users can create their own customized grouping of relevant NARs.
> Managing bundles one at a time will become tedious.
>
> Thanks,
> Mark
>
> On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <jo...@gmail.com> wrote:
>
> > Sivaprasanna - yes absolutely.  That is the core point I think of
> > Bryan's first bullet under the 'NiFi' section.
> > On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <si...@gmail.com>
> > wrote:
> > >
> > > One quick question. With the extension registry, my understanding is that
> > > we would try to reduce the overall NiFi size by offloading certain
> > existing
> > > NAR bundles to the extension registry. So are we expecting the extension
> > > registry to also come up with the ability/mechanism that lets an user to
> > > download these bundles ?
> > >
> > > -
> > > Sivaprasanna
> > >
> > > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <jo...@gmail.com> wrote:
> > >
> > > > Bryan
> > > >
> > > > Very exciting to see this under way!!!  We desperately have to get our
> > > > core nifi build size way down and make it far easier for people to
> > > > contribute new processors.  We have a long line of extensions that
> > > > could be really useful/valuable and this will help unlock that while
> > > > improving the user experience tremendously.
> > > >
> > > > For the doc/concerns noted above we should also add/mention that the
> > > > relationship between nars (dependencies between nars for
> > > > apis/controller services/parent nars/etc..) we want to have a way to
> > > > manage/show that or a user experience for it.  Maybe not a day-1 thing
> > > > but important to call out.
> > > >
> > > > Thanks!
> > > > Joe
> > > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <bb...@gmail.com> wrote:
> > > > >
> > > > > All,
> > > > >
> > > > > We've needed the elusive extension registry for quite some time now,
> > > > > and with NiFi Registry I think we are in a good place to make some
> > > > > progress in this area.
> > > > >
> > > > > I've started looking into adding "extension bundles" to NiFi Registry
> > > > > as the next type of versioned item, along side the existing versioned
> > > > > flows, and I wanted to take a minute to outline how that approach
> > > > > could work before getting too far into it.
> > > > >
> > > > > Also, I'd like to focus this discussion on the design and
> > > > > functionality of the extension registry, and not on how the community
> > > > > is going to use it. Topics like hosting a central registry, changing
> > > > > the build process, restructuring the git repo, releasing NARs, etc,
> > > > > are all important topics, but first we need an extension registry
> > > > > before we can do any of that :)
> > > > >
> > > > > Here is a high-level description of what needs to be done...
> > > > >
> > > > > NiFi Registry
> > > > >
> > > > > - Add a new type of item called an extension bundle, where each
> > bundle
> > > > > can contain one ore extensions or APIs
> > > > >
> > > > > - Support bundles for traditional NiFi (aka NARs) and also bundles
> > for
> > > > > MiNiFi CPP
> > > > >
> > > > > - Ability to upload the binary artifact for a bundle and extract the
> > > > > metadata about the bundle, and metadata about the extensions
> > contained
> > > > > in the bundle (more on this later)
> > > > >
> > > > > - Provide a pluggable storage provider for saving the content of each
> > > > > extension bundle so that we can have different implementations like
> > > > > local fileysystem, S3, and other object stores
> > > > >
> > > > > - Provide a REST API for listing and retrieving available bundles,
> > > > > integrate this into the registry Java client and NiFi CLI
> > > > >
> > > > > NAR Maven Plugin
> > > > >
> > > > > - Generate a descriptor for each component in the NAR which will
> > > > > provide information like the description, tags, restricted or not,
> > > > > etc.
> > > > >
> > > > > - These descriptors will be parsed by NiFi Registry when a NAR is
> > > > > being uploaded so that NiFi Registry will know about the extensions
> > > > > contained with in the NAR
> > > > >
> > > > > NiFi
> > > > >
> > > > > - Provide some type of extension manager experience where users can
> > > > > search, browse, & install bundles that are available in any of the
> > > > > registry clients that have been defined
> > > > >
> > > > > - Introduce a new security policy to control which users are allowed
> > > > > to access the extension manager
> > > > >
> > > > > - Installing a bundle should load the NAR and make the extensions
> > > > > available leveraging the recent work done to auto-load new NARs
> > > > >
> > > > > - Importing versioned flows from registry should provide an easy way
> > > > > to install bundles that are required by the flow, but missing from
> > the
> > > > > local NiFi instance
> > > > >
> > > > >
> > > > > If anyone has any thoughts or concerns about this approach, please
> > let
> > > > me know.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Bryan
> > > >
> >

Re: [DISCUSS] Extension Registry

[Mark Bean <ma...@gmail.com>]

I would like to see a "group" capability in the Registry for NAR bundles.
Then, users can create their own customized grouping of relevant NARs.
Managing bundles one at a time will become tedious.

Thanks,
Mark

On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <jo...@gmail.com> wrote:

> Sivaprasanna - yes absolutely.  That is the core point I think of
> Bryan's first bullet under the 'NiFi' section.
> On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <si...@gmail.com>
> wrote:
> >
> > One quick question. With the extension registry, my understanding is that
> > we would try to reduce the overall NiFi size by offloading certain
> existing
> > NAR bundles to the extension registry. So are we expecting the extension
> > registry to also come up with the ability/mechanism that lets an user to
> > download these bundles ?
> >
> > -
> > Sivaprasanna
> >
> > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <jo...@gmail.com> wrote:
> >
> > > Bryan
> > >
> > > Very exciting to see this under way!!!  We desperately have to get our
> > > core nifi build size way down and make it far easier for people to
> > > contribute new processors.  We have a long line of extensions that
> > > could be really useful/valuable and this will help unlock that while
> > > improving the user experience tremendously.
> > >
> > > For the doc/concerns noted above we should also add/mention that the
> > > relationship between nars (dependencies between nars for
> > > apis/controller services/parent nars/etc..) we want to have a way to
> > > manage/show that or a user experience for it.  Maybe not a day-1 thing
> > > but important to call out.
> > >
> > > Thanks!
> > > Joe
> > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <bb...@gmail.com> wrote:
> > > >
> > > > All,
> > > >
> > > > We've needed the elusive extension registry for quite some time now,
> > > > and with NiFi Registry I think we are in a good place to make some
> > > > progress in this area.
> > > >
> > > > I've started looking into adding "extension bundles" to NiFi Registry
> > > > as the next type of versioned item, along side the existing versioned
> > > > flows, and I wanted to take a minute to outline how that approach
> > > > could work before getting too far into it.
> > > >
> > > > Also, I'd like to focus this discussion on the design and
> > > > functionality of the extension registry, and not on how the community
> > > > is going to use it. Topics like hosting a central registry, changing
> > > > the build process, restructuring the git repo, releasing NARs, etc,
> > > > are all important topics, but first we need an extension registry
> > > > before we can do any of that :)
> > > >
> > > > Here is a high-level description of what needs to be done...
> > > >
> > > > NiFi Registry
> > > >
> > > > - Add a new type of item called an extension bundle, where each
> bundle
> > > > can contain one ore extensions or APIs
> > > >
> > > > - Support bundles for traditional NiFi (aka NARs) and also bundles
> for
> > > > MiNiFi CPP
> > > >
> > > > - Ability to upload the binary artifact for a bundle and extract the
> > > > metadata about the bundle, and metadata about the extensions
> contained
> > > > in the bundle (more on this later)
> > > >
> > > > - Provide a pluggable storage provider for saving the content of each
> > > > extension bundle so that we can have different implementations like
> > > > local fileysystem, S3, and other object stores
> > > >
> > > > - Provide a REST API for listing and retrieving available bundles,
> > > > integrate this into the registry Java client and NiFi CLI
> > > >
> > > > NAR Maven Plugin
> > > >
> > > > - Generate a descriptor for each component in the NAR which will
> > > > provide information like the description, tags, restricted or not,
> > > > etc.
> > > >
> > > > - These descriptors will be parsed by NiFi Registry when a NAR is
> > > > being uploaded so that NiFi Registry will know about the extensions
> > > > contained with in the NAR
> > > >
> > > > NiFi
> > > >
> > > > - Provide some type of extension manager experience where users can
> > > > search, browse, & install bundles that are available in any of the
> > > > registry clients that have been defined
> > > >
> > > > - Introduce a new security policy to control which users are allowed
> > > > to access the extension manager
> > > >
> > > > - Installing a bundle should load the NAR and make the extensions
> > > > available leveraging the recent work done to auto-load new NARs
> > > >
> > > > - Importing versioned flows from registry should provide an easy way
> > > > to install bundles that are required by the flow, but missing from
> the
> > > > local NiFi instance
> > > >
> > > >
> > > > If anyone has any thoughts or concerns about this approach, please
> let
> > > me know.
> > > >
> > > > Thanks,
> > > >
> > > > Bryan
> > >
>

Re: [DISCUSS] Extension Registry

[Joe Witt <jo...@gmail.com>]

Sivaprasanna - yes absolutely.  That is the core point I think of
Bryan's first bullet under the 'NiFi' section.
On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <si...@gmail.com> wrote:
>
> One quick question. With the extension registry, my understanding is that
> we would try to reduce the overall NiFi size by offloading certain existing
> NAR bundles to the extension registry. So are we expecting the extension
> registry to also come up with the ability/mechanism that lets an user to
> download these bundles ?
>
> -
> Sivaprasanna
>
> On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <jo...@gmail.com> wrote:
>
> > Bryan
> >
> > Very exciting to see this under way!!!  We desperately have to get our
> > core nifi build size way down and make it far easier for people to
> > contribute new processors.  We have a long line of extensions that
> > could be really useful/valuable and this will help unlock that while
> > improving the user experience tremendously.
> >
> > For the doc/concerns noted above we should also add/mention that the
> > relationship between nars (dependencies between nars for
> > apis/controller services/parent nars/etc..) we want to have a way to
> > manage/show that or a user experience for it.  Maybe not a day-1 thing
> > but important to call out.
> >
> > Thanks!
> > Joe
> > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <bb...@gmail.com> wrote:
> > >
> > > All,
> > >
> > > We've needed the elusive extension registry for quite some time now,
> > > and with NiFi Registry I think we are in a good place to make some
> > > progress in this area.
> > >
> > > I've started looking into adding "extension bundles" to NiFi Registry
> > > as the next type of versioned item, along side the existing versioned
> > > flows, and I wanted to take a minute to outline how that approach
> > > could work before getting too far into it.
> > >
> > > Also, I'd like to focus this discussion on the design and
> > > functionality of the extension registry, and not on how the community
> > > is going to use it. Topics like hosting a central registry, changing
> > > the build process, restructuring the git repo, releasing NARs, etc,
> > > are all important topics, but first we need an extension registry
> > > before we can do any of that :)
> > >
> > > Here is a high-level description of what needs to be done...
> > >
> > > NiFi Registry
> > >
> > > - Add a new type of item called an extension bundle, where each bundle
> > > can contain one ore extensions or APIs
> > >
> > > - Support bundles for traditional NiFi (aka NARs) and also bundles for
> > > MiNiFi CPP
> > >
> > > - Ability to upload the binary artifact for a bundle and extract the
> > > metadata about the bundle, and metadata about the extensions contained
> > > in the bundle (more on this later)
> > >
> > > - Provide a pluggable storage provider for saving the content of each
> > > extension bundle so that we can have different implementations like
> > > local fileysystem, S3, and other object stores
> > >
> > > - Provide a REST API for listing and retrieving available bundles,
> > > integrate this into the registry Java client and NiFi CLI
> > >
> > > NAR Maven Plugin
> > >
> > > - Generate a descriptor for each component in the NAR which will
> > > provide information like the description, tags, restricted or not,
> > > etc.
> > >
> > > - These descriptors will be parsed by NiFi Registry when a NAR is
> > > being uploaded so that NiFi Registry will know about the extensions
> > > contained with in the NAR
> > >
> > > NiFi
> > >
> > > - Provide some type of extension manager experience where users can
> > > search, browse, & install bundles that are available in any of the
> > > registry clients that have been defined
> > >
> > > - Introduce a new security policy to control which users are allowed
> > > to access the extension manager
> > >
> > > - Installing a bundle should load the NAR and make the extensions
> > > available leveraging the recent work done to auto-load new NARs
> > >
> > > - Importing versioned flows from registry should provide an easy way
> > > to install bundles that are required by the flow, but missing from the
> > > local NiFi instance
> > >
> > >
> > > If anyone has any thoughts or concerns about this approach, please let
> > me know.
> > >
> > > Thanks,
> > >
> > > Bryan
> >

Re: [DISCUSS] Extension Registry

[Sivaprasanna <si...@gmail.com>]

One quick question. With the extension registry, my understanding is that
we would try to reduce the overall NiFi size by offloading certain existing
NAR bundles to the extension registry. So are we expecting the extension
registry to also come up with the ability/mechanism that lets an user to
download these bundles ?

-
Sivaprasanna

On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <jo...@gmail.com> wrote:

> Bryan
>
> Very exciting to see this under way!!!  We desperately have to get our
> core nifi build size way down and make it far easier for people to
> contribute new processors.  We have a long line of extensions that
> could be really useful/valuable and this will help unlock that while
> improving the user experience tremendously.
>
> For the doc/concerns noted above we should also add/mention that the
> relationship between nars (dependencies between nars for
> apis/controller services/parent nars/etc..) we want to have a way to
> manage/show that or a user experience for it.  Maybe not a day-1 thing
> but important to call out.
>
> Thanks!
> Joe
> On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <bb...@gmail.com> wrote:
> >
> > All,
> >
> > We've needed the elusive extension registry for quite some time now,
> > and with NiFi Registry I think we are in a good place to make some
> > progress in this area.
> >
> > I've started looking into adding "extension bundles" to NiFi Registry
> > as the next type of versioned item, along side the existing versioned
> > flows, and I wanted to take a minute to outline how that approach
> > could work before getting too far into it.
> >
> > Also, I'd like to focus this discussion on the design and
> > functionality of the extension registry, and not on how the community
> > is going to use it. Topics like hosting a central registry, changing
> > the build process, restructuring the git repo, releasing NARs, etc,
> > are all important topics, but first we need an extension registry
> > before we can do any of that :)
> >
> > Here is a high-level description of what needs to be done...
> >
> > NiFi Registry
> >
> > - Add a new type of item called an extension bundle, where each bundle
> > can contain one ore extensions or APIs
> >
> > - Support bundles for traditional NiFi (aka NARs) and also bundles for
> > MiNiFi CPP
> >
> > - Ability to upload the binary artifact for a bundle and extract the
> > metadata about the bundle, and metadata about the extensions contained
> > in the bundle (more on this later)
> >
> > - Provide a pluggable storage provider for saving the content of each
> > extension bundle so that we can have different implementations like
> > local fileysystem, S3, and other object stores
> >
> > - Provide a REST API for listing and retrieving available bundles,
> > integrate this into the registry Java client and NiFi CLI
> >
> > NAR Maven Plugin
> >
> > - Generate a descriptor for each component in the NAR which will
> > provide information like the description, tags, restricted or not,
> > etc.
> >
> > - These descriptors will be parsed by NiFi Registry when a NAR is
> > being uploaded so that NiFi Registry will know about the extensions
> > contained with in the NAR
> >
> > NiFi
> >
> > - Provide some type of extension manager experience where users can
> > search, browse, & install bundles that are available in any of the
> > registry clients that have been defined
> >
> > - Introduce a new security policy to control which users are allowed
> > to access the extension manager
> >
> > - Installing a bundle should load the NAR and make the extensions
> > available leveraging the recent work done to auto-load new NARs
> >
> > - Importing versioned flows from registry should provide an easy way
> > to install bundles that are required by the flow, but missing from the
> > local NiFi instance
> >
> >
> > If anyone has any thoughts or concerns about this approach, please let
> me know.
> >
> > Thanks,
> >
> > Bryan
>

Re: [DISCUSS] Extension Registry

[Joe Witt <jo...@gmail.com>]

Bryan

Very exciting to see this under way!!!  We desperately have to get our
core nifi build size way down and make it far easier for people to
contribute new processors.  We have a long line of extensions that
could be really useful/valuable and this will help unlock that while
improving the user experience tremendously.

For the doc/concerns noted above we should also add/mention that the
relationship between nars (dependencies between nars for
apis/controller services/parent nars/etc..) we want to have a way to
manage/show that or a user experience for it.  Maybe not a day-1 thing
but important to call out.

Thanks!
Joe
On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <bb...@gmail.com> wrote:
>
> All,
>
> We've needed the elusive extension registry for quite some time now,
> and with NiFi Registry I think we are in a good place to make some
> progress in this area.
>
> I've started looking into adding "extension bundles" to NiFi Registry
> as the next type of versioned item, along side the existing versioned
> flows, and I wanted to take a minute to outline how that approach
> could work before getting too far into it.
>
> Also, I'd like to focus this discussion on the design and
> functionality of the extension registry, and not on how the community
> is going to use it. Topics like hosting a central registry, changing
> the build process, restructuring the git repo, releasing NARs, etc,
> are all important topics, but first we need an extension registry
> before we can do any of that :)
>
> Here is a high-level description of what needs to be done...
>
> NiFi Registry
>
> - Add a new type of item called an extension bundle, where each bundle
> can contain one ore extensions or APIs
>
> - Support bundles for traditional NiFi (aka NARs) and also bundles for
> MiNiFi CPP
>
> - Ability to upload the binary artifact for a bundle and extract the
> metadata about the bundle, and metadata about the extensions contained
> in the bundle (more on this later)
>
> - Provide a pluggable storage provider for saving the content of each
> extension bundle so that we can have different implementations like
> local fileysystem, S3, and other object stores
>
> - Provide a REST API for listing and retrieving available bundles,
> integrate this into the registry Java client and NiFi CLI
>
> NAR Maven Plugin
>
> - Generate a descriptor for each component in the NAR which will
> provide information like the description, tags, restricted or not,
> etc.
>
> - These descriptors will be parsed by NiFi Registry when a NAR is
> being uploaded so that NiFi Registry will know about the extensions
> contained with in the NAR
>
> NiFi
>
> - Provide some type of extension manager experience where users can
> search, browse, & install bundles that are available in any of the
> registry clients that have been defined
>
> - Introduce a new security policy to control which users are allowed
> to access the extension manager
>
> - Installing a bundle should load the NAR and make the extensions
> available leveraging the recent work done to auto-load new NARs
>
> - Importing versioned flows from registry should provide an easy way
> to install bundles that are required by the flow, but missing from the
> local NiFi instance
>
>
> If anyone has any thoughts or concerns about this approach, please let me know.
>
> Thanks,
>
> Bryan