You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nifi.apache.org by Pierre Villard <pi...@gmail.com> on 2019/03/25 17:32:50 UTC

[DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

All,

Some really nice features have been included since the NiFi Registry 0.3.0
release and I'm wondering if it'd be a good time to consider a 0.4.0
release.

There is a JIRA tagged for 0.4.0 and there are 2 opened pull requests, plus
interesting discussions / JIRAS on the mailing lists.

Are there any reasons to hold off on a 0.4.0 release?  Are there particular
JIRAs that the community considers necessary to have as part of the release?

If not, happy to give it a try at performing RM duties!

Thanks,
Pierre

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

Posted by Andy LoPresto <al...@apache.org>.

Excellent Bryan. Looking forward to this release. Thanks for taking the initiative here. 

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On May 16, 2019, at 12:34 PM, Bryan Bende <bb...@gmail.com> wrote:
> 
> The last few items tagged for 0.4.0 have been merged so we should be
> good to put out an RC for 0.4.0.
> 
> I'll start working on getting everything together and try to get
> something out soon.
> 
> On Fri, May 10, 2019 at 9:00 AM Bryan Bende <bb...@gmail.com> wrote:
>> 
>> I think we should be able to kick 0.4.0 very soon, just a couple of
>> items to get in. Assuming those get reviewed and merged soon we could
>> possibly kick out an RC early next week. I can plan to be RM unless
>> anyone else is interested.
>> 
>> On Thu, Mar 28, 2019 at 3:39 PM Bryan Bende <bb...@gmail.com> wrote:
>>> 
>>> I think Kevin summarized it nicely. Auto-pulling the extensions with a
>>> versioned flow is definitely the ultimate vision, but we need to take
>>> steps to get there.
>>> 
>>> The process of automatically bringing in the extensions is something
>>> that would be implemented on the NiFi side and will require a bit of
>>> thought and design around the user experience.
>>> 
>>> Before we can do any of that, we first need somewhere to store and
>>> retrieve versioned extensions, which is the work that is underway on
>>> the NiFi Registry side.
>>> 
>>> On Thu, Mar 28, 2019 at 3:29 PM Kevin Doran <kd...@apache.org> wrote:
>>>> 
>>>> Hi Ryan,
>>>> 
>>>> Bryan can speak more to this than I can, but, yes, providing the
>>>> experience you describe is the eventual goal! Aside from the NiFi
>>>> Extension Registry support though (the initial bits of which is what
>>>> Bryan is referring to in 0.4.0), getting the full, streamlined
>>>> experience in NiFi will require changes to NiFi as well, for example
>>>> in the logic to import a flow as it now also has to reach back to
>>>> Registry to pull extensions referenced by the flow during import. So
>>>> the changes to NiFi Registry to support hosting NiFi Extensions is
>>>> only half of the puzzle. This is why Bryan also mentioned it might be
>>>> worth holding back 0.4.0 (or releasing it, but marking the new
>>>> extension stuff as experimental), as it is likely the current
>>>> implementation and interfaces will need to change once the NiFi work
>>>> begins.
>>>> 
>>>> Cheers,
>>>> Kevin
>>>> 
>>>> On Thu, Mar 28, 2019 at 2:56 PM Ryan Hendrickson
>>>> <ry...@gmail.com> wrote:
>>>>> 
>>>>> We'd like to connect a NiFi to a Registry, Pull down a Process Group, and
>>>>> get any NARs required for that without having to manually install them on
>>>>> the NiFi.
>>>>> 
>>>>> That's what I'm understanding the Extension part of the 0.4.0 release to
>>>>> be, am I on point, or way off?
>>>>> 
>>>>> Thanks,
>>>>> Ryan
>>>>> 
>>>>> On Wed, Mar 27, 2019 at 4:17 PM Bryan Rosander <br...@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> Hey Pierre,
>>>>>> 
>>>>>> You can version the metadata db in the root dir of the git repo with event
>>>>>> hooks.
>>>>>> 
>>>>>> With an h2 jdbc url of
>>>>>> 
>>>>>> 'jdbc:h2:$NIFI_REGISTRY_HOME/database/nifi-registry-primary;AUTOCOMMIT=OFF;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=TRUE'
>>>>>> 
>>>>>> You can connect to the db in a separate process, we're using that with a
>>>>>> shell event hook to dump the db to sql script when mutations happen (now
>>>>>> tenant events too :) ), storing that in the git repo using
>>>>>> org.h2.tools.Script, and committing it.  Then we can restore from that sql
>>>>>> script on startup before calling the stock startup script.
>>>>>> 
>>>>>> Side benefit is you can see metadata state changes in the git history and
>>>>>> manually inspect the sql.
>>>>>> 
>>>>>> 
>>>>>> On Wed, Mar 27, 2019 at 1:57 PM Pierre Villard <
>>>>>> pierre.villard.fr@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Bryan,
>>>>>>> 
>>>>>>> Thanks for the details. On my side, the main feature I (and others I
>>>>>>> suppose) am looking for is the ability to rebuild the metadata DB from
>>>>>> the
>>>>>>> Git repo. But to be honest, this is not a blocker at all: I can live
>>>>>> with a
>>>>>>> custom Docker image containing this feature.
>>>>>>> 
>>>>>>> So, from my point of view, I don't have any issue with #2 but maybe #1
>>>>>>> would also give access to experimental features and have interesting
>>>>>>> feedback from the community. But we would, indeed, need to make it
>>>>>> crystal
>>>>>>> clear that the APIs could change in ulterior release(s).
>>>>>>> 
>>>>>>> Pierre
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Le mer. 27 mars 2019 à 18:11, Bryan Bende <bb...@gmail.com> a écrit :
>>>>>>> 
>>>>>>>> Ryan,
>>>>>>>> 
>>>>>>>> The short answer to your question is no, there isn't actually anything
>>>>>>>> new in the UI. The registry UI is setup in a generic way to list
>>>>>>>> "versioned items", so versioned extension bundles will just
>>>>>>>> automatically show up in the list with versioned flows.
>>>>>>>> 
>>>>>>>> Let me put together a wiki page that outlines all of the moving parts
>>>>>>>> for the extension registry work and where we are currently at. I'll
>>>>>>>> send the link on this thread when I have something.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> For Pierre and others,
>>>>>>>> 
>>>>>>>> I was thinking more about this discussion of releasing 0.4.0, and I
>>>>>>>> think there are two of approaches we could take...
>>>>>>>> 
>>>>>>>> 1) If there is a desire for the community to release 0.4.0 sooner
>>>>>>>> rather than later, then we can mark all of the extension bundle
>>>>>>>> related APIs as experimental, which would then give us the freedom to
>>>>>>>> continue evolving them until we have the full extension registry
>>>>>>>> experience, and likely mark them as stable in 0.5.0, or whichever
>>>>>>>> future release we decide. The reason for this is that there will
>>>>>>>> likely be API changes needed while building out the NiFi side of
>>>>>>>> things, and I don't want us to get stuck in a spot where we can't
>>>>>>>> change some API since it has been released, and so far the NiFi side
>>>>>>>> of the work hasn't been started yet.
>>>>>>>> 
>>>>>>>> 2) Wait until more of the extension registry work is finalized and use
>>>>>>>> that as the driver of when to release 0.4.0, with the obvious downside
>>>>>>>> that it may take a bit longer to get a release out which then holds up
>>>>>>>> anything else that is not related to extension registry.
>>>>>>>> 
>>>>>>>> So I guess we just have to decide the driving factor for releasing
>>>>>>>> 0.4.0... If its the non-extension related features/fixes, then #1 is
>>>>>>>> probably the best approach. If its the extension related stuff, then
>>>>>>>> I'd vote for #2 since it isn't really ready yet.
>>>>>>>> 
>>>>>>>> -Bryan
>>>>>>>> 
>>>>>>>> On Wed, Mar 27, 2019 at 12:01 PM Ryan Hendrickson
>>>>>>>> <ry...@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> Bryan,
>>>>>>>>>   Is there a new GUI part of the registry for the extension registry
>>>>>>>> work
>>>>>>>>> in master?  I compiled 0.4.0-SNAPSHOT and didn't see anything
>>>>>>>> immediately.
>>>>>>>>> You mentioned docs and sys admin stuff... Could you point me in the
>>>>>>>>> quick-start path if I wanted to try to kick-the-tires a little on
>>>>>> this?
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Ryan
>>>>>>>>> 
>>>>>>>>> On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende <bb...@gmail.com>
>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Mike,
>>>>>>>>>> 
>>>>>>>>>> All record-oriented components are extensions and thus can make use
>>>>>>> of
>>>>>>>>>> versioned components (i.e. nothing related to records is baked into
>>>>>>>>>> the core framework of NiFi).
>>>>>>>>>> 
>>>>>>>>>> The record API is essentially the module named
>>>>>>>>>> nifi-record-serialization-services-api which at runtime is included
>>>>>>> in
>>>>>>>>>> nifi-standard-services-api-nar. Any record oriented processors you
>>>>>>>>>> build are dependent on a specific version of
>>>>>>>>>> nifi-standard-services-api-nar since that is where the Java
>>>>>> interface
>>>>>>>>>> will be that your processor was compiled against.
>>>>>>>>>> 
>>>>>>>>>> Right now, even without extension registry, you could deploy
>>>>>> multiple
>>>>>>>>>> versions of nifi-standard-services-api-nar to a NiFi instance, and
>>>>>>>>>> then deploy multiple versions of your record processing NAR that
>>>>>>>>>> correspond to the versions of your API NAR.
>>>>>>>>>> 
>>>>>>>>>> Going forward with extension registry, we probably want to consider
>>>>>>>>>> breaking up nifi-standard-services-api-nar into smaller API NARs,
>>>>>> as
>>>>>>>>>> well as nifi-standard-bundle into smaller processor bundles. For
>>>>>>>>>> example, there could be a record API NAR and a record processors
>>>>>> NAR,
>>>>>>>>>> that would both be split out of from the standard NARs.
>>>>>>>>>> 
>>>>>>>>>> -Bryan
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen <
>>>>>> mikerthomsen@gmail.com
>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Great news about the extension registry! As we get closer to that
>>>>>>>> being
>>>>>>>>>>> ready, I'd like to add in some discussion about versioning the
>>>>>>>> record API
>>>>>>>>>>> separately. A lot of the custom processors we build now are
>>>>>> Record
>>>>>>>>>>> API-based, and it would be great to be able to decouple them from
>>>>>>> one
>>>>>>>>>>> specific release of NiFi.
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <bb...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Pierre,
>>>>>>>>>>>> 
>>>>>>>>>>>> I think we are definitely close to an 0.4.0 release. A major
>>>>>>> chunk
>>>>>>>> of
>>>>>>>>>>>> extension registry work has already landed in master, and I
>>>>>> still
>>>>>>>> have
>>>>>>>>>> one
>>>>>>>>>>>> other jira that I almost have ready and wanted to include,
>>>>>>>> NIFIREG-233
>>>>>>>>>> for
>>>>>>>>>>>> generating extensions docs. Plus we probably need to make a few
>>>>>>>>>>>> additions/updates to the admin guide.
>>>>>>>>>>>> 
>>>>>>>>>>>> -Bryan
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard <
>>>>>>>>>>>> pierre.villard.fr@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> All,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Some really nice features have been included since the NiFi
>>>>>>>> Registry
>>>>>>>>>>>> 0.3.0
>>>>>>>>>>>>> release and I'm wondering if it'd be a good time to consider
>>>>>> a
>>>>>>>> 0.4.0
>>>>>>>>>>>>> release.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> There is a JIRA tagged for 0.4.0 and there are 2 opened pull
>>>>>>>>>> requests,
>>>>>>>>>>>> plus
>>>>>>>>>>>>> interesting discussions / JIRAS on the mailing lists.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Are there any reasons to hold off on a 0.4.0 release?  Are
>>>>>>> there
>>>>>>>>>>>> particular
>>>>>>>>>>>>> JIRAs that the community considers necessary to have as part
>>>>>> of
>>>>>>>> the
>>>>>>>>>>>>> release?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> If not, happy to give it a try at performing RM duties!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Pierre
>>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> Sent from Gmail Mobile
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

Posted by Bryan Bende <bb...@gmail.com>.

The last few items tagged for 0.4.0 have been merged so we should be
good to put out an RC for 0.4.0.

I'll start working on getting everything together and try to get
something out soon.

On Fri, May 10, 2019 at 9:00 AM Bryan Bende <bb...@gmail.com> wrote:
>
> I think we should be able to kick 0.4.0 very soon, just a couple of
> items to get in. Assuming those get reviewed and merged soon we could
> possibly kick out an RC early next week. I can plan to be RM unless
> anyone else is interested.
>
> On Thu, Mar 28, 2019 at 3:39 PM Bryan Bende <bb...@gmail.com> wrote:
> >
> > I think Kevin summarized it nicely. Auto-pulling the extensions with a
> > versioned flow is definitely the ultimate vision, but we need to take
> > steps to get there.
> >
> > The process of automatically bringing in the extensions is something
> > that would be implemented on the NiFi side and will require a bit of
> > thought and design around the user experience.
> >
> > Before we can do any of that, we first need somewhere to store and
> > retrieve versioned extensions, which is the work that is underway on
> > the NiFi Registry side.
> >
> > On Thu, Mar 28, 2019 at 3:29 PM Kevin Doran <kd...@apache.org> wrote:
> > >
> > > Hi Ryan,
> > >
> > > Bryan can speak more to this than I can, but, yes, providing the
> > > experience you describe is the eventual goal! Aside from the NiFi
> > > Extension Registry support though (the initial bits of which is what
> > > Bryan is referring to in 0.4.0), getting the full, streamlined
> > > experience in NiFi will require changes to NiFi as well, for example
> > > in the logic to import a flow as it now also has to reach back to
> > > Registry to pull extensions referenced by the flow during import. So
> > > the changes to NiFi Registry to support hosting NiFi Extensions is
> > > only half of the puzzle. This is why Bryan also mentioned it might be
> > > worth holding back 0.4.0 (or releasing it, but marking the new
> > > extension stuff as experimental), as it is likely the current
> > > implementation and interfaces will need to change once the NiFi work
> > > begins.
> > >
> > > Cheers,
> > > Kevin
> > >
> > > On Thu, Mar 28, 2019 at 2:56 PM Ryan Hendrickson
> > > <ry...@gmail.com> wrote:
> > > >
> > > > We'd like to connect a NiFi to a Registry, Pull down a Process Group, and
> > > > get any NARs required for that without having to manually install them on
> > > > the NiFi.
> > > >
> > > > That's what I'm understanding the Extension part of the 0.4.0 release to
> > > > be, am I on point, or way off?
> > > >
> > > > Thanks,
> > > > Ryan
> > > >
> > > > On Wed, Mar 27, 2019 at 4:17 PM Bryan Rosander <br...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hey Pierre,
> > > > >
> > > > > You can version the metadata db in the root dir of the git repo with event
> > > > > hooks.
> > > > >
> > > > > With an h2 jdbc url of
> > > > >
> > > > > 'jdbc:h2:$NIFI_REGISTRY_HOME/database/nifi-registry-primary;AUTOCOMMIT=OFF;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=TRUE'
> > > > >
> > > > > You can connect to the db in a separate process, we're using that with a
> > > > > shell event hook to dump the db to sql script when mutations happen (now
> > > > > tenant events too :) ), storing that in the git repo using
> > > > > org.h2.tools.Script, and committing it.  Then we can restore from that sql
> > > > > script on startup before calling the stock startup script.
> > > > >
> > > > > Side benefit is you can see metadata state changes in the git history and
> > > > > manually inspect the sql.
> > > > >
> > > > >
> > > > > On Wed, Mar 27, 2019 at 1:57 PM Pierre Villard <
> > > > > pierre.villard.fr@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Bryan,
> > > > > >
> > > > > > Thanks for the details. On my side, the main feature I (and others I
> > > > > > suppose) am looking for is the ability to rebuild the metadata DB from
> > > > > the
> > > > > > Git repo. But to be honest, this is not a blocker at all: I can live
> > > > > with a
> > > > > > custom Docker image containing this feature.
> > > > > >
> > > > > > So, from my point of view, I don't have any issue with #2 but maybe #1
> > > > > > would also give access to experimental features and have interesting
> > > > > > feedback from the community. But we would, indeed, need to make it
> > > > > crystal
> > > > > > clear that the APIs could change in ulterior release(s).
> > > > > >
> > > > > > Pierre
> > > > > >
> > > > > >
> > > > > >
> > > > > > Le mer. 27 mars 2019 à 18:11, Bryan Bende <bb...@gmail.com> a écrit :
> > > > > >
> > > > > > > Ryan,
> > > > > > >
> > > > > > > The short answer to your question is no, there isn't actually anything
> > > > > > > new in the UI. The registry UI is setup in a generic way to list
> > > > > > > "versioned items", so versioned extension bundles will just
> > > > > > > automatically show up in the list with versioned flows.
> > > > > > >
> > > > > > > Let me put together a wiki page that outlines all of the moving parts
> > > > > > > for the extension registry work and where we are currently at. I'll
> > > > > > > send the link on this thread when I have something.
> > > > > > >
> > > > > > >
> > > > > > > For Pierre and others,
> > > > > > >
> > > > > > > I was thinking more about this discussion of releasing 0.4.0, and I
> > > > > > > think there are two of approaches we could take...
> > > > > > >
> > > > > > > 1) If there is a desire for the community to release 0.4.0 sooner
> > > > > > > rather than later, then we can mark all of the extension bundle
> > > > > > > related APIs as experimental, which would then give us the freedom to
> > > > > > > continue evolving them until we have the full extension registry
> > > > > > > experience, and likely mark them as stable in 0.5.0, or whichever
> > > > > > > future release we decide. The reason for this is that there will
> > > > > > > likely be API changes needed while building out the NiFi side of
> > > > > > > things, and I don't want us to get stuck in a spot where we can't
> > > > > > > change some API since it has been released, and so far the NiFi side
> > > > > > > of the work hasn't been started yet.
> > > > > > >
> > > > > > > 2) Wait until more of the extension registry work is finalized and use
> > > > > > > that as the driver of when to release 0.4.0, with the obvious downside
> > > > > > > that it may take a bit longer to get a release out which then holds up
> > > > > > > anything else that is not related to extension registry.
> > > > > > >
> > > > > > > So I guess we just have to decide the driving factor for releasing
> > > > > > > 0.4.0... If its the non-extension related features/fixes, then #1 is
> > > > > > > probably the best approach. If its the extension related stuff, then
> > > > > > > I'd vote for #2 since it isn't really ready yet.
> > > > > > >
> > > > > > > -Bryan
> > > > > > >
> > > > > > > On Wed, Mar 27, 2019 at 12:01 PM Ryan Hendrickson
> > > > > > > <ry...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Bryan,
> > > > > > > >    Is there a new GUI part of the registry for the extension registry
> > > > > > > work
> > > > > > > > in master?  I compiled 0.4.0-SNAPSHOT and didn't see anything
> > > > > > > immediately.
> > > > > > > > You mentioned docs and sys admin stuff... Could you point me in the
> > > > > > > > quick-start path if I wanted to try to kick-the-tires a little on
> > > > > this?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Ryan
> > > > > > > >
> > > > > > > > On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende <bb...@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Mike,
> > > > > > > > >
> > > > > > > > > All record-oriented components are extensions and thus can make use
> > > > > > of
> > > > > > > > > versioned components (i.e. nothing related to records is baked into
> > > > > > > > > the core framework of NiFi).
> > > > > > > > >
> > > > > > > > > The record API is essentially the module named
> > > > > > > > > nifi-record-serialization-services-api which at runtime is included
> > > > > > in
> > > > > > > > > nifi-standard-services-api-nar. Any record oriented processors you
> > > > > > > > > build are dependent on a specific version of
> > > > > > > > > nifi-standard-services-api-nar since that is where the Java
> > > > > interface
> > > > > > > > > will be that your processor was compiled against.
> > > > > > > > >
> > > > > > > > > Right now, even without extension registry, you could deploy
> > > > > multiple
> > > > > > > > > versions of nifi-standard-services-api-nar to a NiFi instance, and
> > > > > > > > > then deploy multiple versions of your record processing NAR that
> > > > > > > > > correspond to the versions of your API NAR.
> > > > > > > > >
> > > > > > > > > Going forward with extension registry, we probably want to consider
> > > > > > > > > breaking up nifi-standard-services-api-nar into smaller API NARs,
> > > > > as
> > > > > > > > > well as nifi-standard-bundle into smaller processor bundles. For
> > > > > > > > > example, there could be a record API NAR and a record processors
> > > > > NAR,
> > > > > > > > > that would both be split out of from the standard NARs.
> > > > > > > > >
> > > > > > > > > -Bryan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen <
> > > > > mikerthomsen@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Great news about the extension registry! As we get closer to that
> > > > > > > being
> > > > > > > > > > ready, I'd like to add in some discussion about versioning the
> > > > > > > record API
> > > > > > > > > > separately. A lot of the custom processors we build now are
> > > > > Record
> > > > > > > > > > API-based, and it would be great to be able to decouple them from
> > > > > > one
> > > > > > > > > > specific release of NiFi.
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <bb...@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Pierre,
> > > > > > > > > > >
> > > > > > > > > > > I think we are definitely close to an 0.4.0 release. A major
> > > > > > chunk
> > > > > > > of
> > > > > > > > > > > extension registry work has already landed in master, and I
> > > > > still
> > > > > > > have
> > > > > > > > > one
> > > > > > > > > > > other jira that I almost have ready and wanted to include,
> > > > > > > NIFIREG-233
> > > > > > > > > for
> > > > > > > > > > > generating extensions docs. Plus we probably need to make a few
> > > > > > > > > > > additions/updates to the admin guide.
> > > > > > > > > > >
> > > > > > > > > > > -Bryan
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard <
> > > > > > > > > > > pierre.villard.fr@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > All,
> > > > > > > > > > > >
> > > > > > > > > > > > Some really nice features have been included since the NiFi
> > > > > > > Registry
> > > > > > > > > > > 0.3.0
> > > > > > > > > > > > release and I'm wondering if it'd be a good time to consider
> > > > > a
> > > > > > > 0.4.0
> > > > > > > > > > > > release.
> > > > > > > > > > > >
> > > > > > > > > > > > There is a JIRA tagged for 0.4.0 and there are 2 opened pull
> > > > > > > > > requests,
> > > > > > > > > > > plus
> > > > > > > > > > > > interesting discussions / JIRAS on the mailing lists.
> > > > > > > > > > > >
> > > > > > > > > > > > Are there any reasons to hold off on a 0.4.0 release?  Are
> > > > > > there
> > > > > > > > > > > particular
> > > > > > > > > > > > JIRAs that the community considers necessary to have as part
> > > > > of
> > > > > > > the
> > > > > > > > > > > > release?
> > > > > > > > > > > >
> > > > > > > > > > > > If not, happy to give it a try at performing RM duties!
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Pierre
> > > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Sent from Gmail Mobile
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

Posted by Bryan Bende <bb...@gmail.com>.

I think we should be able to kick 0.4.0 very soon, just a couple of
items to get in. Assuming those get reviewed and merged soon we could
possibly kick out an RC early next week. I can plan to be RM unless
anyone else is interested.

On Thu, Mar 28, 2019 at 3:39 PM Bryan Bende <bb...@gmail.com> wrote:
>
> I think Kevin summarized it nicely. Auto-pulling the extensions with a
> versioned flow is definitely the ultimate vision, but we need to take
> steps to get there.
>
> The process of automatically bringing in the extensions is something
> that would be implemented on the NiFi side and will require a bit of
> thought and design around the user experience.
>
> Before we can do any of that, we first need somewhere to store and
> retrieve versioned extensions, which is the work that is underway on
> the NiFi Registry side.
>
> On Thu, Mar 28, 2019 at 3:29 PM Kevin Doran <kd...@apache.org> wrote:
> >
> > Hi Ryan,
> >
> > Bryan can speak more to this than I can, but, yes, providing the
> > experience you describe is the eventual goal! Aside from the NiFi
> > Extension Registry support though (the initial bits of which is what
> > Bryan is referring to in 0.4.0), getting the full, streamlined
> > experience in NiFi will require changes to NiFi as well, for example
> > in the logic to import a flow as it now also has to reach back to
> > Registry to pull extensions referenced by the flow during import. So
> > the changes to NiFi Registry to support hosting NiFi Extensions is
> > only half of the puzzle. This is why Bryan also mentioned it might be
> > worth holding back 0.4.0 (or releasing it, but marking the new
> > extension stuff as experimental), as it is likely the current
> > implementation and interfaces will need to change once the NiFi work
> > begins.
> >
> > Cheers,
> > Kevin
> >
> > On Thu, Mar 28, 2019 at 2:56 PM Ryan Hendrickson
> > <ry...@gmail.com> wrote:
> > >
> > > We'd like to connect a NiFi to a Registry, Pull down a Process Group, and
> > > get any NARs required for that without having to manually install them on
> > > the NiFi.
> > >
> > > That's what I'm understanding the Extension part of the 0.4.0 release to
> > > be, am I on point, or way off?
> > >
> > > Thanks,
> > > Ryan
> > >
> > > On Wed, Mar 27, 2019 at 4:17 PM Bryan Rosander <br...@gmail.com>
> > > wrote:
> > >
> > > > Hey Pierre,
> > > >
> > > > You can version the metadata db in the root dir of the git repo with event
> > > > hooks.
> > > >
> > > > With an h2 jdbc url of
> > > >
> > > > 'jdbc:h2:$NIFI_REGISTRY_HOME/database/nifi-registry-primary;AUTOCOMMIT=OFF;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=TRUE'
> > > >
> > > > You can connect to the db in a separate process, we're using that with a
> > > > shell event hook to dump the db to sql script when mutations happen (now
> > > > tenant events too :) ), storing that in the git repo using
> > > > org.h2.tools.Script, and committing it.  Then we can restore from that sql
> > > > script on startup before calling the stock startup script.
> > > >
> > > > Side benefit is you can see metadata state changes in the git history and
> > > > manually inspect the sql.
> > > >
> > > >
> > > > On Wed, Mar 27, 2019 at 1:57 PM Pierre Villard <
> > > > pierre.villard.fr@gmail.com>
> > > > wrote:
> > > >
> > > > > Bryan,
> > > > >
> > > > > Thanks for the details. On my side, the main feature I (and others I
> > > > > suppose) am looking for is the ability to rebuild the metadata DB from
> > > > the
> > > > > Git repo. But to be honest, this is not a blocker at all: I can live
> > > > with a
> > > > > custom Docker image containing this feature.
> > > > >
> > > > > So, from my point of view, I don't have any issue with #2 but maybe #1
> > > > > would also give access to experimental features and have interesting
> > > > > feedback from the community. But we would, indeed, need to make it
> > > > crystal
> > > > > clear that the APIs could change in ulterior release(s).
> > > > >
> > > > > Pierre
> > > > >
> > > > >
> > > > >
> > > > > Le mer. 27 mars 2019 à 18:11, Bryan Bende <bb...@gmail.com> a écrit :
> > > > >
> > > > > > Ryan,
> > > > > >
> > > > > > The short answer to your question is no, there isn't actually anything
> > > > > > new in the UI. The registry UI is setup in a generic way to list
> > > > > > "versioned items", so versioned extension bundles will just
> > > > > > automatically show up in the list with versioned flows.
> > > > > >
> > > > > > Let me put together a wiki page that outlines all of the moving parts
> > > > > > for the extension registry work and where we are currently at. I'll
> > > > > > send the link on this thread when I have something.
> > > > > >
> > > > > >
> > > > > > For Pierre and others,
> > > > > >
> > > > > > I was thinking more about this discussion of releasing 0.4.0, and I
> > > > > > think there are two of approaches we could take...
> > > > > >
> > > > > > 1) If there is a desire for the community to release 0.4.0 sooner
> > > > > > rather than later, then we can mark all of the extension bundle
> > > > > > related APIs as experimental, which would then give us the freedom to
> > > > > > continue evolving them until we have the full extension registry
> > > > > > experience, and likely mark them as stable in 0.5.0, or whichever
> > > > > > future release we decide. The reason for this is that there will
> > > > > > likely be API changes needed while building out the NiFi side of
> > > > > > things, and I don't want us to get stuck in a spot where we can't
> > > > > > change some API since it has been released, and so far the NiFi side
> > > > > > of the work hasn't been started yet.
> > > > > >
> > > > > > 2) Wait until more of the extension registry work is finalized and use
> > > > > > that as the driver of when to release 0.4.0, with the obvious downside
> > > > > > that it may take a bit longer to get a release out which then holds up
> > > > > > anything else that is not related to extension registry.
> > > > > >
> > > > > > So I guess we just have to decide the driving factor for releasing
> > > > > > 0.4.0... If its the non-extension related features/fixes, then #1 is
> > > > > > probably the best approach. If its the extension related stuff, then
> > > > > > I'd vote for #2 since it isn't really ready yet.
> > > > > >
> > > > > > -Bryan
> > > > > >
> > > > > > On Wed, Mar 27, 2019 at 12:01 PM Ryan Hendrickson
> > > > > > <ry...@gmail.com> wrote:
> > > > > > >
> > > > > > > Bryan,
> > > > > > >    Is there a new GUI part of the registry for the extension registry
> > > > > > work
> > > > > > > in master?  I compiled 0.4.0-SNAPSHOT and didn't see anything
> > > > > > immediately.
> > > > > > > You mentioned docs and sys admin stuff... Could you point me in the
> > > > > > > quick-start path if I wanted to try to kick-the-tires a little on
> > > > this?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Ryan
> > > > > > >
> > > > > > > On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende <bb...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Mike,
> > > > > > > >
> > > > > > > > All record-oriented components are extensions and thus can make use
> > > > > of
> > > > > > > > versioned components (i.e. nothing related to records is baked into
> > > > > > > > the core framework of NiFi).
> > > > > > > >
> > > > > > > > The record API is essentially the module named
> > > > > > > > nifi-record-serialization-services-api which at runtime is included
> > > > > in
> > > > > > > > nifi-standard-services-api-nar. Any record oriented processors you
> > > > > > > > build are dependent on a specific version of
> > > > > > > > nifi-standard-services-api-nar since that is where the Java
> > > > interface
> > > > > > > > will be that your processor was compiled against.
> > > > > > > >
> > > > > > > > Right now, even without extension registry, you could deploy
> > > > multiple
> > > > > > > > versions of nifi-standard-services-api-nar to a NiFi instance, and
> > > > > > > > then deploy multiple versions of your record processing NAR that
> > > > > > > > correspond to the versions of your API NAR.
> > > > > > > >
> > > > > > > > Going forward with extension registry, we probably want to consider
> > > > > > > > breaking up nifi-standard-services-api-nar into smaller API NARs,
> > > > as
> > > > > > > > well as nifi-standard-bundle into smaller processor bundles. For
> > > > > > > > example, there could be a record API NAR and a record processors
> > > > NAR,
> > > > > > > > that would both be split out of from the standard NARs.
> > > > > > > >
> > > > > > > > -Bryan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen <
> > > > mikerthomsen@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Great news about the extension registry! As we get closer to that
> > > > > > being
> > > > > > > > > ready, I'd like to add in some discussion about versioning the
> > > > > > record API
> > > > > > > > > separately. A lot of the custom processors we build now are
> > > > Record
> > > > > > > > > API-based, and it would be great to be able to decouple them from
> > > > > one
> > > > > > > > > specific release of NiFi.
> > > > > > > > >
> > > > > > > > > On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <bb...@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Pierre,
> > > > > > > > > >
> > > > > > > > > > I think we are definitely close to an 0.4.0 release. A major
> > > > > chunk
> > > > > > of
> > > > > > > > > > extension registry work has already landed in master, and I
> > > > still
> > > > > > have
> > > > > > > > one
> > > > > > > > > > other jira that I almost have ready and wanted to include,
> > > > > > NIFIREG-233
> > > > > > > > for
> > > > > > > > > > generating extensions docs. Plus we probably need to make a few
> > > > > > > > > > additions/updates to the admin guide.
> > > > > > > > > >
> > > > > > > > > > -Bryan
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard <
> > > > > > > > > > pierre.villard.fr@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > All,
> > > > > > > > > > >
> > > > > > > > > > > Some really nice features have been included since the NiFi
> > > > > > Registry
> > > > > > > > > > 0.3.0
> > > > > > > > > > > release and I'm wondering if it'd be a good time to consider
> > > > a
> > > > > > 0.4.0
> > > > > > > > > > > release.
> > > > > > > > > > >
> > > > > > > > > > > There is a JIRA tagged for 0.4.0 and there are 2 opened pull
> > > > > > > > requests,
> > > > > > > > > > plus
> > > > > > > > > > > interesting discussions / JIRAS on the mailing lists.
> > > > > > > > > > >
> > > > > > > > > > > Are there any reasons to hold off on a 0.4.0 release?  Are
> > > > > there
> > > > > > > > > > particular
> > > > > > > > > > > JIRAs that the community considers necessary to have as part
> > > > of
> > > > > > the
> > > > > > > > > > > release?
> > > > > > > > > > >
> > > > > > > > > > > If not, happy to give it a try at performing RM duties!
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Pierre
> > > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Sent from Gmail Mobile
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

Posted by Bryan Bende <bb...@gmail.com>.

I think Kevin summarized it nicely. Auto-pulling the extensions with a
versioned flow is definitely the ultimate vision, but we need to take
steps to get there.

The process of automatically bringing in the extensions is something
that would be implemented on the NiFi side and will require a bit of
thought and design around the user experience.

Before we can do any of that, we first need somewhere to store and
retrieve versioned extensions, which is the work that is underway on
the NiFi Registry side.

On Thu, Mar 28, 2019 at 3:29 PM Kevin Doran <kd...@apache.org> wrote:
>
> Hi Ryan,
>
> Bryan can speak more to this than I can, but, yes, providing the
> experience you describe is the eventual goal! Aside from the NiFi
> Extension Registry support though (the initial bits of which is what
> Bryan is referring to in 0.4.0), getting the full, streamlined
> experience in NiFi will require changes to NiFi as well, for example
> in the logic to import a flow as it now also has to reach back to
> Registry to pull extensions referenced by the flow during import. So
> the changes to NiFi Registry to support hosting NiFi Extensions is
> only half of the puzzle. This is why Bryan also mentioned it might be
> worth holding back 0.4.0 (or releasing it, but marking the new
> extension stuff as experimental), as it is likely the current
> implementation and interfaces will need to change once the NiFi work
> begins.
>
> Cheers,
> Kevin
>
> On Thu, Mar 28, 2019 at 2:56 PM Ryan Hendrickson
> <ry...@gmail.com> wrote:
> >
> > We'd like to connect a NiFi to a Registry, Pull down a Process Group, and
> > get any NARs required for that without having to manually install them on
> > the NiFi.
> >
> > That's what I'm understanding the Extension part of the 0.4.0 release to
> > be, am I on point, or way off?
> >
> > Thanks,
> > Ryan
> >
> > On Wed, Mar 27, 2019 at 4:17 PM Bryan Rosander <br...@gmail.com>
> > wrote:
> >
> > > Hey Pierre,
> > >
> > > You can version the metadata db in the root dir of the git repo with event
> > > hooks.
> > >
> > > With an h2 jdbc url of
> > >
> > > 'jdbc:h2:$NIFI_REGISTRY_HOME/database/nifi-registry-primary;AUTOCOMMIT=OFF;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=TRUE'
> > >
> > > You can connect to the db in a separate process, we're using that with a
> > > shell event hook to dump the db to sql script when mutations happen (now
> > > tenant events too :) ), storing that in the git repo using
> > > org.h2.tools.Script, and committing it.  Then we can restore from that sql
> > > script on startup before calling the stock startup script.
> > >
> > > Side benefit is you can see metadata state changes in the git history and
> > > manually inspect the sql.
> > >
> > >
> > > On Wed, Mar 27, 2019 at 1:57 PM Pierre Villard <
> > > pierre.villard.fr@gmail.com>
> > > wrote:
> > >
> > > > Bryan,
> > > >
> > > > Thanks for the details. On my side, the main feature I (and others I
> > > > suppose) am looking for is the ability to rebuild the metadata DB from
> > > the
> > > > Git repo. But to be honest, this is not a blocker at all: I can live
> > > with a
> > > > custom Docker image containing this feature.
> > > >
> > > > So, from my point of view, I don't have any issue with #2 but maybe #1
> > > > would also give access to experimental features and have interesting
> > > > feedback from the community. But we would, indeed, need to make it
> > > crystal
> > > > clear that the APIs could change in ulterior release(s).
> > > >
> > > > Pierre
> > > >
> > > >
> > > >
> > > > Le mer. 27 mars 2019 à 18:11, Bryan Bende <bb...@gmail.com> a écrit :
> > > >
> > > > > Ryan,
> > > > >
> > > > > The short answer to your question is no, there isn't actually anything
> > > > > new in the UI. The registry UI is setup in a generic way to list
> > > > > "versioned items", so versioned extension bundles will just
> > > > > automatically show up in the list with versioned flows.
> > > > >
> > > > > Let me put together a wiki page that outlines all of the moving parts
> > > > > for the extension registry work and where we are currently at. I'll
> > > > > send the link on this thread when I have something.
> > > > >
> > > > >
> > > > > For Pierre and others,
> > > > >
> > > > > I was thinking more about this discussion of releasing 0.4.0, and I
> > > > > think there are two of approaches we could take...
> > > > >
> > > > > 1) If there is a desire for the community to release 0.4.0 sooner
> > > > > rather than later, then we can mark all of the extension bundle
> > > > > related APIs as experimental, which would then give us the freedom to
> > > > > continue evolving them until we have the full extension registry
> > > > > experience, and likely mark them as stable in 0.5.0, or whichever
> > > > > future release we decide. The reason for this is that there will
> > > > > likely be API changes needed while building out the NiFi side of
> > > > > things, and I don't want us to get stuck in a spot where we can't
> > > > > change some API since it has been released, and so far the NiFi side
> > > > > of the work hasn't been started yet.
> > > > >
> > > > > 2) Wait until more of the extension registry work is finalized and use
> > > > > that as the driver of when to release 0.4.0, with the obvious downside
> > > > > that it may take a bit longer to get a release out which then holds up
> > > > > anything else that is not related to extension registry.
> > > > >
> > > > > So I guess we just have to decide the driving factor for releasing
> > > > > 0.4.0... If its the non-extension related features/fixes, then #1 is
> > > > > probably the best approach. If its the extension related stuff, then
> > > > > I'd vote for #2 since it isn't really ready yet.
> > > > >
> > > > > -Bryan
> > > > >
> > > > > On Wed, Mar 27, 2019 at 12:01 PM Ryan Hendrickson
> > > > > <ry...@gmail.com> wrote:
> > > > > >
> > > > > > Bryan,
> > > > > >    Is there a new GUI part of the registry for the extension registry
> > > > > work
> > > > > > in master?  I compiled 0.4.0-SNAPSHOT and didn't see anything
> > > > > immediately.
> > > > > > You mentioned docs and sys admin stuff... Could you point me in the
> > > > > > quick-start path if I wanted to try to kick-the-tires a little on
> > > this?
> > > > > >
> > > > > > Thanks,
> > > > > > Ryan
> > > > > >
> > > > > > On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende <bb...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Mike,
> > > > > > >
> > > > > > > All record-oriented components are extensions and thus can make use
> > > > of
> > > > > > > versioned components (i.e. nothing related to records is baked into
> > > > > > > the core framework of NiFi).
> > > > > > >
> > > > > > > The record API is essentially the module named
> > > > > > > nifi-record-serialization-services-api which at runtime is included
> > > > in
> > > > > > > nifi-standard-services-api-nar. Any record oriented processors you
> > > > > > > build are dependent on a specific version of
> > > > > > > nifi-standard-services-api-nar since that is where the Java
> > > interface
> > > > > > > will be that your processor was compiled against.
> > > > > > >
> > > > > > > Right now, even without extension registry, you could deploy
> > > multiple
> > > > > > > versions of nifi-standard-services-api-nar to a NiFi instance, and
> > > > > > > then deploy multiple versions of your record processing NAR that
> > > > > > > correspond to the versions of your API NAR.
> > > > > > >
> > > > > > > Going forward with extension registry, we probably want to consider
> > > > > > > breaking up nifi-standard-services-api-nar into smaller API NARs,
> > > as
> > > > > > > well as nifi-standard-bundle into smaller processor bundles. For
> > > > > > > example, there could be a record API NAR and a record processors
> > > NAR,
> > > > > > > that would both be split out of from the standard NARs.
> > > > > > >
> > > > > > > -Bryan
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen <
> > > mikerthomsen@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Great news about the extension registry! As we get closer to that
> > > > > being
> > > > > > > > ready, I'd like to add in some discussion about versioning the
> > > > > record API
> > > > > > > > separately. A lot of the custom processors we build now are
> > > Record
> > > > > > > > API-based, and it would be great to be able to decouple them from
> > > > one
> > > > > > > > specific release of NiFi.
> > > > > > > >
> > > > > > > > On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <bb...@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Pierre,
> > > > > > > > >
> > > > > > > > > I think we are definitely close to an 0.4.0 release. A major
> > > > chunk
> > > > > of
> > > > > > > > > extension registry work has already landed in master, and I
> > > still
> > > > > have
> > > > > > > one
> > > > > > > > > other jira that I almost have ready and wanted to include,
> > > > > NIFIREG-233
> > > > > > > for
> > > > > > > > > generating extensions docs. Plus we probably need to make a few
> > > > > > > > > additions/updates to the admin guide.
> > > > > > > > >
> > > > > > > > > -Bryan
> > > > > > > > >
> > > > > > > > > On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard <
> > > > > > > > > pierre.villard.fr@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > All,
> > > > > > > > > >
> > > > > > > > > > Some really nice features have been included since the NiFi
> > > > > Registry
> > > > > > > > > 0.3.0
> > > > > > > > > > release and I'm wondering if it'd be a good time to consider
> > > a
> > > > > 0.4.0
> > > > > > > > > > release.
> > > > > > > > > >
> > > > > > > > > > There is a JIRA tagged for 0.4.0 and there are 2 opened pull
> > > > > > > requests,
> > > > > > > > > plus
> > > > > > > > > > interesting discussions / JIRAS on the mailing lists.
> > > > > > > > > >
> > > > > > > > > > Are there any reasons to hold off on a 0.4.0 release?  Are
> > > > there
> > > > > > > > > particular
> > > > > > > > > > JIRAs that the community considers necessary to have as part
> > > of
> > > > > the
> > > > > > > > > > release?
> > > > > > > > > >
> > > > > > > > > > If not, happy to give it a try at performing RM duties!
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Pierre
> > > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Sent from Gmail Mobile
> > > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

Posted by Kevin Doran <kd...@apache.org>.

Hi Ryan,

Bryan can speak more to this than I can, but, yes, providing the
experience you describe is the eventual goal! Aside from the NiFi
Extension Registry support though (the initial bits of which is what
Bryan is referring to in 0.4.0), getting the full, streamlined
experience in NiFi will require changes to NiFi as well, for example
in the logic to import a flow as it now also has to reach back to
Registry to pull extensions referenced by the flow during import. So
the changes to NiFi Registry to support hosting NiFi Extensions is
only half of the puzzle. This is why Bryan also mentioned it might be
worth holding back 0.4.0 (or releasing it, but marking the new
extension stuff as experimental), as it is likely the current
implementation and interfaces will need to change once the NiFi work
begins.

Cheers,
Kevin

On Thu, Mar 28, 2019 at 2:56 PM Ryan Hendrickson
<ry...@gmail.com> wrote:
>
> We'd like to connect a NiFi to a Registry, Pull down a Process Group, and
> get any NARs required for that without having to manually install them on
> the NiFi.
>
> That's what I'm understanding the Extension part of the 0.4.0 release to
> be, am I on point, or way off?
>
> Thanks,
> Ryan
>
> On Wed, Mar 27, 2019 at 4:17 PM Bryan Rosander <br...@gmail.com>
> wrote:
>
> > Hey Pierre,
> >
> > You can version the metadata db in the root dir of the git repo with event
> > hooks.
> >
> > With an h2 jdbc url of
> >
> > 'jdbc:h2:$NIFI_REGISTRY_HOME/database/nifi-registry-primary;AUTOCOMMIT=OFF;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=TRUE'
> >
> > You can connect to the db in a separate process, we're using that with a
> > shell event hook to dump the db to sql script when mutations happen (now
> > tenant events too :) ), storing that in the git repo using
> > org.h2.tools.Script, and committing it.  Then we can restore from that sql
> > script on startup before calling the stock startup script.
> >
> > Side benefit is you can see metadata state changes in the git history and
> > manually inspect the sql.
> >
> >
> > On Wed, Mar 27, 2019 at 1:57 PM Pierre Villard <
> > pierre.villard.fr@gmail.com>
> > wrote:
> >
> > > Bryan,
> > >
> > > Thanks for the details. On my side, the main feature I (and others I
> > > suppose) am looking for is the ability to rebuild the metadata DB from
> > the
> > > Git repo. But to be honest, this is not a blocker at all: I can live
> > with a
> > > custom Docker image containing this feature.
> > >
> > > So, from my point of view, I don't have any issue with #2 but maybe #1
> > > would also give access to experimental features and have interesting
> > > feedback from the community. But we would, indeed, need to make it
> > crystal
> > > clear that the APIs could change in ulterior release(s).
> > >
> > > Pierre
> > >
> > >
> > >
> > > Le mer. 27 mars 2019 à 18:11, Bryan Bende <bb...@gmail.com> a écrit :
> > >
> > > > Ryan,
> > > >
> > > > The short answer to your question is no, there isn't actually anything
> > > > new in the UI. The registry UI is setup in a generic way to list
> > > > "versioned items", so versioned extension bundles will just
> > > > automatically show up in the list with versioned flows.
> > > >
> > > > Let me put together a wiki page that outlines all of the moving parts
> > > > for the extension registry work and where we are currently at. I'll
> > > > send the link on this thread when I have something.
> > > >
> > > >
> > > > For Pierre and others,
> > > >
> > > > I was thinking more about this discussion of releasing 0.4.0, and I
> > > > think there are two of approaches we could take...
> > > >
> > > > 1) If there is a desire for the community to release 0.4.0 sooner
> > > > rather than later, then we can mark all of the extension bundle
> > > > related APIs as experimental, which would then give us the freedom to
> > > > continue evolving them until we have the full extension registry
> > > > experience, and likely mark them as stable in 0.5.0, or whichever
> > > > future release we decide. The reason for this is that there will
> > > > likely be API changes needed while building out the NiFi side of
> > > > things, and I don't want us to get stuck in a spot where we can't
> > > > change some API since it has been released, and so far the NiFi side
> > > > of the work hasn't been started yet.
> > > >
> > > > 2) Wait until more of the extension registry work is finalized and use
> > > > that as the driver of when to release 0.4.0, with the obvious downside
> > > > that it may take a bit longer to get a release out which then holds up
> > > > anything else that is not related to extension registry.
> > > >
> > > > So I guess we just have to decide the driving factor for releasing
> > > > 0.4.0... If its the non-extension related features/fixes, then #1 is
> > > > probably the best approach. If its the extension related stuff, then
> > > > I'd vote for #2 since it isn't really ready yet.
> > > >
> > > > -Bryan
> > > >
> > > > On Wed, Mar 27, 2019 at 12:01 PM Ryan Hendrickson
> > > > <ry...@gmail.com> wrote:
> > > > >
> > > > > Bryan,
> > > > >    Is there a new GUI part of the registry for the extension registry
> > > > work
> > > > > in master?  I compiled 0.4.0-SNAPSHOT and didn't see anything
> > > > immediately.
> > > > > You mentioned docs and sys admin stuff... Could you point me in the
> > > > > quick-start path if I wanted to try to kick-the-tires a little on
> > this?
> > > > >
> > > > > Thanks,
> > > > > Ryan
> > > > >
> > > > > On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende <bb...@gmail.com>
> > wrote:
> > > > >
> > > > > > Mike,
> > > > > >
> > > > > > All record-oriented components are extensions and thus can make use
> > > of
> > > > > > versioned components (i.e. nothing related to records is baked into
> > > > > > the core framework of NiFi).
> > > > > >
> > > > > > The record API is essentially the module named
> > > > > > nifi-record-serialization-services-api which at runtime is included
> > > in
> > > > > > nifi-standard-services-api-nar. Any record oriented processors you
> > > > > > build are dependent on a specific version of
> > > > > > nifi-standard-services-api-nar since that is where the Java
> > interface
> > > > > > will be that your processor was compiled against.
> > > > > >
> > > > > > Right now, even without extension registry, you could deploy
> > multiple
> > > > > > versions of nifi-standard-services-api-nar to a NiFi instance, and
> > > > > > then deploy multiple versions of your record processing NAR that
> > > > > > correspond to the versions of your API NAR.
> > > > > >
> > > > > > Going forward with extension registry, we probably want to consider
> > > > > > breaking up nifi-standard-services-api-nar into smaller API NARs,
> > as
> > > > > > well as nifi-standard-bundle into smaller processor bundles. For
> > > > > > example, there could be a record API NAR and a record processors
> > NAR,
> > > > > > that would both be split out of from the standard NARs.
> > > > > >
> > > > > > -Bryan
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen <
> > mikerthomsen@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Great news about the extension registry! As we get closer to that
> > > > being
> > > > > > > ready, I'd like to add in some discussion about versioning the
> > > > record API
> > > > > > > separately. A lot of the custom processors we build now are
> > Record
> > > > > > > API-based, and it would be great to be able to decouple them from
> > > one
> > > > > > > specific release of NiFi.
> > > > > > >
> > > > > > > On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <bb...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Pierre,
> > > > > > > >
> > > > > > > > I think we are definitely close to an 0.4.0 release. A major
> > > chunk
> > > > of
> > > > > > > > extension registry work has already landed in master, and I
> > still
> > > > have
> > > > > > one
> > > > > > > > other jira that I almost have ready and wanted to include,
> > > > NIFIREG-233
> > > > > > for
> > > > > > > > generating extensions docs. Plus we probably need to make a few
> > > > > > > > additions/updates to the admin guide.
> > > > > > > >
> > > > > > > > -Bryan
> > > > > > > >
> > > > > > > > On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard <
> > > > > > > > pierre.villard.fr@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > All,
> > > > > > > > >
> > > > > > > > > Some really nice features have been included since the NiFi
> > > > Registry
> > > > > > > > 0.3.0
> > > > > > > > > release and I'm wondering if it'd be a good time to consider
> > a
> > > > 0.4.0
> > > > > > > > > release.
> > > > > > > > >
> > > > > > > > > There is a JIRA tagged for 0.4.0 and there are 2 opened pull
> > > > > > requests,
> > > > > > > > plus
> > > > > > > > > interesting discussions / JIRAS on the mailing lists.
> > > > > > > > >
> > > > > > > > > Are there any reasons to hold off on a 0.4.0 release?  Are
> > > there
> > > > > > > > particular
> > > > > > > > > JIRAs that the community considers necessary to have as part
> > of
> > > > the
> > > > > > > > > release?
> > > > > > > > >
> > > > > > > > > If not, happy to give it a try at performing RM duties!
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Pierre
> > > > > > > > >
> > > > > > > > --
> > > > > > > > Sent from Gmail Mobile
> > > > > > > >
> > > > > >
> > > >
> > >
> >

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

Posted by Ryan Hendrickson <ry...@gmail.com>.

We'd like to connect a NiFi to a Registry, Pull down a Process Group, and
get any NARs required for that without having to manually install them on
the NiFi.

That's what I'm understanding the Extension part of the 0.4.0 release to
be, am I on point, or way off?

Thanks,
Ryan

On Wed, Mar 27, 2019 at 4:17 PM Bryan Rosander <br...@gmail.com>
wrote:

> Hey Pierre,
>
> You can version the metadata db in the root dir of the git repo with event
> hooks.
>
> With an h2 jdbc url of
>
> 'jdbc:h2:$NIFI_REGISTRY_HOME/database/nifi-registry-primary;AUTOCOMMIT=OFF;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=TRUE'
>
> You can connect to the db in a separate process, we're using that with a
> shell event hook to dump the db to sql script when mutations happen (now
> tenant events too :) ), storing that in the git repo using
> org.h2.tools.Script, and committing it.  Then we can restore from that sql
> script on startup before calling the stock startup script.
>
> Side benefit is you can see metadata state changes in the git history and
> manually inspect the sql.
>
>
> On Wed, Mar 27, 2019 at 1:57 PM Pierre Villard <
> pierre.villard.fr@gmail.com>
> wrote:
>
> > Bryan,
> >
> > Thanks for the details. On my side, the main feature I (and others I
> > suppose) am looking for is the ability to rebuild the metadata DB from
> the
> > Git repo. But to be honest, this is not a blocker at all: I can live
> with a
> > custom Docker image containing this feature.
> >
> > So, from my point of view, I don't have any issue with #2 but maybe #1
> > would also give access to experimental features and have interesting
> > feedback from the community. But we would, indeed, need to make it
> crystal
> > clear that the APIs could change in ulterior release(s).
> >
> > Pierre
> >
> >
> >
> > Le mer. 27 mars 2019 à 18:11, Bryan Bende <bb...@gmail.com> a écrit :
> >
> > > Ryan,
> > >
> > > The short answer to your question is no, there isn't actually anything
> > > new in the UI. The registry UI is setup in a generic way to list
> > > "versioned items", so versioned extension bundles will just
> > > automatically show up in the list with versioned flows.
> > >
> > > Let me put together a wiki page that outlines all of the moving parts
> > > for the extension registry work and where we are currently at. I'll
> > > send the link on this thread when I have something.
> > >
> > >
> > > For Pierre and others,
> > >
> > > I was thinking more about this discussion of releasing 0.4.0, and I
> > > think there are two of approaches we could take...
> > >
> > > 1) If there is a desire for the community to release 0.4.0 sooner
> > > rather than later, then we can mark all of the extension bundle
> > > related APIs as experimental, which would then give us the freedom to
> > > continue evolving them until we have the full extension registry
> > > experience, and likely mark them as stable in 0.5.0, or whichever
> > > future release we decide. The reason for this is that there will
> > > likely be API changes needed while building out the NiFi side of
> > > things, and I don't want us to get stuck in a spot where we can't
> > > change some API since it has been released, and so far the NiFi side
> > > of the work hasn't been started yet.
> > >
> > > 2) Wait until more of the extension registry work is finalized and use
> > > that as the driver of when to release 0.4.0, with the obvious downside
> > > that it may take a bit longer to get a release out which then holds up
> > > anything else that is not related to extension registry.
> > >
> > > So I guess we just have to decide the driving factor for releasing
> > > 0.4.0... If its the non-extension related features/fixes, then #1 is
> > > probably the best approach. If its the extension related stuff, then
> > > I'd vote for #2 since it isn't really ready yet.
> > >
> > > -Bryan
> > >
> > > On Wed, Mar 27, 2019 at 12:01 PM Ryan Hendrickson
> > > <ry...@gmail.com> wrote:
> > > >
> > > > Bryan,
> > > >    Is there a new GUI part of the registry for the extension registry
> > > work
> > > > in master?  I compiled 0.4.0-SNAPSHOT and didn't see anything
> > > immediately.
> > > > You mentioned docs and sys admin stuff... Could you point me in the
> > > > quick-start path if I wanted to try to kick-the-tires a little on
> this?
> > > >
> > > > Thanks,
> > > > Ryan
> > > >
> > > > On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende <bb...@gmail.com>
> wrote:
> > > >
> > > > > Mike,
> > > > >
> > > > > All record-oriented components are extensions and thus can make use
> > of
> > > > > versioned components (i.e. nothing related to records is baked into
> > > > > the core framework of NiFi).
> > > > >
> > > > > The record API is essentially the module named
> > > > > nifi-record-serialization-services-api which at runtime is included
> > in
> > > > > nifi-standard-services-api-nar. Any record oriented processors you
> > > > > build are dependent on a specific version of
> > > > > nifi-standard-services-api-nar since that is where the Java
> interface
> > > > > will be that your processor was compiled against.
> > > > >
> > > > > Right now, even without extension registry, you could deploy
> multiple
> > > > > versions of nifi-standard-services-api-nar to a NiFi instance, and
> > > > > then deploy multiple versions of your record processing NAR that
> > > > > correspond to the versions of your API NAR.
> > > > >
> > > > > Going forward with extension registry, we probably want to consider
> > > > > breaking up nifi-standard-services-api-nar into smaller API NARs,
> as
> > > > > well as nifi-standard-bundle into smaller processor bundles. For
> > > > > example, there could be a record API NAR and a record processors
> NAR,
> > > > > that would both be split out of from the standard NARs.
> > > > >
> > > > > -Bryan
> > > > >
> > > > >
> > > > > On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen <
> mikerthomsen@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Great news about the extension registry! As we get closer to that
> > > being
> > > > > > ready, I'd like to add in some discussion about versioning the
> > > record API
> > > > > > separately. A lot of the custom processors we build now are
> Record
> > > > > > API-based, and it would be great to be able to decouple them from
> > one
> > > > > > specific release of NiFi.
> > > > > >
> > > > > > On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <bb...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hi Pierre,
> > > > > > >
> > > > > > > I think we are definitely close to an 0.4.0 release. A major
> > chunk
> > > of
> > > > > > > extension registry work has already landed in master, and I
> still
> > > have
> > > > > one
> > > > > > > other jira that I almost have ready and wanted to include,
> > > NIFIREG-233
> > > > > for
> > > > > > > generating extensions docs. Plus we probably need to make a few
> > > > > > > additions/updates to the admin guide.
> > > > > > >
> > > > > > > -Bryan
> > > > > > >
> > > > > > > On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard <
> > > > > > > pierre.villard.fr@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > All,
> > > > > > > >
> > > > > > > > Some really nice features have been included since the NiFi
> > > Registry
> > > > > > > 0.3.0
> > > > > > > > release and I'm wondering if it'd be a good time to consider
> a
> > > 0.4.0
> > > > > > > > release.
> > > > > > > >
> > > > > > > > There is a JIRA tagged for 0.4.0 and there are 2 opened pull
> > > > > requests,
> > > > > > > plus
> > > > > > > > interesting discussions / JIRAS on the mailing lists.
> > > > > > > >
> > > > > > > > Are there any reasons to hold off on a 0.4.0 release?  Are
> > there
> > > > > > > particular
> > > > > > > > JIRAs that the community considers necessary to have as part
> of
> > > the
> > > > > > > > release?
> > > > > > > >
> > > > > > > > If not, happy to give it a try at performing RM duties!
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Pierre
> > > > > > > >
> > > > > > > --
> > > > > > > Sent from Gmail Mobile
> > > > > > >
> > > > >
> > >
> >
>

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

Posted by Bryan Rosander <br...@gmail.com>.

Hey Pierre,

You can version the metadata db in the root dir of the git repo with event
hooks.

With an h2 jdbc url of
'jdbc:h2:$NIFI_REGISTRY_HOME/database/nifi-registry-primary;AUTOCOMMIT=OFF;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=TRUE'

You can connect to the db in a separate process, we're using that with a
shell event hook to dump the db to sql script when mutations happen (now
tenant events too :) ), storing that in the git repo using
org.h2.tools.Script, and committing it.  Then we can restore from that sql
script on startup before calling the stock startup script.

Side benefit is you can see metadata state changes in the git history and
manually inspect the sql.


On Wed, Mar 27, 2019 at 1:57 PM Pierre Villard <pi...@gmail.com>
wrote:

> Bryan,
>
> Thanks for the details. On my side, the main feature I (and others I
> suppose) am looking for is the ability to rebuild the metadata DB from the
> Git repo. But to be honest, this is not a blocker at all: I can live with a
> custom Docker image containing this feature.
>
> So, from my point of view, I don't have any issue with #2 but maybe #1
> would also give access to experimental features and have interesting
> feedback from the community. But we would, indeed, need to make it crystal
> clear that the APIs could change in ulterior release(s).
>
> Pierre
>
>
>
> Le mer. 27 mars 2019 à 18:11, Bryan Bende <bb...@gmail.com> a écrit :
>
> > Ryan,
> >
> > The short answer to your question is no, there isn't actually anything
> > new in the UI. The registry UI is setup in a generic way to list
> > "versioned items", so versioned extension bundles will just
> > automatically show up in the list with versioned flows.
> >
> > Let me put together a wiki page that outlines all of the moving parts
> > for the extension registry work and where we are currently at. I'll
> > send the link on this thread when I have something.
> >
> >
> > For Pierre and others,
> >
> > I was thinking more about this discussion of releasing 0.4.0, and I
> > think there are two of approaches we could take...
> >
> > 1) If there is a desire for the community to release 0.4.0 sooner
> > rather than later, then we can mark all of the extension bundle
> > related APIs as experimental, which would then give us the freedom to
> > continue evolving them until we have the full extension registry
> > experience, and likely mark them as stable in 0.5.0, or whichever
> > future release we decide. The reason for this is that there will
> > likely be API changes needed while building out the NiFi side of
> > things, and I don't want us to get stuck in a spot where we can't
> > change some API since it has been released, and so far the NiFi side
> > of the work hasn't been started yet.
> >
> > 2) Wait until more of the extension registry work is finalized and use
> > that as the driver of when to release 0.4.0, with the obvious downside
> > that it may take a bit longer to get a release out which then holds up
> > anything else that is not related to extension registry.
> >
> > So I guess we just have to decide the driving factor for releasing
> > 0.4.0... If its the non-extension related features/fixes, then #1 is
> > probably the best approach. If its the extension related stuff, then
> > I'd vote for #2 since it isn't really ready yet.
> >
> > -Bryan
> >
> > On Wed, Mar 27, 2019 at 12:01 PM Ryan Hendrickson
> > <ry...@gmail.com> wrote:
> > >
> > > Bryan,
> > >    Is there a new GUI part of the registry for the extension registry
> > work
> > > in master?  I compiled 0.4.0-SNAPSHOT and didn't see anything
> > immediately.
> > > You mentioned docs and sys admin stuff... Could you point me in the
> > > quick-start path if I wanted to try to kick-the-tires a little on this?
> > >
> > > Thanks,
> > > Ryan
> > >
> > > On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende <bb...@gmail.com> wrote:
> > >
> > > > Mike,
> > > >
> > > > All record-oriented components are extensions and thus can make use
> of
> > > > versioned components (i.e. nothing related to records is baked into
> > > > the core framework of NiFi).
> > > >
> > > > The record API is essentially the module named
> > > > nifi-record-serialization-services-api which at runtime is included
> in
> > > > nifi-standard-services-api-nar. Any record oriented processors you
> > > > build are dependent on a specific version of
> > > > nifi-standard-services-api-nar since that is where the Java interface
> > > > will be that your processor was compiled against.
> > > >
> > > > Right now, even without extension registry, you could deploy multiple
> > > > versions of nifi-standard-services-api-nar to a NiFi instance, and
> > > > then deploy multiple versions of your record processing NAR that
> > > > correspond to the versions of your API NAR.
> > > >
> > > > Going forward with extension registry, we probably want to consider
> > > > breaking up nifi-standard-services-api-nar into smaller API NARs, as
> > > > well as nifi-standard-bundle into smaller processor bundles. For
> > > > example, there could be a record API NAR and a record processors NAR,
> > > > that would both be split out of from the standard NARs.
> > > >
> > > > -Bryan
> > > >
> > > >
> > > > On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen <mikerthomsen@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > Great news about the extension registry! As we get closer to that
> > being
> > > > > ready, I'd like to add in some discussion about versioning the
> > record API
> > > > > separately. A lot of the custom processors we build now are Record
> > > > > API-based, and it would be great to be able to decouple them from
> one
> > > > > specific release of NiFi.
> > > > >
> > > > > On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <bb...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi Pierre,
> > > > > >
> > > > > > I think we are definitely close to an 0.4.0 release. A major
> chunk
> > of
> > > > > > extension registry work has already landed in master, and I still
> > have
> > > > one
> > > > > > other jira that I almost have ready and wanted to include,
> > NIFIREG-233
> > > > for
> > > > > > generating extensions docs. Plus we probably need to make a few
> > > > > > additions/updates to the admin guide.
> > > > > >
> > > > > > -Bryan
> > > > > >
> > > > > > On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard <
> > > > > > pierre.villard.fr@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > All,
> > > > > > >
> > > > > > > Some really nice features have been included since the NiFi
> > Registry
> > > > > > 0.3.0
> > > > > > > release and I'm wondering if it'd be a good time to consider a
> > 0.4.0
> > > > > > > release.
> > > > > > >
> > > > > > > There is a JIRA tagged for 0.4.0 and there are 2 opened pull
> > > > requests,
> > > > > > plus
> > > > > > > interesting discussions / JIRAS on the mailing lists.
> > > > > > >
> > > > > > > Are there any reasons to hold off on a 0.4.0 release?  Are
> there
> > > > > > particular
> > > > > > > JIRAs that the community considers necessary to have as part of
> > the
> > > > > > > release?
> > > > > > >
> > > > > > > If not, happy to give it a try at performing RM duties!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Pierre
> > > > > > >
> > > > > > --
> > > > > > Sent from Gmail Mobile
> > > > > >
> > > >
> >
>

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

Posted by Pierre Villard <pi...@gmail.com>.

Bryan,

Thanks for the details. On my side, the main feature I (and others I
suppose) am looking for is the ability to rebuild the metadata DB from the
Git repo. But to be honest, this is not a blocker at all: I can live with a
custom Docker image containing this feature.

So, from my point of view, I don't have any issue with #2 but maybe #1
would also give access to experimental features and have interesting
feedback from the community. But we would, indeed, need to make it crystal
clear that the APIs could change in ulterior release(s).

Pierre



Le mer. 27 mars 2019 à 18:11, Bryan Bende <bb...@gmail.com> a écrit :

> Ryan,
>
> The short answer to your question is no, there isn't actually anything
> new in the UI. The registry UI is setup in a generic way to list
> "versioned items", so versioned extension bundles will just
> automatically show up in the list with versioned flows.
>
> Let me put together a wiki page that outlines all of the moving parts
> for the extension registry work and where we are currently at. I'll
> send the link on this thread when I have something.
>
>
> For Pierre and others,
>
> I was thinking more about this discussion of releasing 0.4.0, and I
> think there are two of approaches we could take...
>
> 1) If there is a desire for the community to release 0.4.0 sooner
> rather than later, then we can mark all of the extension bundle
> related APIs as experimental, which would then give us the freedom to
> continue evolving them until we have the full extension registry
> experience, and likely mark them as stable in 0.5.0, or whichever
> future release we decide. The reason for this is that there will
> likely be API changes needed while building out the NiFi side of
> things, and I don't want us to get stuck in a spot where we can't
> change some API since it has been released, and so far the NiFi side
> of the work hasn't been started yet.
>
> 2) Wait until more of the extension registry work is finalized and use
> that as the driver of when to release 0.4.0, with the obvious downside
> that it may take a bit longer to get a release out which then holds up
> anything else that is not related to extension registry.
>
> So I guess we just have to decide the driving factor for releasing
> 0.4.0... If its the non-extension related features/fixes, then #1 is
> probably the best approach. If its the extension related stuff, then
> I'd vote for #2 since it isn't really ready yet.
>
> -Bryan
>
> On Wed, Mar 27, 2019 at 12:01 PM Ryan Hendrickson
> <ry...@gmail.com> wrote:
> >
> > Bryan,
> >    Is there a new GUI part of the registry for the extension registry
> work
> > in master?  I compiled 0.4.0-SNAPSHOT and didn't see anything
> immediately.
> > You mentioned docs and sys admin stuff... Could you point me in the
> > quick-start path if I wanted to try to kick-the-tires a little on this?
> >
> > Thanks,
> > Ryan
> >
> > On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende <bb...@gmail.com> wrote:
> >
> > > Mike,
> > >
> > > All record-oriented components are extensions and thus can make use of
> > > versioned components (i.e. nothing related to records is baked into
> > > the core framework of NiFi).
> > >
> > > The record API is essentially the module named
> > > nifi-record-serialization-services-api which at runtime is included in
> > > nifi-standard-services-api-nar. Any record oriented processors you
> > > build are dependent on a specific version of
> > > nifi-standard-services-api-nar since that is where the Java interface
> > > will be that your processor was compiled against.
> > >
> > > Right now, even without extension registry, you could deploy multiple
> > > versions of nifi-standard-services-api-nar to a NiFi instance, and
> > > then deploy multiple versions of your record processing NAR that
> > > correspond to the versions of your API NAR.
> > >
> > > Going forward with extension registry, we probably want to consider
> > > breaking up nifi-standard-services-api-nar into smaller API NARs, as
> > > well as nifi-standard-bundle into smaller processor bundles. For
> > > example, there could be a record API NAR and a record processors NAR,
> > > that would both be split out of from the standard NARs.
> > >
> > > -Bryan
> > >
> > >
> > > On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen <mi...@gmail.com>
> > > wrote:
> > > >
> > > > Great news about the extension registry! As we get closer to that
> being
> > > > ready, I'd like to add in some discussion about versioning the
> record API
> > > > separately. A lot of the custom processors we build now are Record
> > > > API-based, and it would be great to be able to decouple them from one
> > > > specific release of NiFi.
> > > >
> > > > On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <bb...@gmail.com>
> wrote:
> > > >
> > > > > Hi Pierre,
> > > > >
> > > > > I think we are definitely close to an 0.4.0 release. A major chunk
> of
> > > > > extension registry work has already landed in master, and I still
> have
> > > one
> > > > > other jira that I almost have ready and wanted to include,
> NIFIREG-233
> > > for
> > > > > generating extensions docs. Plus we probably need to make a few
> > > > > additions/updates to the admin guide.
> > > > >
> > > > > -Bryan
> > > > >
> > > > > On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard <
> > > > > pierre.villard.fr@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > All,
> > > > > >
> > > > > > Some really nice features have been included since the NiFi
> Registry
> > > > > 0.3.0
> > > > > > release and I'm wondering if it'd be a good time to consider a
> 0.4.0
> > > > > > release.
> > > > > >
> > > > > > There is a JIRA tagged for 0.4.0 and there are 2 opened pull
> > > requests,
> > > > > plus
> > > > > > interesting discussions / JIRAS on the mailing lists.
> > > > > >
> > > > > > Are there any reasons to hold off on a 0.4.0 release?  Are there
> > > > > particular
> > > > > > JIRAs that the community considers necessary to have as part of
> the
> > > > > > release?
> > > > > >
> > > > > > If not, happy to give it a try at performing RM duties!
> > > > > >
> > > > > > Thanks,
> > > > > > Pierre
> > > > > >
> > > > > --
> > > > > Sent from Gmail Mobile
> > > > >
> > >
>

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

Posted by Bryan Bende <bb...@gmail.com>.

Ryan,

The short answer to your question is no, there isn't actually anything
new in the UI. The registry UI is setup in a generic way to list
"versioned items", so versioned extension bundles will just
automatically show up in the list with versioned flows.

Let me put together a wiki page that outlines all of the moving parts
for the extension registry work and where we are currently at. I'll
send the link on this thread when I have something.

For Pierre and others,

I was thinking more about this discussion of releasing 0.4.0, and I
think there are two of approaches we could take...

1) If there is a desire for the community to release 0.4.0 sooner
rather than later, then we can mark all of the extension bundle
related APIs as experimental, which would then give us the freedom to
continue evolving them until we have the full extension registry
experience, and likely mark them as stable in 0.5.0, or whichever
future release we decide. The reason for this is that there will
likely be API changes needed while building out the NiFi side of
things, and I don't want us to get stuck in a spot where we can't
change some API since it has been released, and so far the NiFi side
of the work hasn't been started yet.

2) Wait until more of the extension registry work is finalized and use
that as the driver of when to release 0.4.0, with the obvious downside
that it may take a bit longer to get a release out which then holds up
anything else that is not related to extension registry.

So I guess we just have to decide the driving factor for releasing
0.4.0... If its the non-extension related features/fixes, then #1 is
probably the best approach. If its the extension related stuff, then
I'd vote for #2 since it isn't really ready yet.

-Bryan

On Wed, Mar 27, 2019 at 12:01 PM Ryan Hendrickson
<ry...@gmail.com> wrote:
>
> Bryan,
>    Is there a new GUI part of the registry for the extension registry work
> in master?  I compiled 0.4.0-SNAPSHOT and didn't see anything immediately.
> You mentioned docs and sys admin stuff... Could you point me in the
> quick-start path if I wanted to try to kick-the-tires a little on this?
>
> Thanks,
> Ryan
>
> On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende <bb...@gmail.com> wrote:
>
> > Mike,
> >
> > All record-oriented components are extensions and thus can make use of
> > versioned components (i.e. nothing related to records is baked into
> > the core framework of NiFi).
> >
> > The record API is essentially the module named
> > nifi-record-serialization-services-api which at runtime is included in
> > nifi-standard-services-api-nar. Any record oriented processors you
> > build are dependent on a specific version of
> > nifi-standard-services-api-nar since that is where the Java interface
> > will be that your processor was compiled against.
> >
> > Right now, even without extension registry, you could deploy multiple
> > versions of nifi-standard-services-api-nar to a NiFi instance, and
> > then deploy multiple versions of your record processing NAR that
> > correspond to the versions of your API NAR.
> >
> > Going forward with extension registry, we probably want to consider
> > breaking up nifi-standard-services-api-nar into smaller API NARs, as
> > well as nifi-standard-bundle into smaller processor bundles. For
> > example, there could be a record API NAR and a record processors NAR,
> > that would both be split out of from the standard NARs.
> >
> > -Bryan
> >
> >
> > On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen <mi...@gmail.com>
> > wrote:
> > >
> > > Great news about the extension registry! As we get closer to that being
> > > ready, I'd like to add in some discussion about versioning the record API
> > > separately. A lot of the custom processors we build now are Record
> > > API-based, and it would be great to be able to decouple them from one
> > > specific release of NiFi.
> > >
> > > On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <bb...@gmail.com> wrote:
> > >
> > > > Hi Pierre,
> > > >
> > > > I think we are definitely close to an 0.4.0 release. A major chunk of
> > > > extension registry work has already landed in master, and I still have
> > one
> > > > other jira that I almost have ready and wanted to include, NIFIREG-233
> > for
> > > > generating extensions docs. Plus we probably need to make a few
> > > > additions/updates to the admin guide.
> > > >
> > > > -Bryan
> > > >
> > > > On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard <
> > > > pierre.villard.fr@gmail.com>
> > > > wrote:
> > > >
> > > > > All,
> > > > >
> > > > > Some really nice features have been included since the NiFi Registry
> > > > 0.3.0
> > > > > release and I'm wondering if it'd be a good time to consider a 0.4.0
> > > > > release.
> > > > >
> > > > > There is a JIRA tagged for 0.4.0 and there are 2 opened pull
> > requests,
> > > > plus
> > > > > interesting discussions / JIRAS on the mailing lists.
> > > > >
> > > > > Are there any reasons to hold off on a 0.4.0 release?  Are there
> > > > particular
> > > > > JIRAs that the community considers necessary to have as part of the
> > > > > release?
> > > > >
> > > > > If not, happy to give it a try at performing RM duties!
> > > > >
> > > > > Thanks,
> > > > > Pierre
> > > > >
> > > > --
> > > > Sent from Gmail Mobile
> > > >
> >

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

Posted by Ryan Hendrickson <ry...@gmail.com>.

Bryan,
   Is there a new GUI part of the registry for the extension registry work
in master?  I compiled 0.4.0-SNAPSHOT and didn't see anything immediately.
You mentioned docs and sys admin stuff... Could you point me in the
quick-start path if I wanted to try to kick-the-tires a little on this?

Thanks,
Ryan

On Tue, Mar 26, 2019 at 9:57 AM Bryan Bende <bb...@gmail.com> wrote:

> Mike,
>
> All record-oriented components are extensions and thus can make use of
> versioned components (i.e. nothing related to records is baked into
> the core framework of NiFi).
>
> The record API is essentially the module named
> nifi-record-serialization-services-api which at runtime is included in
> nifi-standard-services-api-nar. Any record oriented processors you
> build are dependent on a specific version of
> nifi-standard-services-api-nar since that is where the Java interface
> will be that your processor was compiled against.
>
> Right now, even without extension registry, you could deploy multiple
> versions of nifi-standard-services-api-nar to a NiFi instance, and
> then deploy multiple versions of your record processing NAR that
> correspond to the versions of your API NAR.
>
> Going forward with extension registry, we probably want to consider
> breaking up nifi-standard-services-api-nar into smaller API NARs, as
> well as nifi-standard-bundle into smaller processor bundles. For
> example, there could be a record API NAR and a record processors NAR,
> that would both be split out of from the standard NARs.
>
> -Bryan
>
>
> On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen <mi...@gmail.com>
> wrote:
> >
> > Great news about the extension registry! As we get closer to that being
> > ready, I'd like to add in some discussion about versioning the record API
> > separately. A lot of the custom processors we build now are Record
> > API-based, and it would be great to be able to decouple them from one
> > specific release of NiFi.
> >
> > On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <bb...@gmail.com> wrote:
> >
> > > Hi Pierre,
> > >
> > > I think we are definitely close to an 0.4.0 release. A major chunk of
> > > extension registry work has already landed in master, and I still have
> one
> > > other jira that I almost have ready and wanted to include, NIFIREG-233
> for
> > > generating extensions docs. Plus we probably need to make a few
> > > additions/updates to the admin guide.
> > >
> > > -Bryan
> > >
> > > On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard <
> > > pierre.villard.fr@gmail.com>
> > > wrote:
> > >
> > > > All,
> > > >
> > > > Some really nice features have been included since the NiFi Registry
> > > 0.3.0
> > > > release and I'm wondering if it'd be a good time to consider a 0.4.0
> > > > release.
> > > >
> > > > There is a JIRA tagged for 0.4.0 and there are 2 opened pull
> requests,
> > > plus
> > > > interesting discussions / JIRAS on the mailing lists.
> > > >
> > > > Are there any reasons to hold off on a 0.4.0 release?  Are there
> > > particular
> > > > JIRAs that the community considers necessary to have as part of the
> > > > release?
> > > >
> > > > If not, happy to give it a try at performing RM duties!
> > > >
> > > > Thanks,
> > > > Pierre
> > > >
> > > --
> > > Sent from Gmail Mobile
> > >
>

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

Posted by Bryan Bende <bb...@gmail.com>.

Mike,

All record-oriented components are extensions and thus can make use of
versioned components (i.e. nothing related to records is baked into
the core framework of NiFi).

The record API is essentially the module named
nifi-record-serialization-services-api which at runtime is included in
nifi-standard-services-api-nar. Any record oriented processors you
build are dependent on a specific version of
nifi-standard-services-api-nar since that is where the Java interface
will be that your processor was compiled against.

Right now, even without extension registry, you could deploy multiple
versions of nifi-standard-services-api-nar to a NiFi instance, and
then deploy multiple versions of your record processing NAR that
correspond to the versions of your API NAR.

Going forward with extension registry, we probably want to consider
breaking up nifi-standard-services-api-nar into smaller API NARs, as
well as nifi-standard-bundle into smaller processor bundles. For
example, there could be a record API NAR and a record processors NAR,
that would both be split out of from the standard NARs.

-Bryan

On Tue, Mar 26, 2019 at 7:55 AM Mike Thomsen <mi...@gmail.com> wrote:
>
> Great news about the extension registry! As we get closer to that being
> ready, I'd like to add in some discussion about versioning the record API
> separately. A lot of the custom processors we build now are Record
> API-based, and it would be great to be able to decouple them from one
> specific release of NiFi.
>
> On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <bb...@gmail.com> wrote:
>
> > Hi Pierre,
> >
> > I think we are definitely close to an 0.4.0 release. A major chunk of
> > extension registry work has already landed in master, and I still have one
> > other jira that I almost have ready and wanted to include, NIFIREG-233 for
> > generating extensions docs. Plus we probably need to make a few
> > additions/updates to the admin guide.
> >
> > -Bryan
> >
> > On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard <
> > pierre.villard.fr@gmail.com>
> > wrote:
> >
> > > All,
> > >
> > > Some really nice features have been included since the NiFi Registry
> > 0.3.0
> > > release and I'm wondering if it'd be a good time to consider a 0.4.0
> > > release.
> > >
> > > There is a JIRA tagged for 0.4.0 and there are 2 opened pull requests,
> > plus
> > > interesting discussions / JIRAS on the mailing lists.
> > >
> > > Are there any reasons to hold off on a 0.4.0 release?  Are there
> > particular
> > > JIRAs that the community considers necessary to have as part of the
> > > release?
> > >
> > > If not, happy to give it a try at performing RM duties!
> > >
> > > Thanks,
> > > Pierre
> > >
> > --
> > Sent from Gmail Mobile
> >

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

Posted by Mike Thomsen <mi...@gmail.com>.

Great news about the extension registry! As we get closer to that being
ready, I'd like to add in some discussion about versioning the record API
separately. A lot of the custom processors we build now are Record
API-based, and it would be great to be able to decouple them from one
specific release of NiFi.

On Mon, Mar 25, 2019 at 2:14 PM Bryan Bende <bb...@gmail.com> wrote:

> Hi Pierre,
>
> I think we are definitely close to an 0.4.0 release. A major chunk of
> extension registry work has already landed in master, and I still have one
> other jira that I almost have ready and wanted to include, NIFIREG-233 for
> generating extensions docs. Plus we probably need to make a few
> additions/updates to the admin guide.
>
> -Bryan
>
> On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard <
> pierre.villard.fr@gmail.com>
> wrote:
>
> > All,
> >
> > Some really nice features have been included since the NiFi Registry
> 0.3.0
> > release and I'm wondering if it'd be a good time to consider a 0.4.0
> > release.
> >
> > There is a JIRA tagged for 0.4.0 and there are 2 opened pull requests,
> plus
> > interesting discussions / JIRAS on the mailing lists.
> >
> > Are there any reasons to hold off on a 0.4.0 release?  Are there
> particular
> > JIRAs that the community considers necessary to have as part of the
> > release?
> >
> > If not, happy to give it a try at performing RM duties!
> >
> > Thanks,
> > Pierre
> >
> --
> Sent from Gmail Mobile
>

Re: [DISCUSS] Closing in on a NiFi Registry 0.4.0 release?

Posted by Bryan Bende <bb...@gmail.com>.

Hi Pierre,

I think we are definitely close to an 0.4.0 release. A major chunk of
extension registry work has already landed in master, and I still have one
other jira that I almost have ready and wanted to include, NIFIREG-233 for
generating extensions docs. Plus we probably need to make a few
additions/updates to the admin guide.

-Bryan

On Mon, Mar 25, 2019 at 1:33 PM Pierre Villard <pi...@gmail.com>
wrote:

> All,
>
> Some really nice features have been included since the NiFi Registry 0.3.0
> release and I'm wondering if it'd be a good time to consider a 0.4.0
> release.
>
> There is a JIRA tagged for 0.4.0 and there are 2 opened pull requests, plus
> interesting discussions / JIRAS on the mailing lists.
>
> Are there any reasons to hold off on a 0.4.0 release?  Are there particular
> JIRAs that the community considers necessary to have as part of the
> release?
>
> If not, happy to give it a try at performing RM duties!
>
> Thanks,
> Pierre
>
-- 
Sent from Gmail Mobile