You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Chad Woodhead <ch...@gmail.com> on 2019/03/19 15:16:43 UTC
Re: Problems with NiFi Registry Conflicts after Processor Upgrades
Hey guys,
I'm going to be upgrading my NiFi clusters to 1.9.1 soon so this topic was
important to me. I've done some testing and wanted to share my results.
*Test environment setup*:
2 NiFi Clusters
1 NiFi Registry
Dev NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0
Cert NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0
*Working upgrade method*:
1. Upgrade Dev NiFi from 1.8.0 to 1.9.1
2. All Dev NiFi versioned flows showed local changes due to new properties
in processors
3. Commit all Dev NiFi changes for all versioned process groups
4. In NiFi Registry, view and see all committed changes from Dev NiFi
5. On Cert NiFi, all versioned PGs show newer version available.
6. On Cert NiFi, change all versioned process groups to use new version.
PGs now show "Flow version is current" (Note: the processors DO NOT become
invalid and they do not show the new properties yet)
7. Upgrade Cert NiFi from 1.8.0 to 1.9.1
8. On Cert NiFi, all versioned process groups still show they are on the
latest version and the processors DO show the new properties now
*Not working upgrade method*:
1. Upgrade Dev NiFi from 1.8.0 to 1.9.1
2. All Dev NiFi versioned flows showed local changes due to new properties
in processors
3. Commit all Dev NiFi changes to all versioned process groups
4. In NiFi Registry, view and see all committed changes from Dev NiFi
5. Upgrade Cert NiFi from 1.8.0 to 1.9.1
6. On Cert NiFi, versioned process groups show "Local changes have been
made and a newer version of this flow is available". When clicking on PG
and selecting "Version", only options are "Show local changes", "Revert
local changes", and "Stop version control".
"Show local changes" allows you to see what has changed (all new properties
in processors).
"Revert local changes" does nothing as these changes are required since
they are new properties from the upgrade.
7. My only 2 options at this point are
-Stop version control of the PG and start it back up, causing me to lose
all history
-Delete the PG on Cert and then re-import from NiFi Registry. This option
really isn't an option when in Prod as I don't want to stop/delete
production flows that need to keep ingesting data.
So as shown above, when upgrading NiFi with versioned flows in NiFi
Registry, the steps are very important and as long pull in the latest
commit to Cert before upgrading Cert, your process groups will work as
expected.
Thanks,
Chad
>
>
> On 11/29/18, 3:36 PM, "Peter Wicks (pwicks)" <pw...@micron.com> wrote:
>
> *External Message* - Use caution before opening links or attachments
>
> Bryan,
>
> I agree, that is probably a solution. Unfortunately, there is no mass
> upgrade option, so we'd have to manually touch ever versioned process group
> (or scripted it).
>
> Thanks,
> Peter
>
> -----Original Message-----
> From: Bryan Bende [mailto:bbende@gmail.com]
> Sent: Thursday, November 29, 2018 1:29 PM
> To: users@nifi.apache.org
> Subject: [EXT] Re: Problems with NiFi Registry Conflicts after
> Processor Upgrades
>
> Peter,
>
> I feel like this came up before, and unfortunately I'm not sure there
> is currently a solution.
>
> I think ultimately there needs to be some kind of force upgrade so you
> can ignore the local changes and take whatever is available.
>
> The only thing I can think of, but haven't tried, is if you had
> upgraded the PG in the second instance before upgrading NiFi itself, it
> would bring in the new properties that are not valid in that version and
> the processor would show as invalid, then upgrade NiFi and it would be
> valid again.
>
> -Bryan
> On Thu, Nov 29, 2018 at 3:13 PM Peter Wicks (pwicks) <
> pwicks@micron.com> wrote:
> >
> > Ran into a NiFi Registry issue while upgrading our instances to NiFi
> 1.8.0. ExecuteSQL had a number of new properties added to it in 1.8.0, so
> after upgrading our, our versioned processor groups show as having local
> changes, which is good. We went ahead and checked the changes into the
> registry.
> >
> >
> >
> > Enter the second instance... we upgraded a second instance. It also
> see's local changes, but now the processor group is in conflict, because we
> have local (identical) changes, and we have a newer version checked in. If
> you try to revert the local changes so you can sync things up... you can't,
> because these are properties on the Processor, and the default values
> automatically come back. So our second processor group is in conflict and
> we haven't found a way to bring it back in sync without deleting it and re
> loading it from the registry. Help would be appreciated.
> >
> >
> >
> > Thanks,
> >
> > Peter
>
>
>
Re: Problems with NiFi Registry Conflicts after Processor Upgrades
Posted by Kevin Doran <kd...@apache.org>.
Chad,
I wanted to echo Bryan and say thanks for sharing the details of an
upgrade process that works. Until we have improved NiFi to handle the
upgrades regardless of order of steps, this is a helpful reference for
the community for a sequence that can work.
Cheers,
Kevin
On Tue, Mar 19, 2019 at 11:40 AM Bryan Bende <bb...@gmail.com> wrote:
>
> Chad,
>
> Thanks for reporting your experience and glad to hear that there is a
> good process to follow.
>
> We do want to make this situation better and there is a JIRA to track
> the issue [1].
>
> Thanks,
>
> Bryan
>
> [1] https://issues.apache.org/jira/browse/NIFI-6028
>
>
>
> On Tue, Mar 19, 2019 at 11:23 AM Chad Woodhead <ch...@gmail.com> wrote:
> >
> > Hey guys,
> >
> > I'm going to be upgrading my NiFi clusters to 1.9.1 soon so this topic was important to me. I've done some testing and wanted to share my results.
> >
> > Test environment setup:
> > 2 NiFi Clusters
> > 1 NiFi Registry
> >
> > Dev NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0
> > Cert NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0
> >
> > Working upgrade method:
> > 1. Upgrade Dev NiFi from 1.8.0 to 1.9.1
> > 2. All Dev NiFi versioned flows showed local changes due to new properties in processors
> > 3. Commit all Dev NiFi changes for all versioned process groups
> > 4. In NiFi Registry, view and see all committed changes from Dev NiFi
> > 5. On Cert NiFi, all versioned PGs show newer version available.
> > 6. On Cert NiFi, change all versioned process groups to use new version. PGs now show "Flow version is current" (Note: the processors DO NOT become invalid and they do not show the new properties yet)
> > 7. Upgrade Cert NiFi from 1.8.0 to 1.9.1
> > 8. On Cert NiFi, all versioned process groups still show they are on the latest version and the processors DO show the new properties now
> >
> > Not working upgrade method:
> > 1. Upgrade Dev NiFi from 1.8.0 to 1.9.1
> > 2. All Dev NiFi versioned flows showed local changes due to new properties in processors
> > 3. Commit all Dev NiFi changes to all versioned process groups
> > 4. In NiFi Registry, view and see all committed changes from Dev NiFi
> > 5. Upgrade Cert NiFi from 1.8.0 to 1.9.1
> > 6. On Cert NiFi, versioned process groups show "Local changes have been made and a newer version of this flow is available". When clicking on PG and selecting "Version", only options are "Show local changes", "Revert local changes", and "Stop version control".
> > "Show local changes" allows you to see what has changed (all new properties in processors).
> > "Revert local changes" does nothing as these changes are required since they are new properties from the upgrade.
> > 7. My only 2 options at this point are
> > -Stop version control of the PG and start it back up, causing me to lose all history
> > -Delete the PG on Cert and then re-import from NiFi Registry. This option really isn't an option when in Prod as I don't want to stop/delete production flows that need to keep ingesting data.
> >
> > So as shown above, when upgrading NiFi with versioned flows in NiFi Registry, the steps are very important and as long pull in the latest commit to Cert before upgrading Cert, your process groups will work as expected.
> >
> > Thanks,
> > Chad
> >
> >>
> >>
> >>
> >> On 11/29/18, 3:36 PM, "Peter Wicks (pwicks)" <pw...@micron.com> wrote:
> >>
> >> *External Message* - Use caution before opening links or attachments
> >>
> >> Bryan,
> >>
> >> I agree, that is probably a solution. Unfortunately, there is no mass upgrade option, so we'd have to manually touch ever versioned process group (or scripted it).
> >>
> >> Thanks,
> >> Peter
> >>
> >> -----Original Message-----
> >> From: Bryan Bende [mailto:bbende@gmail.com]
> >> Sent: Thursday, November 29, 2018 1:29 PM
> >> To: users@nifi.apache.org
> >> Subject: [EXT] Re: Problems with NiFi Registry Conflicts after Processor Upgrades
> >>
> >> Peter,
> >>
> >> I feel like this came up before, and unfortunately I'm not sure there is currently a solution.
> >>
> >> I think ultimately there needs to be some kind of force upgrade so you can ignore the local changes and take whatever is available.
> >>
> >> The only thing I can think of, but haven't tried, is if you had upgraded the PG in the second instance before upgrading NiFi itself, it would bring in the new properties that are not valid in that version and the processor would show as invalid, then upgrade NiFi and it would be valid again.
> >>
> >> -Bryan
> >> On Thu, Nov 29, 2018 at 3:13 PM Peter Wicks (pwicks) <pw...@micron.com> wrote:
> >> >
> >> > Ran into a NiFi Registry issue while upgrading our instances to NiFi 1.8.0. ExecuteSQL had a number of new properties added to it in 1.8.0, so after upgrading our, our versioned processor groups show as having local changes, which is good. We went ahead and checked the changes into the registry.
> >> >
> >> >
> >> >
> >> > Enter the second instance... we upgraded a second instance. It also see's local changes, but now the processor group is in conflict, because we have local (identical) changes, and we have a newer version checked in. If you try to revert the local changes so you can sync things up... you can't, because these are properties on the Processor, and the default values automatically come back. So our second processor group is in conflict and we haven't found a way to bring it back in sync without deleting it and re loading it from the registry. Help would be appreciated.
> >> >
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > Peter
> >>
> >>
Re: Problems with NiFi Registry Conflicts after Processor Upgrades
Posted by Bryan Bende <bb...@gmail.com>.
Chad,
Thanks for reporting your experience and glad to hear that there is a
good process to follow.
We do want to make this situation better and there is a JIRA to track
the issue [1].
Thanks,
Bryan
[1] https://issues.apache.org/jira/browse/NIFI-6028
On Tue, Mar 19, 2019 at 11:23 AM Chad Woodhead <ch...@gmail.com> wrote:
>
> Hey guys,
>
> I'm going to be upgrading my NiFi clusters to 1.9.1 soon so this topic was important to me. I've done some testing and wanted to share my results.
>
> Test environment setup:
> 2 NiFi Clusters
> 1 NiFi Registry
>
> Dev NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0
> Cert NiFi on 1.8.0 versioning flows with NiFi Registry 0.2.0
>
> Working upgrade method:
> 1. Upgrade Dev NiFi from 1.8.0 to 1.9.1
> 2. All Dev NiFi versioned flows showed local changes due to new properties in processors
> 3. Commit all Dev NiFi changes for all versioned process groups
> 4. In NiFi Registry, view and see all committed changes from Dev NiFi
> 5. On Cert NiFi, all versioned PGs show newer version available.
> 6. On Cert NiFi, change all versioned process groups to use new version. PGs now show "Flow version is current" (Note: the processors DO NOT become invalid and they do not show the new properties yet)
> 7. Upgrade Cert NiFi from 1.8.0 to 1.9.1
> 8. On Cert NiFi, all versioned process groups still show they are on the latest version and the processors DO show the new properties now
>
> Not working upgrade method:
> 1. Upgrade Dev NiFi from 1.8.0 to 1.9.1
> 2. All Dev NiFi versioned flows showed local changes due to new properties in processors
> 3. Commit all Dev NiFi changes to all versioned process groups
> 4. In NiFi Registry, view and see all committed changes from Dev NiFi
> 5. Upgrade Cert NiFi from 1.8.0 to 1.9.1
> 6. On Cert NiFi, versioned process groups show "Local changes have been made and a newer version of this flow is available". When clicking on PG and selecting "Version", only options are "Show local changes", "Revert local changes", and "Stop version control".
> "Show local changes" allows you to see what has changed (all new properties in processors).
> "Revert local changes" does nothing as these changes are required since they are new properties from the upgrade.
> 7. My only 2 options at this point are
> -Stop version control of the PG and start it back up, causing me to lose all history
> -Delete the PG on Cert and then re-import from NiFi Registry. This option really isn't an option when in Prod as I don't want to stop/delete production flows that need to keep ingesting data.
>
> So as shown above, when upgrading NiFi with versioned flows in NiFi Registry, the steps are very important and as long pull in the latest commit to Cert before upgrading Cert, your process groups will work as expected.
>
> Thanks,
> Chad
>
>>
>>
>>
>> On 11/29/18, 3:36 PM, "Peter Wicks (pwicks)" <pw...@micron.com> wrote:
>>
>> *External Message* - Use caution before opening links or attachments
>>
>> Bryan,
>>
>> I agree, that is probably a solution. Unfortunately, there is no mass upgrade option, so we'd have to manually touch ever versioned process group (or scripted it).
>>
>> Thanks,
>> Peter
>>
>> -----Original Message-----
>> From: Bryan Bende [mailto:bbende@gmail.com]
>> Sent: Thursday, November 29, 2018 1:29 PM
>> To: users@nifi.apache.org
>> Subject: [EXT] Re: Problems with NiFi Registry Conflicts after Processor Upgrades
>>
>> Peter,
>>
>> I feel like this came up before, and unfortunately I'm not sure there is currently a solution.
>>
>> I think ultimately there needs to be some kind of force upgrade so you can ignore the local changes and take whatever is available.
>>
>> The only thing I can think of, but haven't tried, is if you had upgraded the PG in the second instance before upgrading NiFi itself, it would bring in the new properties that are not valid in that version and the processor would show as invalid, then upgrade NiFi and it would be valid again.
>>
>> -Bryan
>> On Thu, Nov 29, 2018 at 3:13 PM Peter Wicks (pwicks) <pw...@micron.com> wrote:
>> >
>> > Ran into a NiFi Registry issue while upgrading our instances to NiFi 1.8.0. ExecuteSQL had a number of new properties added to it in 1.8.0, so after upgrading our, our versioned processor groups show as having local changes, which is good. We went ahead and checked the changes into the registry.
>> >
>> >
>> >
>> > Enter the second instance... we upgraded a second instance. It also see's local changes, but now the processor group is in conflict, because we have local (identical) changes, and we have a newer version checked in. If you try to revert the local changes so you can sync things up... you can't, because these are properties on the Processor, and the default values automatically come back. So our second processor group is in conflict and we haven't found a way to bring it back in sync without deleting it and re loading it from the registry. Help would be appreciated.
>> >
>> >
>> >
>> > Thanks,
>> >
>> > Peter
>>
>>