You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Simon Bence <si...@gmail.com> on 2022/12/01 12:42:28 UTC
Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows
Hi Josef,
I would like to inform you that there is an adjustment for the ticket I mentioned in my previous mail, which is needed for the proper behaviour of the function in every case. The change is in PR phase but I hope it will be merged soon: https://github.com/apache/nifi/pull/6741 <https://github.com/apache/nifi/pull/6741> I would not recommend to go forward without this additional change!
Regards,
Bence
> On 2022. Nov 29., at 16:21, Simon Bence <si...@gmail.com> wrote:
>
> Hi Josef,
>
> Your welcome! In your case it looks like the information is lost during transformations of the internal representation.
>
> For technical details, here you can take a look on the code changes: https://github.com/apache/nifi/commit/df2147829742129c039b37c5d6f4f11aa54785a2 <https://github.com/apache/nifi/commit/df2147829742129c039b37c5d6f4f11aa54785a2>
>
> Regards,
> Bence
>
>> On 2022. Nov 29., at 16:08, <Josef.Zahner1@swisscom.com <ma...@swisscom.com>> <Josef.Zahner1@swisscom.com <ma...@swisscom.com>> wrote:
>>
>> Hi Bence, Joe
>>
>> Thank you guys for your fast response.
>>
>> @Bence you are right, the whole issue seems to be related to nested flows which we try to sync. We are relying heavily on this as we are doing the integration on one NiFi system and then sync it to production via the NiFi Registry. Our parent flow contains multiple nested flows.
>>
>> Ok, as NIFI-10874 will be included in the next release we will skip the 1.19.0 and wait for 1.19.1 / 1.20.0. Sadly it will probably take more than a few days until we see the next release... Do you have an idea why the flow with the nested flow could lose the storageLocation? Because, we just change a small thing in one of the nested flows and suddenly one of the other nested flows have been losing it’s storageLocation in the parent flow and from this point the whole flow was broken. Now we are scared to change anything as it could brake anytime again.
>>
>> Cheers Josef
>>
>>
>>
>> From: Simon Bence <simonbence.dev@gmail.com <ma...@gmail.com>>
>> Reply to: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
>> Date: Tuesday, 29 November 2022 at 13:31
>> To: "users@nifi.apache.org <ma...@nifi.apache.org>" <users@nifi.apache.org <ma...@nifi.apache.org>>
>> Subject: Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows
>>
>> Hi Josef,
>>
>> Thanks for your patience!
>>
>> I took a deeper look on what you were writing. In general, this is a sign for the case where the registry client cannot find the nested flow. As far as I understood in your case this happens when you are having a sync. Based on this you may be hitting https://issues.apache.org/jira/browse/NIFI-10874 <https://issues.apache.org/jira/browse/NIFI-10874> which is not part of NiFi 1.19.0. The next release should contain the fix that addresses your issue.
>>
>> Regards,
>> Bence
>>
>>
>>> On 2022. Nov 28., at 17:48, Simon Bence <simonbence.dev@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Hi Josef,
>>>
>>> Thank you for raising the attention to this.
>>>
>>> The storageLocation is a new concept to generalise the information used to locate nested flows in a versioned flow. In general it can be there and does not cause issues, as of now NiFi falls back the previous way to locate nested flows. As Joe mentioned, there is a fix in 1.19, but I need to double check if it relates to this situation. Please give me some time, I will find you back.
>>>
>>> Regards,
>>> Bence
>>>
>>>
>>>> On 2022. Nov 28., at 17:17, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
>>>>
>>>> Josef
>>>>
>>>> Sorry for the challenges you've hit there. I do think in 1.18 we had a bug/regression as we refactored our registry client. That issue should be resolved in 1.19 which just went live thanks to https://issues.apache.org/jira/browse/NIFI-10787 <https://issues.apache.org/jira/browse/NIFI-10787>. However, I am not positive if this will solve the scenario you've hit now but please if able try it out.
>>>>
>>>> Thanks
>>>>
>>>> On Mon, Nov 28, 2022 at 8:50 AM <Josef.Zahner1@swisscom.com <ma...@swisscom.com>> wrote:
>>>> Hi guys
>>>>
>>>>
>>>>
>>>> We had the following bug twice already and it broke the whole sync with the NiFi registry for the Flow/PG. First time was directly after we have upgraded from NiFi 1.15.3 to 1.18.0, but we ignored it as we thought it could be because of the upgrade, however it occurred again after a few NiFi Registry commits on NiFi 1.18.0… The error was the following when we tried to change the flow version or when we tried to start from scratch with that version from the NiFi Registry, so the version was broken in the NiFi Registry:
>>>>
>>>>
>>>>
>>>> <image001.png>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> We investigated the last NiFi Registry Commit and we saw in our GIT repo (we sync the “flow_storage” to GIT) on line 4847 that NiFi removed a “storageLocation” from another PG which makes no sense at all. We changed nothing there and especially why should NiFi remove only the storageLocation line… We have one specialty, as we have nested NiFi Registry flows, so one of the flows where the storageLocation has been removed was such a nested flow.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> <image002.png>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Luckily we were able to resolve the error. We tried to add the line and commit it to GIT plus we dropped the the database to repopulate the DB, however it was broken again after a commit from NiFi. So we tried to manually create a new fake version on bucket.yml in the corresponding bucket folder and added as well the line in the snapshot again. We dropped then the DB and restarted NiFi Registry and voilà it was working again.
>>>>
>>>>
>>>>
>>>> However it was a nightmare to get it working again as the flow was completely broken, we couldn’t checkout the affected version at all.
>>>>
>>>>
>>>>
>>>> Any thoughts on this? Shall I fill a Jira Ticket? The problem is, we can’t really reproduce it, it looks like it happens randomly. As you could imagine, we can’t share our template as it contains a lot of confidential material.
>>>>
>>>>
>>>>
>>>> Cheers Josef
>>>>
>
Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows
Posted by Kevin Doran <kd...@apache.org>.
I believe as soon as this is merged Joe Witt is planning to prepare the
1.19.1 RC candidate and open the vote. We should have a release by next
week unless something comes up
On Dec 1, 2022 at 07:44:29, Josef.Zahner1@swisscom.com wrote:
> Hi Bence
>
>
>
> Ok got it, thanks a lot, So we will try to sync as less as possible. Let’s
> hope that we will see a NiFi 1.19.1 soon…
>
>
>
> Cheers Josef
>
>
>
> *From: *Simon Bence <si...@gmail.com>
> *Reply to: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Thursday, 1 December 2022 at 13:42
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: NiFi Registry Bug which brakes the flow sync with NiFi
> 1.18.0 (and same version of the registry) on nested flows
>
>
>
> Hi Josef,
>
>
>
> I would like to inform you that there is an adjustment for the ticket I
> mentioned in my previous mail, which is needed for the proper behaviour of
> the function in every case. The change is in PR phase but I hope it will be
> merged soon: https://github.com/apache/nifi/pull/6741 I would not
> recommend to go forward without this additional change!
>
>
>
> Regards,
>
> Bence
>
>
>
> On 2022. Nov 29., at 16:21, Simon Bence <si...@gmail.com> wrote:
>
>
>
> Hi Josef,
>
>
>
> Your welcome! In your case it looks like the information is lost during
> transformations of the internal representation.
>
>
>
> For technical details, here you can take a look on the code changes:
> https://github.com/apache/nifi/commit/df2147829742129c039b37c5d6f4f11aa54785a2
>
>
>
> Regards,
>
> Bence
>
>
>
> On 2022. Nov 29., at 16:08, <Jo...@swisscom.com> <
> Josef.Zahner1@swisscom.com> wrote:
>
>
>
> Hi Bence, Joe
>
>
>
> Thank you guys for your fast response.
>
>
>
> @Bence you are right, the whole issue seems to be related to nested flows
> which we try to sync. We are relying heavily on this as we are doing the
> integration on one NiFi system and then sync it to production via the NiFi
> Registry. Our parent flow contains multiple nested flows.
>
>
>
> Ok, as NIFI-10874 will be included in the next release we will skip the
> 1.19.0 and wait for 1.19.1 / 1.20.0. Sadly it will probably take more than
> a few days until we see the next release... Do you have an idea why the
> flow with the nested flow could lose the storageLocation? Because, we just
> change a small thing in one of the nested flows and suddenly one of the
> other nested flows have been losing it’s storageLocation in the parent flow
> and from this point the whole flow was broken. Now we are scared to change
> anything as it could brake anytime again.
>
>
>
> Cheers Josef
>
>
>
>
>
>
>
> *From: *Simon Bence <si...@gmail.com>
> *Reply to: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Tuesday, 29 November 2022 at 13:31
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: NiFi Registry Bug which brakes the flow sync with NiFi
> 1.18.0 (and same version of the registry) on nested flows
>
>
>
> Hi Josef,
>
>
>
> Thanks for your patience!
>
>
>
> I took a deeper look on what you were writing. In general, this is a sign
> for the case where the registry client cannot find the nested flow. As far
> as I understood in your case this happens when you are having a sync. Based
> on this you may be hitting
> https://issues.apache.org/jira/browse/NIFI-10874 which is not part of
> NiFi 1.19.0. The next release should contain the fix that addresses your
> issue.
>
>
>
> Regards,
>
> Bence
>
>
>
>
> On 2022. Nov 28., at 17:48, Simon Bence <si...@gmail.com> wrote:
>
> Hi Josef,
>
> Thank you for raising the attention to this.
>
> The storageLocation is a new concept to generalise the information used to
> locate nested flows in a versioned flow. In general it can be there and
> does not cause issues, as of now NiFi falls back the previous way to locate
> nested flows. As Joe mentioned, there is a fix in 1.19, but I need to
> double check if it relates to this situation. Please give me some time, I
> will find you back.
>
> Regards,
> Bence
>
>
>
> On 2022. Nov 28., at 17:17, Joe Witt <jo...@gmail.com> wrote:
>
> Josef
>
> Sorry for the challenges you've hit there. I do think in 1.18 we had a
> bug/regression as we refactored our registry client. That issue should be
> resolved in 1.19 which just went live thanks to
> https://issues.apache.org/jira/browse/NIFI-10787. However, I am not
> positive if this will solve the scenario you've hit now but please if able
> try it out.
>
> Thanks
>
> On Mon, Nov 28, 2022 at 8:50 AM <Jo...@swisscom.com> wrote:
> Hi guys
>
>
>
> We had the following bug twice already and it broke the whole sync with
> the NiFi registry for the Flow/PG. First time was directly after we have
> upgraded from NiFi 1.15.3 to 1.18.0, but we ignored it as we
> thought it could be because of the upgrade, however it occurred again after
> a few NiFi Registry commits on NiFi 1.18.0… The error was the following
> when we tried to change the flow version or when we tried to start from
> scratch with that version from the NiFi Registry, so the version was broken
> in the NiFi Registry:
>
>
>
> <image001.png>
>
>
>
>
>
> We investigated the last NiFi Registry Commit and we saw in our GIT repo
> (we sync the “flow_storage” to GIT) on line 4847 that NiFi removed a
> “storageLocation” from another PG which makes no sense at all. We changed
> nothing there and especially why should NiFi remove only the
> storageLocation line… We have one specialty, as we have nested NiFi
> Registry flows, so one of the flows where the storageLocation has
> been removed was such a nested flow.
>
>
>
>
>
> <image002.png>
>
>
>
>
>
> Luckily we were able to resolve the error. We tried to add the line and
> commit it to GIT plus we dropped the the database to repopulate the DB,
> however it was broken again after a commit from NiFi. So we tried to
> manually create a new fake version on bucket.yml in the corresponding
> bucket folder and added as well the line in the snapshot again. We dropped
> then the DB and restarted NiFi Registry and voilà it was working again.
>
>
>
> However it was a nightmare to get it working again as the flow was
> completely broken, we couldn’t checkout the affected version at all.
>
>
>
> Any thoughts on this? Shall I fill a Jira Ticket? The problem is, we can’t
> really reproduce it, it looks like it happens randomly. As you could
> imagine, we can’t share our template as it contains a lot of
> confidential material.
>
>
>
> Cheers Josef
>
>
>
>
>
Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows
Posted by Jo...@swisscom.com.
Hi Bence
Ok got it, thanks a lot, So we will try to sync as less as possible. Let’s hope that we will see a NiFi 1.19.1 soon…
Cheers Josef
From: Simon Bence <si...@gmail.com>
Reply to: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, 1 December 2022 at 13:42
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows
Hi Josef,
I would like to inform you that there is an adjustment for the ticket I mentioned in my previous mail, which is needed for the proper behaviour of the function in every case. The change is in PR phase but I hope it will be merged soon: https://github.com/apache/nifi/pull/6741 I would not recommend to go forward without this additional change!
Regards,
Bence
On 2022. Nov 29., at 16:21, Simon Bence <si...@gmail.com>> wrote:
Hi Josef,
Your welcome! In your case it looks like the information is lost during transformations of the internal representation.
For technical details, here you can take a look on the code changes: https://github.com/apache/nifi/commit/df2147829742129c039b37c5d6f4f11aa54785a2
Regards,
Bence
On 2022. Nov 29., at 16:08, <Jo...@swisscom.com>> <Jo...@swisscom.com>> wrote:
Hi Bence, Joe
Thank you guys for your fast response.
@Bence you are right, the whole issue seems to be related to nested flows which we try to sync. We are relying heavily on this as we are doing the integration on one NiFi system and then sync it to production via the NiFi Registry. Our parent flow contains multiple nested flows.
Ok, as NIFI-10874 will be included in the next release we will skip the 1.19.0 and wait for 1.19.1 / 1.20.0. Sadly it will probably take more than a few days until we see the next release... Do you have an idea why the flow with the nested flow could lose the storageLocation? Because, we just change a small thing in one of the nested flows and suddenly one of the other nested flows have been losing it’s storageLocation in the parent flow and from this point the whole flow was broken. Now we are scared to change anything as it could brake anytime again.
Cheers Josef
From: Simon Bence <si...@gmail.com>>
Reply to: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Tuesday, 29 November 2022 at 13:31
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 (and same version of the registry) on nested flows
Hi Josef,
Thanks for your patience!
I took a deeper look on what you were writing. In general, this is a sign for the case where the registry client cannot find the nested flow. As far as I understood in your case this happens when you are having a sync. Based on this you may be hitting https://issues.apache.org/jira/browse/NIFI-10874 which is not part of NiFi 1.19.0. The next release should contain the fix that addresses your issue.
Regards,
Bence
On 2022. Nov 28., at 17:48, Simon Bence <si...@gmail.com>> wrote:
Hi Josef,
Thank you for raising the attention to this.
The storageLocation is a new concept to generalise the information used to locate nested flows in a versioned flow. In general it can be there and does not cause issues, as of now NiFi falls back the previous way to locate nested flows. As Joe mentioned, there is a fix in 1.19, but I need to double check if it relates to this situation. Please give me some time, I will find you back.
Regards,
Bence
On 2022. Nov 28., at 17:17, Joe Witt <jo...@gmail.com>> wrote:
Josef
Sorry for the challenges you've hit there. I do think in 1.18 we had a bug/regression as we refactored our registry client. That issue should be resolved in 1.19 which just went live thanks to https://issues.apache.org/jira/browse/NIFI-10787. However, I am not positive if this will solve the scenario you've hit now but please if able try it out.
Thanks
On Mon, Nov 28, 2022 at 8:50 AM <Jo...@swisscom.com>> wrote:
Hi guys
We had the following bug twice already and it broke the whole sync with the NiFi registry for the Flow/PG. First time was directly after we have upgraded from NiFi 1.15.3 to 1.18.0, but we ignored it as we thought it could be because of the upgrade, however it occurred again after a few NiFi Registry commits on NiFi 1.18.0… The error was the following when we tried to change the flow version or when we tried to start from scratch with that version from the NiFi Registry, so the version was broken in the NiFi Registry:
<image001.png>
We investigated the last NiFi Registry Commit and we saw in our GIT repo (we sync the “flow_storage” to GIT) on line 4847 that NiFi removed a “storageLocation” from another PG which makes no sense at all. We changed nothing there and especially why should NiFi remove only the storageLocation line… We have one specialty, as we have nested NiFi Registry flows, so one of the flows where the storageLocation has been removed was such a nested flow.
<image002.png>
Luckily we were able to resolve the error. We tried to add the line and commit it to GIT plus we dropped the the database to repopulate the DB, however it was broken again after a commit from NiFi. So we tried to manually create a new fake version on bucket.yml in the corresponding bucket folder and added as well the line in the snapshot again. We dropped then the DB and restarted NiFi Registry and voilà it was working again.
However it was a nightmare to get it working again as the flow was completely broken, we couldn’t checkout the affected version at all.
Any thoughts on this? Shall I fill a Jira Ticket? The problem is, we can’t really reproduce it, it looks like it happens randomly. As you could imagine, we can’t share our template as it contains a lot of confidential material.
Cheers Josef