You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Shawn Weeks <sw...@weeksconsulting.us> on 2020/08/10 17:00:10 UTC
Data Provenance Stops Working
I’m running a three node NiFi Cluster on AWS EC2s using integrated Zookeeper and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is 1.8.0_242. For some reason after a period of time Data Provenance goes blank, old records are no longer queryable and new data provenance doesn’t appear to get written. No Lucene or other exceptions are logged and restarting the node causes data provenance to go back to being written however old data provenance does not re-appear. No exceptions appear when querying data provenance. All tests have been run as the initial admin user and roles and permissions appear to be correct. I’ve also checked selinux audit logs to see if something is being blocked but nothing appears.
WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30 seconds for rollover, 100mb rollover size, 2 query threads, 2 index threads, compress on rollover, and don’t sync always.
Thanks
Shawn Weeks
Re: Data Provenance Stops Working
Posted by Joe Witt <jo...@gmail.com>.
Which I hope to have RC1 of 1.12 up today to start the review/vote
process. If you two (Shawn, Wyll) could test it against what you're seeing
it would be very helpful!
Thanks
On Mon, Aug 10, 2020 at 10:43 AM Mark Payne <ma...@hotmail.com> wrote:
> OK, so that would be indicative of NIFI-7346. So the data is being written
> into the ether, and then it’s being indexed by Lucene (which is why you see
> the data being written to that Lucene subdirectory). But even when you
> search, it can’t find the event files themselves. So this should all be
> addressed in the upcoming release.
>
> Thanks
> -Mark
>
>
> On Aug 10, 2020, at 1:40 PM, Wyll Ingersoll <
> wyllys.ingersoll@keepertech.com> wrote:
>
> Great, thanks! Looking forward to trying it out on 1.12
>
> I can't see any provenance data with either method (right-click on
> processor or hamburger menu).
>
> When I login to the server, I do see data being written to the
> provenance_repository directory in a sub-dir that looks like
> "lucene-8-index-1560864167766" and which appears to have current data, I
> just cant view anything from the UI.
>
>
> ------------------------------
> *From:* Mark Payne <ma...@hotmail.com>
> *Sent:* Monday, August 10, 2020 1:37 PM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: Data Provenance Stops Working
>
> Shawn / Wyll,
>
> I think you’re probably running into NIFI-7346 [1], which basically says
> there’s a case in which NiFi may “age off” old data even when it’s still
> the file that’s being actively written to. In Linux/OSX this results in
> simply deleting the file, and then anything else written to it disappears
> into the ether. Of course, now the file never exceeds the max size, since
> it’ll be 0 bytes forever, os it never rolls over again. So when this
> happens, no more provenance data gets created until NiFi is restarted.
>
> It’s also possible that you’re hitting NIFI-7375 [2]. This Jira only
> affects you if you get to provenance by right-clicking on a Processor and
> clicking View Provenance (i.e., not if you go to the Hamburger Menu in the
> top-right corner and go to Provenance from there and search that way). If
> this is the problem, once you right-click and go to View Provenance, you
> can actually click the Search icon (magnifying glass) in that empty
> Provenance Results panel and click Search and then it will actually bring
> back the results. So that’s obnoxious but it’s a workaround that may help.
>
> The good news is that both of these have been addressed for 1.12.0, which
> sounds like it should be coming out very soon!
>
> Thanks
> -Mark
>
> [1] https://issues.apache.org/jira/browse/NIFI-7346
> [2] https://issues.apache.org/jira/browse/NIFI-7375
>
>
> On Aug 10, 2020, at 1:26 PM, Joe Witt <jo...@gmail.com> wrote:
>
> shawn - i believe it is related to our default settings and have phoned a
> friend to jump in here when able. but default retention and default
> sharding i *think* can result in this. You can generate a thread dump
> before and after the locked state to see what it is stuck/sitting on. That
> will help here
>
> Thanks
>
> On Mon, Aug 10, 2020 at 10:24 AM Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
> Out of the box even the initial admin user has to be granted permission I
> think, mine worked fine for several months since 1.11.1 was released and
> just started having an issues a couple of weeks ago. I’ve increasing the
> retention time a bit to see if that improves the situation a bit.
>
> Thanks
> Shawn Weeks
>
> *From: *Wyll Ingersoll <wy...@keepertech.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Monday, August 10, 2020 at 12:22 PM
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Data Provenance Stops Working
>
> I run 1.11.4 in a cluster on AWS also and have a similar issue with the
> provenance data, I can't ever view it. It's probably somehow misconfigured
> but I haven't been able to figure it out.
> ------------------------------
> *From:* Andy LoPresto <al...@apache.org>
> *Sent:* Monday, August 10, 2020 1:11 PM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: Data Provenance Stops Working
>
> Shawn,
>
> I don’t know if this is specifically related, but there were a number of
> critical issues discovered in the 1.11.x release line that have been fixed
> in 1.11.4. I would not recommend running any prior version on that release
> line.
>
> 1.12.0 should be coming imminently, so if you are going to upgrade anyway,
> you may want to wait a week or so and get the newest bits with hundreds of
> new features, but for stability alone, I would strongly recommend 1.11.4.
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
> On Aug 10, 2020, at 10:00 AM, Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
> I’m running a three node NiFi Cluster on AWS EC2s using integrated
> Zookeeper and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is
> 1.8.0_242. For some reason after a period of time Data Provenance goes
> blank, old records are no longer queryable and new data provenance doesn’t
> appear to get written. No Lucene or other exceptions are logged and
> restarting the node causes data provenance to go back to being written
> however old data provenance does not re-appear. No exceptions appear when
> querying data provenance. All tests have been run as the initial admin user
> and roles and permissions appear to be correct. I’ve also checked selinux
> audit logs to see if something is being blocked but nothing appears.
>
> WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30
> seconds for rollover, 100mb rollover size, 2 query threads, 2 index
> threads, compress on rollover, and don’t sync always.
>
> Thanks
> Shawn Weeks
>
>
>
Re: Data Provenance Stops Working
Posted by Mark Payne <ma...@hotmail.com>.
OK, so that would be indicative of NIFI-7346. So the data is being written into the ether, and then it’s being indexed by Lucene (which is why you see the data being written to that Lucene subdirectory). But even when you search, it can’t find the event files themselves. So this should all be addressed in the upcoming release.
Thanks
-Mark
On Aug 10, 2020, at 1:40 PM, Wyll Ingersoll <wy...@keepertech.com>> wrote:
Great, thanks! Looking forward to trying it out on 1.12
I can't see any provenance data with either method (right-click on processor or hamburger menu).
When I login to the server, I do see data being written to the provenance_repository directory in a sub-dir that looks like "lucene-8-index-1560864167766" and which appears to have current data, I just cant view anything from the UI.
________________________________
From: Mark Payne <ma...@hotmail.com>>
Sent: Monday, August 10, 2020 1:37 PM
To: users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>
Subject: Re: Data Provenance Stops Working
Shawn / Wyll,
I think you’re probably running into NIFI-7346 [1], which basically says there’s a case in which NiFi may “age off” old data even when it’s still the file that’s being actively written to. In Linux/OSX this results in simply deleting the file, and then anything else written to it disappears into the ether. Of course, now the file never exceeds the max size, since it’ll be 0 bytes forever, os it never rolls over again. So when this happens, no more provenance data gets created until NiFi is restarted.
It’s also possible that you’re hitting NIFI-7375 [2]. This Jira only affects you if you get to provenance by right-clicking on a Processor and clicking View Provenance (i.e., not if you go to the Hamburger Menu in the top-right corner and go to Provenance from there and search that way). If this is the problem, once you right-click and go to View Provenance, you can actually click the Search icon (magnifying glass) in that empty Provenance Results panel and click Search and then it will actually bring back the results. So that’s obnoxious but it’s a workaround that may help.
The good news is that both of these have been addressed for 1.12.0, which sounds like it should be coming out very soon!
Thanks
-Mark
[1] https://issues.apache.org/jira/browse/NIFI-7346
[2] https://issues.apache.org/jira/browse/NIFI-7375
On Aug 10, 2020, at 1:26 PM, Joe Witt <jo...@gmail.com>> wrote:
shawn - i believe it is related to our default settings and have phoned a friend to jump in here when able. but default retention and default sharding i *think* can result in this. You can generate a thread dump before and after the locked state to see what it is stuck/sitting on. That will help here
Thanks
On Mon, Aug 10, 2020 at 10:24 AM Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Out of the box even the initial admin user has to be granted permission I think, mine worked fine for several months since 1.11.1 was released and just started having an issues a couple of weeks ago. I’ve increasing the retention time a bit to see if that improves the situation a bit.
Thanks
Shawn Weeks
From: Wyll Ingersoll <wy...@keepertech.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Monday, August 10, 2020 at 12:22 PM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Data Provenance Stops Working
I run 1.11.4 in a cluster on AWS also and have a similar issue with the provenance data, I can't ever view it. It's probably somehow misconfigured but I haven't been able to figure it out.
________________________________
From: Andy LoPresto <al...@apache.org>>
Sent: Monday, August 10, 2020 1:11 PM
To: users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>
Subject: Re: Data Provenance Stops Working
Shawn,
I don’t know if this is specifically related, but there were a number of critical issues discovered in the 1.11.x release line that have been fixed in 1.11.4. I would not recommend running any prior version on that release line.
1.12.0 should be coming imminently, so if you are going to upgrade anyway, you may want to wait a week or so and get the newest bits with hundreds of new features, but for stability alone, I would strongly recommend 1.11.4.
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 10, 2020, at 10:00 AM, Shawn Weeks <sw...@weeksconsulting.us>> wrote:
I’m running a three node NiFi Cluster on AWS EC2s using integrated Zookeeper and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is 1.8.0_242. For some reason after a period of time Data Provenance goes blank, old records are no longer queryable and new data provenance doesn’t appear to get written. No Lucene or other exceptions are logged and restarting the node causes data provenance to go back to being written however old data provenance does not re-appear. No exceptions appear when querying data provenance. All tests have been run as the initial admin user and roles and permissions appear to be correct. I’ve also checked selinux audit logs to see if something is being blocked but nothing appears.
WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30 seconds for rollover, 100mb rollover size, 2 query threads, 2 index threads, compress on rollover, and don’t sync always.
Thanks
Shawn Weeks
Re: Data Provenance Stops Working
Posted by Wyll Ingersoll <wy...@keepertech.com>.
Great, thanks! Looking forward to trying it out on 1.12
I can't see any provenance data with either method (right-click on processor or hamburger menu).
When I login to the server, I do see data being written to the provenance_repository directory in a sub-dir that looks like "lucene-8-index-1560864167766" and which appears to have current data, I just cant view anything from the UI.
________________________________
From: Mark Payne <ma...@hotmail.com>
Sent: Monday, August 10, 2020 1:37 PM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Data Provenance Stops Working
Shawn / Wyll,
I think you’re probably running into NIFI-7346 [1], which basically says there’s a case in which NiFi may “age off” old data even when it’s still the file that’s being actively written to. In Linux/OSX this results in simply deleting the file, and then anything else written to it disappears into the ether. Of course, now the file never exceeds the max size, since it’ll be 0 bytes forever, os it never rolls over again. So when this happens, no more provenance data gets created until NiFi is restarted.
It’s also possible that you’re hitting NIFI-7375 [2]. This Jira only affects you if you get to provenance by right-clicking on a Processor and clicking View Provenance (i.e., not if you go to the Hamburger Menu in the top-right corner and go to Provenance from there and search that way). If this is the problem, once you right-click and go to View Provenance, you can actually click the Search icon (magnifying glass) in that empty Provenance Results panel and click Search and then it will actually bring back the results. So that’s obnoxious but it’s a workaround that may help.
The good news is that both of these have been addressed for 1.12.0, which sounds like it should be coming out very soon!
Thanks
-Mark
[1] https://issues.apache.org/jira/browse/NIFI-7346
[2] https://issues.apache.org/jira/browse/NIFI-7375
On Aug 10, 2020, at 1:26 PM, Joe Witt <jo...@gmail.com>> wrote:
shawn - i believe it is related to our default settings and have phoned a friend to jump in here when able. but default retention and default sharding i *think* can result in this. You can generate a thread dump before and after the locked state to see what it is stuck/sitting on. That will help here
Thanks
On Mon, Aug 10, 2020 at 10:24 AM Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Out of the box even the initial admin user has to be granted permission I think, mine worked fine for several months since 1.11.1 was released and just started having an issues a couple of weeks ago. I’ve increasing the retention time a bit to see if that improves the situation a bit.
Thanks
Shawn Weeks
From: Wyll Ingersoll <wy...@keepertech.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Monday, August 10, 2020 at 12:22 PM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Data Provenance Stops Working
I run 1.11.4 in a cluster on AWS also and have a similar issue with the provenance data, I can't ever view it. It's probably somehow misconfigured but I haven't been able to figure it out.
________________________________
From: Andy LoPresto <al...@apache.org>>
Sent: Monday, August 10, 2020 1:11 PM
To: users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>
Subject: Re: Data Provenance Stops Working
Shawn,
I don’t know if this is specifically related, but there were a number of critical issues discovered in the 1.11.x release line that have been fixed in 1.11.4. I would not recommend running any prior version on that release line.
1.12.0 should be coming imminently, so if you are going to upgrade anyway, you may want to wait a week or so and get the newest bits with hundreds of new features, but for stability alone, I would strongly recommend 1.11.4.
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 10, 2020, at 10:00 AM, Shawn Weeks <sw...@weeksconsulting.us>> wrote:
I’m running a three node NiFi Cluster on AWS EC2s using integrated Zookeeper and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is 1.8.0_242. For some reason after a period of time Data Provenance goes blank, old records are no longer queryable and new data provenance doesn’t appear to get written. No Lucene or other exceptions are logged and restarting the node causes data provenance to go back to being written however old data provenance does not re-appear. No exceptions appear when querying data provenance. All tests have been run as the initial admin user and roles and permissions appear to be correct. I’ve also checked selinux audit logs to see if something is being blocked but nothing appears.
WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30 seconds for rollover, 100mb rollover size, 2 query threads, 2 index threads, compress on rollover, and don’t sync always.
Thanks
Shawn Weeks
Re: Data Provenance Stops Working
Posted by Pierre Villard <pi...@gmail.com>.
Darren - I'm using NiFi 1.11.4 for a while now (with OOTB configuration)
and I do have provenance data for the flows I'm running.
Le lun. 10 août 2020 à 23:24, Wyll Ingersoll <
wyllys.ingersoll@keepertech.com> a écrit :
> Ah! That fixed my problem!
>
> I am running a secure/authenticated configuration. My user had the
> correct policy permissions when viewed from the hamburger menu, but not on
> the access-policies for the individual processors/canvas.
>
> Very confusing! One would think if the admin user had the permissions,
> they would apply everywhere, but apparently not.
>
> Anyway, now I can see provenance data. Thanks for the tip!
> ------------------------------
> *From:* Shawn Weeks <sw...@weeksconsulting.us>
> *Sent:* Monday, August 10, 2020 5:19 PM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: Data Provenance Stops Working
>
>
> That sounds like a permission issue. Are you running in secure mode? If so
> right click on the main canvas and go to access policies and make sure your
> current user is allowed provenance access, you’ll also have to go to the
> policies section in the system menu in the upper right hand corner and do
> the same.
>
>
>
> Thanks
>
> Shawn
>
>
>
> *From: *Darren Govoni <da...@ontrenet.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Monday, August 10, 2020 at 3:52 PM
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Data Provenance Stops Working
>
>
>
> I also use 1.11.4 and out of the box there IS NO provenance data
> whatsoever. It just doesn't work if you install and run nifi as is.
>
> Sent from my Verizon, Samsung Galaxy smartphone
>
> Get Outlook for Android <https://aka.ms/ghei36>
>
>
> ------------------------------
>
> *From:* Shawn Weeks <sw...@weeksconsulting.us>
> *Sent:* Monday, August 10, 2020 2:23:19 PM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: Data Provenance Stops Working
>
>
>
> It sounds like if I expand the retention time a lot, say 30 days the issue
> should be less bad?
>
>
>
> Thanks
>
> Shawn
>
>
>
> *From: *Mark Payne <ma...@hotmail.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Monday, August 10, 2020 at 12:37 PM
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Data Provenance Stops Working
>
>
>
> Shawn / Wyll,
>
>
>
> I think you’re probably running into NIFI-7346 [1], which basically says
> there’s a case in which NiFi may “age off” old data even when it’s still
> the file that’s being actively written to. In Linux/OSX this results in
> simply deleting the file, and then anything else written to it disappears
> into the ether. Of course, now the file never exceeds the max size, since
> it’ll be 0 bytes forever, os it never rolls over again. So when this
> happens, no more provenance data gets created until NiFi is restarted.
>
>
>
> It’s also possible that you’re hitting NIFI-7375 [2]. This Jira only
> affects you if you get to provenance by right-clicking on a Processor and
> clicking View Provenance (i.e., not if you go to the Hamburger Menu in the
> top-right corner and go to Provenance from there and search that way). If
> this is the problem, once you right-click and go to View Provenance, you
> can actually click the Search icon (magnifying glass) in that empty
> Provenance Results panel and click Search and then it will actually bring
> back the results. So that’s obnoxious but it’s a workaround that may help.
>
>
>
> The good news is that both of these have been addressed for 1.12.0, which
> sounds like it should be coming out very soon!
>
>
>
> Thanks
>
> -Mark
>
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-7346
>
> [2] https://issues.apache.org/jira/browse/NIFI-7375
>
>
>
>
>
> On Aug 10, 2020, at 1:26 PM, Joe Witt <jo...@gmail.com> wrote:
>
>
>
> shawn - i believe it is related to our default settings and have phoned a
> friend to jump in here when able. but default retention and default
> sharding i *think* can result in this. You can generate a thread dump
> before and after the locked state to see what it is stuck/sitting on. That
> will help here
>
>
>
> Thanks
>
>
>
> On Mon, Aug 10, 2020 at 10:24 AM Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
> Out of the box even the initial admin user has to be granted permission I
> think, mine worked fine for several months since 1.11.1 was released and
> just started having an issues a couple of weeks ago. I’ve increasing the
> retention time a bit to see if that improves the situation a bit.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From: *Wyll Ingersoll <wy...@keepertech.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Monday, August 10, 2020 at 12:22 PM
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Data Provenance Stops Working
>
>
>
> I run 1.11.4 in a cluster on AWS also and have a similar issue with the
> provenance data, I can't ever view it. It's probably somehow misconfigured
> but I haven't been able to figure it out.
> ------------------------------
>
> *From:* Andy LoPresto <al...@apache.org>
> *Sent:* Monday, August 10, 2020 1:11 PM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: Data Provenance Stops Working
>
>
>
> Shawn,
>
>
>
> I don’t know if this is specifically related, but there were a number of
> critical issues discovered in the 1.11.x release line that have been fixed
> in 1.11.4. I would not recommend running any prior version on that release
> line.
>
>
>
> 1.12.0 should be coming imminently, so if you are going to upgrade anyway,
> you may want to wait a week or so and get the newest bits with hundreds of
> new features, but for stability alone, I would strongly recommend 1.11.4.
>
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
>
>
>
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> He/Him
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
>
> On Aug 10, 2020, at 10:00 AM, Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
>
>
> I’m running a three node NiFi Cluster on AWS EC2s using integrated
> Zookeeper and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is
> 1.8.0_242. For some reason after a period of time Data Provenance goes
> blank, old records are no longer queryable and new data provenance doesn’t
> appear to get written. No Lucene or other exceptions are logged and
> restarting the node causes data provenance to go back to being written
> however old data provenance does not re-appear. No exceptions appear when
> querying data provenance. All tests have been run as the initial admin user
> and roles and permissions appear to be correct. I’ve also checked selinux
> audit logs to see if something is being blocked but nothing appears.
>
>
>
> WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30
> seconds for rollover, 100mb rollover size, 2 query threads, 2 index
> threads, compress on rollover, and don’t sync always.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
>
>
Re: Data Provenance Stops Working
Posted by Wyll Ingersoll <wy...@keepertech.com>.
Ah! That fixed my problem!
I am running a secure/authenticated configuration. My user had the correct policy permissions when viewed from the hamburger menu, but not on the access-policies for the individual processors/canvas.
Very confusing! One would think if the admin user had the permissions, they would apply everywhere, but apparently not.
Anyway, now I can see provenance data. Thanks for the tip!
________________________________
From: Shawn Weeks <sw...@weeksconsulting.us>
Sent: Monday, August 10, 2020 5:19 PM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Data Provenance Stops Working
That sounds like a permission issue. Are you running in secure mode? If so right click on the main canvas and go to access policies and make sure your current user is allowed provenance access, you’ll also have to go to the policies section in the system menu in the upper right hand corner and do the same.
Thanks
Shawn
From: Darren Govoni <da...@ontrenet.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Monday, August 10, 2020 at 3:52 PM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Data Provenance Stops Working
I also use 1.11.4 and out of the box there IS NO provenance data whatsoever. It just doesn't work if you install and run nifi as is.
Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android<https://aka.ms/ghei36>
________________________________
From: Shawn Weeks <sw...@weeksconsulting.us>
Sent: Monday, August 10, 2020 2:23:19 PM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Data Provenance Stops Working
It sounds like if I expand the retention time a lot, say 30 days the issue should be less bad?
Thanks
Shawn
From: Mark Payne <ma...@hotmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Monday, August 10, 2020 at 12:37 PM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Data Provenance Stops Working
Shawn / Wyll,
I think you’re probably running into NIFI-7346 [1], which basically says there’s a case in which NiFi may “age off” old data even when it’s still the file that’s being actively written to. In Linux/OSX this results in simply deleting the file, and then anything else written to it disappears into the ether. Of course, now the file never exceeds the max size, since it’ll be 0 bytes forever, os it never rolls over again. So when this happens, no more provenance data gets created until NiFi is restarted.
It’s also possible that you’re hitting NIFI-7375 [2]. This Jira only affects you if you get to provenance by right-clicking on a Processor and clicking View Provenance (i.e., not if you go to the Hamburger Menu in the top-right corner and go to Provenance from there and search that way). If this is the problem, once you right-click and go to View Provenance, you can actually click the Search icon (magnifying glass) in that empty Provenance Results panel and click Search and then it will actually bring back the results. So that’s obnoxious but it’s a workaround that may help.
The good news is that both of these have been addressed for 1.12.0, which sounds like it should be coming out very soon!
Thanks
-Mark
[1] https://issues.apache.org/jira/browse/NIFI-7346
[2] https://issues.apache.org/jira/browse/NIFI-7375
On Aug 10, 2020, at 1:26 PM, Joe Witt <jo...@gmail.com>> wrote:
shawn - i believe it is related to our default settings and have phoned a friend to jump in here when able. but default retention and default sharding i *think* can result in this. You can generate a thread dump before and after the locked state to see what it is stuck/sitting on. That will help here
Thanks
On Mon, Aug 10, 2020 at 10:24 AM Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Out of the box even the initial admin user has to be granted permission I think, mine worked fine for several months since 1.11.1 was released and just started having an issues a couple of weeks ago. I’ve increasing the retention time a bit to see if that improves the situation a bit.
Thanks
Shawn Weeks
From: Wyll Ingersoll <wy...@keepertech.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Monday, August 10, 2020 at 12:22 PM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Data Provenance Stops Working
I run 1.11.4 in a cluster on AWS also and have a similar issue with the provenance data, I can't ever view it. It's probably somehow misconfigured but I haven't been able to figure it out.
________________________________
From: Andy LoPresto <al...@apache.org>>
Sent: Monday, August 10, 2020 1:11 PM
To: users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>
Subject: Re: Data Provenance Stops Working
Shawn,
I don’t know if this is specifically related, but there were a number of critical issues discovered in the 1.11.x release line that have been fixed in 1.11.4. I would not recommend running any prior version on that release line.
1.12.0 should be coming imminently, so if you are going to upgrade anyway, you may want to wait a week or so and get the newest bits with hundreds of new features, but for stability alone, I would strongly recommend 1.11.4.
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 10, 2020, at 10:00 AM, Shawn Weeks <sw...@weeksconsulting.us>> wrote:
I’m running a three node NiFi Cluster on AWS EC2s using integrated Zookeeper and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is 1.8.0_242. For some reason after a period of time Data Provenance goes blank, old records are no longer queryable and new data provenance doesn’t appear to get written. No Lucene or other exceptions are logged and restarting the node causes data provenance to go back to being written however old data provenance does not re-appear. No exceptions appear when querying data provenance. All tests have been run as the initial admin user and roles and permissions appear to be correct. I’ve also checked selinux audit logs to see if something is being blocked but nothing appears.
WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30 seconds for rollover, 100mb rollover size, 2 query threads, 2 index threads, compress on rollover, and don’t sync always.
Thanks
Shawn Weeks
Re: Data Provenance Stops Working
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
That sounds like a permission issue. Are you running in secure mode? If so right click on the main canvas and go to access policies and make sure your current user is allowed provenance access, you’ll also have to go to the policies section in the system menu in the upper right hand corner and do the same.
Thanks
Shawn
From: Darren Govoni <da...@ontrenet.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Monday, August 10, 2020 at 3:52 PM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Data Provenance Stops Working
I also use 1.11.4 and out of the box there IS NO provenance data whatsoever. It just doesn't work if you install and run nifi as is.
Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android<https://aka.ms/ghei36>
________________________________
From: Shawn Weeks <sw...@weeksconsulting.us>
Sent: Monday, August 10, 2020 2:23:19 PM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Data Provenance Stops Working
It sounds like if I expand the retention time a lot, say 30 days the issue should be less bad?
Thanks
Shawn
From: Mark Payne <ma...@hotmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Monday, August 10, 2020 at 12:37 PM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Data Provenance Stops Working
Shawn / Wyll,
I think you’re probably running into NIFI-7346 [1], which basically says there’s a case in which NiFi may “age off” old data even when it’s still the file that’s being actively written to. In Linux/OSX this results in simply deleting the file, and then anything else written to it disappears into the ether. Of course, now the file never exceeds the max size, since it’ll be 0 bytes forever, os it never rolls over again. So when this happens, no more provenance data gets created until NiFi is restarted.
It’s also possible that you’re hitting NIFI-7375 [2]. This Jira only affects you if you get to provenance by right-clicking on a Processor and clicking View Provenance (i.e., not if you go to the Hamburger Menu in the top-right corner and go to Provenance from there and search that way). If this is the problem, once you right-click and go to View Provenance, you can actually click the Search icon (magnifying glass) in that empty Provenance Results panel and click Search and then it will actually bring back the results. So that’s obnoxious but it’s a workaround that may help.
The good news is that both of these have been addressed for 1.12.0, which sounds like it should be coming out very soon!
Thanks
-Mark
[1] https://issues.apache.org/jira/browse/NIFI-7346
[2] https://issues.apache.org/jira/browse/NIFI-7375
On Aug 10, 2020, at 1:26 PM, Joe Witt <jo...@gmail.com>> wrote:
shawn - i believe it is related to our default settings and have phoned a friend to jump in here when able. but default retention and default sharding i *think* can result in this. You can generate a thread dump before and after the locked state to see what it is stuck/sitting on. That will help here
Thanks
On Mon, Aug 10, 2020 at 10:24 AM Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Out of the box even the initial admin user has to be granted permission I think, mine worked fine for several months since 1.11.1 was released and just started having an issues a couple of weeks ago. I’ve increasing the retention time a bit to see if that improves the situation a bit.
Thanks
Shawn Weeks
From: Wyll Ingersoll <wy...@keepertech.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Monday, August 10, 2020 at 12:22 PM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Data Provenance Stops Working
I run 1.11.4 in a cluster on AWS also and have a similar issue with the provenance data, I can't ever view it. It's probably somehow misconfigured but I haven't been able to figure it out.
________________________________
From: Andy LoPresto <al...@apache.org>>
Sent: Monday, August 10, 2020 1:11 PM
To: users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>
Subject: Re: Data Provenance Stops Working
Shawn,
I don’t know if this is specifically related, but there were a number of critical issues discovered in the 1.11.x release line that have been fixed in 1.11.4. I would not recommend running any prior version on that release line.
1.12.0 should be coming imminently, so if you are going to upgrade anyway, you may want to wait a week or so and get the newest bits with hundreds of new features, but for stability alone, I would strongly recommend 1.11.4.
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 10, 2020, at 10:00 AM, Shawn Weeks <sw...@weeksconsulting.us>> wrote:
I’m running a three node NiFi Cluster on AWS EC2s using integrated Zookeeper and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is 1.8.0_242. For some reason after a period of time Data Provenance goes blank, old records are no longer queryable and new data provenance doesn’t appear to get written. No Lucene or other exceptions are logged and restarting the node causes data provenance to go back to being written however old data provenance does not re-appear. No exceptions appear when querying data provenance. All tests have been run as the initial admin user and roles and permissions appear to be correct. I’ve also checked selinux audit logs to see if something is being blocked but nothing appears.
WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30 seconds for rollover, 100mb rollover size, 2 query threads, 2 index threads, compress on rollover, and don’t sync always.
Thanks
Shawn Weeks
Re: Data Provenance Stops Working
Posted by Darren Govoni <da...@ontrenet.com>.
I also use 1.11.4 and out of the box there IS NO provenance data whatsoever. It just doesn't work if you install and run nifi as is.
Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android<https://aka.ms/ghei36>
________________________________
From: Shawn Weeks <sw...@weeksconsulting.us>
Sent: Monday, August 10, 2020 2:23:19 PM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Data Provenance Stops Working
It sounds like if I expand the retention time a lot, say 30 days the issue should be less bad?
Thanks
Shawn
From: Mark Payne <ma...@hotmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Monday, August 10, 2020 at 12:37 PM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Data Provenance Stops Working
Shawn / Wyll,
I think you’re probably running into NIFI-7346 [1], which basically says there’s a case in which NiFi may “age off” old data even when it’s still the file that’s being actively written to. In Linux/OSX this results in simply deleting the file, and then anything else written to it disappears into the ether. Of course, now the file never exceeds the max size, since it’ll be 0 bytes forever, os it never rolls over again. So when this happens, no more provenance data gets created until NiFi is restarted.
It’s also possible that you’re hitting NIFI-7375 [2]. This Jira only affects you if you get to provenance by right-clicking on a Processor and clicking View Provenance (i.e., not if you go to the Hamburger Menu in the top-right corner and go to Provenance from there and search that way). If this is the problem, once you right-click and go to View Provenance, you can actually click the Search icon (magnifying glass) in that empty Provenance Results panel and click Search and then it will actually bring back the results. So that’s obnoxious but it’s a workaround that may help.
The good news is that both of these have been addressed for 1.12.0, which sounds like it should be coming out very soon!
Thanks
-Mark
[1] https://issues.apache.org/jira/browse/NIFI-7346
[2] https://issues.apache.org/jira/browse/NIFI-7375
On Aug 10, 2020, at 1:26 PM, Joe Witt <jo...@gmail.com>> wrote:
shawn - i believe it is related to our default settings and have phoned a friend to jump in here when able. but default retention and default sharding i *think* can result in this. You can generate a thread dump before and after the locked state to see what it is stuck/sitting on. That will help here
Thanks
On Mon, Aug 10, 2020 at 10:24 AM Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Out of the box even the initial admin user has to be granted permission I think, mine worked fine for several months since 1.11.1 was released and just started having an issues a couple of weeks ago. I’ve increasing the retention time a bit to see if that improves the situation a bit.
Thanks
Shawn Weeks
From: Wyll Ingersoll <wy...@keepertech.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Monday, August 10, 2020 at 12:22 PM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Data Provenance Stops Working
I run 1.11.4 in a cluster on AWS also and have a similar issue with the provenance data, I can't ever view it. It's probably somehow misconfigured but I haven't been able to figure it out.
________________________________
From: Andy LoPresto <al...@apache.org>>
Sent: Monday, August 10, 2020 1:11 PM
To: users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>
Subject: Re: Data Provenance Stops Working
Shawn,
I don’t know if this is specifically related, but there were a number of critical issues discovered in the 1.11.x release line that have been fixed in 1.11.4. I would not recommend running any prior version on that release line.
1.12.0 should be coming imminently, so if you are going to upgrade anyway, you may want to wait a week or so and get the newest bits with hundreds of new features, but for stability alone, I would strongly recommend 1.11.4.
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 10, 2020, at 10:00 AM, Shawn Weeks <sw...@weeksconsulting.us>> wrote:
I’m running a three node NiFi Cluster on AWS EC2s using integrated Zookeeper and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is 1.8.0_242. For some reason after a period of time Data Provenance goes blank, old records are no longer queryable and new data provenance doesn’t appear to get written. No Lucene or other exceptions are logged and restarting the node causes data provenance to go back to being written however old data provenance does not re-appear. No exceptions appear when querying data provenance. All tests have been run as the initial admin user and roles and permissions appear to be correct. I’ve also checked selinux audit logs to see if something is being blocked but nothing appears.
WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30 seconds for rollover, 100mb rollover size, 2 query threads, 2 index threads, compress on rollover, and don’t sync always.
Thanks
Shawn Weeks
Re: Data Provenance Stops Working
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
It sounds like if I expand the retention time a lot, say 30 days the issue should be less bad?
Thanks
Shawn
From: Mark Payne <ma...@hotmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Monday, August 10, 2020 at 12:37 PM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Data Provenance Stops Working
Shawn / Wyll,
I think you’re probably running into NIFI-7346 [1], which basically says there’s a case in which NiFi may “age off” old data even when it’s still the file that’s being actively written to. In Linux/OSX this results in simply deleting the file, and then anything else written to it disappears into the ether. Of course, now the file never exceeds the max size, since it’ll be 0 bytes forever, os it never rolls over again. So when this happens, no more provenance data gets created until NiFi is restarted.
It’s also possible that you’re hitting NIFI-7375 [2]. This Jira only affects you if you get to provenance by right-clicking on a Processor and clicking View Provenance (i.e., not if you go to the Hamburger Menu in the top-right corner and go to Provenance from there and search that way). If this is the problem, once you right-click and go to View Provenance, you can actually click the Search icon (magnifying glass) in that empty Provenance Results panel and click Search and then it will actually bring back the results. So that’s obnoxious but it’s a workaround that may help.
The good news is that both of these have been addressed for 1.12.0, which sounds like it should be coming out very soon!
Thanks
-Mark
[1] https://issues.apache.org/jira/browse/NIFI-7346
[2] https://issues.apache.org/jira/browse/NIFI-7375
On Aug 10, 2020, at 1:26 PM, Joe Witt <jo...@gmail.com>> wrote:
shawn - i believe it is related to our default settings and have phoned a friend to jump in here when able. but default retention and default sharding i *think* can result in this. You can generate a thread dump before and after the locked state to see what it is stuck/sitting on. That will help here
Thanks
On Mon, Aug 10, 2020 at 10:24 AM Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Out of the box even the initial admin user has to be granted permission I think, mine worked fine for several months since 1.11.1 was released and just started having an issues a couple of weeks ago. I’ve increasing the retention time a bit to see if that improves the situation a bit.
Thanks
Shawn Weeks
From: Wyll Ingersoll <wy...@keepertech.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Monday, August 10, 2020 at 12:22 PM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Data Provenance Stops Working
I run 1.11.4 in a cluster on AWS also and have a similar issue with the provenance data, I can't ever view it. It's probably somehow misconfigured but I haven't been able to figure it out.
________________________________
From: Andy LoPresto <al...@apache.org>>
Sent: Monday, August 10, 2020 1:11 PM
To: users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>
Subject: Re: Data Provenance Stops Working
Shawn,
I don’t know if this is specifically related, but there were a number of critical issues discovered in the 1.11.x release line that have been fixed in 1.11.4. I would not recommend running any prior version on that release line.
1.12.0 should be coming imminently, so if you are going to upgrade anyway, you may want to wait a week or so and get the newest bits with hundreds of new features, but for stability alone, I would strongly recommend 1.11.4.
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 10, 2020, at 10:00 AM, Shawn Weeks <sw...@weeksconsulting.us>> wrote:
I’m running a three node NiFi Cluster on AWS EC2s using integrated Zookeeper and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is 1.8.0_242. For some reason after a period of time Data Provenance goes blank, old records are no longer queryable and new data provenance doesn’t appear to get written. No Lucene or other exceptions are logged and restarting the node causes data provenance to go back to being written however old data provenance does not re-appear. No exceptions appear when querying data provenance. All tests have been run as the initial admin user and roles and permissions appear to be correct. I’ve also checked selinux audit logs to see if something is being blocked but nothing appears.
WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30 seconds for rollover, 100mb rollover size, 2 query threads, 2 index threads, compress on rollover, and don’t sync always.
Thanks
Shawn Weeks
Re: Data Provenance Stops Working
Posted by Mark Payne <ma...@hotmail.com>.
Shawn / Wyll,
I think you’re probably running into NIFI-7346 [1], which basically says there’s a case in which NiFi may “age off” old data even when it’s still the file that’s being actively written to. In Linux/OSX this results in simply deleting the file, and then anything else written to it disappears into the ether. Of course, now the file never exceeds the max size, since it’ll be 0 bytes forever, os it never rolls over again. So when this happens, no more provenance data gets created until NiFi is restarted.
It’s also possible that you’re hitting NIFI-7375 [2]. This Jira only affects you if you get to provenance by right-clicking on a Processor and clicking View Provenance (i.e., not if you go to the Hamburger Menu in the top-right corner and go to Provenance from there and search that way). If this is the problem, once you right-click and go to View Provenance, you can actually click the Search icon (magnifying glass) in that empty Provenance Results panel and click Search and then it will actually bring back the results. So that’s obnoxious but it’s a workaround that may help.
The good news is that both of these have been addressed for 1.12.0, which sounds like it should be coming out very soon!
Thanks
-Mark
[1] https://issues.apache.org/jira/browse/NIFI-7346
[2] https://issues.apache.org/jira/browse/NIFI-7375
On Aug 10, 2020, at 1:26 PM, Joe Witt <jo...@gmail.com>> wrote:
shawn - i believe it is related to our default settings and have phoned a friend to jump in here when able. but default retention and default sharding i *think* can result in this. You can generate a thread dump before and after the locked state to see what it is stuck/sitting on. That will help here
Thanks
On Mon, Aug 10, 2020 at 10:24 AM Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Out of the box even the initial admin user has to be granted permission I think, mine worked fine for several months since 1.11.1 was released and just started having an issues a couple of weeks ago. I’ve increasing the retention time a bit to see if that improves the situation a bit.
Thanks
Shawn Weeks
From: Wyll Ingersoll <wy...@keepertech.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Monday, August 10, 2020 at 12:22 PM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: Data Provenance Stops Working
I run 1.11.4 in a cluster on AWS also and have a similar issue with the provenance data, I can't ever view it. It's probably somehow misconfigured but I haven't been able to figure it out.
________________________________
From: Andy LoPresto <al...@apache.org>>
Sent: Monday, August 10, 2020 1:11 PM
To: users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>
Subject: Re: Data Provenance Stops Working
Shawn,
I don’t know if this is specifically related, but there were a number of critical issues discovered in the 1.11.x release line that have been fixed in 1.11.4. I would not recommend running any prior version on that release line.
1.12.0 should be coming imminently, so if you are going to upgrade anyway, you may want to wait a week or so and get the newest bits with hundreds of new features, but for stability alone, I would strongly recommend 1.11.4.
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 10, 2020, at 10:00 AM, Shawn Weeks <sw...@weeksconsulting.us>> wrote:
I’m running a three node NiFi Cluster on AWS EC2s using integrated Zookeeper and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is 1.8.0_242. For some reason after a period of time Data Provenance goes blank, old records are no longer queryable and new data provenance doesn’t appear to get written. No Lucene or other exceptions are logged and restarting the node causes data provenance to go back to being written however old data provenance does not re-appear. No exceptions appear when querying data provenance. All tests have been run as the initial admin user and roles and permissions appear to be correct. I’ve also checked selinux audit logs to see if something is being blocked but nothing appears.
WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30 seconds for rollover, 100mb rollover size, 2 query threads, 2 index threads, compress on rollover, and don’t sync always.
Thanks
Shawn Weeks
Re: Data Provenance Stops Working
Posted by Joe Witt <jo...@gmail.com>.
shawn - i believe it is related to our default settings and have phoned a
friend to jump in here when able. but default retention and default
sharding i *think* can result in this. You can generate a thread dump
before and after the locked state to see what it is stuck/sitting on. That
will help here
Thanks
On Mon, Aug 10, 2020 at 10:24 AM Shawn Weeks <sw...@weeksconsulting.us>
wrote:
> Out of the box even the initial admin user has to be granted permission I
> think, mine worked fine for several months since 1.11.1 was released and
> just started having an issues a couple of weeks ago. I’ve increasing the
> retention time a bit to see if that improves the situation a bit.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From: *Wyll Ingersoll <wy...@keepertech.com>
> *Reply-To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Date: *Monday, August 10, 2020 at 12:22 PM
> *To: *"users@nifi.apache.org" <us...@nifi.apache.org>
> *Subject: *Re: Data Provenance Stops Working
>
>
>
> I run 1.11.4 in a cluster on AWS also and have a similar issue with the
> provenance data, I can't ever view it. It's probably somehow misconfigured
> but I haven't been able to figure it out.
> ------------------------------
>
> *From:* Andy LoPresto <al...@apache.org>
> *Sent:* Monday, August 10, 2020 1:11 PM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: Data Provenance Stops Working
>
>
>
> Shawn,
>
>
>
> I don’t know if this is specifically related, but there were a number of
> critical issues discovered in the 1.11.x release line that have been fixed
> in 1.11.4. I would not recommend running any prior version on that release
> line.
>
>
>
> 1.12.0 should be coming imminently, so if you are going to upgrade anyway,
> you may want to wait a week or so and get the newest bits with hundreds of
> new features, but for stability alone, I would strongly recommend 1.11.4.
>
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
>
>
>
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> He/Him
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
>
> On Aug 10, 2020, at 10:00 AM, Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
>
>
> I’m running a three node NiFi Cluster on AWS EC2s using integrated
> Zookeeper and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is
> 1.8.0_242. For some reason after a period of time Data Provenance goes
> blank, old records are no longer queryable and new data provenance doesn’t
> appear to get written. No Lucene or other exceptions are logged and
> restarting the node causes data provenance to go back to being written
> however old data provenance does not re-appear. No exceptions appear when
> querying data provenance. All tests have been run as the initial admin user
> and roles and permissions appear to be correct. I’ve also checked selinux
> audit logs to see if something is being blocked but nothing appears.
>
>
>
> WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30
> seconds for rollover, 100mb rollover size, 2 query threads, 2 index
> threads, compress on rollover, and don’t sync always.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
Re: Data Provenance Stops Working
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
Out of the box even the initial admin user has to be granted permission I think, mine worked fine for several months since 1.11.1 was released and just started having an issues a couple of weeks ago. I’ve increasing the retention time a bit to see if that improves the situation a bit.
Thanks
Shawn Weeks
From: Wyll Ingersoll <wy...@keepertech.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Monday, August 10, 2020 at 12:22 PM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: Data Provenance Stops Working
I run 1.11.4 in a cluster on AWS also and have a similar issue with the provenance data, I can't ever view it. It's probably somehow misconfigured but I haven't been able to figure it out.
________________________________
From: Andy LoPresto <al...@apache.org>
Sent: Monday, August 10, 2020 1:11 PM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Data Provenance Stops Working
Shawn,
I don’t know if this is specifically related, but there were a number of critical issues discovered in the 1.11.x release line that have been fixed in 1.11.4. I would not recommend running any prior version on that release line.
1.12.0 should be coming imminently, so if you are going to upgrade anyway, you may want to wait a week or so and get the newest bits with hundreds of new features, but for stability alone, I would strongly recommend 1.11.4.
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 10, 2020, at 10:00 AM, Shawn Weeks <sw...@weeksconsulting.us>> wrote:
I’m running a three node NiFi Cluster on AWS EC2s using integrated Zookeeper and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is 1.8.0_242. For some reason after a period of time Data Provenance goes blank, old records are no longer queryable and new data provenance doesn’t appear to get written. No Lucene or other exceptions are logged and restarting the node causes data provenance to go back to being written however old data provenance does not re-appear. No exceptions appear when querying data provenance. All tests have been run as the initial admin user and roles and permissions appear to be correct. I’ve also checked selinux audit logs to see if something is being blocked but nothing appears.
WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30 seconds for rollover, 100mb rollover size, 2 query threads, 2 index threads, compress on rollover, and don’t sync always.
Thanks
Shawn Weeks
Re: Data Provenance Stops Working
Posted by Wyll Ingersoll <wy...@keepertech.com>.
I run 1.11.4 in a cluster on AWS also and have a similar issue with the provenance data, I can't ever view it. It's probably somehow misconfigured but I haven't been able to figure it out.
________________________________
From: Andy LoPresto <al...@apache.org>
Sent: Monday, August 10, 2020 1:11 PM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Data Provenance Stops Working
Shawn,
I don’t know if this is specifically related, but there were a number of critical issues discovered in the 1.11.x release line that have been fixed in 1.11.4. I would not recommend running any prior version on that release line.
1.12.0 should be coming imminently, so if you are going to upgrade anyway, you may want to wait a week or so and get the newest bits with hundreds of new features, but for stability alone, I would strongly recommend 1.11.4.
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Aug 10, 2020, at 10:00 AM, Shawn Weeks <sw...@weeksconsulting.us>> wrote:
I’m running a three node NiFi Cluster on AWS EC2s using integrated Zookeeper and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is 1.8.0_242. For some reason after a period of time Data Provenance goes blank, old records are no longer queryable and new data provenance doesn’t appear to get written. No Lucene or other exceptions are logged and restarting the node causes data provenance to go back to being written however old data provenance does not re-appear. No exceptions appear when querying data provenance. All tests have been run as the initial admin user and roles and permissions appear to be correct. I’ve also checked selinux audit logs to see if something is being blocked but nothing appears.
WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30 seconds for rollover, 100mb rollover size, 2 query threads, 2 index threads, compress on rollover, and don’t sync always.
Thanks
Shawn Weeks
Re: Data Provenance Stops Working
Posted by Andy LoPresto <al...@apache.org>.
Shawn,
I don’t know if this is specifically related, but there were a number of critical issues discovered in the 1.11.x release line that have been fixed in 1.11.4. I would not recommend running any prior version on that release line.
1.12.0 should be coming imminently, so if you are going to upgrade anyway, you may want to wait a week or so and get the newest bits with hundreds of new features, but for stability alone, I would strongly recommend 1.11.4.
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes <https://cwiki.apache.org/confluence/display/NIFI/Release+Notes>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Aug 10, 2020, at 10:00 AM, Shawn Weeks <sw...@weeksconsulting.us> wrote:
>
> I’m running a three node NiFi Cluster on AWS EC2s using integrated Zookeeper and SSL Enabled. Version is 1.11.1, OS is RedHat 7.7, Java is 1.8.0_242. For some reason after a period of time Data Provenance goes blank, old records are no longer queryable and new data provenance doesn’t appear to get written. No Lucene or other exceptions are logged and restarting the node causes data provenance to go back to being written however old data provenance does not re-appear. No exceptions appear when querying data provenance. All tests have been run as the initial admin user and roles and permissions appear to be correct. I’ve also checked selinux audit logs to see if something is being blocked but nothing appears.
>
> WriteAheadProvenanceRepository, max storage is set to 24 hours, 1GB, 30 seconds for rollover, 100mb rollover size, 2 query threads, 2 index threads, compress on rollover, and don’t sync always.
>
> Thanks
> Shawn Weeks