You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Giulia Scalaberni <gi...@genomedics.it> on 2019/08/01 07:57:22 UTC
"Deadlock" data provenance after few days
Hi,
Both on my 1.9.2 and 1.9.0 standalone installations data provenance is
working correctly for nearly a month scheduled once a day then on the
server all the data provenance in all individual processors and in the
general reporting stays empty, even when I send new data through all the
processors manually.
When opening a data provenance menu I see a very short popup menu appearing,
but it closes for me too fast to see what is going on.
On disk I still have more than 100GB available.
The problem goes away when I restart the Nifi instance, and pops up again
after nearly a month.
Does anyone has the same behavior? Have you find a solution?
Thank you in advance.
Have a nice day,
Giulia
Re: "Deadlock" data provenance after few days
Posted by Jo...@swisscom.com.
Hi guys,
I’m jumping in here, we see the same as Giulia, just in another timeframe. We are using the default provenance settings (nothing changed except the path) with 4GB space on the provenance partition. Are there any suggestions what to change? Of course as I’m checking it now we should lift up the two values below to fit better to the partition size of 4GB.
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.index.shard.size =500 MB
What else?
[user@abc ~]$ du -h /provenance_repo/
144K /provenance_repo/toc
273M /provenance_repo/index-1564098506268
400M /provenance_repo/
Thanks in advance,
Josef
From: Mark Payne <ma...@hotmail.com>
Reply to: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Monday, 19 August 2019 at 15:49
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: "Deadlock" data provenance after few days
Giulia,
Looking at the thread dump, it appears that there are no problems with deadlocking or anything like that in Provenance. As Joe mentioned, the settings may need to be adjusted a bit. Can you share what you have in nifi.properties in the Provenance section? Also, it would be helpful to provide the results of running the following command on your provenance repo directory:
du -h provenance_repository/
The default settings that are used for provenance are intended to avoid using a huge amount of disk space. However, they are not ideal for production. The max storage size is set to 1 GB with the shard size set to 500 MB. As a result, depending on timing, it may be possible that the provenance index directories occupy all - or nearly all - of the 1 GB storage size allocated and as a result, all of the event data is aged off. If this happens, you'll see everything functioning properly except that provenance events won't be shown.
Thanks
-Mark
On Aug 19, 2019, at 7:59 AM, Joe Witt <jo...@gmail.com>> wrote:
Just to clarify when you say locked fo you mean that you just cannot get any provenance results but the flow still functions? If so I think your prov settings need to be adjusted. Ill review the thread output when not on a phone.
Thanks
On Mon, Aug 19, 2019 at 4:24 AM Giulia Scalaberni <gi...@genomedics.it>> wrote:
Hi all,
As requested by Joe Witt, here I attach the dump log after the lock.
My Nifi after nearly 30 days of scheduled activity has locked as usual…
BG,
Giulia
Da: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Inviato: giovedì 1 agosto 2019 15:31
A: users@nifi.apache.org<ma...@nifi.apache.org>
Oggetto: Re: "Deadlock" data provenance after few days
Giulia,
When you're experiencing this condition can you capture a thread dump and share the logs (bootstrap/app)? To create this you can run <where-nifi-installed>/bin/nifi.sh dump threaddump-locked-prov.log
Thanks
On Thu, Aug 1, 2019 at 3:57 AM Giulia Scalaberni <gi...@genomedics.it>> wrote:
Hi,
Both on my 1.9.2 and 1.9.0 standalone installations data provenance is working correctly for nearly a month scheduled once a day then on the server all the data provenance in all individual processors and in the general reporting stays empty, even when I send new data through all the processors manually.
When opening a data provenance menu I see a very short popup menu appearing, but it closes for me too fast to see what is going on.
On disk I still have more than 100GB available.
The problem goes away when I restart the Nifi instance, and pops up again after nearly a month.
Does anyone has the same behavior? Have you find a solution?
Thank you in advance.
Have a nice day,
Giulia
Re: "Deadlock" data provenance after few days
Posted by Mark Payne <ma...@hotmail.com>.
Giulia,
Looking at the thread dump, it appears that there are no problems with deadlocking or anything like that in Provenance. As Joe mentioned, the settings may need to be adjusted a bit. Can you share what you have in nifi.properties in the Provenance section? Also, it would be helpful to provide the results of running the following command on your provenance repo directory:
du -h provenance_repository/
The default settings that are used for provenance are intended to avoid using a huge amount of disk space. However, they are not ideal for production. The max storage size is set to 1 GB with the shard size set to 500 MB. As a result, depending on timing, it may be possible that the provenance index directories occupy all - or nearly all - of the 1 GB storage size allocated and as a result, all of the event data is aged off. If this happens, you'll see everything functioning properly except that provenance events won't be shown.
Thanks
-Mark
On Aug 19, 2019, at 7:59 AM, Joe Witt <jo...@gmail.com>> wrote:
Just to clarify when you say locked fo you mean that you just cannot get any provenance results but the flow still functions? If so I think your prov settings need to be adjusted. Ill review the thread output when not on a phone.
Thanks
On Mon, Aug 19, 2019 at 4:24 AM Giulia Scalaberni <gi...@genomedics.it>> wrote:
Hi all,
As requested by Joe Witt, here I attach the dump log after the lock.
My Nifi after nearly 30 days of scheduled activity has locked as usual…
BG,
Giulia
Da: Joe Witt [mailto:joe.witt@gmail.com<ma...@gmail.com>]
Inviato: giovedì 1 agosto 2019 15:31
A: users@nifi.apache.org<ma...@nifi.apache.org>
Oggetto: Re: "Deadlock" data provenance after few days
Giulia,
When you're experiencing this condition can you capture a thread dump and share the logs (bootstrap/app)? To create this you can run <where-nifi-installed>/bin/nifi.sh dump threaddump-locked-prov.log
Thanks
On Thu, Aug 1, 2019 at 3:57 AM Giulia Scalaberni <gi...@genomedics.it>> wrote:
Hi,
Both on my 1.9.2 and 1.9.0 standalone installations data provenance is working correctly for nearly a month scheduled once a day then on the server all the data provenance in all individual processors and in the general reporting stays empty, even when I send new data through all the processors manually.
When opening a data provenance menu I see a very short popup menu appearing, but it closes for me too fast to see what is going on.
On disk I still have more than 100GB available.
The problem goes away when I restart the Nifi instance, and pops up again after nearly a month.
Does anyone has the same behavior? Have you find a solution?
Thank you in advance.
Have a nice day,
Giulia
Re: "Deadlock" data provenance after few days
Posted by Joe Witt <jo...@gmail.com>.
Just to clarify when you say locked fo you mean that you just cannot get
any provenance results but the flow still functions? If so I think your
prov settings need to be adjusted. Ill review the thread output when not
on a phone.
Thanks
On Mon, Aug 19, 2019 at 4:24 AM Giulia Scalaberni <
giulia.scalaberni@genomedics.it> wrote:
> Hi all,
>
> As requested by Joe Witt, here I attach the dump log after the lock.
>
> My Nifi after nearly 30 days of scheduled activity has locked as usual…
>
>
>
> BG,
>
> Giulia
>
>
>
> *Da:* Joe Witt [mailto:joe.witt@gmail.com]
> *Inviato:* giovedì 1 agosto 2019 15:31
> *A:* users@nifi.apache.org
> *Oggetto:* Re: "Deadlock" data provenance after few days
>
>
>
> Giulia,
>
>
>
> When you're experiencing this condition can you capture a thread dump and
> share the logs (bootstrap/app)? To create this you can run
> <where-nifi-installed>/bin/nifi.sh dump threaddump-locked-prov.log
>
>
>
> Thanks
>
>
>
> On Thu, Aug 1, 2019 at 3:57 AM Giulia Scalaberni <
> giulia.scalaberni@genomedics.it> wrote:
>
> Hi,
>
> Both on my 1.9.2 and 1.9.0 standalone installations data provenance is
> working correctly for nearly a month scheduled once a day then on the
> server all the data provenance in all individual processors and in the
> general reporting stays empty, even when I send new data through all the
> processors manually.
>
> When opening a data provenance menu I see a very short popup menu
> appearing, but it closes for me too fast to see what is going on.
>
> On disk I still have more than 100GB available.
>
> The problem goes away when I restart the Nifi instance, and pops up again
> after nearly a month.
>
>
>
>
>
> Does anyone has the same behavior? Have you find a solution?
>
>
>
> Thank you in advance.
>
>
>
> Have a nice day,
>
> Giulia
>
>
R: "Deadlock" data provenance after few days
Posted by Giulia Scalaberni <gi...@genomedics.it>.
Hi all,
As requested by Joe Witt, here I attach the dump log after the lock.
My Nifi after nearly 30 days of scheduled activity has locked as usual…
BG,
Giulia
Da: Joe Witt [mailto:joe.witt@gmail.com]
Inviato: giovedì 1 agosto 2019 15:31
A: users@nifi.apache.org
Oggetto: Re: "Deadlock" data provenance after few days
Giulia,
When you're experiencing this condition can you capture a thread dump and share the logs (bootstrap/app)? To create this you can run <where-nifi-installed>/bin/nifi.sh dump threaddump-locked-prov.log
Thanks
On Thu, Aug 1, 2019 at 3:57 AM Giulia Scalaberni <gi...@genomedics.it> wrote:
Hi,
Both on my 1.9.2 and 1.9.0 standalone installations data provenance is working correctly for nearly a month scheduled once a day then on the server all the data provenance in all individual processors and in the general reporting stays empty, even when I send new data through all the processors manually.
When opening a data provenance menu I see a very short popup menu appearing, but it closes for me too fast to see what is going on.
On disk I still have more than 100GB available.
The problem goes away when I restart the Nifi instance, and pops up again after nearly a month.
Does anyone has the same behavior? Have you find a solution?
Thank you in advance.
Have a nice day,
Giulia
Re: "Deadlock" data provenance after few days
Posted by Erik Anderson <ea...@pobox.com>.
Giulia, not sure if this is helpful or not.
We had similar issues with NiFi 1.9.x but it was because we upgraded 1.9.0 -> 1.9.1 -> 1.9.2. The culprit was ever running 1.9.1. We saw issues if the node ever ran 1.9.1 on any of our NiFi instances then the data providence was corrupted.
"Corrupted" = Providence Events didnt show in the UI, or there was a long delay before the events showed up, or... strange behaviors.
In short what we did was:
1) We created a new 1.9.2 install, used the nifi-toolkit to copy the flow from the repository, and started it again.
2) Set the logback.xml rollingPolicy's totalSizeCap to 10G and 5G
3) Appended a .gz extention in the logback.xml APP_FILE, USER_FILE, BOOTSTRAP_FILE fileNamePattern settings
Its been running months without issues.
I also recommend putting the provenance_repository directory on its own disk partition. If the root partition fills its game over for the NiFi instance. (In reality we put the nifi/content_repository, nifi/conf, nifi/flowfile_repository, nifi/provenance_repository each on their own disk partitions)
Erik Anderson
Bloomberg
On Thu, Aug 1, 2019, at 9:31 AM, Joe Witt wrote:
> Giulia,
>
> When you're experiencing this condition can you capture a thread dump and share the logs (bootstrap/app)? To create this you can run <where-nifi-installed>/bin/nifi.sh dump threaddump-locked-prov.log
>
> Thanks
>
> On Thu, Aug 1, 2019 at 3:57 AM Giulia Scalaberni <gi...@genomedics.it> wrote:
>> Hi,____
>> Both on my 1.9.2 and 1.9.0 standalone installations data provenance is working correctly for nearly a month scheduled once a day then on the server all the data provenance in all individual processors and in the general reporting stays empty, even when I send new data through all the processors manually. ____
>> When opening a data provenance menu I see a very short popup menu appearing, but it closes for me too fast to see what is going on.____
>> On disk I still have more than 100GB available.____
>> The problem goes away when I restart the Nifi instance, and pops up again after nearly a month.____
>> *__ __*
>> *__ __*
>> Does anyone has the same behavior? Have you find a solution?____
>> __ __
>> Thank you in advance.____
>> __ __
>> Have a nice day,____
>> Giulia____
>> ____
Re: "Deadlock" data provenance after few days
Posted by Joe Witt <jo...@gmail.com>.
Giulia,
When you're experiencing this condition can you capture a thread dump and
share the logs (bootstrap/app)? To create this you can run
<where-nifi-installed>/bin/nifi.sh dump threaddump-locked-prov.log
Thanks
On Thu, Aug 1, 2019 at 3:57 AM Giulia Scalaberni <
giulia.scalaberni@genomedics.it> wrote:
> Hi,
>
> Both on my 1.9.2 and 1.9.0 standalone installations data provenance is
> working correctly for nearly a month scheduled once a day then on the
> server all the data provenance in all individual processors and in the
> general reporting stays empty, even when I send new data through all the
> processors manually.
>
> When opening a data provenance menu I see a very short popup menu
> appearing, but it closes for me too fast to see what is going on.
>
> On disk I still have more than 100GB available.
>
> The problem goes away when I restart the Nifi instance, and pops up again
> after nearly a month.
>
>
>
>
>
> Does anyone has the same behavior? Have you find a solution?
>
>
>
> Thank you in advance.
>
>
>
> Have a nice day,
>
> Giulia
>
>