You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Panos Geo <pa...@outlook.com> on 2017/11/24 15:46:51 UTC
NiFi stops processing further flowfiles
Hello all,
We are running NiFi in a docker container that has roughly 500 processors running. These include consuming messages (MQTT), interaction with a database and some data processing with execute scripts. Our NiFi version is v.1.4.0 and this instance has 18GB of RAM with 12GB allocated to Java heap.
Quite recently we increased the load of the incoming messages (1000messages/minute) that lead to more files being queued and some backpressure. We then have experienced what appears to be some kind of deadlock. NiFi stops processing any flowfiles and the processors running get stuck (there is a thread running constantly; we see 1 at the top right corner of the processor). If we try to stop these processors the icon gets changed to stop, but we now have 2 threads stuck at the top right corner of the processor. We cannot start these processors again and no other data flow is being processed unless we restart NiFi. We have experienced this multiple times over the past week.
I am attaching a nifi dump in case it helps. Our nifi.properties is fairly standard (we just have smaller retention periods for content and provenance).
Any thoughts? What can we do to debug this further?
Many thanks,
Panos
Re: NiFi stops processing further flowfiles
Posted by Panos Geo <pa...@outlook.com>.
Hello Juan,
Thanks for your reply. I'll keep an eye out on the upper left corner of the UI indeed 😉
Cheers,
Panos
________________________________
From: Juan Sequeiros <he...@gmail.com>
Sent: Friday, November 24, 2017 6:59 PM
To: users@nifi.apache.org
Subject: Re: NiFi stops processing further flowfiles
Hi Panos,
The scheduling strategy defaults to timer driven on processors and I've always avoided using event driven so if you are not purposely using it I would just leave that setting alone. It even warns you that it is experimental.
Your thread count is shown on the UI or can be called through RestApi /nifi-api/flow/status
Thanks,
Juan
On Fri, Nov 24, 2017 at 1:24 PM Panos Geo <pa...@outlook.com>> wrote:
Thanks for your reply Juan.
I have been using NiFi for almost 2 years now and didn't know about this setting. I 'll try to increase it and see what happens 😊
Does it also make sense to increase the "Maximum Event Driven Thread Count"?
Finally, is there a way to monitor thread usage within NiFi?
Many thanks,
Panos
________________________________
From: Juan Sequeiros <he...@gmail.com>>
Sent: Friday, November 24, 2017 4:50 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi stops processing further flowfiles
Good afternoon,
My quick guess is you are running out of threads.
If your system can handle it increase the default "max thread" count found:
Left side controller settings > Maximum timer driven thread count
On Fri, Nov 24, 2017 at 10:47 AM Panos Geo <pa...@outlook.com>> wrote:
Hello all,
We are running NiFi in a docker container that has roughly 500 processors running. These include consuming messages (MQTT), interaction with a database and some data processing with execute scripts. Our NiFi version is v.1.4.0 and this instance has 18GB of RAM with 12GB allocated to Java heap.
Quite recently we increased the load of the incoming messages (1000messages/minute) that lead to more files being queued and some backpressure. We then have experienced what appears to be some kind of deadlock. NiFi stops processing any flowfiles and the processors running get stuck (there is a thread running constantly; we see 1 at the top right corner of the processor). If we try to stop these processors the icon gets changed to stop, but we now have 2 threads stuck at the top right corner of the processor. We cannot start these processors again and no other data flow is being processed unless we restart NiFi. We have experienced this multiple times over the past week.
I am attaching a nifi dump in case it helps. Our nifi.properties is fairly standard (we just have smaller retention periods for content and provenance).
Any thoughts? What can we do to debug this further?
Many thanks,
Panos
Re: NiFi stops processing further flowfiles
Posted by Joe Witt <jo...@gmail.com>.
Panos
Thanks - we should change the default that is used at some point. The old
one has been stable/solid for a long time though so switching it will need
to be a deliberate choice as well. Was thinking it made sense to do at the
next major release but perhaps at one of the upcoming minors it would be
good too. The new repo honors the old data structure so migration should
be trivial.
Thanks
On Wed, Nov 29, 2017 at 1:04 AM, Panos Geo <pa...@outlook.com>
wrote:
> Hello Joe,
>
>
> A quick message to say that we switched to 'WriteAheadProvenanceRepository'
> and we don't see NiFi getting stuck anymore. Perhaps this would help other
> people too 😉
>
>
> Many thanks,
>
> Panos
>
>
> ------------------------------
> *From:* Joe Witt <jo...@gmail.com>
> *Sent:* Saturday, November 25, 2017 3:05 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi stops processing further flowfiles
>
> Panos
>
> In looking at the previously attached stack dump it looks as though the
> provenance repository might be in a live lock state with one of its
> threads. The getSize call it is sitting on has been seen before and fixed
> for some cases but perhaps not all. There is a whole new provenance
> repository you could switch to though which offers far faster and possibly
> more stable behavior called the 'WriteAheadProvenanceRepository'. You
> can switch to it and it will honor the old data I believe. To switch you
> just edit your nifi.properties file and where it said '
> PersistentProvenanceRepository' you change that classname to '
> WriteAheadProvenanceRepository'.
>
> Thanks
> Joe
>
> On Fri, Nov 24, 2017 at 1:59 PM, Juan Sequeiros <he...@gmail.com>
> wrote:
>
> Hi Panos,
>
> The scheduling strategy defaults to timer driven on processors and I've
> always avoided using event driven so if you are not purposely using it I
> would just leave that setting alone. It even warns you that it is
> experimental.
>
> Your thread count is shown on the UI or can be called through RestApi
> /nifi-api/flow/status
>
> Thanks,
>
> Juan
>
> On Fri, Nov 24, 2017 at 1:24 PM Panos Geo <pa...@outlook.com>
> wrote:
>
> Thanks for your reply Juan.
>
>
> I have been using NiFi for almost 2 years now and didn't know about this
> setting. I 'll try to increase it and see what happens 😊
>
>
> Does it also make sense to increase the "Maximum Event Driven Thread
> Count"?
>
>
> Finally, is there a way to monitor thread usage within NiFi?
>
>
> Many thanks,
>
> Panos
>
>
>
> ------------------------------
> *From:* Juan Sequeiros <he...@gmail.com>
> *Sent:* Friday, November 24, 2017 4:50 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi stops processing further flowfiles
>
> Good afternoon,
>
> My quick guess is you are running out of threads.
> If your system can handle it increase the default "max thread" count found:
>
> Left side controller settings > Maximum timer driven thread count
>
> On Fri, Nov 24, 2017 at 10:47 AM Panos Geo <pa...@outlook.com>
> wrote:
>
> Hello all,
>
> We are running NiFi in a docker container that has roughly 500 processors
> running. These include consuming messages (MQTT), interaction with a
> database and some data processing with execute scripts. Our NiFi version is
> v.1.4.0 and this instance has 18GB of RAM with 12GB allocated to Java heap.
>
> Quite recently we increased the load of the incoming messages
> (1000messages/minute) that lead to more files being queued and some
> backpressure. We then have experienced what appears to be some kind of
> deadlock. NiFi stops processing any flowfiles and the processors running
> get stuck (there is a thread running constantly; we see 1 at the top right
> corner of the processor). If we try to stop these processors the icon gets
> changed to stop, but we now have 2 threads stuck at the top right corner of
> the processor. We cannot start these processors again and no other data
> flow is being processed unless we restart NiFi. We have experienced this
> multiple times over the past week.
>
> I am attaching a nifi dump in case it helps. Our nifi.properties is fairly
> standard (we just have smaller retention periods for content and
> provenance).
>
> Any thoughts? What can we do to debug this further?
>
> Many thanks,
> Panos
>
>
>
Re: NiFi stops processing further flowfiles
Posted by Panos Geo <pa...@outlook.com>.
Hello Joe,
A quick message to say that we switched to 'WriteAheadProvenanceRepository' and we don't see NiFi getting stuck anymore. Perhaps this would help other people too 😉
Many thanks,
Panos
________________________________
From: Joe Witt <jo...@gmail.com>
Sent: Saturday, November 25, 2017 3:05 AM
To: users@nifi.apache.org
Subject: Re: NiFi stops processing further flowfiles
Panos
In looking at the previously attached stack dump it looks as though the provenance repository might be in a live lock state with one of its threads. The getSize call it is sitting on has been seen before and fixed for some cases but perhaps not all. There is a whole new provenance repository you could switch to though which offers far faster and possibly more stable behavior called the 'WriteAheadProvenanceRepository'. You can switch to it and it will honor the old data I believe. To switch you just edit your nifi.properties file and where it said 'PersistentProvenanceRepository' you change that classname to 'WriteAheadProvenanceRepository'.
Thanks
Joe
On Fri, Nov 24, 2017 at 1:59 PM, Juan Sequeiros <he...@gmail.com>> wrote:
Hi Panos,
The scheduling strategy defaults to timer driven on processors and I've always avoided using event driven so if you are not purposely using it I would just leave that setting alone. It even warns you that it is experimental.
Your thread count is shown on the UI or can be called through RestApi /nifi-api/flow/status
Thanks,
Juan
On Fri, Nov 24, 2017 at 1:24 PM Panos Geo <pa...@outlook.com>> wrote:
Thanks for your reply Juan.
I have been using NiFi for almost 2 years now and didn't know about this setting. I 'll try to increase it and see what happens 😊
Does it also make sense to increase the "Maximum Event Driven Thread Count"?
Finally, is there a way to monitor thread usage within NiFi?
Many thanks,
Panos
________________________________
From: Juan Sequeiros <he...@gmail.com>>
Sent: Friday, November 24, 2017 4:50 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi stops processing further flowfiles
Good afternoon,
My quick guess is you are running out of threads.
If your system can handle it increase the default "max thread" count found:
Left side controller settings > Maximum timer driven thread count
On Fri, Nov 24, 2017 at 10:47 AM Panos Geo <pa...@outlook.com>> wrote:
Hello all,
We are running NiFi in a docker container that has roughly 500 processors running. These include consuming messages (MQTT), interaction with a database and some data processing with execute scripts. Our NiFi version is v.1.4.0 and this instance has 18GB of RAM with 12GB allocated to Java heap.
Quite recently we increased the load of the incoming messages (1000messages/minute) that lead to more files being queued and some backpressure. We then have experienced what appears to be some kind of deadlock. NiFi stops processing any flowfiles and the processors running get stuck (there is a thread running constantly; we see 1 at the top right corner of the processor). If we try to stop these processors the icon gets changed to stop, but we now have 2 threads stuck at the top right corner of the processor. We cannot start these processors again and no other data flow is being processed unless we restart NiFi. We have experienced this multiple times over the past week.
I am attaching a nifi dump in case it helps. Our nifi.properties is fairly standard (we just have smaller retention periods for content and provenance).
Any thoughts? What can we do to debug this further?
Many thanks,
Panos
Re: NiFi stops processing further flowfiles
Posted by Panos Geo <pa...@outlook.com>.
Hello Joe,
Many thanks for your reply and for checking the stack dump. I was trying hard to remember where I've read about the new implementation of the provenance repo. Thanks for the pointer. We'll check it out to see if it yields more stable behaviour.
Cheers,
Panos
________________________________
From: Joe Witt <jo...@gmail.com>
Sent: Saturday, November 25, 2017 3:05 AM
To: users@nifi.apache.org
Subject: Re: NiFi stops processing further flowfiles
Panos
In looking at the previously attached stack dump it looks as though the provenance repository might be in a live lock state with one of its threads. The getSize call it is sitting on has been seen before and fixed for some cases but perhaps not all. There is a whole new provenance repository you could switch to though which offers far faster and possibly more stable behavior called the 'WriteAheadProvenanceRepository'. You can switch to it and it will honor the old data I believe. To switch you just edit your nifi.properties file and where it said 'PersistentProvenanceRepository' you change that classname to 'WriteAheadProvenanceRepository'.
Thanks
Joe
On Fri, Nov 24, 2017 at 1:59 PM, Juan Sequeiros <he...@gmail.com>> wrote:
Hi Panos,
The scheduling strategy defaults to timer driven on processors and I've always avoided using event driven so if you are not purposely using it I would just leave that setting alone. It even warns you that it is experimental.
Your thread count is shown on the UI or can be called through RestApi /nifi-api/flow/status
Thanks,
Juan
On Fri, Nov 24, 2017 at 1:24 PM Panos Geo <pa...@outlook.com>> wrote:
Thanks for your reply Juan.
I have been using NiFi for almost 2 years now and didn't know about this setting. I 'll try to increase it and see what happens 😊
Does it also make sense to increase the "Maximum Event Driven Thread Count"?
Finally, is there a way to monitor thread usage within NiFi?
Many thanks,
Panos
________________________________
From: Juan Sequeiros <he...@gmail.com>>
Sent: Friday, November 24, 2017 4:50 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi stops processing further flowfiles
Good afternoon,
My quick guess is you are running out of threads.
If your system can handle it increase the default "max thread" count found:
Left side controller settings > Maximum timer driven thread count
On Fri, Nov 24, 2017 at 10:47 AM Panos Geo <pa...@outlook.com>> wrote:
Hello all,
We are running NiFi in a docker container that has roughly 500 processors running. These include consuming messages (MQTT), interaction with a database and some data processing with execute scripts. Our NiFi version is v.1.4.0 and this instance has 18GB of RAM with 12GB allocated to Java heap.
Quite recently we increased the load of the incoming messages (1000messages/minute) that lead to more files being queued and some backpressure. We then have experienced what appears to be some kind of deadlock. NiFi stops processing any flowfiles and the processors running get stuck (there is a thread running constantly; we see 1 at the top right corner of the processor). If we try to stop these processors the icon gets changed to stop, but we now have 2 threads stuck at the top right corner of the processor. We cannot start these processors again and no other data flow is being processed unless we restart NiFi. We have experienced this multiple times over the past week.
I am attaching a nifi dump in case it helps. Our nifi.properties is fairly standard (we just have smaller retention periods for content and provenance).
Any thoughts? What can we do to debug this further?
Many thanks,
Panos
Re: NiFi stops processing further flowfiles
Posted by Joe Witt <jo...@gmail.com>.
Panos
In looking at the previously attached stack dump it looks as though the
provenance repository might be in a live lock state with one of its
threads. The getSize call it is sitting on has been seen before and fixed
for some cases but perhaps not all. There is a whole new provenance
repository you could switch to though which offers far faster and possibly
more stable behavior called the 'WriteAheadProvenanceRepository'. You can
switch to it and it will honor the old data I believe. To switch you just
edit your nifi.properties file and where it said
'PersistentProvenanceRepository' you change that classname to
'WriteAheadProvenanceRepository'.
Thanks
Joe
On Fri, Nov 24, 2017 at 1:59 PM, Juan Sequeiros <he...@gmail.com> wrote:
> Hi Panos,
>
> The scheduling strategy defaults to timer driven on processors and I've
> always avoided using event driven so if you are not purposely using it I
> would just leave that setting alone. It even warns you that it is
> experimental.
>
> Your thread count is shown on the UI or can be called through RestApi
> /nifi-api/flow/status
>
> Thanks,
>
> Juan
>
> On Fri, Nov 24, 2017 at 1:24 PM Panos Geo <pa...@outlook.com>
> wrote:
>
>> Thanks for your reply Juan.
>>
>>
>> I have been using NiFi for almost 2 years now and didn't know about this
>> setting. I 'll try to increase it and see what happens 😊
>>
>>
>> Does it also make sense to increase the "Maximum Event Driven Thread
>> Count"?
>>
>>
>> Finally, is there a way to monitor thread usage within NiFi?
>>
>>
>> Many thanks,
>>
>> Panos
>>
>>
>>
>> ------------------------------
>> *From:* Juan Sequeiros <he...@gmail.com>
>> *Sent:* Friday, November 24, 2017 4:50 PM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: NiFi stops processing further flowfiles
>>
>> Good afternoon,
>>
>> My quick guess is you are running out of threads.
>> If your system can handle it increase the default "max thread" count
>> found:
>>
>> Left side controller settings > Maximum timer driven thread count
>>
>> On Fri, Nov 24, 2017 at 10:47 AM Panos Geo <pa...@outlook.com>
>> wrote:
>>
>> Hello all,
>>
>> We are running NiFi in a docker container that has roughly 500 processors
>> running. These include consuming messages (MQTT), interaction with a
>> database and some data processing with execute scripts. Our NiFi version is
>> v.1.4.0 and this instance has 18GB of RAM with 12GB allocated to Java heap.
>>
>> Quite recently we increased the load of the incoming messages
>> (1000messages/minute) that lead to more files being queued and some
>> backpressure. We then have experienced what appears to be some kind of
>> deadlock. NiFi stops processing any flowfiles and the processors running
>> get stuck (there is a thread running constantly; we see 1 at the top right
>> corner of the processor). If we try to stop these processors the icon gets
>> changed to stop, but we now have 2 threads stuck at the top right corner of
>> the processor. We cannot start these processors again and no other data
>> flow is being processed unless we restart NiFi. We have experienced this
>> multiple times over the past week.
>>
>> I am attaching a nifi dump in case it helps. Our nifi.properties is
>> fairly standard (we just have smaller retention periods for content and
>> provenance).
>>
>> Any thoughts? What can we do to debug this further?
>>
>> Many thanks,
>> Panos
>>
>>
Re: NiFi stops processing further flowfiles
Posted by Juan Sequeiros <he...@gmail.com>.
Hi Panos,
The scheduling strategy defaults to timer driven on processors and I've
always avoided using event driven so if you are not purposely using it I
would just leave that setting alone. It even warns you that it is
experimental.
Your thread count is shown on the UI or can be called through RestApi
/nifi-api/flow/status
Thanks,
Juan
On Fri, Nov 24, 2017 at 1:24 PM Panos Geo <pa...@outlook.com>
wrote:
> Thanks for your reply Juan.
>
>
> I have been using NiFi for almost 2 years now and didn't know about this
> setting. I 'll try to increase it and see what happens 😊
>
>
> Does it also make sense to increase the "Maximum Event Driven Thread
> Count"?
>
>
> Finally, is there a way to monitor thread usage within NiFi?
>
>
> Many thanks,
>
> Panos
>
>
>
> ------------------------------
> *From:* Juan Sequeiros <he...@gmail.com>
> *Sent:* Friday, November 24, 2017 4:50 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: NiFi stops processing further flowfiles
>
> Good afternoon,
>
> My quick guess is you are running out of threads.
> If your system can handle it increase the default "max thread" count found:
>
> Left side controller settings > Maximum timer driven thread count
>
> On Fri, Nov 24, 2017 at 10:47 AM Panos Geo <pa...@outlook.com>
> wrote:
>
> Hello all,
>
> We are running NiFi in a docker container that has roughly 500 processors
> running. These include consuming messages (MQTT), interaction with a
> database and some data processing with execute scripts. Our NiFi version is
> v.1.4.0 and this instance has 18GB of RAM with 12GB allocated to Java heap.
>
> Quite recently we increased the load of the incoming messages
> (1000messages/minute) that lead to more files being queued and some
> backpressure. We then have experienced what appears to be some kind of
> deadlock. NiFi stops processing any flowfiles and the processors running
> get stuck (there is a thread running constantly; we see 1 at the top right
> corner of the processor). If we try to stop these processors the icon gets
> changed to stop, but we now have 2 threads stuck at the top right corner of
> the processor. We cannot start these processors again and no other data
> flow is being processed unless we restart NiFi. We have experienced this
> multiple times over the past week.
>
> I am attaching a nifi dump in case it helps. Our nifi.properties is fairly
> standard (we just have smaller retention periods for content and
> provenance).
>
> Any thoughts? What can we do to debug this further?
>
> Many thanks,
> Panos
>
>
Re: NiFi stops processing further flowfiles
Posted by Panos Geo <pa...@outlook.com>.
Thanks for your reply Juan.
I have been using NiFi for almost 2 years now and didn't know about this setting. I 'll try to increase it and see what happens 😊
Does it also make sense to increase the "Maximum Event Driven Thread Count"?
Finally, is there a way to monitor thread usage within NiFi?
Many thanks,
Panos
________________________________
From: Juan Sequeiros <he...@gmail.com>
Sent: Friday, November 24, 2017 4:50 PM
To: users@nifi.apache.org
Subject: Re: NiFi stops processing further flowfiles
Good afternoon,
My quick guess is you are running out of threads.
If your system can handle it increase the default "max thread" count found:
Left side controller settings > Maximum timer driven thread count
On Fri, Nov 24, 2017 at 10:47 AM Panos Geo <pa...@outlook.com>> wrote:
Hello all,
We are running NiFi in a docker container that has roughly 500 processors running. These include consuming messages (MQTT), interaction with a database and some data processing with execute scripts. Our NiFi version is v.1.4.0 and this instance has 18GB of RAM with 12GB allocated to Java heap.
Quite recently we increased the load of the incoming messages (1000messages/minute) that lead to more files being queued and some backpressure. We then have experienced what appears to be some kind of deadlock. NiFi stops processing any flowfiles and the processors running get stuck (there is a thread running constantly; we see 1 at the top right corner of the processor). If we try to stop these processors the icon gets changed to stop, but we now have 2 threads stuck at the top right corner of the processor. We cannot start these processors again and no other data flow is being processed unless we restart NiFi. We have experienced this multiple times over the past week.
I am attaching a nifi dump in case it helps. Our nifi.properties is fairly standard (we just have smaller retention periods for content and provenance).
Any thoughts? What can we do to debug this further?
Many thanks,
Panos
Re: NiFi stops processing further flowfiles
Posted by Juan Sequeiros <he...@gmail.com>.
Good afternoon,
My quick guess is you are running out of threads.
If your system can handle it increase the default "max thread" count found:
Left side controller settings > Maximum timer driven thread count
On Fri, Nov 24, 2017 at 10:47 AM Panos Geo <pa...@outlook.com>
wrote:
> Hello all,
>
> We are running NiFi in a docker container that has roughly 500 processors
> running. These include consuming messages (MQTT), interaction with a
> database and some data processing with execute scripts. Our NiFi version is
> v.1.4.0 and this instance has 18GB of RAM with 12GB allocated to Java heap.
>
> Quite recently we increased the load of the incoming messages
> (1000messages/minute) that lead to more files being queued and some
> backpressure. We then have experienced what appears to be some kind of
> deadlock. NiFi stops processing any flowfiles and the processors running
> get stuck (there is a thread running constantly; we see 1 at the top right
> corner of the processor). If we try to stop these processors the icon gets
> changed to stop, but we now have 2 threads stuck at the top right corner of
> the processor. We cannot start these processors again and no other data
> flow is being processed unless we restart NiFi. We have experienced this
> multiple times over the past week.
>
> I am attaching a nifi dump in case it helps. Our nifi.properties is fairly
> standard (we just have smaller retention periods for content and
> provenance).
>
> Any thoughts? What can we do to debug this further?
>
> Many thanks,
> Panos
>
>