You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Joe Gresock <jg...@gmail.com> on 2019/06/09 21:35:15 UTC
Keeping NiFi 1.9.2 console available
I posted about this a while back on 1.6.0, but as far as I can tell it has
only gotten worse in 1.9.2.
I have a cluster of 7 nifi nodes running on CentOS 6 VMs. When no data is
being processed, the UI console is very responsive. However, once I start
processing lots of data, the CPU gets maxed by "user" processes and the
console can no longer be used (either one or more node gets disconnected
due to timeouts talking to the rest of the cluster, or the UI simply goes
away).
The impression I get is that threads are so prioritized for the processes
that nothing can be spared for interacting with the console, or keeping the
nodes connected to each other.
Now, my latest idea would be to decrease the number of threads that my
processors use, to try to give more CPU to the cluster overhead.. does this
seem like a reasonable approach?
A separate observation is that in 1.6.0, it usually took about a minute for
all of the nodes to connect enough that the UI was available, starting from
when the last node became officially "up". However, in 1.9.2, this takes
about 15 minutes for the UI to become usable, during which time the UI
claims no nodes are connected.
I'm happy to provide thread dumps or other debugging output if anyone has
an idea of how to improve this behavior.
Thanks!
Joe
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon>
Virus-free.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: Keeping NiFi 1.9.2 console available
Posted by Joe Gresock <jg...@gmail.com>.
In htop, nifi is using between 50-95% of the CPU during heavy processing.
Once the UI comes back up, I can see that the G1 Young Generation fired
about 200 times since I started the cluster up 45 minutes ago, and it says
this was between 3-7 minutes of GC time (if I'm reading that right).
After the load died down, the heap has between 24-58% used, of a 24GB heap.
I tried out decreasing the max number of timer threads, per [1], which I
suspect was a big part of my problem: We had our max timer driven threads
set to 150, which is way more than the recommended 2-4 times number of
cores. I decreased this to 32 and saw the above results. Similar behavior
happened as before, but to a lesser extent, so I suspect I need to decrease
the max threads again and keep trying.
[1]
https://community.hortonworks.com/articles/221808/understanding-nifi-max-thread-pools-and-processor.html
On Mon, Jun 10, 2019 at 12:54 PM Joe Witt <jo...@gmail.com> wrote:
> ...also how does the heap and garbage collection look during this?
>
> On Sun, Jun 9, 2019, 5:58 PM Joe Witt <jo...@gmail.com> wrote:
>
>> Joe
>>
>> When you view top or other tools what is dominating the cpu?
>>
>> thanks
>> joe
>>
>> On Sun, Jun 9, 2019, 5:35 PM Joe Gresock <jg...@gmail.com> wrote:
>>
>>> I posted about this a while back on 1.6.0, but as far as I can tell it
>>> has only gotten worse in 1.9.2.
>>>
>>> I have a cluster of 7 nifi nodes running on CentOS 6 VMs. When no data
>>> is being processed, the UI console is very responsive. However, once I
>>> start processing lots of data, the CPU gets maxed by "user" processes and
>>> the console can no longer be used (either one or more node gets
>>> disconnected due to timeouts talking to the rest of the cluster, or the UI
>>> simply goes away).
>>>
>>> The impression I get is that threads are so prioritized for the
>>> processes that nothing can be spared for interacting with the console, or
>>> keeping the nodes connected to each other.
>>>
>>> Now, my latest idea would be to decrease the number of threads that my
>>> processors use, to try to give more CPU to the cluster overhead.. does this
>>> seem like a reasonable approach?
>>>
>>> A separate observation is that in 1.6.0, it usually took about a minute
>>> for all of the nodes to connect enough that the UI was available, starting
>>> from when the last node became officially "up". However, in 1.9.2, this
>>> takes about 15 minutes for the UI to become usable, during which time the
>>> UI claims no nodes are connected.
>>>
>>> I'm happy to provide thread dumps or other debugging output if anyone
>>> has an idea of how to improve this behavior.
>>>
>>> Thanks!
>>> Joe
>>>
>>>
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free.
>>> www.avast.com
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
>>> <#m_-5640816275201314745_m_-4644020240664756686_m_-5737275650488698501_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>
>>
--
I know what it is to be in need, and I know what it is to have plenty. I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want. I can do
all this through him who gives me strength. *-Philippians 4:12-13*
Re: Keeping NiFi 1.9.2 console available
Posted by Joe Witt <jo...@gmail.com>.
...also how does the heap and garbage collection look during this?
On Sun, Jun 9, 2019, 5:58 PM Joe Witt <jo...@gmail.com> wrote:
> Joe
>
> When you view top or other tools what is dominating the cpu?
>
> thanks
> joe
>
> On Sun, Jun 9, 2019, 5:35 PM Joe Gresock <jg...@gmail.com> wrote:
>
>> I posted about this a while back on 1.6.0, but as far as I can tell it
>> has only gotten worse in 1.9.2.
>>
>> I have a cluster of 7 nifi nodes running on CentOS 6 VMs. When no data
>> is being processed, the UI console is very responsive. However, once I
>> start processing lots of data, the CPU gets maxed by "user" processes and
>> the console can no longer be used (either one or more node gets
>> disconnected due to timeouts talking to the rest of the cluster, or the UI
>> simply goes away).
>>
>> The impression I get is that threads are so prioritized for the processes
>> that nothing can be spared for interacting with the console, or keeping the
>> nodes connected to each other.
>>
>> Now, my latest idea would be to decrease the number of threads that my
>> processors use, to try to give more CPU to the cluster overhead.. does this
>> seem like a reasonable approach?
>>
>> A separate observation is that in 1.6.0, it usually took about a minute
>> for all of the nodes to connect enough that the UI was available, starting
>> from when the last node became officially "up". However, in 1.9.2, this
>> takes about 15 minutes for the UI to become usable, during which time the
>> UI claims no nodes are connected.
>>
>> I'm happy to provide thread dumps or other debugging output if anyone has
>> an idea of how to improve this behavior.
>>
>> Thanks!
>> Joe
>>
>>
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free.
>> www.avast.com
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
>> <#m_-4644020240664756686_m_-5737275650488698501_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
>
Re: Keeping NiFi 1.9.2 console available
Posted by Joe Witt <jo...@gmail.com>.
Joe
When you view top or other tools what is dominating the cpu?
thanks
joe
On Sun, Jun 9, 2019, 5:35 PM Joe Gresock <jg...@gmail.com> wrote:
> I posted about this a while back on 1.6.0, but as far as I can tell it has
> only gotten worse in 1.9.2.
>
> I have a cluster of 7 nifi nodes running on CentOS 6 VMs. When no data is
> being processed, the UI console is very responsive. However, once I start
> processing lots of data, the CPU gets maxed by "user" processes and the
> console can no longer be used (either one or more node gets disconnected
> due to timeouts talking to the rest of the cluster, or the UI simply goes
> away).
>
> The impression I get is that threads are so prioritized for the processes
> that nothing can be spared for interacting with the console, or keeping the
> nodes connected to each other.
>
> Now, my latest idea would be to decrease the number of threads that my
> processors use, to try to give more CPU to the cluster overhead.. does this
> seem like a reasonable approach?
>
> A separate observation is that in 1.6.0, it usually took about a minute
> for all of the nodes to connect enough that the UI was available, starting
> from when the last node became officially "up". However, in 1.9.2, this
> takes about 15 minutes for the UI to become usable, during which time the
> UI claims no nodes are connected.
>
> I'm happy to provide thread dumps or other debugging output if anyone has
> an idea of how to improve this behavior.
>
> Thanks!
> Joe
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
> <#m_-5737275650488698501_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>