You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Elli Schwarz <el...@yahoo.com> on 2015/10/26 15:57:40 UTC

Determine which Nifi processor eating CPU usage

Hello,
We have a nifi flow with many custom processors (and many processor groups). We suspect that one or more processors are eating up CPU usage, so we're wondering if there's an easy way to tell which processor has a heavy load on the CPU. There are tables to see processors in order of number of flow files or bytes in/out, etc, but not based on CPU usage. In fact, I can't find a way to see a table of all processors that have active threads. All that we know is that the top command has nifi running at 800%, and we're doing trial and error by turning off processors until we hit the one that makes CPU utilization go down.

We did see an earlier post about processors that poll can be eating up CPU cycles, but that doesn't seem to be the case here. Once in the past we had a custom processor with a bug that caused it to eat CPU cycles, but we discovered the issue not through Nifi but because we happened to be examining the code.
Thank you!
-Elli


Re: Determine which Nifi processor eating CPU usage

Posted by Joe Witt <jo...@gmail.com>.
Elli,

In the majority of cases what Mark suggested will help you pinpoint
the offending processors.

If it is not clear which are the offending processes/extension then it
can require some fairly involved digging.  For such things I have
found the following articles provide good examples of how to
potentially work through it:

  https://blogs.oracle.com/jiechen/entry/analysis_against_jvm_thread_dump
  https://blogs.manageengine.com/application-performance-2/appmanager/2011/02/09/identify-java-code-consuming-high-cpu-in-linux-linking-jvm-thread-and-linux-pid.html

And for those really special times you can also, assuming linux, run
'perf top'.  This command is *amazing* albeit reveals very low level
details which aren't always easy to correlate to higher level user
code.

Thanks
Joe

On Mon, Oct 26, 2015 at 11:14 AM, Oleg Zhurakousky
<oz...@hortonworks.com> wrote:
> Unfortunately I can’t seem to even see how it would be possible for NiFi to
> tell you that since your custom Processors are running within the same JVM
> as NiFi.
> Having said that the 800% tells me that you probably have some processor
> with custom thread pool where each thread is spinning in the loop with a lot
> of misses on the functionality it expects to perform.
> For example:
> while (true) {
> if (someCondition){
>             // do something
> }
> }
> The above will definitely eat 100% of your CPU if ’someCondition’ never
> happens and if you have something like this running in multiple threads on 8
> cores there is your 800%.
> That could be your code or some library you are using.
>
> There is also a slight chance that your the code executed by multiple
> threads is actually doing something very CPU intensive
>
> Hope that helps
>
> Oleg
>
>
> On Oct 26, 2015, at 10:57 AM, Elli Schwarz <el...@yahoo.com>
> wrote:
>
> Hello,
>
> We have a nifi flow with many custom processors (and many processor groups).
> We suspect that one or more processors are eating up CPU usage, so we're
> wondering if there's an easy way to tell which processor has a heavy load on
> the CPU. There are tables to see processors in order of number of flow files
> or bytes in/out, etc, but not based on CPU usage. In fact, I can't find a
> way to see a table of all processors that have active threads. All that we
> know is that the top command has nifi running at 800%, and we're doing trial
> and error by turning off processors until we hit the one that makes CPU
> utilization go down.
>
> We did see an earlier post about processors that poll can be eating up CPU
> cycles, but that doesn't seem to be the case here. Once in the past we had a
> custom processor with a bug that caused it to eat CPU cycles, but we
> discovered the issue not through Nifi but because we happened to be
> examining the code.
>
> Thank you!
>
> -Elli
>
>

Determine which Nifi processor eating CPU usage

Posted by Elli Schwarz <el...@yahoo.com>.
Hello,
We have a nifi flow with many custom processors (and many processor groups). We suspect that one or more processors are eating up CPU usage, so we're wondering if there's an easy way to tell which processor has a heavy load on the CPU. There are tables to see processors in order of number of flow files or bytes in/out, etc, but not based on CPU usage. In fact, I can't find a way to see a table of all processors that have active threads. All that we know is that the top command has nifi running at 800%, and we're doing trial and error by turning off processors until we hit the one that makes CPU utilization go down.

We did see an earlier post about processors that poll can be eating up CPU cycles, but that doesn't seem to be the case here. Once in the past we had a custom processor with a bug that caused it to eat CPU cycles, but we discovered the issue not through Nifi but because we happened to be examining the code.
Thank you!
-Elli


Re: Determine which Nifi processor eating CPU usage

Posted by xm...@gmail.com.
This brings up NiFi best practices question: whether a developer can use spawn  threads in processors or controller services? 
If there are any guidelines , it would be nice to document in developer guide. 
Sumanth 


Sent from my iPad

> On Oct 26, 2015, at 8:14 AM, Oleg Zhurakousky <oz...@hortonworks.com> wrote:
> 
> Unfortunately I can’t seem to even see how it would be possible for NiFi to tell you that since your custom Processors are running within the same JVM as NiFi.
> Having said that the 800% tells me that you probably have some processor with custom thread pool where each thread is spinning in the loop with a lot of misses on the functionality it expects to perform.
> For example:
> while (true) {
> if (someCondition){
>             // do something 
> } 
> }
> The above will definitely eat 100% of your CPU if ’someCondition’ never happens and if you have something like this running in multiple threads on 8 cores there is your 800%.
> That could be your code or some library you are using.
> 
> There is also a slight chance that your the code executed by multiple threads is actually doing something very CPU intensive 
> 
> Hope that helps
> 
> Oleg
> 
> 
>> On Oct 26, 2015, at 10:57 AM, Elli Schwarz <el...@yahoo.com> wrote:
>> 
>> Hello,
>> 
>> We have a nifi flow with many custom processors (and many processor groups). We suspect that one or more processors are eating up CPU usage, so we're wondering if there's an easy way to tell which processor has a heavy load on the CPU. There are tables to see processors in order of number of flow files or bytes in/out, etc, but not based on CPU usage. In fact, I can't find a way to see a table of all processors that have active threads. All that we know is that the top command has nifi running at 800%, and we're doing trial and error by turning off processors until we hit the one that makes CPU utilization go down.
>> 
>> We did see an earlier post about processors that poll can be eating up CPU cycles, but that doesn't seem to be the case here. Once in the past we had a custom processor with a bug that caused it to eat CPU cycles, but we discovered the issue not through Nifi but because we happened to be examining the code.
>> 
>> Thank you!
>> 
>> -Elli
>