You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Mike Harding <mi...@gmail.com> on 2016/07/14 14:34:03 UTC
nifi process running at 114% of node CPU !!?
Hi All,
The node in my cluster running nifi crashed due to a CPU overload event.
After restarting I analysed the CPU consumption and found that nifi was the
issue. As you can see below it was running at 133% CPU:
PID USER PR NI VIRT RES SHR S *%CPU* %MEM TIME+ COMMAND
2031 nifi 20 0 3392960 990.7m 34124 S *113.4* 12.4 1179:47 java]
I have only one data pipe line setup that is receiving data through a
HandleHTTPRequest processor and after playing around and turning other
processors off in the pipe it was only when I stopped this process that the
CPU dropped significantly to around 10% CPU.
Its receiving around 67KB of data every 5 minutes from multiple requests
from a up stream web app.
Has any one else seen this behaviour and or know whether there are ways of
managing the CPU usage of HandleHTTPRequest ?
Thanks,
Mike
Re: nifi process running at 114% of node CPU !!?
Posted by Mike Harding <mi...@gmail.com>.
Hi Mike,
Thanks for creating the JIRA - I'm unable to create a JIRA as I'm away on
holiday at the minute.
Mike
On Fri, 15 Jul 2016 at 14:30, Michael Moser <mo...@gmail.com> wrote:
> I went ahead and created NIFI-2268 for this, since it was fresh in my
> mind. ListenHTTP calls ProcessContext.yield() whenever it doesn't have
> work to do, so HandleHttpRequest could do the same.
>
> -- Mike
>
>
> On Thu, Jul 14, 2016 at 7:00 PM, Joe Witt <jo...@gmail.com> wrote:
>
>> Mike,
>>
>> If you don't mind could you file a JIRA for this. Frankly it sounds
>> like a bug to me. We should consider making a default scheduling
>> period of something a bit slower. Frankly just dialing back to 100 ms
>> would be sufficient most likely. If you agree this is a bug please
>> file one here: https://issues.apache.org/jira/browse/NIFI
>>
>> If you could attach a template of the flow that behaves badly and the
>> one that behaves better that would be ideal but if not just a good
>> description should do.
>>
>> Thanks
>> Joe
>>
>> On Thu, Jul 14, 2016 at 6:57 PM, Mike Harding <mi...@gmail.com>
>> wrote:
>> > Thanks all - I checked the logs and there is nothing I can see thats
>> seems
>> > erroneous. I increased the number of threads for the processor and
>> added the
>> > 10 second scheduling and it has dropped dramatically from 2.5M tasks to
>> 300
>> > over 5 minute period. CPU for the nifi java process is now running at
>> 8-10%
>> > CPU.
>> >
>> > I don't think I saw this issue when using HTTPListen processor which I
>> > recently from to HttpRequestHandle.
>> >
>> > Cheers,
>> > Mike
>> >
>> > On 14 July 2016 at 16:41, Aldrin Piri <al...@gmail.com> wrote:
>> >>
>> >> Mike,
>> >>
>> >> To add some context, while NiFi will intelligently schedule processors
>> to
>> >> execute, given HandleHTTPRequest's function as a listener, it is
>> constantly
>> >> scheduled to run, checking for a request to handle. I assume by
>> number of
>> >> tasks, you mean the rolling count over the last 5 minutes. As
>> mentioned by
>> >> Andy, you can tamper this rate by increasing the run scheduld if the
>> >> handling of the HTTP requests with a slight latency is acceptable to
>> you and
>> >> your needs.
>> >>
>> >>
>> >> On Thu, Jul 14, 2016 at 11:05 AM Andy LoPresto <al...@apache.org>
>> >> wrote:
>> >>>
>> >>> Mike,
>> >>>
>> >>> You can adjust the processor properties for the HandleHTTPRequest
>> >>> processor in the scheduling tab.
>> >>>
>> >>> “Concurrent tasks” limits the number of threads this processor will
>> use
>> >>> (default is 1)
>> >>> “Run schedule” determines the frequency that this processor will be
>> run
>> >>> (default is ‘0 sec’ which means continuously)
>> >>>
>> >>> If you are only getting requests on a much slower schedule, you could
>> >>> reduce the run schedule to ~10 seconds and see if this is better for
>> you. I
>> >>> have not encountered NiFi running at such high CPU percentage with
>> that
>> >>> little data.
>> >>>
>> >>> As for the high number of tasks, that is definitely an anomaly.
>> >>> Configuration best practices [1] currently recommend increasing the
>> limit to
>> >>> the 10k range, but 2.5M for a single processor is unusual. Can you
>> inspect
>> >>> the logs (located in $NIFI_HOME/logs) to see if there are errors or
>> more
>> >>> insight there?
>> >>>
>> >>> [1]
>> >>>
>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices
>> >>>
>> >>>
>> >>>
>> >>> Andy LoPresto
>> >>> alopresto@apache.org
>> >>> alopresto.apache@gmail.com
>> >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>> >>>
>> >>> On Jul 14, 2016, at 10:36 AM, Mike Harding <mi...@gmail.com>
>> >>> wrote:
>> >>>
>> >>> ps - also noticed it seems to generate a lot of tasks, currently 2.5M
>> >>> compared to other processes in the pipeline which reports 10s of
>> tasks.
>> >>>
>> >>> Mike
>> >>>
>> >>> On 14 July 2016 at 15:34, Mike Harding <mi...@gmail.com>
>> wrote:
>> >>>>
>> >>>>
>> >>>> Hi All,
>> >>>>
>> >>>> The node in my cluster running nifi crashed due to a CPU overload
>> event.
>> >>>> After restarting I analysed the CPU consumption and found that nifi
>> was the
>> >>>> issue. As you can see below it was running at 133% CPU:
>> >>>>
>> >>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> >>>> COMMAND
>> >>>>
>> >>>> 2031 nifi 20 0 3392960 990.7m 34124 S 113.4 12.4 1179:47
>> java]
>> >>>>
>> >>>> I have only one data pipe line setup that is receiving data through a
>> >>>> HandleHTTPRequest processor and after playing around and turning
>> other
>> >>>> processors off in the pipe it was only when I stopped this process
>> that the
>> >>>> CPU dropped significantly to around 10% CPU.
>> >>>>
>> >>>> Its receiving around 67KB of data every 5 minutes from multiple
>> requests
>> >>>> from a up stream web app.
>> >>>>
>> >>>> Has any one else seen this behaviour and or know whether there are
>> ways
>> >>>> of managing the CPU usage of HandleHTTPRequest ?
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> Mike
>> >>>
>> >>>
>> >>>
>> >
>>
>
>
Re: nifi process running at 114% of node CPU !!?
Posted by Michael Moser <mo...@gmail.com>.
I went ahead and created NIFI-2268 for this, since it was fresh in my
mind. ListenHTTP calls ProcessContext.yield() whenever it doesn't have
work to do, so HandleHttpRequest could do the same.
-- Mike
On Thu, Jul 14, 2016 at 7:00 PM, Joe Witt <jo...@gmail.com> wrote:
> Mike,
>
> If you don't mind could you file a JIRA for this. Frankly it sounds
> like a bug to me. We should consider making a default scheduling
> period of something a bit slower. Frankly just dialing back to 100 ms
> would be sufficient most likely. If you agree this is a bug please
> file one here: https://issues.apache.org/jira/browse/NIFI
>
> If you could attach a template of the flow that behaves badly and the
> one that behaves better that would be ideal but if not just a good
> description should do.
>
> Thanks
> Joe
>
> On Thu, Jul 14, 2016 at 6:57 PM, Mike Harding <mi...@gmail.com>
> wrote:
> > Thanks all - I checked the logs and there is nothing I can see thats
> seems
> > erroneous. I increased the number of threads for the processor and added
> the
> > 10 second scheduling and it has dropped dramatically from 2.5M tasks to
> 300
> > over 5 minute period. CPU for the nifi java process is now running at
> 8-10%
> > CPU.
> >
> > I don't think I saw this issue when using HTTPListen processor which I
> > recently from to HttpRequestHandle.
> >
> > Cheers,
> > Mike
> >
> > On 14 July 2016 at 16:41, Aldrin Piri <al...@gmail.com> wrote:
> >>
> >> Mike,
> >>
> >> To add some context, while NiFi will intelligently schedule processors
> to
> >> execute, given HandleHTTPRequest's function as a listener, it is
> constantly
> >> scheduled to run, checking for a request to handle. I assume by number
> of
> >> tasks, you mean the rolling count over the last 5 minutes. As
> mentioned by
> >> Andy, you can tamper this rate by increasing the run scheduld if the
> >> handling of the HTTP requests with a slight latency is acceptable to
> you and
> >> your needs.
> >>
> >>
> >> On Thu, Jul 14, 2016 at 11:05 AM Andy LoPresto <al...@apache.org>
> >> wrote:
> >>>
> >>> Mike,
> >>>
> >>> You can adjust the processor properties for the HandleHTTPRequest
> >>> processor in the scheduling tab.
> >>>
> >>> “Concurrent tasks” limits the number of threads this processor will use
> >>> (default is 1)
> >>> “Run schedule” determines the frequency that this processor will be run
> >>> (default is ‘0 sec’ which means continuously)
> >>>
> >>> If you are only getting requests on a much slower schedule, you could
> >>> reduce the run schedule to ~10 seconds and see if this is better for
> you. I
> >>> have not encountered NiFi running at such high CPU percentage with that
> >>> little data.
> >>>
> >>> As for the high number of tasks, that is definitely an anomaly.
> >>> Configuration best practices [1] currently recommend increasing the
> limit to
> >>> the 10k range, but 2.5M for a single processor is unusual. Can you
> inspect
> >>> the logs (located in $NIFI_HOME/logs) to see if there are errors or
> more
> >>> insight there?
> >>>
> >>> [1]
> >>>
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices
> >>>
> >>>
> >>>
> >>> Andy LoPresto
> >>> alopresto@apache.org
> >>> alopresto.apache@gmail.com
> >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >>>
> >>> On Jul 14, 2016, at 10:36 AM, Mike Harding <mi...@gmail.com>
> >>> wrote:
> >>>
> >>> ps - also noticed it seems to generate a lot of tasks, currently 2.5M
> >>> compared to other processes in the pipeline which reports 10s of tasks.
> >>>
> >>> Mike
> >>>
> >>> On 14 July 2016 at 15:34, Mike Harding <mi...@gmail.com> wrote:
> >>>>
> >>>>
> >>>> Hi All,
> >>>>
> >>>> The node in my cluster running nifi crashed due to a CPU overload
> event.
> >>>> After restarting I analysed the CPU consumption and found that nifi
> was the
> >>>> issue. As you can see below it was running at 133% CPU:
> >>>>
> >>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> >>>> COMMAND
> >>>>
> >>>> 2031 nifi 20 0 3392960 990.7m 34124 S 113.4 12.4 1179:47
> java]
> >>>>
> >>>> I have only one data pipe line setup that is receiving data through a
> >>>> HandleHTTPRequest processor and after playing around and turning other
> >>>> processors off in the pipe it was only when I stopped this process
> that the
> >>>> CPU dropped significantly to around 10% CPU.
> >>>>
> >>>> Its receiving around 67KB of data every 5 minutes from multiple
> requests
> >>>> from a up stream web app.
> >>>>
> >>>> Has any one else seen this behaviour and or know whether there are
> ways
> >>>> of managing the CPU usage of HandleHTTPRequest ?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Mike
> >>>
> >>>
> >>>
> >
>
Re: nifi process running at 114% of node CPU !!?
Posted by Joe Witt <jo...@gmail.com>.
Mike,
If you don't mind could you file a JIRA for this. Frankly it sounds
like a bug to me. We should consider making a default scheduling
period of something a bit slower. Frankly just dialing back to 100 ms
would be sufficient most likely. If you agree this is a bug please
file one here: https://issues.apache.org/jira/browse/NIFI
If you could attach a template of the flow that behaves badly and the
one that behaves better that would be ideal but if not just a good
description should do.
Thanks
Joe
On Thu, Jul 14, 2016 at 6:57 PM, Mike Harding <mi...@gmail.com> wrote:
> Thanks all - I checked the logs and there is nothing I can see thats seems
> erroneous. I increased the number of threads for the processor and added the
> 10 second scheduling and it has dropped dramatically from 2.5M tasks to 300
> over 5 minute period. CPU for the nifi java process is now running at 8-10%
> CPU.
>
> I don't think I saw this issue when using HTTPListen processor which I
> recently from to HttpRequestHandle.
>
> Cheers,
> Mike
>
> On 14 July 2016 at 16:41, Aldrin Piri <al...@gmail.com> wrote:
>>
>> Mike,
>>
>> To add some context, while NiFi will intelligently schedule processors to
>> execute, given HandleHTTPRequest's function as a listener, it is constantly
>> scheduled to run, checking for a request to handle. I assume by number of
>> tasks, you mean the rolling count over the last 5 minutes. As mentioned by
>> Andy, you can tamper this rate by increasing the run scheduld if the
>> handling of the HTTP requests with a slight latency is acceptable to you and
>> your needs.
>>
>>
>> On Thu, Jul 14, 2016 at 11:05 AM Andy LoPresto <al...@apache.org>
>> wrote:
>>>
>>> Mike,
>>>
>>> You can adjust the processor properties for the HandleHTTPRequest
>>> processor in the scheduling tab.
>>>
>>> “Concurrent tasks” limits the number of threads this processor will use
>>> (default is 1)
>>> “Run schedule” determines the frequency that this processor will be run
>>> (default is ‘0 sec’ which means continuously)
>>>
>>> If you are only getting requests on a much slower schedule, you could
>>> reduce the run schedule to ~10 seconds and see if this is better for you. I
>>> have not encountered NiFi running at such high CPU percentage with that
>>> little data.
>>>
>>> As for the high number of tasks, that is definitely an anomaly.
>>> Configuration best practices [1] currently recommend increasing the limit to
>>> the 10k range, but 2.5M for a single processor is unusual. Can you inspect
>>> the logs (located in $NIFI_HOME/logs) to see if there are errors or more
>>> insight there?
>>>
>>> [1]
>>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices
>>>
>>>
>>>
>>> Andy LoPresto
>>> alopresto@apache.org
>>> alopresto.apache@gmail.com
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>>
>>> On Jul 14, 2016, at 10:36 AM, Mike Harding <mi...@gmail.com>
>>> wrote:
>>>
>>> ps - also noticed it seems to generate a lot of tasks, currently 2.5M
>>> compared to other processes in the pipeline which reports 10s of tasks.
>>>
>>> Mike
>>>
>>> On 14 July 2016 at 15:34, Mike Harding <mi...@gmail.com> wrote:
>>>>
>>>>
>>>> Hi All,
>>>>
>>>> The node in my cluster running nifi crashed due to a CPU overload event.
>>>> After restarting I analysed the CPU consumption and found that nifi was the
>>>> issue. As you can see below it was running at 133% CPU:
>>>>
>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>>>> COMMAND
>>>>
>>>> 2031 nifi 20 0 3392960 990.7m 34124 S 113.4 12.4 1179:47 java]
>>>>
>>>> I have only one data pipe line setup that is receiving data through a
>>>> HandleHTTPRequest processor and after playing around and turning other
>>>> processors off in the pipe it was only when I stopped this process that the
>>>> CPU dropped significantly to around 10% CPU.
>>>>
>>>> Its receiving around 67KB of data every 5 minutes from multiple requests
>>>> from a up stream web app.
>>>>
>>>> Has any one else seen this behaviour and or know whether there are ways
>>>> of managing the CPU usage of HandleHTTPRequest ?
>>>>
>>>> Thanks,
>>>>
>>>> Mike
>>>
>>>
>>>
>
Re: nifi process running at 114% of node CPU !!?
Posted by Mike Harding <mi...@gmail.com>.
Thanks all - I checked the logs and there is nothing I can see thats seems
erroneous. I increased the number of threads for the processor and added
the 10 second scheduling and it has dropped dramatically from 2.5M tasks to
300 over 5 minute period. CPU for the nifi java process is now running at
8-10% CPU.
I don't think I saw this issue when using HTTPListen processor which I
recently from to HttpRequestHandle.
Cheers,
Mike
On 14 July 2016 at 16:41, Aldrin Piri <al...@gmail.com> wrote:
> Mike,
>
> To add some context, while NiFi will intelligently schedule processors to
> execute, given HandleHTTPRequest's function as a listener, it is
> constantly scheduled to run, checking for a request to handle. I assume by
> number of tasks, you mean the rolling count over the last 5 minutes. As
> mentioned by Andy, you can tamper this rate by increasing the run scheduld
> if the handling of the HTTP requests with a slight latency is acceptable to
> you and your needs.
>
>
> On Thu, Jul 14, 2016 at 11:05 AM Andy LoPresto <al...@apache.org>
> wrote:
>
>> Mike,
>>
>> You can adjust the processor properties for the HandleHTTPRequest
>> processor in the scheduling tab.
>>
>> “Concurrent tasks” limits the number of threads this processor will use
>> (default is 1)
>> “Run schedule” determines the frequency that this processor will be run
>> (default is ‘0 sec’ which means continuously)
>>
>> If you are only getting requests on a much slower schedule, you could
>> reduce the run schedule to ~10 seconds and see if this is better for you. I
>> have not encountered NiFi running at such high CPU percentage with that
>> little data.
>>
>> As for the high number of tasks, that is definitely an anomaly.
>> Configuration best practices [1] currently recommend increasing the limit
>> to the 10k range, but 2.5M for a single processor is unusual. Can you
>> inspect the logs (located in $NIFI_HOME/logs) to see if there are errors or
>> more insight there?
>>
>> [1]
>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices
>>
>>
>>
>> Andy LoPresto
>> alopresto@apache.org
>> *alopresto.apache@gmail.com <al...@gmail.com>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Jul 14, 2016, at 10:36 AM, Mike Harding <mi...@gmail.com>
>> wrote:
>>
>> ps - also noticed it seems to generate a lot of tasks, currently 2.5M
>> compared to other processes in the pipeline which reports 10s of tasks.
>>
>> Mike
>>
>> On 14 July 2016 at 15:34, Mike Harding <mi...@gmail.com> wrote:
>>
>>>
>>> Hi All,
>>>
>>> The node in my cluster running nifi crashed due to a CPU overload event.
>>> After restarting I analysed the CPU consumption and found that nifi was the
>>> issue. As you can see below it was running at 133% CPU:
>>>
>>> PID USER PR NI VIRT RES SHR S *%CPU* %MEM TIME+
>>> COMMAND
>>>
>>> 2031 nifi 20 0 3392960 990.7m 34124 S *113.4* 12.4 1179:47
>>> java]
>>>
>>> I have only one data pipe line setup that is receiving data through a
>>> HandleHTTPRequest processor and after playing around and turning other
>>> processors off in the pipe it was only when I stopped this process that the
>>> CPU dropped significantly to around 10% CPU.
>>>
>>> Its receiving around 67KB of data every 5 minutes from multiple requests
>>> from a up stream web app.
>>>
>>> Has any one else seen this behaviour and or know whether there are ways
>>> of managing the CPU usage of HandleHTTPRequest ?
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>
>>
>>
Re: nifi process running at 114% of node CPU !!?
Posted by Aldrin Piri <al...@gmail.com>.
Mike,
To add some context, while NiFi will intelligently schedule processors to
execute, given HandleHTTPRequest's function as a listener, it is constantly
scheduled to run, checking for a request to handle. I assume by number of
tasks, you mean the rolling count over the last 5 minutes. As mentioned by
Andy, you can tamper this rate by increasing the run scheduld if the
handling of the HTTP requests with a slight latency is acceptable to you
and your needs.
On Thu, Jul 14, 2016 at 11:05 AM Andy LoPresto <al...@apache.org> wrote:
> Mike,
>
> You can adjust the processor properties for the HandleHTTPRequest
> processor in the scheduling tab.
>
> “Concurrent tasks” limits the number of threads this processor will use
> (default is 1)
> “Run schedule” determines the frequency that this processor will be run
> (default is ‘0 sec’ which means continuously)
>
> If you are only getting requests on a much slower schedule, you could
> reduce the run schedule to ~10 seconds and see if this is better for you. I
> have not encountered NiFi running at such high CPU percentage with that
> little data.
>
> As for the high number of tasks, that is definitely an anomaly.
> Configuration best practices [1] currently recommend increasing the limit
> to the 10k range, but 2.5M for a single processor is unusual. Can you
> inspect the logs (located in $NIFI_HOME/logs) to see if there are errors or
> more insight there?
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices
>
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Jul 14, 2016, at 10:36 AM, Mike Harding <mi...@gmail.com> wrote:
>
> ps - also noticed it seems to generate a lot of tasks, currently 2.5M
> compared to other processes in the pipeline which reports 10s of tasks.
>
> Mike
>
> On 14 July 2016 at 15:34, Mike Harding <mi...@gmail.com> wrote:
>
>>
>> Hi All,
>>
>> The node in my cluster running nifi crashed due to a CPU overload event.
>> After restarting I analysed the CPU consumption and found that nifi was the
>> issue. As you can see below it was running at 133% CPU:
>>
>> PID USER PR NI VIRT RES SHR S *%CPU* %MEM TIME+
>> COMMAND
>>
>> 2031 nifi 20 0 3392960 990.7m 34124 S *113.4* 12.4 1179:47
>> java]
>>
>> I have only one data pipe line setup that is receiving data through a
>> HandleHTTPRequest processor and after playing around and turning other
>> processors off in the pipe it was only when I stopped this process that the
>> CPU dropped significantly to around 10% CPU.
>>
>> Its receiving around 67KB of data every 5 minutes from multiple requests
>> from a up stream web app.
>>
>> Has any one else seen this behaviour and or know whether there are ways
>> of managing the CPU usage of HandleHTTPRequest ?
>>
>> Thanks,
>>
>> Mike
>>
>
>
>
Re: nifi process running at 114% of node CPU !!?
Posted by Andy LoPresto <al...@apache.org>.
Mike,
You can adjust the processor properties for the HandleHTTPRequest processor in the scheduling tab.
“Concurrent tasks” limits the number of threads this processor will use (default is 1)
“Run schedule” determines the frequency that this processor will be run (default is ‘0 sec’ which means continuously)
If you are only getting requests on a much slower schedule, you could reduce the run schedule to ~10 seconds and see if this is better for you. I have not encountered NiFi running at such high CPU percentage with that little data.
As for the high number of tasks, that is definitely an anomaly. Configuration best practices [1] currently recommend increasing the limit to the 10k range, but 2.5M for a single processor is unusual. Can you inspect the logs (located in $NIFI_HOME/logs) to see if there are errors or more insight there?
[1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jul 14, 2016, at 10:36 AM, Mike Harding <mi...@gmail.com> wrote:
>
> ps - also noticed it seems to generate a lot of tasks, currently 2.5M compared to other processes in the pipeline which reports 10s of tasks.
>
> Mike
>
> On 14 July 2016 at 15:34, Mike Harding <mikeyharding@gmail.com <ma...@gmail.com>> wrote:
>
> Hi All,
>
> The node in my cluster running nifi crashed due to a CPU overload event. After restarting I analysed the CPU consumption and found that nifi was the issue. As you can see below it was running at 133% CPU:
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 2031 nifi 20 0 3392960 990.7m 34124 S 113.4 12.4 1179:47 java]
>
> I have only one data pipe line setup that is receiving data through a HandleHTTPRequest processor and after playing around and turning other processors off in the pipe it was only when I stopped this process that the CPU dropped significantly to around 10% CPU.
>
> Its receiving around 67KB of data every 5 minutes from multiple requests from a up stream web app.
>
> Has any one else seen this behaviour and or know whether there are ways of managing the CPU usage of HandleHTTPRequest ?
>
> Thanks,
>
> Mike
>
>
Re: nifi process running at 114% of node CPU !!?
Posted by Mike Harding <mi...@gmail.com>.
ps - also noticed it seems to generate a lot of tasks, currently 2.5M
compared to other processes in the pipeline which reports 10s of tasks.
Mike
On 14 July 2016 at 15:34, Mike Harding <mi...@gmail.com> wrote:
> Hi All,
>
> The node in my cluster running nifi crashed due to a CPU overload event.
> After restarting I analysed the CPU consumption and found that nifi was the
> issue. As you can see below it was running at 133% CPU:
>
> PID USER PR NI VIRT RES SHR S *%CPU* %MEM TIME+
> COMMAND
>
> 2031 nifi 20 0 3392960 990.7m 34124 S *113.4* 12.4 1179:47
> java]
>
> I have only one data pipe line setup that is receiving data through a
> HandleHTTPRequest processor and after playing around and turning other
> processors off in the pipe it was only when I stopped this process that the
> CPU dropped significantly to around 10% CPU.
>
> Its receiving around 67KB of data every 5 minutes from multiple requests
> from a up stream web app.
>
> Has any one else seen this behaviour and or know whether there are ways of
> managing the CPU usage of HandleHTTPRequest ?
>
> Thanks,
>
> Mike
>