You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Adam Lamar <ad...@gmail.com> on 2015/11/13 06:27:00 UTC
High CPU usage in FileSystemRepository.java
Hi everybody!
I'm following up from my previous thread about high CPU usage in GetSQS.
I ran into high CPU usage while developing a patch for that processor,
and while investigating with "top", I noticed one NiFi thread in
particular showed high CPU usage, even after turning off all processors
and restarting NiFi.
A jstack showed this thread was busy at FileSystemRepository.java line
1287 [1]. Since that is a continue statement, it suggests that the
thread was churning in the surrounding for loop.
I didn't debug any further, but I did add a sleep statement just before
the continue, and CPU usage dropped wildly, settling around 2-4%.
I hope this is useful information, and I'm happy to debug further if needed.
Cheers,
Adam
[1]
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L1287
Re: High CPU usage in FileSystemRepository.java
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
Adam, thanks for doing all the research and pointing out where the problem is. It is definitely a bug.
Joe I’ve raised the https://issues.apache.org/jira/browse/NIFI-1200
Cheers
Oleg
On Nov 19, 2015, at 1:18 PM, Joe Percivall <jo...@yahoo.com>> wrote:
Hello Adam,
Are you still seeing high cpu usage?
Sorry no has gotten back to you sooner, we are all working very hard to get 0.4.0 out.
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall<http://linkedin.com/in/Percivall>
e: joepercivall@yahoo.com
On Friday, November 13, 2015 4:10 PM, Adam Lamar <ad...@gmail.com> wrote:
Mark,
For this development system, I'm running the packaged OpenJDK from
Ubuntu 14.04:
$ java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-0ubuntu1.14.04.1)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
Interestingly, I tried another system running the Oracle JDK (same
version) and didn't see the same issue. Though it does seem to benefit
from the additional sleep in that loop, but just barely (maybe 1% CPU
difference - I could be making that up).
I hadn't uncommented those values, but I tried with no noticeable
difference on the OpenJDK system.
Hope that helps,
Adam
On 11/13/15 5:28 AM, Mark Payne wrote:
Adam,
What version of Java are you running?
Do you have the following lines from conf/bootstrap.conf uncommented, or are they still commented out?
java.arg.7=-XX:ReservedCodeCacheSize=256m
java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m
java.arg.9=-XX:+UseCodeCacheFlushing
java.arg.11=-XX:PermSize=128M
java.arg.12=-XX:MaxPermSize=128M
Thanks
-Mark
On Nov 13, 2015, at 12:28 AM, Joe Witt <jo...@gmail.com> wrote:
sorry - i see now :-)
Thanks for the analysis. Will dig in.
Joe
On Fri, Nov 13, 2015 at 12:28 AM, Joe Witt <jo...@gmail.com> wrote:
Adam,
Are you on a recent master build?
Thanks
Joe
On Fri, Nov 13, 2015 at 12:27 AM, Adam Lamar <ad...@gmail.com> wrote:
Hi everybody!
I'm following up from my previous thread about high CPU usage in GetSQS. I
ran into high CPU usage while developing a patch for that processor, and
while investigating with "top", I noticed one NiFi thread in particular
showed high CPU usage, even after turning off all processors and restarting
NiFi.
A jstack showed this thread was busy at FileSystemRepository.java line 1287
[1]. Since that is a continue statement, it suggests that the thread was
churning in the surrounding for loop.
I didn't debug any further, but I did add a sleep statement just before the
continue, and CPU usage dropped wildly, settling around 2-4%.
I hope this is useful information, and I'm happy to debug further if needed.
Cheers,
Adam
[1]
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L1287
Re: High CPU usage in FileSystemRepository.java
Posted by Joe Percivall <jo...@yahoo.com>.
Hello Adam,
Are you still seeing high cpu usage?
Sorry no has gotten back to you sooner, we are all working very hard to get 0.4.0 out.
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com
On Friday, November 13, 2015 4:10 PM, Adam Lamar <ad...@gmail.com> wrote:
Mark,
For this development system, I'm running the packaged OpenJDK from
Ubuntu 14.04:
$ java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-0ubuntu1.14.04.1)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
Interestingly, I tried another system running the Oracle JDK (same
version) and didn't see the same issue. Though it does seem to benefit
from the additional sleep in that loop, but just barely (maybe 1% CPU
difference - I could be making that up).
I hadn't uncommented those values, but I tried with no noticeable
difference on the OpenJDK system.
Hope that helps,
Adam
On 11/13/15 5:28 AM, Mark Payne wrote:
> Adam,
>
> What version of Java are you running?
>
> Do you have the following lines from conf/bootstrap.conf uncommented, or are they still commented out?
>
> java.arg.7=-XX:ReservedCodeCacheSize=256m
> java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m
> java.arg.9=-XX:+UseCodeCacheFlushing
> java.arg.11=-XX:PermSize=128M
> java.arg.12=-XX:MaxPermSize=128M
>
> Thanks
> -Mark
>
>
>> On Nov 13, 2015, at 12:28 AM, Joe Witt <jo...@gmail.com> wrote:
>>
>> sorry - i see now :-)
>>
>> Thanks for the analysis. Will dig in.
>>
>> Joe
>>
>> On Fri, Nov 13, 2015 at 12:28 AM, Joe Witt <jo...@gmail.com> wrote:
>>> Adam,
>>>
>>> Are you on a recent master build?
>>>
>>> Thanks
>>> Joe
>>>
>>> On Fri, Nov 13, 2015 at 12:27 AM, Adam Lamar <ad...@gmail.com> wrote:
>>>> Hi everybody!
>>>>
>>>> I'm following up from my previous thread about high CPU usage in GetSQS. I
>>>> ran into high CPU usage while developing a patch for that processor, and
>>>> while investigating with "top", I noticed one NiFi thread in particular
>>>> showed high CPU usage, even after turning off all processors and restarting
>>>> NiFi.
>>>>
>>>> A jstack showed this thread was busy at FileSystemRepository.java line 1287
>>>> [1]. Since that is a continue statement, it suggests that the thread was
>>>> churning in the surrounding for loop.
>>>>
>>>> I didn't debug any further, but I did add a sleep statement just before the
>>>> continue, and CPU usage dropped wildly, settling around 2-4%.
>>>>
>>>> I hope this is useful information, and I'm happy to debug further if needed.
>>>>
>>>> Cheers,
>>>> Adam
>>>>
>>>> [1]
>>>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L1287
Re: High CPU usage in FileSystemRepository.java
Posted by Adam Lamar <ad...@gmail.com>.
Mark,
For this development system, I'm running the packaged OpenJDK from
Ubuntu 14.04:
$ java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-0ubuntu1.14.04.1)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
Interestingly, I tried another system running the Oracle JDK (same
version) and didn't see the same issue. Though it does seem to benefit
from the additional sleep in that loop, but just barely (maybe 1% CPU
difference - I could be making that up).
I hadn't uncommented those values, but I tried with no noticeable
difference on the OpenJDK system.
Hope that helps,
Adam
On 11/13/15 5:28 AM, Mark Payne wrote:
> Adam,
>
> What version of Java are you running?
>
> Do you have the following lines from conf/bootstrap.conf uncommented, or are they still commented out?
>
> java.arg.7=-XX:ReservedCodeCacheSize=256m
> java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m
> java.arg.9=-XX:+UseCodeCacheFlushing
> java.arg.11=-XX:PermSize=128M
> java.arg.12=-XX:MaxPermSize=128M
>
> Thanks
> -Mark
>
>
>> On Nov 13, 2015, at 12:28 AM, Joe Witt <jo...@gmail.com> wrote:
>>
>> sorry - i see now :-)
>>
>> Thanks for the analysis. Will dig in.
>>
>> Joe
>>
>> On Fri, Nov 13, 2015 at 12:28 AM, Joe Witt <jo...@gmail.com> wrote:
>>> Adam,
>>>
>>> Are you on a recent master build?
>>>
>>> Thanks
>>> Joe
>>>
>>> On Fri, Nov 13, 2015 at 12:27 AM, Adam Lamar <ad...@gmail.com> wrote:
>>>> Hi everybody!
>>>>
>>>> I'm following up from my previous thread about high CPU usage in GetSQS. I
>>>> ran into high CPU usage while developing a patch for that processor, and
>>>> while investigating with "top", I noticed one NiFi thread in particular
>>>> showed high CPU usage, even after turning off all processors and restarting
>>>> NiFi.
>>>>
>>>> A jstack showed this thread was busy at FileSystemRepository.java line 1287
>>>> [1]. Since that is a continue statement, it suggests that the thread was
>>>> churning in the surrounding for loop.
>>>>
>>>> I didn't debug any further, but I did add a sleep statement just before the
>>>> continue, and CPU usage dropped wildly, settling around 2-4%.
>>>>
>>>> I hope this is useful information, and I'm happy to debug further if needed.
>>>>
>>>> Cheers,
>>>> Adam
>>>>
>>>> [1]
>>>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L1287
Re: High CPU usage in FileSystemRepository.java
Posted by Mark Payne <ma...@hotmail.com>.
Adam,
What version of Java are you running?
Do you have the following lines from conf/bootstrap.conf uncommented, or are they still commented out?
java.arg.7=-XX:ReservedCodeCacheSize=256m
java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m
java.arg.9=-XX:+UseCodeCacheFlushing
java.arg.11=-XX:PermSize=128M
java.arg.12=-XX:MaxPermSize=128M
Thanks
-Mark
> On Nov 13, 2015, at 12:28 AM, Joe Witt <jo...@gmail.com> wrote:
>
> sorry - i see now :-)
>
> Thanks for the analysis. Will dig in.
>
> Joe
>
> On Fri, Nov 13, 2015 at 12:28 AM, Joe Witt <jo...@gmail.com> wrote:
>> Adam,
>>
>> Are you on a recent master build?
>>
>> Thanks
>> Joe
>>
>> On Fri, Nov 13, 2015 at 12:27 AM, Adam Lamar <ad...@gmail.com> wrote:
>>> Hi everybody!
>>>
>>> I'm following up from my previous thread about high CPU usage in GetSQS. I
>>> ran into high CPU usage while developing a patch for that processor, and
>>> while investigating with "top", I noticed one NiFi thread in particular
>>> showed high CPU usage, even after turning off all processors and restarting
>>> NiFi.
>>>
>>> A jstack showed this thread was busy at FileSystemRepository.java line 1287
>>> [1]. Since that is a continue statement, it suggests that the thread was
>>> churning in the surrounding for loop.
>>>
>>> I didn't debug any further, but I did add a sleep statement just before the
>>> continue, and CPU usage dropped wildly, settling around 2-4%.
>>>
>>> I hope this is useful information, and I'm happy to debug further if needed.
>>>
>>> Cheers,
>>> Adam
>>>
>>> [1]
>>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L1287
Re: High CPU usage in FileSystemRepository.java
Posted by Joe Witt <jo...@gmail.com>.
sorry - i see now :-)
Thanks for the analysis. Will dig in.
Joe
On Fri, Nov 13, 2015 at 12:28 AM, Joe Witt <jo...@gmail.com> wrote:
> Adam,
>
> Are you on a recent master build?
>
> Thanks
> Joe
>
> On Fri, Nov 13, 2015 at 12:27 AM, Adam Lamar <ad...@gmail.com> wrote:
>> Hi everybody!
>>
>> I'm following up from my previous thread about high CPU usage in GetSQS. I
>> ran into high CPU usage while developing a patch for that processor, and
>> while investigating with "top", I noticed one NiFi thread in particular
>> showed high CPU usage, even after turning off all processors and restarting
>> NiFi.
>>
>> A jstack showed this thread was busy at FileSystemRepository.java line 1287
>> [1]. Since that is a continue statement, it suggests that the thread was
>> churning in the surrounding for loop.
>>
>> I didn't debug any further, but I did add a sleep statement just before the
>> continue, and CPU usage dropped wildly, settling around 2-4%.
>>
>> I hope this is useful information, and I'm happy to debug further if needed.
>>
>> Cheers,
>> Adam
>>
>> [1]
>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L1287
Re: High CPU usage in FileSystemRepository.java
Posted by Joe Witt <jo...@gmail.com>.
Adam,
Are you on a recent master build?
Thanks
Joe
On Fri, Nov 13, 2015 at 12:27 AM, Adam Lamar <ad...@gmail.com> wrote:
> Hi everybody!
>
> I'm following up from my previous thread about high CPU usage in GetSQS. I
> ran into high CPU usage while developing a patch for that processor, and
> while investigating with "top", I noticed one NiFi thread in particular
> showed high CPU usage, even after turning off all processors and restarting
> NiFi.
>
> A jstack showed this thread was busy at FileSystemRepository.java line 1287
> [1]. Since that is a continue statement, it suggests that the thread was
> churning in the surrounding for loop.
>
> I didn't debug any further, but I did add a sleep statement just before the
> continue, and CPU usage dropped wildly, settling around 2-4%.
>
> I hope this is useful information, and I'm happy to debug further if needed.
>
> Cheers,
> Adam
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L1287