You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Josef Zahner (Jira)" <ji...@apache.org> on 2019/11/27 12:10:00 UTC
[jira] [Updated] (NIFI-6908) PutKudu 1.10.0 Memory Leak
[ https://issues.apache.org/jira/browse/NIFI-6908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josef Zahner updated NIFI-6908:
-------------------------------
Description:
PutKudu 1.10.0 eats up all the heap memory and garbage collection can't anymore free up memory after a few hours.
We have an NiFi 8-Node cluster (31GB java max memory configured) with a streaming source which generates constantly about 2'500 flowfiles/2.5GB data in 5 minutes. In our example the streaming source was running on "nifi-05" (green line). As you can see between 00:00 and 04:00 the memory grows and grows and at the end the node became instable and the dreaded "java.lang.OutOfMemoryError: Java heap space" message appeared. We tried to do a manual garbage collection with visualvm profiler, but it didn't helped.
!memory_leak.png!
We are sure that the PutKudu is the culprit, as we have now taken the codebase from PutKudu 1.9.2 and use it now in NiFi 1.10.0 without any leaks at all.
Like this our cluster crashed within 5-6 hours with our current load as the memory was completely full.
was:
PutKudu 1.10.0 eats up all the heap memory and garbage collection can't anymore free up memory after a few hours.
We have an NiFi 8-Node cluster (31GB java max memory configured) with a streaming source which generates constantly about 2'500 flowfiles/2.5GB data in 5 minutes. In our example the streaming source was running on "nifi-05" (green line). As you can see between 00:00 and 04:00 the memory grows and grows and at the end the node became instable and the dreaded "java.lang.OutOfMemoryError: Java heap space" message appeared. With tried to do a manual garbage collection with visualvm profiler, but it didn't helped.
!memory_leak.png!
We are sure that the PutKudu is the culprit, as we have now taken the codebase from PutKudu 1.9.2 and use it now in NiFi 1.10.0 without any leaks at all.
Like this our cluster crashed within 5-6 hours with our current load as the memory was completely full.
> PutKudu 1.10.0 Memory Leak
> --------------------------
>
> Key: NIFI-6908
> URL: https://issues.apache.org/jira/browse/NIFI-6908
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.10.0
> Environment: NiFi 1.10.0 8-Node Cluster; Kudu 1.10.0
> Reporter: Josef Zahner
> Priority: Blocker
> Labels: heap, kudu, oom
> Attachments: memory_leak.png
>
>
> PutKudu 1.10.0 eats up all the heap memory and garbage collection can't anymore free up memory after a few hours.
> We have an NiFi 8-Node cluster (31GB java max memory configured) with a streaming source which generates constantly about 2'500 flowfiles/2.5GB data in 5 minutes. In our example the streaming source was running on "nifi-05" (green line). As you can see between 00:00 and 04:00 the memory grows and grows and at the end the node became instable and the dreaded "java.lang.OutOfMemoryError: Java heap space" message appeared. We tried to do a manual garbage collection with visualvm profiler, but it didn't helped.
> !memory_leak.png!
> We are sure that the PutKudu is the culprit, as we have now taken the codebase from PutKudu 1.9.2 and use it now in NiFi 1.10.0 without any leaks at all.
> Like this our cluster crashed within 5-6 hours with our current load as the memory was completely full.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)