You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "Gresock, Joseph" <jo...@lmco.com> on 2016/02/15 15:33:34 UTC
OutofMemory
Devs,
We've been seeing some OutOfMemoryErrors on the NCM of our 10-node cluster recently. The flow has ~600 processors, and the NCM runs on a VM with 8GB RAM. We have 6G allocated to the Nifi JVM on this node.
The specific log message we see is:
WARN [Process NCM Request-6] org.apache.nifi.io.socket.SocketListener Dispatching socket request encountered exception due to: java.lang.OutOfMemoryError: Java heap space
First, I'm hoping there's some advice on how to avoid this in the first place, but barring that, is there a way to configure Nifi to auto-restart the NCM when it gets this error? I seem to remember seeing this in the past, but I couldn't find anything in bootstrap.conf or nifi.properties that looked related.
Thanks,
Joe
Re: EXTERNAL: Re: OutofMemory
Posted by Aldrin Piri <al...@gmail.com>.
Joe,
As you suspect, you would want to increase time. The buffer property is
the overall force exerted on the heap and the snapshot time correlates to
how long it takes for that buffer to be filled.
I think we can find some ways to handle this a bit better and have created
a ticket: https://issues.apache.org/jira/browse/NIFI-1515
Thanks for reporting the issue!
On Mon, Feb 15, 2016 at 10:44 AM, Gresock, Joseph <jo...@lmco.com>
wrote:
> Thanks Corey. I'll try that, but I'm wondering what the
> nifi.components.status.snapshot.frequency does. Would decreasing that time
> (1 min) make it take snapshots more frequently, and would this have the
> effect of decreasing or increasing the memory used? I did decrease the
> nifi.components.status.repository.buffer.size to 180 and I'll see how that
> behaves.
>
>
>
> Joe
>
> -----Original Message-----
> From: Corey Flowers [mailto:cflowers@onyxpoint.com]
> Sent: Monday, February 15, 2016 10:32 AM
> To: dev@nifi.apache.org
> Subject: EXTERNAL: Re: OutofMemory
>
> Hey Joseph,
>
> I have a couple of clusters in the 600-1200 range and they have
> 16-32 GB jvm heap sizes respectfully. Really it depends on what processors
> you are using and your volumes. One thing that may help a little is to
> decrease your number of stored statics in the graph.
> There are two properties in the conf file, one is set to 1440 and the
> other is a time, which is every 1 min. I believe these stats are stored in
> the heap space. Devs correct me if I am wrong. You could lessen the time
> and amount to buy you a little space. I don't think this is a solution,
> really it is more of a band-aide.
>
> Good luck!
>
>
>
>
>
> Sent from my iPhone
>
> > On Feb 15, 2016, at 9:33 AM, Gresock, Joseph <jo...@lmco.com>
> wrote:
> >
> > Devs,
> >
> > We've been seeing some OutOfMemoryErrors on the NCM of our 10-node
> cluster recently. The flow has ~600 processors, and the NCM runs on a VM
> with 8GB RAM. We have 6G allocated to the Nifi JVM on this node.
> >
> > The specific log message we see is:
> >
> > WARN [Process NCM Request-6] org.apache.nifi.io.socket.SocketListener
> > Dispatching socket request encountered exception due to:
> > java.lang.OutOfMemoryError: Java heap space
> >
> > First, I'm hoping there's some advice on how to avoid this in the first
> place, but barring that, is there a way to configure Nifi to auto-restart
> the NCM when it gets this error? I seem to remember seeing this in the
> past, but I couldn't find anything in bootstrap.conf or nifi.properties
> that looked related.
> >
> > Thanks,
> > Joe
>
RE: EXTERNAL: Re: OutofMemory
Posted by "Gresock, Joseph" <jo...@lmco.com>.
Thanks Corey. I'll try that, but I'm wondering what the nifi.components.status.snapshot.frequency does. Would decreasing that time (1 min) make it take snapshots more frequently, and would this have the effect of decreasing or increasing the memory used? I did decrease the nifi.components.status.repository.buffer.size to 180 and I'll see how that behaves.
Joe
-----Original Message-----
From: Corey Flowers [mailto:cflowers@onyxpoint.com]
Sent: Monday, February 15, 2016 10:32 AM
To: dev@nifi.apache.org
Subject: EXTERNAL: Re: OutofMemory
Hey Joseph,
I have a couple of clusters in the 600-1200 range and they have
16-32 GB jvm heap sizes respectfully. Really it depends on what processors you are using and your volumes. One thing that may help a little is to decrease your number of stored statics in the graph.
There are two properties in the conf file, one is set to 1440 and the other is a time, which is every 1 min. I believe these stats are stored in the heap space. Devs correct me if I am wrong. You could lessen the time and amount to buy you a little space. I don't think this is a solution, really it is more of a band-aide.
Good luck!
Sent from my iPhone
> On Feb 15, 2016, at 9:33 AM, Gresock, Joseph <jo...@lmco.com> wrote:
>
> Devs,
>
> We've been seeing some OutOfMemoryErrors on the NCM of our 10-node cluster recently. The flow has ~600 processors, and the NCM runs on a VM with 8GB RAM. We have 6G allocated to the Nifi JVM on this node.
>
> The specific log message we see is:
>
> WARN [Process NCM Request-6] org.apache.nifi.io.socket.SocketListener
> Dispatching socket request encountered exception due to:
> java.lang.OutOfMemoryError: Java heap space
>
> First, I'm hoping there's some advice on how to avoid this in the first place, but barring that, is there a way to configure Nifi to auto-restart the NCM when it gets this error? I seem to remember seeing this in the past, but I couldn't find anything in bootstrap.conf or nifi.properties that looked related.
>
> Thanks,
> Joe
Re: OutofMemory
Posted by Corey Flowers <cf...@onyxpoint.com>.
Hey Joseph,
I have a couple of clusters in the 600-1200 range and they have
16-32 GB jvm heap sizes respectfully. Really it depends on what
processors you are using and your volumes. One thing that may help a
little is to decrease your number of stored statics in the graph.
There are two properties in the conf file, one is set to 1440 and the
other is a time, which is every 1 min. I believe these stats are
stored in the heap space. Devs correct me if I am wrong. You could
lessen the time and amount to buy you a little space. I don't think
this is a solution, really it is more of a band-aide.
Good luck!
Sent from my iPhone
> On Feb 15, 2016, at 9:33 AM, Gresock, Joseph <jo...@lmco.com> wrote:
>
> Devs,
>
> We've been seeing some OutOfMemoryErrors on the NCM of our 10-node cluster recently. The flow has ~600 processors, and the NCM runs on a VM with 8GB RAM. We have 6G allocated to the Nifi JVM on this node.
>
> The specific log message we see is:
>
> WARN [Process NCM Request-6] org.apache.nifi.io.socket.SocketListener Dispatching socket request encountered exception due to: java.lang.OutOfMemoryError: Java heap space
>
> First, I'm hoping there's some advice on how to avoid this in the first place, but barring that, is there a way to configure Nifi to auto-restart the NCM when it gets this error? I seem to remember seeing this in the past, but I couldn't find anything in bootstrap.conf or nifi.properties that looked related.
>
> Thanks,
> Joe
Re: OutofMemory
Posted by Matthew Clarke <ma...@gmail.com>.
Joseph,
An inspection of a heap dump created when the out of memory
condition occurs will shed some light on the specific cause of the issue.
The following command can be used to get a heap dump.:
jmap -dump:format=b,file=nifi-heap-dump.bin <PID>
you can get the PID using the following command:
ps -ef|grep org.apache.nifi.NiFi
One common cause of out of memory on the NCM is related to the amount of
components status retained. You can edit the nifi.properties file to
decrease the amount of stats retained or change the duration of those stats
collected:
# Component Status Repository
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min
Either decrease the buffer size (will retain less stats for each component)
or decrease frequency of which stats are collected.
Thanks,
Matt
On Mon, Feb 15, 2016 at 9:33 AM, Gresock, Joseph <jo...@lmco.com>
wrote:
> Devs,
>
> We've been seeing some OutOfMemoryErrors on the NCM of our 10-node cluster
> recently. The flow has ~600 processors, and the NCM runs on a VM with 8GB
> RAM. We have 6G allocated to the Nifi JVM on this node.
>
> The specific log message we see is:
>
> WARN [Process NCM Request-6] org.apache.nifi.io.socket.SocketListener
> Dispatching socket request encountered exception due to:
> java.lang.OutOfMemoryError: Java heap space
>
> First, I'm hoping there's some advice on how to avoid this in the first
> place, but barring that, is there a way to configure Nifi to auto-restart
> the NCM when it gets this error? I seem to remember seeing this in the
> past, but I couldn't find anything in bootstrap.conf or nifi.properties
> that looked related.
>
> Thanks,
> Joe
>