You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Dmitry Goldenberg <dg...@hexastax.com> on 2016/03/29 20:31:03 UTC
OutOfMemory / PermGen space
Hi,
Started getting OOM once I've added PutHDFS to a flow.
I'm running on a CentOS VM with 2GB of memory allocated. I've munged the
nifi startup script to do -Xms1024m -Xmx1024m". Any recommendations on
this?
Thanks,
- Dmitry
2016-03-29 11:25:54,200 ERROR [Flow Service Tasks Thread-2]
o.a.nifi.controller.StandardFlowService Unable to save flow controller
configuration due to: java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.findBootstrapClass(Native Method) ~[na:1.7.0_67]
at java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:1070)
~[na:1.7.0_67]
at java.lang.ClassLoader.loadClass(ClassLoader.java:414) ~[na:1.7.0_67]
at java.lang.ClassLoader.loadClass(ClassLoader.java:412) ~[na:1.7.0_67]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
~[na:1.7.0_67]
at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_67]
at
org.apache.nifi.nar.NarThreadContextClassLoader.loadClass(NarThreadContextClassLoader.java:96)
~[nifi-nar-utils-0.5.1.jar:0.5.1]
at
javax.xml.transform.FactoryFinder.getProviderClass(FactoryFinder.java:114)
~[na:1.7.0_67]
at javax.xml.transform.FactoryFinder.newInstance(FactoryFinder.java:182)
~[na:1.7.0_67]
at javax.xml.transform.FactoryFinder.find(FactoryFinder.java:296)
~[na:1.7.0_67]
at
javax.xml.transform.TransformerFactory.newInstance(TransformerFactory.java:101)
~[na:1.7.0_67]
at
org.apache.nifi.controller.StandardFlowSerializer.serialize(StandardFlowSerializer.java:102)
~[na:na]
at
org.apache.nifi.controller.FlowController.serialize(FlowController.java:1262)
~[na:na]
at
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.save(StandardXMLFlowConfigurationDAO.java:128)
~[na:na]
at
org.apache.nifi.controller.StandardFlowService$SaveReportingTask.run(StandardFlowService.java:842)
~[na:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_67]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_67]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_67]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_67]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_67]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_67]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
Re: OutOfMemory / PermGen space
Posted by Matthew Clarke <ma...@gmail.com>.
Dmitry,
What version of Java are you running? My guess is you are running
Java 7 and in that case I suggest you uncomment out the java related lines
in the bootstrap.conf file:
java.arg.7=-XX:ReservedCodeCacheSize=256m
java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m
java.arg.9=-XX:+UseCodeCacheFlushing
java.arg.11=-XX:PermSize=128M
java.arg.12=-XX:MaxPermSize=128M
Restart you NiFi after making these changes.
If you running Java 8, PermGen memory space just uses heap space
unlike Java 7. so in that case try increasing the min and max
sizeconfigured for your heap in the same bootstrap.conf file:
java.arg.2=-Xms4096m
java.arg.3=-Xmx4096m
Again any changes made to all but teh logback.xml config file require a
restart to take affect.
Matt
On Tue, Mar 29, 2016 at 2:31 PM, Dmitry Goldenberg <dgoldenberg@hexastax.com
> wrote:
> Hi,
>
> Started getting OOM once I've added PutHDFS to a flow.
>
> I'm running on a CentOS VM with 2GB of memory allocated. I've munged the
> nifi startup script to do -Xms1024m -Xmx1024m". Any recommendations on
> this?
>
> Thanks,
> - Dmitry
>
> 2016-03-29 11:25:54,200 ERROR [Flow Service Tasks Thread-2]
> o.a.nifi.controller.StandardFlowService Unable to save flow controller
> configuration due to: java.lang.OutOfMemoryError: PermGen space
> java.lang.OutOfMemoryError: PermGen space
> at java.lang.ClassLoader.findBootstrapClass(Native Method) ~[na:1.7.0_67]
> at java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:1070)
> ~[na:1.7.0_67]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:414) ~[na:1.7.0_67]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:412) ~[na:1.7.0_67]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> ~[na:1.7.0_67]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_67]
> at
> org.apache.nifi.nar.NarThreadContextClassLoader.loadClass(NarThreadContextClassLoader.java:96)
> ~[nifi-nar-utils-0.5.1.jar:0.5.1]
> at
> javax.xml.transform.FactoryFinder.getProviderClass(FactoryFinder.java:114)
> ~[na:1.7.0_67]
> at javax.xml.transform.FactoryFinder.newInstance(FactoryFinder.java:182)
> ~[na:1.7.0_67]
> at javax.xml.transform.FactoryFinder.find(FactoryFinder.java:296)
> ~[na:1.7.0_67]
> at
> javax.xml.transform.TransformerFactory.newInstance(TransformerFactory.java:101)
> ~[na:1.7.0_67]
> at
> org.apache.nifi.controller.StandardFlowSerializer.serialize(StandardFlowSerializer.java:102)
> ~[na:na]
> at
> org.apache.nifi.controller.FlowController.serialize(FlowController.java:1262)
> ~[na:na]
> at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.save(StandardXMLFlowConfigurationDAO.java:128)
> ~[na:na]
> at
> org.apache.nifi.controller.StandardFlowService$SaveReportingTask.run(StandardFlowService.java:842)
> ~[na:na]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> [na:1.7.0_67]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> [na:1.7.0_67]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> [na:1.7.0_67]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [na:1.7.0_67]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_67]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_67]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
>
>
Re: OutOfMemory / PermGen space
Posted by Joe Witt <jo...@gmail.com>.
Dmitry,
This indicates you're using Java 7 most likely. Look here [1] which
points to [2].
Specifically you'll want to update conf/bootstrap.properties
To have these lines uncommented
java.arg.11=-XX:PermSize=128M
java.arg.12=-XX:MaxPermSize=128M
You can choose different values than 128M as you like. But the
defaults are too low given how many classes we load. In Java 8 folks
don't have to mess with this as the function of the permgen moved into
the heap.
[1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system-requirements
[2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#bootstrap_properties
Thanks
Joe
On Tue, Mar 29, 2016 at 12:31 PM, Dmitry Goldenberg
<dg...@hexastax.com> wrote:
> Hi,
>
> Started getting OOM once I've added PutHDFS to a flow.
>
> I'm running on a CentOS VM with 2GB of memory allocated. I've munged the
> nifi startup script to do -Xms1024m -Xmx1024m". Any recommendations on
> this?
>
> Thanks,
> - Dmitry
>
> 2016-03-29 11:25:54,200 ERROR [Flow Service Tasks Thread-2]
> o.a.nifi.controller.StandardFlowService Unable to save flow controller
> configuration due to: java.lang.OutOfMemoryError: PermGen space
> java.lang.OutOfMemoryError: PermGen space
> at java.lang.ClassLoader.findBootstrapClass(Native Method) ~[na:1.7.0_67]
> at java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:1070)
> ~[na:1.7.0_67]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:414) ~[na:1.7.0_67]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:412) ~[na:1.7.0_67]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> ~[na:1.7.0_67]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_67]
> at
> org.apache.nifi.nar.NarThreadContextClassLoader.loadClass(NarThreadContextClassLoader.java:96)
> ~[nifi-nar-utils-0.5.1.jar:0.5.1]
> at
> javax.xml.transform.FactoryFinder.getProviderClass(FactoryFinder.java:114)
> ~[na:1.7.0_67]
> at javax.xml.transform.FactoryFinder.newInstance(FactoryFinder.java:182)
> ~[na:1.7.0_67]
> at javax.xml.transform.FactoryFinder.find(FactoryFinder.java:296)
> ~[na:1.7.0_67]
> at
> javax.xml.transform.TransformerFactory.newInstance(TransformerFactory.java:101)
> ~[na:1.7.0_67]
> at
> org.apache.nifi.controller.StandardFlowSerializer.serialize(StandardFlowSerializer.java:102)
> ~[na:na]
> at
> org.apache.nifi.controller.FlowController.serialize(FlowController.java:1262)
> ~[na:na]
> at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.save(StandardXMLFlowConfigurationDAO.java:128)
> ~[na:na]
> at
> org.apache.nifi.controller.StandardFlowService$SaveReportingTask.run(StandardFlowService.java:842)
> ~[na:na]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> [na:1.7.0_67]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> [na:1.7.0_67]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> [na:1.7.0_67]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [na:1.7.0_67]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_67]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_67]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
>