Re: PutHDFS irrecoverably stops NiFi from loading workflow

[Chris Teoh <ch...@gmail.com>]

Hey all,

So here's the error:-
ERROR [NiFi Web Server-381780] org.apache.nifi.NiFi An Unknown Error
Occurred in Thread Thread[NiFi Web Server-381780,5,main]:
java.lang.OutOfMemoryError: PermGen space

The processors continue to run as I can see them continue to write entries
into the nifi-app.log file.

This error seemed to persist even when I set the PermGen values to 2048M.

In this scenario is there a way to remove a processor when the web UI is
unavailable?

Kind Regards
Chris

On Mon, 31 Aug 2015 at 02:27 Joe Witt <jo...@gmail.com> wrote:

> Not from a NiFi perspective.  Java 8 works great.  As a community
> though we plan to keep supporting Java 7 users as best we can until
> NiFi 1.0.0 happens.
>
> On Sun, Aug 30, 2015 at 12:10 PM, Rick Braddy <rb...@softnas.com> wrote:
> > Is there any reason to just use Java 8 so it's not an issue?
> >
> >
> >
> >> On Aug 30, 2015, at 5:54 AM, Joe Witt <jo...@gmail.com> wrote:
> >>
> >> Chris
> >>
> >> Alan nailed it.  In bootstrap.conf just uncomment these two lines and
> >> restart nifi.
> >>
> >> #java.arg.11=-XX:PermSize=128M
> >> #java.arg.12=-XX:MaxPermSize=128M
> >>
> >>
> >> But you are not the first run to run into this stumbling block so have
> >> created this jira [1].
> >>
> >> Thanks
> >> Joe
> >>
> >> [1] https://issues.apache.org/jira/browse/NIFI-909
> >>
> >>> On Sun, Aug 30, 2015 at 2:58 AM, Alan Jackoway <al...@cloudera.com>
> wrote:
> >>> Have you tried setting the PermGen size with something like
> >>> -XX:MaxPermSize=512m ? If you look in the file bootstrap.conf you
> should see
> >>> an example of how to set it.
> >>>
> >>> I ask because that error doesn't look specific to PutHDFS to me. If
> the real
> >>> problem is PermGen space, you may go through a lot of work trying to
> remove
> >>> that processor and then have this happen again when you add something
> else.
> >>>
> >>> Let us know if bootstrap.conf works for you,
> >>> Alan
> >>>
> >>>> On Aug 30, 2015 7:21 AM, "Chris Teoh" <ch...@gmail.com> wrote:
> >>>>
> >>>> Hey folks,
> >>>>
> >>>> I added PutHDFS into my workflow, pointed it to core-site.xml and
> >>>> hdfs-site.xml and it crashed. Restarting doesn't seem to help as the
> browser
> >>>> will say NiFi is loading, and then fails to load.
> >>>>
> >>>> How do I remove the PutHDFS processor without the web UI?
> >>>>
> >>>> Errors in the log are:-
> >>>> o.a.n.p.PersistentProvenanceRepository Failed to rollover Provenance
> >>>> repository due to java.lang.OutOfMemoryError: PermGen space
> >>>>
> >>>> o.apache.nifi.processors.hadoop.PutHDFS
> >>>> [PutHDFS[id=4cc42c80-e37f-4122-a6f2-0c3b61ffc00b]] Failed to write to
> HDFS
> >>>> due to {} java.lang.OutOfMemoryError: PermGen space at
> >>>> java.lang.ClassLoader.defineClass1(Native Method) ~[na:1.7.0_67] at
> >>>> java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> ~[na:1.7.0_67] at
> >>>>
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> >>>> ~[na:1.7.0_67] at
> >>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> ~[na:1.7.0_67]
> >>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> ~[na:1.7.0_67]
> >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> ~[na:1.7.0_67] at
> >>>> java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_67]
> at
> >>>> java.security.AccessController.doPrivileged(Native Method)
> ~[na:1.7.0_67] at
> >>>> java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> ~[na:1.7.0_67] at
> >>>> java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[na:1.7.0_67]
> at
> >>>> java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_67]
> at
> >>>>
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:755)
> >>>> ~[hadoop-hdfs-2.6.0.jar: na] at
> >>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.7.0_67]
> >>>> at
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>>> ~[na:1.7.0_67] at
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>> ~[na:1.7.0_67] at java.lang.reflect.Method.invoke(Method.java:606)
> >>>> ~[na:1.7.0_67] at
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> >>>> ~[hadoop-common-2.6.0.jar:na] at
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> >>>> ~[hadoop-common-2.6.0.jar:na] at
> com.sun.proxy.$Proxy156.getFileInfo(Unknown
> >>>> Source) ~[na:na] at
> >>>> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
> >>>> ~[hadoop-hdfs-2.6.0.jar:na] at
> >>>>
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
> >>>> ~[hadoop-hdfs-2.6.0.jar:na] at
> >>>>
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
> >>>> ~[hadoop-hdfs-2.6.0.jar:na] at
> >>>>
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> >>>> ~[hadoop-common-2.6.0.jar:na] at
> >>>>
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
> >>>> ~[hadoop-hdfs-2.6.0.jar:na] at
> >>>> org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:232)
> >>>> ~[nifi-hdfs-processors-0.2.1.jar:0.2.1] at
> >>>>
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> >>>> ~[nifi-api-0.2.1.jar:0.2.1] at
> >>>>
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077)
> >>>> ~[na:na] at
> >>>>
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127)
> >>>> ~[na:na] at
> >>>>
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49)
> >>>> ~[na:na] at
> >>>>
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
> >>>> ~[na:na] at
> >>>>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> ~[na:1.7.0_67] at
> >>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> >>>> ~[na:1.7.0_67]
>