You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Chris Teoh <ch...@gmail.com> on 2015/12/04 15:13:07 UTC
Recovering from heap memory exceeded
Hey folks,
I'm trying to recover NiFi from a state where the processors are stuck
running a process (in reality the process isn't running) even after a NiFi
restart.
I was running SplitText on log files to return a lot of FlowFiles which
caused an out of memory error.
The result is a lot of the following errors and the stuck processors.
Expiration Action
org.apache.nifi.provenance.lucene.DeleteIndexAction@4f65b2f0 on Provenance
Event file ./provenance_repository/3981676.prov.gz due to
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed;
will not perform additional Expiration Actions on this file at this time
How do I recover from this? I had to delete all the repositories to make
this come back to a known good state. Is this the only way or can I recover
the state?
Kind Regards
Chris
Re: Recovering from heap memory exceeded
Posted by Joe Witt <jo...@gmail.com>.
Chris,
I think there are some important details missing. In the case of auto
resume being false on restart there is no opportunity for those
processors to be stuck as they'll never receive a thread. Can you
share a thread dump? Feel free to create a JIRA for what you're
seeing and attach the thread dump. You can create one by running
<nifi-install-dir>/bin/nifi.sh dump Then you can find the output in
the logs/nifi-bootstrap.log i believe it is.
Thanks
Joe
On Fri, Dec 4, 2015 at 8:46 PM, Chris Teoh <ch...@gmail.com> wrote:
> Hi Mark,
>
> I did try that but the processors indicated their process was still running
> and stuck.
>
> Kind regards
> Chris
>
> On Sat, 5 Dec 2015 at 2:49 AM, Mark Payne <ma...@hotmail.com> wrote:
>>
>> Chris,
>>
>> Before you restart NiFi, if you look in the conf/nifi.properties file,
>> there is a property called
>> "nifi.flowcontroller.autoResumeState". If this is set to false, then on
>> restart, the processors will
>> be stopped instead of running. This will keep you from getting back into
>> the OutOfMemoryError state
>> and give you a chance to update the flow so that you can avoid getting
>> back into this state.
>>
>> Thanks
>> -Mark
>>
>>
>> > On Dec 4, 2015, at 9:13 AM, Chris Teoh <ch...@gmail.com> wrote:
>> >
>> > Hey folks,
>> >
>> > I'm trying to recover NiFi from a state where the processors are stuck
>> > running a process (in reality the process isn't running) even after a NiFi
>> > restart.
>> >
>> > I was running SplitText on log files to return a lot of FlowFiles which
>> > caused an out of memory error.
>> >
>> > The result is a lot of the following errors and the stuck processors.
>> >
>> > Expiration Action
>> > org.apache.nifi.provenance.lucene.DeleteIndexAction@4f65b2f0 on Provenance
>> > Event file ./provenance_repository/3981676.prov.gz due to
>> > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed;
>> > will not perform additional Expiration Actions on this file at this time
>> >
>> > How do I recover from this? I had to delete all the repositories to make
>> > this come back to a known good state. Is this the only way or can I recover
>> > the state?
>> >
>> > Kind Regards
>> > Chris
>>
>
Re: Recovering from heap memory exceeded
Posted by Chris Teoh <ch...@gmail.com>.
Hi Mark,
I did try that but the processors indicated their process was still running
and stuck.
Kind regards
Chris
On Sat, 5 Dec 2015 at 2:49 AM, Mark Payne <ma...@hotmail.com> wrote:
> Chris,
>
> Before you restart NiFi, if you look in the conf/nifi.properties file,
> there is a property called
> "nifi.flowcontroller.autoResumeState". If this is set to false, then on
> restart, the processors will
> be stopped instead of running. This will keep you from getting back into
> the OutOfMemoryError state
> and give you a chance to update the flow so that you can avoid getting
> back into this state.
>
> Thanks
> -Mark
>
>
> > On Dec 4, 2015, at 9:13 AM, Chris Teoh <ch...@gmail.com> wrote:
> >
> > Hey folks,
> >
> > I'm trying to recover NiFi from a state where the processors are stuck
> running a process (in reality the process isn't running) even after a NiFi
> restart.
> >
> > I was running SplitText on log files to return a lot of FlowFiles which
> caused an out of memory error.
> >
> > The result is a lot of the following errors and the stuck processors.
> >
> > Expiration Action
> org.apache.nifi.provenance.lucene.DeleteIndexAction@4f65b2f0 on
> Provenance Event file ./provenance_repository/3981676.prov.gz due to
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed;
> will not perform additional Expiration Actions on this file at this time
> >
> > How do I recover from this? I had to delete all the repositories to make
> this come back to a known good state. Is this the only way or can I recover
> the state?
> >
> > Kind Regards
> > Chris
>
>
Re: Recovering from heap memory exceeded
Posted by Mark Payne <ma...@hotmail.com>.
Chris,
Before you restart NiFi, if you look in the conf/nifi.properties file, there is a property called
"nifi.flowcontroller.autoResumeState". If this is set to false, then on restart, the processors will
be stopped instead of running. This will keep you from getting back into the OutOfMemoryError state
and give you a chance to update the flow so that you can avoid getting back into this state.
Thanks
-Mark
> On Dec 4, 2015, at 9:13 AM, Chris Teoh <ch...@gmail.com> wrote:
>
> Hey folks,
>
> I'm trying to recover NiFi from a state where the processors are stuck running a process (in reality the process isn't running) even after a NiFi restart.
>
> I was running SplitText on log files to return a lot of FlowFiles which caused an out of memory error.
>
> The result is a lot of the following errors and the stuck processors.
>
> Expiration Action org.apache.nifi.provenance.lucene.DeleteIndexAction@4f65b2f0 on Provenance Event file ./provenance_repository/3981676.prov.gz due to org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed; will not perform additional Expiration Actions on this file at this time
>
> How do I recover from this? I had to delete all the repositories to make this come back to a known good state. Is this the only way or can I recover the state?
>
> Kind Regards
> Chris